When I started building my AI technical interviewer, I did what most people do: I threw a big system prompt at the LLM and told it to "act like an interviewer, ask coding questions, give hints when the candidate is stuck, and score them at the end."
It worked... about 80% of the time.
The other 20% is what this post is about.
Here's what kept happening. Mid-interview, the model would:
None of this was a prompting skill issue. I rewrote that system prompt probably fifteen times. The real issue is structural: an LLM generating the next thing to say has no actual memory of "what phase are we in," unless you spoon-feed it that context perfectly, every single turn, forever. And even then, it can just... decide to do something else. It's a language model, not a state tracker. Treating it like one is where things fall apart.
For a casual chatbot, that's fine — mild chaos is charming. For a product where someone's actual hiring decision depends on the interview going through every stage correctly, "mild chaos" is a support ticket and an angry candidate.
I pulled the structure of the interview out of the LLM entirely and put it into a plain old finite state machine.
Something like:
SETUP → GREETING → QUESTION_ASKED → CANDIDATE_CODING →
HINT_CHECK → EVALUATING → SCORING → DONE
The state machine — not the model — decides what phase we're in and what's allowed to happen next. The LLM only gets called inside a state, to do the one thing that state needs: generate a question, generate a hint, or generate a scorecard. It never gets to decide "we're done now" or "let's skip to scoring." That's not its job anymore.
Practically, this meant:
The LLM still does all the hard, "actually intelligent" work — writing a good question, phrasing a helpful hint, writing a fair evaluation. It's just not allowed to drive the car anymore. It's a really good passenger with really good opinions.
class InterviewState(Enum):
GREETING = "greeting"
QUESTION_ASKED = "question_asked"
CANDIDATE_CODING = "candidate_coding"
HINT_CHECK = "hint_check"
EVALUATING = "evaluating"
SCORING = "scoring"
DONE = "done"
def transition(current_state, event):
if current_state == InterviewState.CANDIDATE_CODING:
if event == "idle_35s":
return InterviewState.HINT_CHECK
if event == "code_submitted":
return InterviewState.EVALUATING
Compare that to letting the model implicitly track state through conversation history alone — there's no guarantee, no test coverage, and no way to catch a bad transition before it reaches the user.
I expected this to make the product feel more robotic. It did the opposite.
Because the structure is now guaranteed, I could actually let the LLM be more creative and natural within each state, without worrying about it going off the rails. Constraining the skeleton let me loosen up the muscle. Counterintuitive, but it checks out — a lot of "unpredictable AI" complaints aren't really about the model being too creative, they're about the model having too much control over things it was never designed to control.
If you're building something where an LLM is orchestrating a multi-step process — not just answering one-off questions — ask yourself: does the model actually need to decide what happens next, or does it just need to generate good content within a step someone else decides?
Most of the time, in my experience, it's the second one. And the moment I stopped asking the LLM to be both the actor and the director, my "why did the interview just end after one question" bugs basically disappeared overnight.
I write about the messier, more practical side of building AI products — the stuff that doesn't make it into the demo. If you're curious, I built this exact system as a live product: AI Technical Interviewer. Happy to talk through the architecture more in the comments.