{"slug": "ai-engineering-interview-questions-retrieval-evaluation-and-everything-that", "title": "AI engineering interview questions: retrieval, evaluation and everything that breaks", "summary": "In a blog post, a developer discusses AI engineering interview questions, emphasizing that candidates are now evaluated on their ability to diagnose retrieval versus generation failures, build robust evaluation sets, handle structured output, and mitigate prompt injection. The post highlights that senior-level answers focus on systematic debugging and production-ready practices, such as treating model output as untrusted input and maintaining regression test sets from real failures.", "body_md": "The field is young enough that nobody has ten years of experience, so interviewers weight engineering judgement heavily over tool familiarity. The questions are less about what you have used and more about how you would know it was working.\n\nAlmost every candidate has now built something with a language model. That means the demo no longer differentiates anyone, and interviews have moved to the parts that are hard: making retrieval actually relevant, knowing whether a change made things worse, and handling text you did not write.\n\nEveryone can draw the pipeline: embed the documents, embed the question, find the nearest, put them in the context. Interviewers have seen it. The questions are all about what makes it work badly.\n\n**Junior answer:** I would improve the instructions to the model, try a larger model, and adjust the temperature to make it more accurate and less prone to making things up.\n\n**Senior answer:** First I would find out whether retrieval or generation is failing, because they need opposite fixes and guessing wastes a week. I would take a sample of the bad answers and look at what was actually retrieved. If the right passage was never retrieved, no amount of instruction tuning will help and the work is in chunking or hybrid search. If the right passage was retrieved and the answer still contradicted it, that is a generation and grounding problem. Before changing anything I would build a small set of question and expected-answer pairs, so I can tell whether the next change helped or just moved the failures somewhere I was not looking.\n\n*Splitting retrieval failure from generation failure is the single most useful diagnostic in this area, and most candidates skip straight to the model.*\n\nIf there is one topic that separates people who have shipped this from people who have prototyped it, this is it. Non-deterministic output means you cannot rely on the usual assertion-based safety net, and the temptation is to test by trying a few things and feeling good about them.\n\nWhat a strong answer describes is a fixed evaluation set built from real failures, run on every change, with results compared rather than eyeballed. Then the honest caveats: automated judging is itself unreliable and needs spot-checking, and an evaluation set drifts away from real usage unless you keep feeding production failures back into it.\n\nSay that you would add every reported bad answer to the set as a regression case and you have described the practice that actually keeps these systems from decaying.\n\nA practical question because it is what most production integrations need. The model produces text; your code needs a shape it can rely on.\n\nThe expected answer has three parts: constrain the output where the platform supports it, validate against a schema on the way in rather than trusting it, and have a defined path when validation fails. That last part is where the follow-up goes. Retrying forever is a cost and rate-limit incident waiting to happen, so a bounded retry and then a fallback is the answer that shows you have run this in production.\n\nThe framing that landsTreat model output as untrusted input, exactly like a request body from the internet. Validate it, bound it, and never let it reach anything consequential unchecked. That single sentence answers a surprising number of questions in this area.\n\nIncreasingly asked, and the answers separate people quickly.\n\nThe core problem is that instructions and data arrive through the same channel, so text from a document, a web page or a user message can contain something that reads as an instruction. There is no complete fix through wording alone, and a candidate who claims otherwise has not thought about it hard.\n\nWhat actually reduces risk is architectural: keep untrusted content clearly separated from your own instructions, do not grant the system capabilities it does not need, require confirmation before anything consequential or irreversible, and validate any tool call before executing it. The mindset here is the same one a [security round](https://practicedepth.com/blog/security-interview-questions-senior-engineers) tests, applied to a component that is non-deterministic.\n\nFrequently the most practical section, and the one where product-minded engineers do well.\n\nCost scales with tokens, so long context is a per-request bill, not a one-off design choice. Latency scales with output length in particular, which is why streaming exists and why it changes perceived speed far more than raw model speed. The senior framing is routing: not every request needs the most capable option, and using a cheaper path for the easy majority is usually the largest available win.\n\nA judgement question, and the wrong answer is reaching for fine-tuning first.\n\nThe distinction to draw is between knowledge and behaviour. If the system needs facts it does not have, that is retrieval, and fine-tuning is an expensive way to bake in something that will be stale next month. If it needs a consistent format or style, that is a case for tuning, once you have exhausted cheaper options. Being explicit that you would start with the cheapest reversible change is the signal.\n\nMuch less than you would think. The field moves fast enough that interviewers weight reasoning about retrieval quality, evaluation and failure handling well above tool familiarity.\n\nFor an AI engineering role, usually not. Embeddings and why similarity is not relevance, yes. Gradient descent, rarely, unless the role is actually a modelling role.\n\nTalk about what went wrong and what you measured. A small project with a real evaluation set and a documented failure mode is stronger evidence than a large one with neither.\n\nThis area is almost entirely judgement under follow-up, which is exactly what a focused practice round exposes. [Practise an AI engineering interview](https://practicedepth.com/app/practice/interview?focus=ai-engineering&mode=senior&utm_source=devto&utm_medium=article&utm_campaign=organic)", "url": "https://wpnews.pro/news/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that", "canonical_source": "https://dev.to/moputa/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that-breaks-62n", "published_at": "2026-08-14 08:37:00+00:00", "updated_at": "2026-08-14 09:05:26.230966+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-products", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that", "markdown": "https://wpnews.pro/news/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that.md", "text": "https://wpnews.pro/news/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that.txt", "jsonld": "https://wpnews.pro/news/ai-engineering-interview-questions-retrieval-evaluation-and-everything-that.jsonld"}}