The signals that mattered when you hired ML engineers in 2022 barely predict who ships reliable AI systems in 2026. "Trained a model on Kaggle" and "knows PyTorch" tell you almost nothing about whether someone can put an agent in front of real users without lighting your token budget on fire.
After 200+ projects, here's the evaluation framework we actually use.
There are three distinct roles people lump together as "AI engineer," and mixing them up is the #1 hiring mistake:
Most teams post a research-engineer job description and then wonder why the candidate can't ship a support agent. Hire for the tier that matches the work.
1. Production deployment history. Ask for a system they shipped that real users hit, and what broke at 2am. Demos prove the happy path once; production is the ten-thousandth weird input under a cost ceiling. Anyone can build the demo.
2. Cost awareness. "How would you cut the inference bill on this by half?" A strong answer names model routing, caching, and prompt/token discipline immediately. If cost never comes up, they've never run anything at scale.
3. Evaluation framework design. The single best predictor. Ask how they'd know a prompt change made the system better. If the answer isn't a golden test set of real input/output pairs, they've been guessing, and guessing doesn't ship v2.
4. Architecture decision-making. When to fine-tune vs RAG vs prompt, when pgvector-on-Postgres is enough vs a dedicated vector DB. Good engineers reach for the boring, cheap option first.
5. Agent system experience. Have they built something with bounded action space, max-call limits, and circuit breakers? Unbounded agent loops are how a $40 demo becomes a $4,000 bill.
6. Security and safety awareness. Prompt injection, data leakage through context, output validation. If they've never thought about a user pasting ignore previous instructions
, that's a gap.
7. AI-first methodology. Do they use AI to build (codegen, review, test generation)? The 10-20x speed difference in 2026 is mostly workflow, not raw skill.
Skip the LeetCode. Give a scoped, realistic problem — "design a support-triage agent for this product" — and watch how they reason about evals, cost, and failure modes out loud. The thinking is the signal.
Build in-house only when AI is your core product, you already have senior ML talent, and your iteration cycle is under 48 hours. Otherwise the realistic move for a first production deployment is a partner who ships in weeks and hands off to a single senior hire who shadows the build — so you own the architecture when the engagement ends. We break the full tradeoff, tier salary bands, and a candidate scorecard down in the original guide on groovyweb.co.
The short version: stop screening for who knows AI and start screening for who has shipped and maintained it. Evals, cost-awareness, and production scars beat any framework name on a resume.
Originally published on Groovy Web.