# Evaluating ML-Based Hiring Tools: An Engineer's Checklist > Source: > Published: 2026-09-16 13:05:43+00:00 If your company is buying an ML-based hiring tool, there is a decent chance someone will forward the technical due-diligence to an engineer. This is a checklist for that engineer. The business folks will evaluate the demo; your job is to evaluate the model, the data path, and the integration surface — the parts a demo is specifically designed to hide. The first thing to establish: can the system produce a per-decision explanation, and is that explanation faithful to the model, or is it a post-hoc rationalization bolted on for the UI? Ask what method generates the "why this candidate ranked here" output. If the answer is a real feature-attribution approach tied to defined criteria, good. If the answer is vague — "the model considers many factors" — you are likely looking at a black box with a narrative layer. A useful probe: ask them to show two candidates who ranked closely and explain the delta. Faithful explanations produce a crisp, criteria-linked difference. Rationalizations get mushy. Why you care: a keyword matcher with a neural-net press release cannot explain rankings in criteria terms, because there is no model to explain. The explainability question is the fastest way to expose capability theater. Ask what the model was trained on. Models trained on a company's historical hires learn to replicate historical hiring patterns — including whatever bias was in them. "The model is bias-free by design" is a sentence no serious ML practitioner says; bias is measured, not designed away. What you want to see: •An independent bias audit (not a self-assessment), reasonably recent. •Reported four-fifths / adverse-impact ratios (lowest group pass rate ÷ highest; ≥ 0.80 is the conventional threshold). •A clear statement of whether the tool scores against criteria you define or patterns it infers from your past hires. Prefer the former; the latter quietly automates the status quo. This is not academic. Under NYC Local Law 144 the audit is legally required, and the EU AI Act puts hiring models in its high-risk tier. The compliance obligation sits with the employer, so your engineering sign-off has legal weight. The most common fate of a recruiting tool is shelfware, and the most common cause is integration that looked fine on a partners page and fell apart in production. "API available" is not integration. Check for: •Bidirectional sync with your specific ATS version, documented, ideally with a reference customer on the same stack. •Webhook / event support vs. polling, and rate limits that survive your actual application volume. •Where the source of truth lives when the two systems disagree. Weight this heavily. One healthcare staffing firm re-ran a selection with integration weighted double after their previous tool never synced; adoption went from 25% to 85% at 90 days. Your leverage is highest before the contract exists and drops to near zero afterward. Get, in writing: •Full export of candidate data, scores, and audit logs in standard formats. •No per-export fees, no proprietary-format lock-in. •A defined data-handoff on exit. Lock-in economics are how a mediocre tool becomes a three-year hostage situation. Contract for the exit while you can. Ignore vendor accuracy numbers. "95% accurate" against an undefined benchmark is unfalsifiable. Define your own measure before the pilot: agreement rate between the tool's rankings and your best recruiters' judgments, on your candidates, on live roles, run in parallel with the current process for ~30 days. Track agreement rate, time saved, override rate, and pass-rate stability across groups. •Per-decision explanations are faithful and criteria-linked •Training-data provenance disclosed •Independent bias audit with four-fifths ratios •Scores against defined criteria, not inferred patterns •Documented bidirectional ATS integration on your version •Contractual data + audit-log export, standard formats, no fees •30-day parallel pilot with a pre-defined success metric If you want the business-side version of this — the 7 vendor questions framed for a procurement call rather than a code review — it pairs well with the checklist above. I work on a recruiting AI platform, and my bias is toward tools that can survive this checklist. If you want to see one built for it, [here is how we approach explainability, auditing, and export](https://www.hiremore.ai/blog/ai-recruitment-tools-look-before-buying?utm_source=devto&utm_medium=syndication&utm_campaign=blog21_tools)