cd /news/ai-safety/hirewatch-evaluating-llm-compliance-… · home › topics › ai-safety › article
[ARTICLE · art-173014] src=arxiv.org ↗ pub= topic=ai-safety verified=true sentiment=↓ negative

HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure

A new arXiv paper (2611.24970v1) introduces HireWatch, a 200-item bank of lawful and unlawful interview questions drawn from U.S. federal statutes, regulations and agency guidance, and finds that LLM legal compliance in hiring is brittle. Across roughly 100,000 trials with 18 closed and open-weight LLMs, the researchers report that models generally avoid unlawful questions in low-pressure contexts but legal violations increase sharply under higher contextual pressure, and explicit prompts to comply with law improve compliance rates without eliminating violations. The authors released code and data at github.com/hnghiem-nlp/hirewatch_repo.

recommendation

Test hiring-related LLM prompts against a lawful/unlawful question bank under escalating contextual pressure, not just low-pressure defaults, and treat law-compliance instructions as a partial mitigation rather than a guarantee.

more in playbook: Evals and testing for LLM apps
by read1 min views3 publishedNov 20, 2026

arXiv:2611.24970v1 Announce Type: new Abstract: The deployment of large language models (LLMs) in hiring pipelines is growing, yet existing evaluations generally overlook their compliance with legal rules. To fill this gap, we introduce HireWatch, an evaluation that assesses whether LLMs comply with U.S. laws that govern the questions that job candidates may be asked. We construct a 200-item bank of lawful and unlawful interview questions, whose legal classification is based on U.S. federal statutes, regulations, and agency guidance. We instruct LLMs to select questions from this bank in preparing for job interviews, across a set of various occupations and contexts, and report the fraction of unlawful questions that LLMs select. Evaluation of 18 closed and open-weight LLMs across ~100,000 trials under different types of contextual pressure reveals that LLMs' legal compliance is brittle. While LLMs generally avoid selecting unlawful questions in low-pressure contexts, legal violations increase sharply under higher pressure. Explicit prompts to comply with law can improve compliance rates, but do not altogether eliminate legal violations. We publicly release our code and data at https://github.com/hnghiem-nlp/hirewatch_repo.

── more in #ai-safety 4 stories · sorted by recency
── more on @hirewatch 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/hirewatch-evaluating…] indexed:0 read:1min 2026-11-20 · —