{"slug": "show-hn-langdrift-test-ai-agents-across-languages", "title": "Show HN: LangDrift – test AI agents across languages", "summary": "Rubén González released LangDrift, an open-source tool for testing whether AI agents preserve behavior across languages, including tool selection and arguments. The tool runs YAML scenarios against any HTTP agent endpoint and reports failures by locale, with a demo showing agents failing to call tools in Swahili and Chinese. It is designed for teams who already test agents in English and want to catch behavioral drift in other languages.", "body_md": "Locale-aware evals for AI agent behavior.\n\n**Project page:** [rubenglez.dev/langdrift](https://rubenglez.dev/langdrift)\n\nLangDrift checks whether an AI agent preserves behavior across languages: tool selection, tool arguments, response script, and failure modes. It is built for teams who already test their agent in English and want to know what changes when the same intent arrives in French, Arabic, Chinese, Basque, Swahili, or any other locale.\n\nThe core question:\n\nDoes your agent still choose the right tool when the same user intent arrives in another language?\n\n**No API key required.** Clone the repo, start the fake agent, and run a scenario:\n\n```\ngit clone https://github.com/RubenGlez/langdrift.git\ncd langdrift\npnpm install\npnpm fake-agent\n```\n\nIn another terminal:\n\n```\nnode ./src/cli.ts run ./examples/scenarios/support-routing.yaml --target http://127.0.0.1:3011/api/agent\n```\n\n**Testing your own agent?** Install globally and point it at your endpoint:\n\n```\nnpm install -g langdrift\nlangdrift run ./my-scenario.yaml --target http://localhost:3010/api/agent\n```\n\nThe fake agent intentionally drops tool calls for Swahili (`sw`\n\n) and Chinese (`zh`\n\n), so the demo shows the core failure mode without using an API key:\n\n```\nLocale  Passed  Failure       Detail\nen      1/1     -             create_refund_ticket\nsw      0/1     no_tool_call  expected create_refund_ticket, got no tool calls\nzh      0/1     no_tool_call  expected create_refund_ticket, got no tool calls\n\nResult: failed, 2 of 12 locales failed\n```\n\nAI localization is moving from translated strings to localized behavior. For an agent, a localized experience is only correct if the agent preserves intent, tool selection, and tool arguments across languages.\n\nIn the included benchmark, English often passed while equivalent prompts in Basque, Yoruba, Swahili, Chinese, Welsh, and Mongolian triggered missing tool calls, wrong tool arguments, or different tool-use behavior. The strongest signal is that several of these weaknesses recur across three independently trained models, which is harder to dismiss than any single per-locale rate. This is not a universal language ranking; it is a reproducible demonstration that agent behavior can drift across locales.\n\nRead the full methodology, results, limitations, and supporting papers in [RESEARCH.md](/RubenGlez/langdrift/blob/main/RESEARCH.md).\n\n- Runs YAML scenarios with per-locale user inputs.\n- Sends each locale to any HTTP agent target.\n- Checks tool calls, shallow arguments, forbidden tools, ordered tool-call sequences, and response script.\n- Reports pass/fail by locale with failure mode classification.\n- Emits terminal, JSON, or markdown reports.\n- Exits non-zero on failure, so it works in CI.\n- Supports repeated runs with\n`--iterations N`\n\n. - Supports directory-level scenario runs for locale x scenario matrices.\n- Provides\n`lint`\n\nand LLM-assisted`translate`\n\ncommands for scenario maintenance.\n\nFailure modes include `no_tool_call`\n\n, `wrong_tool`\n\n, `wrong_argument`\n\n, `missing_argument`\n\n, `forbidden_tool`\n\n, `wrong_sequence`\n\n, and `wrong_language`\n\n.\n\n```\nnpm install -g langdrift\n```\n\nRequires Node >= 22. The published package ships compiled JavaScript, so no build step or type-stripping flags are needed to run the installed CLI.\n\nWorking from a clone instead runs the TypeScript source directly via Node's native type stripping (`node ./src/cli.ts`\n\n), which is how the \"30 seconds\" demo above works without a build.\n\nCreate a starter scenario:\n\n```\nlangdrift init ./my-scenario.yaml --template support\n```\n\nEdit the generated YAML:\n\n```\nid: refund_request\nagent: support\n\nlocales:\n  en:\n    input: \"I was charged twice for my subscription. Can you refund one charge?\"\n    expect:\n      toolCall:\n        name: create_refund_ticket\n        arguments:\n          reason: duplicate_charge\n      noToolCall:\n        name: escalate_to_human\n\n  fr:\n    input: \"J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?\"\n    expect:\n      toolCall:\n        name: create_refund_ticket\n        arguments:\n          reason: duplicate_charge\n      responseLanguage: fr\n```\n\nScenario files are a **strict 2-space-indented subset of YAML**, not full YAML. Use exactly two spaces per level (no tabs), and keep every value on one line — block scalars (`input: |`\n\n), flow mappings, and multi-line strings are not supported. Out-of-subset input is rejected with a line-numbered error rather than parsed loosely. Run `langdrift lint`\n\nto catch these and other issues early.\n\nRun it against your agent:\n\n```\nlangdrift run ./my-scenario.yaml --target http://127.0.0.1:3010/api/agent\n```\n\nEmit JSON for tooling:\n\n```\nlangdrift run ./my-scenario.yaml --target http://127.0.0.1:3010/api/agent --format json\n```\n\nRun a directory of scenarios:\n\n```\nlangdrift run ./scenarios --target http://127.0.0.1:3010/api/agent --iterations 3 --format markdown\n```\n\nArgument values are matched with scalar-normalized equality, so a JSON number `2`\n\nmatches `\"2\"`\n\nand a boolean `true`\n\nmatches `\"true\"`\n\n.\n\nTool arguments must be **canonical identifiers or enums**, not free text. Agents tend to echo the user's language into free-text argument values, so an assertion like `reason: duplicate_charge`\n\nwill report a false `wrong_argument`\n\nwhen a localized request produces `reason: \"doble cargo\"`\n\n. Model your tools to emit canonical values, or accept several with `oneOf`\n\n:\n\n```\nexpect:\n  toolCall:\n    name: create_refund_ticket\n    arguments:\n      reason:\n        oneOf: [duplicate_charge, double_charge]\n```\n\n`oneOf`\n\nis an inline list and must contain at least one value; `langdrift lint`\n\nreports an error otherwise.\n\n`noToolCall`\n\nfails the locale if the agent calls a tool it should not. Forbid one tool with `name`\n\n, or several with `anyOf`\n\n:\n\n```\nexpect:\n  toolCall:\n    name: create_refund_ticket\n  noToolCall:\n    anyOf: [escalate_to_human, contact_seller]\n```\n\n`responseLanguage`\n\nis a **script-family check**, not language detection. It confirms a reply uses the script a locale is written in (for example, that an `ar`\n\nreply is in Arabic script). Know its limits before relying on it:\n\n**It cannot distinguish languages that share a script.** A`fr`\n\nassertion passes for any Latin-script reply;`ar`\n\ncannot be told apart from`fa`\n\nor`ur`\n\n;`zh`\n\naccepts any Han text. The one Han exception is`ja`\n\n, which additionally requires kana, so pure-Chinese text does not pass`responseLanguage: ja`\n\n.**The thresholds are asymmetric.** A non-Latin locale passes when at least 10% of letters are in its script; a Latin locale fails only when more than 50% of letters are non-Latin. The measured ratio is included in the failure/pass`detail`\n\nso near-misses are visible.**For a locale whose script LangDrift cannot determine, the check passes** rather than guessing.`langdrift lint`\n\nwarns when a`responseLanguage`\n\nvalue is not script-determinable, since the check can then never fail.\n\nLangDrift makes a `POST`\n\nrequest to your agent for each locale.\n\nRequest:\n\n```\n{\n  \"locale\": \"fr\",\n  \"input\": \"J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?\",\n  \"scenarioId\": \"refund_request\",\n  \"agent\": \"support\"\n}\n```\n\n`agent`\n\nis the scenario's `agent:`\n\nfield, sent as routing metadata; agents that serve one workflow can ignore it.\n\nResponse:\n\n```\n{\n  \"text\": \"Je peux vous aider avec ce remboursement.\",\n  \"toolCalls\": [\n    {\n      \"name\": \"create_refund_ticket\",\n      \"arguments\": {\n        \"reason\": \"duplicate_charge\"\n      }\n    }\n  ],\n  \"structured\": null\n}\n```\n\nResponse fields:\n\n| Field | Type | Description |\n|---|---|---|\n`text` |\nstring | Agent text reply. Missing defaults to `\"\"` . |\n`toolCalls` |\narray | Tool calls made by the agent. Each item must have `name` ; `arguments` is optional. Missing defaults to `[]` . |\n`structured` |\nany | Optional structured output. Missing defaults to `null` . |\n\nExtra response fields are ignored. Non-2xx responses fail the locale.\n\nSee [docs/integrations.md](/RubenGlez/langdrift/blob/main/docs/integrations.md) for OpenAI SDK, Vercel AI SDK, LangChain, Anthropic, and Fastify examples.\n\n```\nlangdrift init [scenario.yaml] [--template support|ecommerce|scheduling|generic]\nlangdrift run <scenario.yaml|dir> --target <url> [--iterations N] [--format text|json|markdown] [--min-pass-rate N] [--allow-fail <locale>]\nlangdrift lint <scenario.yaml|dir>\nlangdrift translate <scenario.yaml> [--locales fr,ar,zh,...] [--write]\n```\n\nUseful CI flags:\n\n`--min-pass-rate N`\n\n: fail only if the overall pass rate is below`N`\n\n.`--allow-fail <locale>`\n\n: keep reporting a known weak locale without letting it fail the build. A value that matches no locale prints a warning.`--format markdown`\n\n: write a table suitable for GitHub Actions summaries or PR comments.`--timeout MS`\n\n: per-request timeout (default 30000). A hung locale is recorded as`target_error`\n\ninstead of stalling the run.\n\nA transport failure (network error, non-2xx, malformed or non-JSON response, or timeout) is classified as `target_error`\n\n, distinct from the behavioral `no_tool_call`\n\nmode, so an agent outage is not mistaken for locale drift.\n\nSee [docs/ci.md](/RubenGlez/langdrift/blob/main/docs/ci.md) for GitHub Actions examples.\n\nThe repo includes two local agents:\n\n`pnpm fake-agent`\n\n: deterministic demo agent, no API key required.`pnpm agent`\n\n: model-backed agent for OpenAI, Anthropic, DeepSeek, or any OpenAI-compatible API.\n\n```\n# OpenAI\nOPENAI_API_KEY=... pnpm agent\n\n# Anthropic\nANTHROPIC_API_KEY=... MODEL_PROVIDER=anthropic MODEL_NAME=claude-haiku-4-5-20251001 pnpm agent\n\n# DeepSeek\nDEEPSEEK_API_KEY=... MODEL_PROVIDER=deepseek MODEL_NAME=deepseek-chat pnpm agent\n\n# Choose domain: support (default), ecommerce, scheduling\nDOMAIN=ecommerce OPENAI_API_KEY=... pnpm agent\n```\n\nThen run a scenario:\n\n```\nlangdrift run ./examples/scenarios/support-routing.yaml --target http://127.0.0.1:3010/api/agent\n```\n\n**Behavior over text.** LangDrift checks tool calls and structured behavior, not whether a reply sounds fluent.**Deterministic assertions first.** No LLM-as-judge in the core loop; failures are explainable and CI-friendly.**HTTP contract over framework lock-in.** Any agent that can accept one POST request can be tested.**Small, inspectable core.** Zero runtime dependencies, TypeScript source, Node >= 22.**CLI, not a library.** The published package exposes the`langdrift`\n\ncommand only; there is no importable JavaScript API. To reuse the internals, work from a clone of the TypeScript source.**Demo without API keys.** The fake agent makes the failure mode visible locally before connecting a real model.\n\n[RESEARCH.md](/RubenGlez/langdrift/blob/main/RESEARCH.md): full experiment, results, limitations, and supporting research.[docs/ci.md](/RubenGlez/langdrift/blob/main/docs/ci.md): CI integration examples.[docs/integrations.md](/RubenGlez/langdrift/blob/main/docs/integrations.md): agent adapter examples.", "url": "https://wpnews.pro/news/show-hn-langdrift-test-ai-agents-across-languages", "canonical_source": "https://github.com/RubenGlez/langdrift", "published_at": "2026-07-09 11:02:56+00:00", "updated_at": "2026-07-09 11:12:51.521312+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "natural-language-processing"], "entities": ["Rubén González", "LangDrift"], "alternates": {"html": "https://wpnews.pro/news/show-hn-langdrift-test-ai-agents-across-languages", "markdown": "https://wpnews.pro/news/show-hn-langdrift-test-ai-agents-across-languages.md", "text": "https://wpnews.pro/news/show-hn-langdrift-test-ai-agents-across-languages.txt", "jsonld": "https://wpnews.pro/news/show-hn-langdrift-test-ai-agents-across-languages.jsonld"}}