Show HN: LangDrift – test AI agents across languages Rubén González released LangDrift, an open-source tool for testing whether AI agents preserve behavior across languages, including tool selection and arguments. The tool runs YAML scenarios against any HTTP agent endpoint and reports failures by locale, with a demo showing agents failing to call tools in Swahili and Chinese. It is designed for teams who already test agents in English and want to catch behavioral drift in other languages. Locale-aware evals for AI agent behavior. Project page: rubenglez.dev/langdrift https://rubenglez.dev/langdrift LangDrift checks whether an AI agent preserves behavior across languages: tool selection, tool arguments, response script, and failure modes. It is built for teams who already test their agent in English and want to know what changes when the same intent arrives in French, Arabic, Chinese, Basque, Swahili, or any other locale. The core question: Does your agent still choose the right tool when the same user intent arrives in another language? No API key required. Clone the repo, start the fake agent, and run a scenario: git clone https://github.com/RubenGlez/langdrift.git cd langdrift pnpm install pnpm fake-agent In another terminal: node ./src/cli.ts run ./examples/scenarios/support-routing.yaml --target http://127.0.0.1:3011/api/agent Testing your own agent? Install globally and point it at your endpoint: npm install -g langdrift langdrift run ./my-scenario.yaml --target http://localhost:3010/api/agent The fake agent intentionally drops tool calls for Swahili sw and Chinese zh , so the demo shows the core failure mode without using an API key: Locale Passed Failure Detail en 1/1 - create refund ticket sw 0/1 no tool call expected create refund ticket, got no tool calls zh 0/1 no tool call expected create refund ticket, got no tool calls Result: failed, 2 of 12 locales failed AI localization is moving from translated strings to localized behavior. For an agent, a localized experience is only correct if the agent preserves intent, tool selection, and tool arguments across languages. In the included benchmark, English often passed while equivalent prompts in Basque, Yoruba, Swahili, Chinese, Welsh, and Mongolian triggered missing tool calls, wrong tool arguments, or different tool-use behavior. The strongest signal is that several of these weaknesses recur across three independently trained models, which is harder to dismiss than any single per-locale rate. This is not a universal language ranking; it is a reproducible demonstration that agent behavior can drift across locales. Read the full methodology, results, limitations, and supporting papers in RESEARCH.md /RubenGlez/langdrift/blob/main/RESEARCH.md . - Runs YAML scenarios with per-locale user inputs. - Sends each locale to any HTTP agent target. - Checks tool calls, shallow arguments, forbidden tools, ordered tool-call sequences, and response script. - Reports pass/fail by locale with failure mode classification. - Emits terminal, JSON, or markdown reports. - Exits non-zero on failure, so it works in CI. - Supports repeated runs with --iterations N . - Supports directory-level scenario runs for locale x scenario matrices. - Provides lint and LLM-assisted translate commands for scenario maintenance. Failure modes include no tool call , wrong tool , wrong argument , missing argument , forbidden tool , wrong sequence , and wrong language . npm install -g langdrift Requires Node = 22. The published package ships compiled JavaScript, so no build step or type-stripping flags are needed to run the installed CLI. Working from a clone instead runs the TypeScript source directly via Node's native type stripping node ./src/cli.ts , which is how the "30 seconds" demo above works without a build. Create a starter scenario: langdrift init ./my-scenario.yaml --template support Edit the generated YAML: id: refund request agent: support locales: en: input: "I was charged twice for my subscription. Can you refund one charge?" expect: toolCall: name: create refund ticket arguments: reason: duplicate charge noToolCall: name: escalate to human fr: input: "J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?" expect: toolCall: name: create refund ticket arguments: reason: duplicate charge responseLanguage: fr Scenario files are a strict 2-space-indented subset of YAML , not full YAML. Use exactly two spaces per level no tabs , and keep every value on one line — block scalars input: | , flow mappings, and multi-line strings are not supported. Out-of-subset input is rejected with a line-numbered error rather than parsed loosely. Run langdrift lint to catch these and other issues early. Run it against your agent: langdrift run ./my-scenario.yaml --target http://127.0.0.1:3010/api/agent Emit JSON for tooling: langdrift run ./my-scenario.yaml --target http://127.0.0.1:3010/api/agent --format json Run a directory of scenarios: langdrift run ./scenarios --target http://127.0.0.1:3010/api/agent --iterations 3 --format markdown Argument values are matched with scalar-normalized equality, so a JSON number 2 matches "2" and a boolean true matches "true" . Tool arguments must be canonical identifiers or enums , not free text. Agents tend to echo the user's language into free-text argument values, so an assertion like reason: duplicate charge will report a false wrong argument when a localized request produces reason: "doble cargo" . Model your tools to emit canonical values, or accept several with oneOf : expect: toolCall: name: create refund ticket arguments: reason: oneOf: duplicate charge, double charge oneOf is an inline list and must contain at least one value; langdrift lint reports an error otherwise. noToolCall fails the locale if the agent calls a tool it should not. Forbid one tool with name , or several with anyOf : expect: toolCall: name: create refund ticket noToolCall: anyOf: escalate to human, contact seller responseLanguage is a script-family check , not language detection. It confirms a reply uses the script a locale is written in for example, that an ar reply is in Arabic script . Know its limits before relying on it: It cannot distinguish languages that share a script. A fr assertion passes for any Latin-script reply; ar cannot be told apart from fa or ur ; zh accepts any Han text. The one Han exception is ja , which additionally requires kana, so pure-Chinese text does not pass responseLanguage: ja . The thresholds are asymmetric. A non-Latin locale passes when at least 10% of letters are in its script; a Latin locale fails only when more than 50% of letters are non-Latin. The measured ratio is included in the failure/pass detail so near-misses are visible. For a locale whose script LangDrift cannot determine, the check passes rather than guessing. langdrift lint warns when a responseLanguage value is not script-determinable, since the check can then never fail. LangDrift makes a POST request to your agent for each locale. Request: { "locale": "fr", "input": "J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?", "scenarioId": "refund request", "agent": "support" } agent is the scenario's agent: field, sent as routing metadata; agents that serve one workflow can ignore it. Response: { "text": "Je peux vous aider avec ce remboursement.", "toolCalls": { "name": "create refund ticket", "arguments": { "reason": "duplicate charge" } } , "structured": null } Response fields: | Field | Type | Description | |---|---|---| text | string | Agent text reply. Missing defaults to "" . | toolCalls | array | Tool calls made by the agent. Each item must have name ; arguments is optional. Missing defaults to . | structured | any | Optional structured output. Missing defaults to null . | Extra response fields are ignored. Non-2xx responses fail the locale. See docs/integrations.md /RubenGlez/langdrift/blob/main/docs/integrations.md for OpenAI SDK, Vercel AI SDK, LangChain, Anthropic, and Fastify examples. langdrift init scenario.yaml --template support|ecommerce|scheduling|generic langdrift run