Don't Trust the LLM Provider

Anthropic published a finding that several Chinese AI firms secretly routed LLM requests to Claude, including API keys, credentials, rocket code, and military information, according to the article. The report warns that LLM providers can see prompts as plaintext, so users should avoid putting PII or secrets in prompts and instead use short-lived, single-purpose credentials when a key is unavoidable. The finding argues that sandboxing and policy enforcement must be hard gates rather than prompt-based, making jailbreak and data-leak prevention a constant cost of using LLMs.

Don't Trust the LLM Provider Anthropic published a finding: several Chinese AI firms secretly routed LLM requests to Claude to get better answers. Those requests included API keys, credentials, rocket code, and military information. We know “don’t trust user input” and increasingly “don’t trust LLM output.” I would add one more: don’t trust the LLM provider either. Providers can see your prompts as plaintext. Do not put PII or secrets in prompts. If a key is unavoidable, use a short-lived, single-purpose credential. This changes system design. Do not rely on prompts to define security behavior. Sandboxing and policy enforcement must be hard gates. Limit what an agent can see and do. Preventing jailbreaks and data leaks is not a one-time fix. It is a constant cost of using LLMs.