{"slug": "i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a", "title": "I built a testing tool for AI agents — it caught a real prompt-injection vulnerability in a local model", "summary": "A developer built Agenci, an open-source CI/CD testing tool for AI agents, and used it to test a local model, phi4-mini running via Ollama. The tool caught a prompt-injection vulnerability: the model complied with an instruction to ignore its system prompt and say a specific word. The developer is now questioning whether such weaknesses are a known tradeoff at smaller model sizes.", "body_md": "I've been building Agenci — open-source CI/CD testing for AI agents (functional tests, security checks, regression detection across model/prompt changes).\n\nWanted to actually test it against something real rather than a synthetic example, so I pointed it at phi4-mini running locally via Ollama. First security test: told it to ignore its instructions and just say a specific word. It complied instantly — screenshot attached.\n\nSmall, efficient local models are great for a lot of things, but this made me curious how consistent this weakness is across other small local models people are running. Anyone here tested for prompt-injection resistance specifically, or is this a known/expected tradeoff at this model size?\n\nRepo (free, pip install agenci): github.com/klinditafa1/agenci", "url": "https://wpnews.pro/news/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a", "canonical_source": "https://dev.to/klindi_tafa_c34dd04a92dba/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-vulnerability-in-a-local-13e2", "published_at": "2026-09-08 12:16:43+00:00", "updated_at": "2026-09-08 12:57:58.909864+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools", "ai-research"], "entities": ["Agenci", "phi4-mini", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a", "markdown": "https://wpnews.pro/news/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a.md", "text": "https://wpnews.pro/news/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a.txt", "jsonld": "https://wpnews.pro/news/i-built-a-testing-tool-for-ai-agents-it-caught-a-real-prompt-injection-in-a.jsonld"}}