{"slug": "your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can", "title": "Your vendor review asks where the data lives. It doesn't ask what the agent can do.", "summary": "A developer published a 35-question AI vendor and agent security questionnaire that extends standard vendor reviews beyond data-hosting and SOC 2 checks to cover what an agent can actually do, including its tool permissions, approval enforcement, token scoping, prompt-injection handling, and audit logging. The kit scores each answer 0-2 with weight-3 questions, where any red flag on a high-weight item halts the rollout, and the questions are tagged against the OWASP Top 10 for LLM Applications 2025. The author also released a free GitHub Action, Agent PreShip Guard, that checks repositories for some of the same issues before shipping.", "body_md": "Most vendor security reviews were written for tools that store data. AI agents also *act*: they read content nobody vetted, hold credentials, and call tools. A questionnaire that stops at \"where is data hosted?\" and \"do you have SOC 2?\" can pass a product whose agent holds one broad admin token and decides on its own when to ask a human.\n\nHere are the questions I'd add before switching on any AI agent or AI feature in a SaaS tool, grouped by what can actually go wrong. Each comes with the answer you want and the answer that should stop the rollout.\n\n**Ask:** List every action the agent can take on its own: send, delete, pay, deploy, change permissions.\n\n**Good:** a complete tool list with a consequence level, and high-consequence tools off or approval-gated by default.\n\n**Red flag:** \"The agent can do anything the user can do.\"\n\n**Ask:** Where is approval enforced: in code outside the model, or by the model deciding to ask?\n\n**Good:** the backend checks each high-consequence call, and the model can't skip it.\n\n**Red flag:** approval depends on a prompt rule or the model's judgment. OWASP's 2025 LLM guidance says controls like authorization checks should not be delegated to the model.\n\n**Ask:** Are the agent's tokens scoped per integration and per task, and short-lived?\n\n**Red flag:** one admin token, or the user's full OAuth scope for everything.\n\n**Ask:** Does the agent act with the requesting user's permissions or a shared service identity?\n\n**Red flag:** a shared superuser identity for everyone in the workspace.\n\n**Ask:** How does the product handle instructions hidden in web pages, emails, files or tool output it reads?\n\n**Good:** untrusted content is separated and labeled, and injected instructions can't trigger tools without approval. They test this.\n\n**Red flag:** \"Our model is trained to ignore malicious instructions,\" offered as the only control.\n\n**Ask:** Can the agent leak data through links or markdown images it outputs?\n\n**Good:** outbound URLs are allowlisted or stripped, and they've tested image-markdown exfiltration.\n\n**Ask:** Is every agent action logged with who asked, the tool call, arguments and result, and can you export that to your SIEM?\n\n**Red flag:** only chat transcripts are kept.\n\n**Ask:** Can an admin disable the agent and revoke its credentials in minutes?\n\n**Red flag:** it takes a support ticket.\n\n**Ask:** Is your data used to train or fine-tune any model, theirs or a provider's, and where does the contract say so? Which model providers receive it, and with what retention?\n\n**Red flag:** \"We may use data to improve our services,\" and no subprocessor list.\n\nGive each answer 0 (red flag), 1 (partial) or 2 (good), and weight the questions above as 3. A total score is useful for comparing vendors, but a 0 on any weight-3 question is a stop, no matter how good the total looks. Ask for evidence (a doc link, a DPA clause, a report scope) rather than yes or no.\n\nI turned the full version into a ready-to-send kit: 35 questions in 10 sections (including supply chain, incident response, cost limits and contract terms), each with good-answer guidance, red flags and weights, plus a scoring sheet and a request email: [AI Vendor & Agent Security Questionnaire](https://setrov.gumroad.com/l/hmyoz). The questions are my own work, tagged against the OWASP Top 10 for LLM Applications 2025 ([https://genai.owasp.org/llm-top-10/](https://genai.owasp.org/llm-top-10/)); they're not a standard or certification.\n\nIf you're on the other side, building the agent, the free [Agent PreShip Guard](https://github.com/elaysuu/agent-preship-guard) GitHub Action checks a repo for some of the same issues before you ship.", "url": "https://wpnews.pro/news/your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can", "canonical_source": "https://dev.to/setrov/your-vendor-review-asks-where-the-data-lives-it-doesnt-ask-what-the-agent-can-do-13bc", "published_at": "2026-10-29 10:09:36+00:00", "updated_at": "2026-10-29 10:17:21.084723+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools"], "entities": ["OWASP", "Agent PreShip Guard", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can", "markdown": "https://wpnews.pro/news/your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can.md", "text": "https://wpnews.pro/news/your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can.txt", "jsonld": "https://wpnews.pro/news/your-vendor-review-asks-where-the-data-lives-it-doesn-t-ask-what-the-agent-can.jsonld"}}