# Your vendor review asks where the data lives. It doesn't ask what the agent can do.

> Source: <https://dev.to/setrov/your-vendor-review-asks-where-the-data-lives-it-doesnt-ask-what-the-agent-can-do-13bc>
> Published: 2026-10-29 10:09:36+00:00

Most vendor security reviews were written for tools that store data. AI agents also *act*: they read content nobody vetted, hold credentials, and call tools. A questionnaire that stops at "where is data hosted?" and "do you have SOC 2?" can pass a product whose agent holds one broad admin token and decides on its own when to ask a human.

Here are the questions I'd add before switching on any AI agent or AI feature in a SaaS tool, grouped by what can actually go wrong. Each comes with the answer you want and the answer that should stop the rollout.

**Ask:** List every action the agent can take on its own: send, delete, pay, deploy, change permissions.

**Good:** a complete tool list with a consequence level, and high-consequence tools off or approval-gated by default.

**Red flag:** "The agent can do anything the user can do."

**Ask:** Where is approval enforced: in code outside the model, or by the model deciding to ask?

**Good:** the backend checks each high-consequence call, and the model can't skip it.

**Red flag:** approval depends on a prompt rule or the model's judgment. OWASP's 2025 LLM guidance says controls like authorization checks should not be delegated to the model.

**Ask:** Are the agent's tokens scoped per integration and per task, and short-lived?

**Red flag:** one admin token, or the user's full OAuth scope for everything.

**Ask:** Does the agent act with the requesting user's permissions or a shared service identity?

**Red flag:** a shared superuser identity for everyone in the workspace.

**Ask:** How does the product handle instructions hidden in web pages, emails, files or tool output it reads?

**Good:** untrusted content is separated and labeled, and injected instructions can't trigger tools without approval. They test this.

**Red flag:** "Our model is trained to ignore malicious instructions," offered as the only control.

**Ask:** Can the agent leak data through links or markdown images it outputs?

**Good:** outbound URLs are allowlisted or stripped, and they've tested image-markdown exfiltration.

**Ask:** Is every agent action logged with who asked, the tool call, arguments and result, and can you export that to your SIEM?

**Red flag:** only chat transcripts are kept.

**Ask:** Can an admin disable the agent and revoke its credentials in minutes?

**Red flag:** it takes a support ticket.

**Ask:** Is your data used to train or fine-tune any model, theirs or a provider's, and where does the contract say so? Which model providers receive it, and with what retention?

**Red flag:** "We may use data to improve our services," and no subprocessor list.

Give each answer 0 (red flag), 1 (partial) or 2 (good), and weight the questions above as 3. A total score is useful for comparing vendors, but a 0 on any weight-3 question is a stop, no matter how good the total looks. Ask for evidence (a doc link, a DPA clause, a report scope) rather than yes or no.

I turned the full version into a ready-to-send kit: 35 questions in 10 sections (including supply chain, incident response, cost limits and contract terms), each with good-answer guidance, red flags and weights, plus a scoring sheet and a request email: [AI Vendor & Agent Security Questionnaire](https://setrov.gumroad.com/l/hmyoz). The questions are my own work, tagged against the OWASP Top 10 for LLM Applications 2025 ([https://genai.owasp.org/llm-top-10/](https://genai.owasp.org/llm-top-10/)); they're not a standard or certification.

If you're on the other side, building the agent, the free [Agent PreShip Guard](https://github.com/elaysuu/agent-preship-guard) GitHub Action checks a repo for some of the same issues before you ship.