Most vendor security reviews were written for tools that store data. AI agents also act: they read content nobody vetted, hold credentials, and call tools. A questionnaire that stops at "where is data hosted?" and "do you have SOC 2?" can pass a product whose agent holds one broad admin token and decides on its own when to ask a human.
Here are the questions I'd add before switching on any AI agent or AI feature in a SaaS tool, grouped by what can actually go wrong. Each comes with the answer you want and the answer that should stop the rollout.
Ask: List every action the agent can take on its own: send, delete, pay, deploy, change permissions.
Good: a complete tool list with a consequence level, and high-consequence tools off or approval-gated by default.
Red flag: "The agent can do anything the user can do."
Ask: Where is approval enforced: in code outside the model, or by the model deciding to ask?
Good: the backend checks each high-consequence call, and the model can't skip it.
Red flag: approval depends on a prompt rule or the model's judgment. OWASP's 2025 LLM guidance says controls like authorization checks should not be delegated to the model.
Ask: Are the agent's tokens scoped per integration and per task, and short-lived?
Red flag: one admin token, or the user's full OAuth scope for everything.
Ask: Does the agent act with the requesting user's permissions or a shared service identity?
Red flag: a shared superuser identity for everyone in the workspace.
Ask: How does the product handle instructions hidden in web pages, emails, files or tool output it reads?
Good: untrusted content is separated and labeled, and injected instructions can't trigger tools without approval. They test this.
Red flag: "Our model is trained to ignore malicious instructions," offered as the only control.
Ask: Can the agent leak data through links or markdown images it outputs?
Good: outbound URLs are allowlisted or stripped, and they've tested image-markdown exfiltration.
Ask: Is every agent action logged with who asked, the tool call, arguments and result, and can you export that to your SIEM?
Red flag: only chat transcripts are kept.
Ask: Can an admin disable the agent and revoke its credentials in minutes?
Red flag: it takes a support ticket.
Ask: Is your data used to train or fine-tune any model, theirs or a provider's, and where does the contract say so? Which model providers receive it, and with what retention?
Red flag: "We may use data to improve our services," and no subprocessor list.
Give each answer 0 (red flag), 1 (partial) or 2 (good), and weight the questions above as 3. A total score is useful for comparing vendors, but a 0 on any weight-3 question is a stop, no matter how good the total looks. Ask for evidence (a doc link, a DPA clause, a report scope) rather than yes or no.
I turned the full version into a ready-to-send kit: 35 questions in 10 sections (including supply chain, incident response, cost limits and contract terms), each with good-answer guidance, red flags and weights, plus a scoring sheet and a request email: AI Vendor & Agent Security Questionnaire. The questions are my own work, tagged against the OWASP Top 10 for LLM Applications 2025 (https://genai.owasp.org/llm-top-10/); they're not a standard or certification.
If you're on the other side, building the agent, the free Agent PreShip Guard GitHub Action checks a repo for some of the same issues before you ship.