The Only GenAI Security Glossary You’ll Actually Need

A new GenAI security glossary compiles the core vocabulary and attack/defense terms used in real design reviews, red-team reports, and the OWASP, NIST, and MITRE frameworks. The glossary covers concepts including tokens, context windows, embeddings, retrieval-augmented generation, fine-tuning, agents, system prompts, hallucinations, and the Model Context Protocol, and notes that prompt injection ranks first on the OWASP Top 10 for LLM Applications 2025.

Every AI security conversation you’ll walk into this year already assumes you know this vocabulary. Someone will say “that’s a confused deputy problem,” or “did you check for the lethal trifecta,” or “this is basically excessive agency,” and if you have to nod along without really knowing, you lose the room. Not because the ideas are hard, but because nobody handed you the dictionary. This is that dictionary. This is not an exhaustive AI encyclopedia , but the practical subset: enough core GenAI vocabulary to follow the conversation, plus every security/attack/defense term that actually shows up in real design reviews, red-team reports, and the OWASP/NIST/MITRE frameworks people cite in the room. Each entry is deliberately short, covering what it is and why it matters or how it gets attacked, so you can read this once, keep it as a reference, and never again be the person in the meeting quietly Googling a term under the table. The smallest unit an LLM reads or writes; not quite a word, not quite a character. Cost, context-window limits, and several attacks token smuggling, homoglyph evasion are all denominated in tokens, so understanding tokenization is a prerequisite to understanding filter bypasses. The total tokens prompt + history + retrieved data + output a model can attend to in one call. It’s also the single trust boundary attackers exploit hardest anything that lands inside it, from any source, gets treated with equal authority by the model. A dense vector representation of text/image/audio capturing semantic meaning, used for search, RAG retrieval, and similarity scoring. Embeddings can leak the original content back out embedding inversion and are rarely encrypted at rest in vector stores, making them an underrated data-exfiltration surface. Fetching relevant external documents at query time and injecting them into the prompt so the model answers with current, grounded data instead of relying solely on training-time knowledge. Every retrieved chunk is untrusted input by default — this is the single biggest on-ramp for indirect prompt injection. Techniques for adapting a base model’s weights to a task or aligning its behavior to preferences, ranging from full retraining to lightweight adapter layers LoRA to preference optimization RLHF/DPO . A few hundred poisoned training examples can implant a backdoor that survives normal evals — the attack surface moves from “what you send the model” to “what shaped the model.” A system that loops perceive → reason → act, calling tools, retaining memory, and making multi-step decisions with limited human-in-the-loop review. This is where GenAI security stops being theoretical: an agent with tool access can do things, not just say things — the blast radius of a single successful attack jumps from “bad text” to “wire transfer” or “code execution.” The instructions set by the application developer that define the model’s role, boundaries, and behavior, prepended before user input. It is not a security boundary — it’s a strong suggestion the model weighs alongside everything else in context, which is why “just tell it not to” never survives red-teaming. A model generating fluent, confident output that is factually wrong or unsupported by its inputs. Security-relevant because attackers exploit this deliberately slopsquatting: hallucinated package names later registered as malware and because ungrounded output in an agentic pipeline can trigger real actions on false premises. An open standard letting an LLM/agent discover and call external tools and data sources through a common interface, replacing bespoke per-tool integrations. It also standardizes a new trust boundary, a malicious or compromised MCP server can inject instructions via tool descriptions, not just tool outputs. Crafting input that overrides or hijacks a model’s instructions, causing it to ignore its system prompt or developer intent. It’s 1 on the OWASP Top 10 for LLM Applications 2025/2026 because it’s the root cause behind most downstream failures i.e. jailbreaks, data leaks, and unauthorized tool calls all typically start here. Direct injection comes from the user’s own input; indirect injection is planted in third-party content the model later retrieves or reads a webpage, a document, an email, a tool result . Indirect injection is the harder problem, the victim never typed the malicious instruction, so input filtering on the user-facing side does nothing. A prompt engineered to bypass a model’s safety training and produce content or actions it was aligned to refuse role-play personas, “DAN”-style framing, multi-turn erosion . Jailbreaks target the model’s alignment layer specifically, distinct from prompt injection which targets the application’s instruction-following — the two are often chained together. Extracting the hidden system prompt itself, usually via “repeat everything above” or translation/encoding tricks. Beyond embarrassment, a leaked system prompt often reveals internal tool names, business logic, and guardrail wording an attacker can then reverse-engineer around. Splitting or encoding a malicious instruction base64, unicode homoglyphs, zero-width characters, translation round-trips so it evades keyword-based input filters but still gets parsed correctly by the model’s tokenizer. It’s the reason regex/keyword guardrails alone are considered close to worthless against a motivated adversary. Building up to a disallowed request across several benign-looking turns rather than asking directly, exploiting the model’s tendency to maintain conversational consistency. Guardrails scoped to a single message miss this entirely — detection has to reason over the whole session. A string of seemingly gibberish tokens appended to a prompt, discovered via gradient-based optimization e.g., GCG , that reliably breaks a model’s alignment. Notable because it’s an automated, transferable attack; one found against an open-weight model often works against closed commercial models too. Treating LLM output as trusted and passing it downstream unsanitized — into a SQL query, a shell command, rendered HTML, or another tool call. This is the classic injection vulnerability class wearing an AI costume; the fix is the same as always parameterize, sandbox, encode , just easier to forget because “the model wrote it.” Rendering model output directly into a webpage or chat UI without escaping, letting an attacker’s injected prompt produce a script can’t use tag as medium won’t allow it . payload that executes in the victim’s browser. It’s a normal XSS finding whose payload delivery mechanism happens to be an LLM. Granting an agent more permissions, tool access, or autonomy than the task requires, so a single successful manipulation prompt injection, tool poisoning can cascade into a high-impact action. It jumped to 3 on the 2026 OWASP LLM Top 10 precisely because agentic deployments keep shipping with broad, unscoped tool grants “to be safe” — the opposite of what actually helps. In multi-agent or multi-step pipelines, one agent’s hallucinated output becomes another agent’s trusted input, compounding the error at each hop until the final action is confidently wrong. Traditional single-turn evals don’t catch this — you have to test the pipeline, not the model. Injecting false or malicious content into an agent’s persistent memory episodic, semantic, or vector store so it influences future sessions long after the original attack input is gone. This defeats the usual mitigation of “just don’t trust this one input”, the poison is now part of the agent’s ground truth. Manipulating a tool’s description, documentation, or response so the agent is tricked into calling it with attacker-chosen parameters, or manipulating the agent into misusing a legitimate tool outside its intended purpose. Since agents select tools based on natural-language descriptions, a poisoned description is effectively a prompt injection with a name and a permission set attached. An agent acting with its own high privileges on behalf of a lower-privileged or malicious requester, because it can’t distinguish “the user asked me to do X” from “content I read told me to do X.” This is the signature agentic-AI failure mode where traditional IAM’s one-principal-one-session model has no answer for an agent wearing multiple hats in one call. An agent taking a harmful, unauthorized, or out-of-scope action autonomously, without a human ever approving that specific step, typically from an accumulated chain of smaller manipulations. It’s the reason “human in the loop” gets redesigned as “human at the checkpoints that matter” — full-time supervision doesn’t scale to agent speed. Exploiting the trust between cooperating agents in a multi-agent system — one compromised or malicious agent feeding poisoned instructions to its peers, who treat peer messages as authoritative. Multi-agent architectures multiply the attack surface faster than they multiply the defenses, since most guardrail tooling was built for a single agent talking to a single human. The specific, high-risk combination of 1 access to private/sensitive data, 2 exposure to untrusted external content, and 3 the ability to communicate externally send email, make API calls, post data all present in the same agent. When all three hold simultaneously, indirect prompt injection stops being a nuisance and becomes a direct data-exfiltration path; removing any one leg breaks the chain. The AI-era denial-of-service: an attacker or a runaway agent loop driving token consumption up until the API bill, not the server, is what falls over. Because inference is metered per token, a cost-spike alert is often the earliest signal that an agent has been hijacked or is stuck in a loop — earlier than any functional error. An agent stuck re-planning, re-calling tools, or retrying without a termination condition, burning cost and potentially taking repeated unwanted actions each cycle. It’s a design failure in the perceive-plan-act-reflect loop, not an attack per se, but attackers can deliberately trigger one to run up cost or exhaust rate limits protecting other tenants. Corrupting training, fine-tuning, or RAG-ingested data so the model learns a wrong fact, a backdoor trigger, or a biased behavior. Distinguish from prompt injection: poisoning corrupts the knowledge , injection corrupts a single interaction — poisoning is patient and persistent, injection is per-request. Querying a deployed model in ways that reconstruct or infer specifics of its training data, including memorized PII, from ordinary-looking authorized access. It’s why “the model never saw the raw data after training” is not the same claim as “the data can’t leak” — regurgitation and membership inference are real, measured phenomena, not hypotheticals. Determining whether a specific record was part of a model’s training set, purely by observing its outputs/confidence on that record versus unseen data. A privacy violation in its own right e.g., “was this patient’s record used to train this model” even without extracting the record’s content. Reconstructing the original text or something close enough to be sensitive from its vector embedding alone. Since embeddings are often stored, cached, and shared more loosely than raw text “it’s just numbers” , this closes a leakage path teams frequently assume doesn’t exist. A multi-tenant RAG system retrieving and surfacing another tenant’s documents due to broken metadata filtering, shared indexes, or missing row-level security in the vector store. It’s the AI-era version of a classic IDOR/broken-access-control bug, except the “record” is a semantic match, not a primary key, which makes it easier to miss in testing. An LLM fabricating or misattributing a source citation that looks authoritative, either from hallucination or from an attacker deliberately planting fake-but-convincing “sources” in retrievable content. Dangerous specifically because citations are the UX signal users are trained to trust as proof of grounding. Exploiting insecure deserialization in model checkpoint formats .pkl, .pt, unsafe .bin to achieve arbitrary code execution the moment a victim loads the file — no inference required. This is why pickle-based formats are being phased out in favor of safetensors, which stores only tensor data and cannot execute code on load. Compromise introduced anywhere upstream of your deployment: a poisoned pretrained model on a public hub, a typosquatted model/package name, a compromised training dataset, or a malicious dependency in the training pipeline. Model provenance and signing comparable to SBOM/SLSA for software is the emerging control, often called an MLBOM Model/ML Bill of Materials . Theft of a trained model’s weights themselves, treated as the crown-jewel IP asset for frontier labs — distinct from stealing training data or extracting behavior via queries. It’s a top concern in frontier-AI governance because weights, once out, can’t be revoked or patched the way a compromised credential can. An input, output, or tool-call filter classifier, regex, policy engine, or a second LLM that checks content against a safety/policy boundary before it’s acted on. Critically, a guardrail is a probabilistic filter , not a security boundary. It reduces likelihood, it doesn’t guarantee prevention, and treating it as the latter is a design mistake that shows up in almost every agentic incident post-mortem. Using one LLM to score or evaluate another model’s or its own output against a rubric, at a scale human review can’t match. It introduces its own attack surface, judge prompt injection, where content in the response being graded manipulates the judge into awarding a passing score it shouldn’t. A curated, versioned set of test inputs with known-good expected outputs or rubrics used to regression-test model or prompt changes before shipping. Replaces “vibes-based” manual spot-checking with a repeatable gate . The AI-era equivalent of a unit-test suite, and just as neglected under deadline pressure. Structured, adversarial testing of a model or system by people or automated agents deliberately trying to break its safety, security, or policy boundaries before real attackers do. OWASP’s GenAI Red-Teaming Guide and NIST’s AI RMF both treat this as a required, recurring practice, not a one-time pre-launch checkbox. Extending “never trust, always verify” to agent-to-agent and agent-to-tool calls: every action is authenticated, authorized, and scoped per-call rather than inherited once from a long-lived session. It directly counters the confused-deputy problem by refusing to let an agent’s identity silently stand in for the user’s. The architectural split between deciding whether an agent’s action is allowed PDP, evaluated against policy and enforcing that decision at the point the action would execute PEP . This is what turns “the system prompt said not to” into an actual technical control an agent can’t reason its way around. A manifest of everything that went into a model or AI system like datasets, base model, fine-tuning data, dependencies, evaluation results that enabling provenance checks and incident scoping. It answers “were we exposed by that compromised model on HuggingFace” in minutes instead of days. A unique, silent marker embedded in a system prompt or document specifically so that if it later appears in model output or an exfiltration channel, you have proof the content was leaked or the prompt was extracted. Cheap, low-effort detection control for prompt leaking and indirect-injection exfiltration. An emerging CVSS-style scoring framework specifically for AI/agentic risks, weighting factors like autonomy level and blast radius that CVSS’s software-vulnerability model doesn’t capture. Useful because a plain CVSS score consistently under-rates agentic findings . It has no field for “this agent can independently take five more actions after the initial compromise.” The most cited LLM-specific risk taxonomy prompt injection, sensitive information disclosure, supply chain, excessive agency, improper output handling, etc. , now in active 2025/2026 revision. The default lingua franca for describing an LLM application finding to another security team. Extensions of the above for agent-specific risks memory poisoning, tool misuse, rogue actions and for the newer “skills” layer packaged instructions + code that inherit the host agent’s privileges . Use these when the finding is about autonomy and delegation , not just single-turn model behavior. The protocol-specific risk list for Model Context Protocol deployments: malicious server trust, tool-description injection, credential leakage through the protocol layer. Relevant the moment your architecture includes any MCP server, first-party or third-party. A knowledge base of adversary tactics and techniques specifically against AI/ML systems, structured like ATT&CK but for model extraction, evasion, poisoning, and inference attacks. Use it when you need attacker TTPs, not just risk categories — it maps to how a real campaign against an AI system would actually unfold. The U.S. government’s voluntary risk-management framework Govern, Map, Measure, Manage for trustworthy AI, with AI 600–1 as the generative-AI-specific profile. The framework of choice when the audience is governance/risk rather than engineering — it speaks in organizational controls, not attack payloads. The international management-system standard for AI, structured like ISO 27001 but for AI governance i.e. policy, risk assessment, lifecycle controls, auditability. Increasingly what enterprise customers and regulators ask for as certifiable proof of a mature AI governance program. The Cloud Security Alliance’s threat-modeling framework purpose-built for multi-agent/agentic systems, enumerating threats layer by layer through the agent stack and across agent-to-agent trust boundaries. Reach for this instead of plain STRIDE when the system under review is the multi-agent orchestration, not a single model behind an API. A practitioner-oriented control framework mapping security controls to specific stages of the AI/ML lifecycle data, model, serving, monitoring , useful for platform teams operationalizing the above frameworks into concrete checklists. The Only GenAI Security Glossary You’ll Actually Need https://pub.towardsai.net/the-only-genai-security-glossary-youll-actually-need-bf29c679010b was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.