llm-inference-glossary.md

wpnews.pro

A living reference for engineers building, optimizing, and serving large language models.

Last Updated: August 2026|Contributions welcome

Core Architecture & Concepts KV Cache & Memory Management Quantization Batching & Scheduling Sampling & Decoding Strategies Speculative Decoding & Draft Models Distributed Inference Kernel Optimization & Hardware Model Serving & Deployment Advanced Optimization Techniques Hardware & Accelerators Model Formats & Serialization Serving Frameworks & Runtimes Advanced Attention Variants Context Management & Long Context Model Merging & Composition Compilation & Graph Optimization Inference-Time Agents & Advanced Reasoning Observability, Cost & Operations Benchmarking & Evaluation Safety, Alignment & Inference-Time Techniques Low-Level GPU/Hardware Primitives

Term	Definition
Autoregressive Generation
The process by which LLMs generate tokens one at a time, with each new token conditioned on all previously generated tokens.
Prefill Phase
The initial forward pass that processes the input prompt to compute key-value (KV) cache entries for all prompt tokens. Also called the "context phase."
Decode Phase
The generation phase where tokens are produced one at a time, each depending on the previous token and the KV cache.
Prompt
The input text provided to the model to condition its generation.
Completion
The output text generated by the model in response to a prompt.
Token
The atomic unit of text for the model, produced by a tokenizer. Can be a word, subword, or character.
Tokenizer
The component that converts between raw text and token IDs using algorithms like BPE, WordPiece, or SentencePiece.
Embedding
A dense vector representation of a token in a high-dimensional space, learned during training.
Logits
The raw, unnormalized output scores from the final linear layer of the model, one per token in the vocabulary.
Softmax
The function that converts logits into a probability distribution over the vocabulary.
Attention Mechanism
The core operation in transformers that computes weighted sums of values based on query-key similarity scores.
Self-Attention
Attention where queries, keys, and values all come from the same sequence.
Multi-Head Attention (MHA)
Running multiple attention operations in parallel with different learned projections.
Grouped Query Attention (GQA)
A variant where multiple query heads share the same key/value heads, reducing KV cache size.
Multi-Query Attention (MQA)
An extreme form of GQA where all query heads share a single key head and single value head.
Feed-Forward Network (FFN)
The position-wise fully connected sublayer in each transformer block, typically using SwiGLU, GeGLU, or GELU activations.
Layer Normalization
Normalization technique applied per layer to stabilize training and inference.
RMSNorm
Root Mean Square Layer Normalization, a simplified variant without mean-centering, commonly used in LLaMA and Mistral.
Rotary Position Embedding (RoPE)
A position encoding method that rotates query/key vectors by a position-dependent angle, enabling better length generalization.
ALiBi (Attention with Linear Biases)
Position encoding that adds linear biases to attention scores instead of explicit embeddings.
Sliding Window Attention
Attention restricted to a fixed-size local window, used in models like Mistral to handle long contexts efficiently.
Mixture of Experts (MoE)
Architecture where only a subset of parameters (experts) are activated per token, enabling massive scale without proportional compute increase.
Top-k Routing
In MoE, selecting the k highest-scoring experts for each token.
MLA (Multi-head Latent Attention)
DeepSeek's attention mechanism that compresses KV cache via low-rank key-value joint compression.

Term	Definition
KV Cache
Stored key and value tensors from previous tokens, eliminating redundant recomputation during autoregressive generation.
KV Cache Size
Memory footprint = `2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_size` .
PagedAttention
vLLM's memory management system that partitions KV cache into fixed-size blocks (pages), enabling efficient sharing and dynamic allocation.
Block Table
A mapping from logical token positions to physical memory blocks in PagedAttention.
Copy-on-Write (CoW)
Memory optimization where KV cache blocks are shared between sequences and only copied when modified, used for beam search and parallel sampling.
Prefix Caching
Reusing KV cache computations for shared prompt prefixes across multiple requests.
KV Cache Eviction
Strategies to remove less important tokens from cache when memory is constrained (e.g., H2O, Heavy Hitter Oracle).
KV Cache Quantization
Compressing cached keys/values to lower precision (INT8, FP8, INT4) to reduce memory usage.
KV Cache Off
Moving KV cache to CPU/disk memory when GPU memory is exhausted, trading latency for capacity.
FlashAttention
IO-aware exact attention algorithm that reduces HBM reads/writes by using tiling and recomputation in SRAM.
FlashAttention-2
Improved version with better parallelism, work partitioning, and reduced non-matmul FLOPs.
FlashAttention-3
Further optimized for Hopper architecture with FP8 support, warp-specialization, and interleaved block-wise GEMMs.
FlashDecoding
Extension of FlashAttention optimized for the decode phase with small batch sizes and long sequences.
FlashInfer
A unified library for attention kernels supporting various attention variants and sparse patterns.
RadixAttention
SGLang's attention mechanism that automatically reuses KV cache across multiple calls by treating the cache as a radix tree.

Term	Definition
Quantization
Reducing numerical precision of model weights/activations to decrease memory and increase throughput.
Post-Training Quantization (PTQ)
Quantizing a pre-trained model without retraining.
Quantization-Aware Training (QAT)
Training with simulated quantization to learn robust low-precision representations.
Weight-Only Quantization
Quantizing only model weights while keeping activations in higher precision.
Weight-Activation Quantization
Quantizing both weights and activations.
INT8
8-bit integer quantization, typically using symmetric or asymmetric schemes.
INT4/INT3/INT2
Extreme weight quantization packing multiple weights per byte.
FP8 (E4M3/E5M2)
8-bit floating point formats defined in the FP8 standard (E4M3 for weights/activations, E5M2 for gradients).
NF4 (Normal Float 4)
A 4-bit data type with non-uniform bins optimized for normally distributed weights, used in QLoRA.
AWQ (Activation-Aware Weight Quantization)
PTQ method that protects salient weight channels based on activation magnitudes.
GPTQ
One-shot weight quantization using approximate second-order information for layer-wise optimal quantization.
SmoothQuant
Migration of quantization difficulty from activations to weights via per-channel scaling.
LLM.int8()
Mixed-precision decomposition that keeps outlier features in FP16 while quantizing the rest to INT8.
GGUF/GGML
File formats and quantization schemes for efficient CPU inference (Q4_0, Q5_K_M, Q8_0, etc.).
QLoRA
Quantized LoRA fine-tuning where the base model is quantized to NF4 while LoRA adapters remain in FP16.
BitsAndBytes
Library implementing LLM.int8() and 4-bit quantization for easy integration.
Dynamic Quantization
Quantizing on-the-fly during inference based on runtime activation statistics.
Static Quantization
Pre-computing quantization parameters from calibration data.
Group-wise Quantization
Applying separate quantization parameters to groups of weights (e.g., 128 consecutive weights).
Channel-wise Quantization
Separate quantization parameters per output channel.
Token-wise Quantization
Separate quantization parameters per token position for activations.

| Term | Definition |

|---|---|
Static Batching |

Grouping multiple requests into a fixed batch before inference; all requests wait for the longest one. | Dynamic Batching | Continuously adding new requests to a running batch as others complete. | Continuous Batching (In-flight Batching) | Dynamically adding/removing requests from a running batch at every iteration, maximizing GPU utilization. | Iteration-level Scheduling | Scheduling decisions made at each generation step rather than per-request. | Request Scheduling | Policies (FCFS, shortest-job-first, priority-based) for ordering incoming requests. | Preemption | Pausing a request mid-generation to free GPU resources for higher-priority requests, with state saved for resumption. | Swap-out/Swap-in | Moving KV cache of preempted requests between GPU and CPU memory. | Chunked Prefill | Breaking long prefill computations into smaller chunks and interleaving them with decode steps to prevent decode starvation. | Prompt Chunking | Processing long prompts in segments to bound prefill latency. | Microbatching | Splitting a batch into smaller sub-batches for pipeline parallelism. | Max Batch Size | The upper limit on concurrent sequences, constrained by memory and latency SLOs. | Max Sequence Length | The maximum context window supported by the model or deployment. | Token Budget | A limit on total tokens (prefill + decode) a request can consume. |

Term	Definition
Greedy Decoding
Always selecting the token with the highest probability. Deterministic but often suboptimal.
Temperature Scaling
Dividing logits by a temperature T before softmax; T < 1 makes distribution sharper (more deterministic), T > 1 makes it flatter (more random).
Top-k Sampling
Restricting sampling to the k most likely tokens, setting others' probabilities to zero.
Top-p (Nucleus) Sampling
Sampling from the smallest set of tokens whose cumulative probability exceeds p.
Min-p Sampling
Sampling from tokens with probability ≥ min_p × max_probability, adaptive to the distribution shape.
Typical Sampling
Sampling from tokens with information content close to the conditional entropy of the distribution.
Repetition Penalty
Downscaling logits of tokens that have already appeared in the sequence to reduce repetition.
Frequency Penalty
Penalizing tokens proportionally to their frequency in the generated text.
Presence Penalty
Applying a fixed penalty to tokens that have appeared at least once.
Beam Search
Maintaining k candidate sequences and expanding the most promising ones at each step.
Diverse Beam Search
Modified beam search that encourages diversity among beam candidates.
Contrastive Search
Decoding that penalizes tokens too similar to previous context, improving coherence.
Speculative Sampling
A verification-based sampling method used in speculative decoding to accept/reject draft tokens.
Watermarked Sampling
Embedding statistical watermarks into generated text for detection/attribution.
Structured Decoding / Constrained Decoding
Enforcing output to follow a grammar, regex, or JSON schema during generation.
Guided Decoding
Frameworks like Outlines, Guidance, or lm-format-enforcer for structured generation.
Logit Bias
Adding fixed values to specific token logits to encourage/discourage their selection.
EOS (End-of-Sequence) Token
Special token signaling the model to stop generating.
Stop Sequences
User-defined strings that trigger generation termination.
Max New Tokens
Hard limit on the number of tokens to generate.

Term	Definition
Speculative Decoding
Using a small draft model to predict multiple future tokens, which are then verified in parallel by the target model.
Draft Model
A smaller, faster model that generates candidate token sequences for verification.
Target Model
The full-size model that verifies draft tokens.
Verification
The process of checking draft tokens against the target model's true distribution.
Acceptance Rate
The fraction of draft tokens accepted by the target model.
Lookahead Decoding
Speculative decoding using the target model itself via Jacobi iteration, eliminating the need for a separate draft model.
Medusa
Method that adds multiple decoding heads to a model for parallel prediction of future tokens.
EAGLE
Enhanced speculative decoding using auto-regressive heads at the feature level for higher acceptance rates.
Prompt Lookup Decoding
Using n-gram matches from the prompt/context as draft tokens instead of a model.
Restoration Loss
Training objective for draft models to match the target model's distribution.
Tree Attention
Attention pattern that verifies multiple speculative token sequences simultaneously in a tree structure.

| Term | Definition |

|---|---|
Tensor Parallelism (TP) |

Splitting individual layers across multiple GPUs by partitioning weight matrices (column-wise or row-wise). | Pipeline Parallelism (PP) | Distributing consecutive layers across different GPUs, with activations passed between stages. | Data Parallelism (DP) | Replicating the model across GPUs and processing different requests on each replica. | Sequence Parallelism | Distributing the sequence dimension across GPUs, useful for long-context inference. | Expert Parallelism (EP) | In MoE models, distributing different experts across GPUs while keeping shared layers replicated. | ZeRO (Zero Redundancy Optimizer) | Memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks. | All-Reduce | Collective communication operation where tensors are summed/averaged across all GPUs. | All-Gather | Collective operation where each GPU receives the full concatenation of tensors from all GPUs. | Reduce-Scatter | Reducing and then scattering results across GPUs. | Point-to-Point Communication | Direct data transfer between specific GPU pairs, used in pipeline parallelism. | Communication Overlap | Hiding communication latency by overlapping with computation (e.g., using CUDA streams). | NCCL (NVIDIA Collective Communications Library) | Library optimized for multi-GPU collective operations. | Ray | Distributed computing framework commonly used for serving LLMs across clusters. | vLLM | High-throughput LLM serving engine featuring PagedAttention and continuous batching. | TensorRT-LLM | NVIDIA's optimized inference library with kernel fusion, quantization, and multi-GPU support. | DeepSpeed-Inference | Microsoft's inference optimization framework with ZeRO partitioning and custom kernels. | Hugging Face TGI (Text Generation Inference) | Production-ready serving stack with flash attention and safetensors support. | SGLang | Efficient serving framework with structured generation and RadixAttention. | LMDeploy | Efficient LLM serving with persistent batching and blocked KV cache. |

Term	Definition
Kernel Fusion
Combining multiple operations into a single GPU kernel to reduce kernel launch overhead and memory traffic.
Operator Fusion
Merging mathematically equivalent operations (e.g., bias-add + activation + layernorm).
Flash-Decoding
Fused attention kernel optimized for the memory-bound decode phase.
CUTLASS
NVIDIA's CUDA template library for high-performance GEMM and convolution kernels.
cuBLAS
NVIDIA's optimized BLAS library for dense linear algebra.
Triton
Open-source Python-like language from OpenAI for writing custom GPU kernels.
Marlin
Optimized INT4/INT8 GEMM kernel for weight-only quantization on Ampere+ GPUs.
FasterTransformer
NVIDIA's transformer optimization library (now largely superseded by TensorRT-LLM).
CUDA Graphs
Recording and replaying sequences of GPU operations to eliminate CPU launch overhead.
Stream-K
Work decomposition strategy for GEMMs that improves GPU utilization for small batch sizes.
Warp-specialization
Assigning different warps to different phases of computation (e.g., GEMM + softmax in FlashAttention-3).
Tensor Cores
Specialized GPU units for mixed-precision matrix multiply-accumulate operations.
Memory-Bound vs. Compute-Bound
Classification of whether performance is limited by memory bandwidth or FLOPS. Decode is typically memory-bound; prefill is compute-bound.
Arithmetic Intensity
Ratio of FLOPs to bytes of memory traffic; determines whether a kernel is compute or memory bound.
Roofline Model
Performance analysis model plotting achievable performance against arithmetic intensity.
HBM (High Bandwidth Memory)
GPU memory technology; bandwidth is often the bottleneck for inference.
L2 Cache
On-GPU cache layer between HBM and SM registers; important for FlashAttention's tiling strategy.
Shared Memory / SRAM
Fast, programmable on-chip memory within each Streaming Multiprocessor (SM).
Register Pressure
The demand for GPU registers; high pressure can limit occupancy.
Occupancy
The ratio of active warps to maximum possible warps on an SM.
Wave Quantization
Performance loss when the problem size doesn't evenly divide the GPU's wave size.

Term	Definition
Throughput
Number of tokens or requests processed per unit time (tokens/sec or requests/sec).
Latency
Time from request submission to first token (TTFT) or between consecutive tokens (TBT/ITL).
Time to First Token (TTFT)
Latency from request arrival to the first generated token. Dominated by prefill.
Time Between Tokens (TBT) / Inter-Token Latency (ITL)
Time between consecutive generated tokens. Dominated by decode.
End-to-End Latency
Total time from request to complete response.
Latency SLO
Service Level Objective specifying maximum acceptable latency.
Goodput
The rate of requests that meet their latency SLOs.
Request Rate
Incoming requests per second (RPS).
Auto-scaling
Dynamically adjusting the number of serving instances based on load.
Load Balancing
Distributing requests across multiple serving instances.
Model Sharding
Splitting a model across multiple devices for serving.
Model Replication
Creating multiple copies of a model for parallel request handling.
Disaggregated Serving
Separating prefill and decode onto different GPU pools, each optimized for their respective workload.
Prefix-Aware Routing
Routing requests to instances that already have the prefix cached.
Streaming Response
Sending generated tokens to the client as they are produced, rather than waiting for completion.
OpenAI API Compatible
Serving endpoint implementing the OpenAI chat/completions API format.
gRPC
High-performance RPC framework often used for model serving.
REST API
HTTP-based API for model inference.
Model Registry
Centralized storage for model versions and artifacts.
A/B Testing
Serving different model versions to compare performance.
Shadow Mode
Running a new model in parallel without serving its outputs, for validation.
Canary Deployment
Gradually rolling out a new model version to a subset of traffic.
KV Cache Store
External storage system for KV caches (e.g., Redis, custom stores) for multi-turn conversations.
Function Calling / Tool Use
Model capability to invoke external tools/APIs during generation.
RAG (Retrieval-Augmented Generation)
Augmenting prompts with retrieved documents to ground generation in external knowledge.

| Term | Definition |

|---|---|
LoRA (Low-Rank Adaptation) |

Fine-tuning method that trains low-rank decomposition matrices instead of full weights, enabling efficient adaptation. | QLoRA | LoRA with a quantized base model, enabling fine-tuning on consumer GPUs. | DoRA (Weight-Decomposed Low-Rank Adaptation) | Decomposing weights into magnitude and direction for more stable LoRA training. | Adapter Layers | Small bottleneck layers inserted into a frozen pretrained model for task-specific adaptation. | Prompt Tuning | Learning soft prompt embeddings while keeping the model frozen. | Prefix Tuning | Learning continuous prefix vectors prepended to keys and values in attention layers. | P-Tuning v2 | Scaling prefix tuning to deeper layers for better performance. | Distillation | Training a smaller student model to mimic a larger teacher model's behavior. | Cascade Inference | Using smaller models for easier inputs and larger models only when needed. | Early Exit | Allowing the model to exit at intermediate layers for simple inputs. | Sparse Attention | Attention patterns that skip certain token pairs (e.g., Longformer, BigBird, Ring Attention). | Ring Attention | Distributed attention for extremely long sequences by partitioning along the sequence dimension in a ring topology. | Striped Attention | Attention pattern that alternates between local and global attention. | H2O (Heavy Hitter Oracle) | KV cache eviction policy that retains tokens with high accumulated attention scores. | StreamingLLM | Enabling infinite-length generation by maintaining attention sinks (initial tokens) and a rolling KV cache. | LM-Infinite | Framework for handling infinite context via segment-based processing. | Activation Checkpointing | Recomputing activations during backward pass to save memory (relevant for training, sometimes inference). | Gradient Checkpointing | Same as activation checkpointing. | CPU Off | Moving model weights or KV cache to CPU memory when GPU memory is full. | Disk Off | Using disk storage for model weights when CPU/GPU memory is insufficient. | Model Compression | General term for techniques reducing model size (pruning, quantization, distillation). | Pruning | Removing less important weights or neurons from a model. | Structured Pruning | Removing entire structures (heads, layers, channels) for hardware efficiency. | Unstructured Pruning | Removing individual weights, requiring sparse matrix support. | SparseGPT | One-shot pruning method for GPT-style models using approximate Hessian information. | Wanda | Pruning weights based on the product of weight magnitude and activation norm. | Mamba / State Space Models (SSMs) | Alternative architectures to transformers with linear-time sequence modeling and no KV cache. | RWKV | RNN-like architecture combining transformer parallelizability with RNN-like memory efficiency. | RetNet | Architecture proposing retention mechanism as an alternative to attention for efficient decoding. | Mixture of Depths | Dynamically skipping layers for less important tokens. | LayerSkip | Training models to be robust to layer skipping for early-exit inference. |

| Term | Definition |

|---|---|
TPU (Tensor Processing Unit) |

Google's ASIC optimized for matrix operations; TPU v4/v5p/v5e used for LLM training/inference. | Groq LPU (Language Processing Unit) | Groq's deterministic tensor streaming processor with no external HBM, enabling extremely low and predictable latency. | AWS Inferentia / Trainium | Amazon's custom AI chips (Inf2, Trn1) for cost-efficient inference. | SambaNova SN40L | Reconfigurable dataflow accelerator with large on-chip memory for model weights. | Cerebras Wafer-Scale Engine | Massive wafer-scale chip designed for training and inference of extremely large models. | Qualcomm AI Stack / NPU | Mobile/edge inference on Snapdragon NPUs via ONNX/QNN. | Apple Neural Engine (ANE) | Dedicated neural processing unit in Apple Silicon for on-device inference. | NVIDIA H100/H200/Blackwell | GPU generations with Transformer Engine, FP8 support, and NVLink/NVSwitch for scale-out. | NVLink / NVSwitch | High-bandwidth interconnect for multi-GPU communication within a node. | InfiniBand / RoCE | High-speed networking for inter-node GPU clusters. | Transformer Engine | NVIDIA library and hardware feature for automatic FP8 precision management. |

Term	Definition
Safetensors
Hugging Face's secure tensor format (no pickle, memory-mappable, faster ).
Pickle / PyTorch .bin
Traditional PyTorch serialization format (security risk due to arbitrary code execution).
ONNX (Open Neural Network Exchange)
Cross-platform model format for portable inference.
TensorRT Engine
NVIDIA's compiled, optimized model format for deployment.
Core ML
Apple's format for on-device inference on iOS/macOS.
TFLite (TensorFlow Lite)
Lightweight format for mobile and edge devices.
OpenVINO IR
Intel's intermediate representation for optimized inference on Intel hardware.
GGUF (GPT-Generated Unified Format)
Successor to GGML; supports metadata, multiple quantization types, and CPU inference.

Term	Definition
llama.cpp
C/C++ implementation of LLaMA optimized for CPU and Apple Silicon inference.
Ollama
User-friendly wrapper around llama.cpp for local model running.
Triton Inference Server
NVIDIA's multi-framework serving platform with dynamic batching and model ensemble.
ONNX Runtime
Cross-platform inference engine supporting ONNX models with hardware accelerators.
OpenVINO
Intel's toolkit for optimizing and deploying deep learning models.
TorchServe / TensorFlow Serving
Framework-native serving solutions.
MLflow / BentoML
Model packaging and serving orchestration platforms.
KServe
Kubernetes-native model serving platform.
Ray Serve
Scalable model serving built on Ray distributed framework.
Modal / Beam / Replicate
Serverless GPU inference platforms.

Term	Definition
Linear Attention
Attention approximations reducing complexity from O(n²) to O(n) via kernel feature maps.
Performer
Uses FAVOR+ (Fast Attention Via Orthogonal Random Features) for linear-time attention.
RFA (Random Feature Attention)
Approximates softmax attention using random Fourier features.
CosFormer
Linear attention using cosine-based reweighting for locality bias.
Local Attention / Strided Attention
Restricting attention to local windows or fixed strides.
Dilated Attention
Skipping tokens at regular intervals within attention windows.
Factorized Attention
Decomposing full attention into multiple cheaper patterns.

Term	Definition
Context Compression
Techniques to summarize or compress long contexts into shorter representations.
Hierarchical Attention
Multi-scale attention operating on token, sentence, and paragraph levels.
Memory-Augmented Networks
External memory modules the model can read/write during generation.
MemGPT
OS-inspired virtual context management paging memory in/out of context.
LoRA-XS / LongLoRA
Efficient fine-tuning specifically for extending context windows.
YaRN (Yet another RoPE extension method)
Interpolation/extrapolation technique for extending RoPE-based models to longer contexts.
NTK-Aware Scaling
Non-linear interpolation of RoPE base frequencies for context extension.
PI (Positional Interpolation)
Linearly interpolating position indices to fit longer sequences into trained context limits.
Self-Extend
Training-free context extension by grouping positions.

Term	Definition
Model Soups
Averaging weights of multiple fine-tuned models.
Task Arithmetic
Adding/subtracting task vectors (fine-tuned minus pre-trained weights) to compose capabilities.
TIES-Merging
Trimming, electing sign, and disjoint merging to resolve interference between models.
DARE (Drop And REscale)
Dropping a large fraction of delta parameters before merging to reduce interference.
SLERP (Spherical Linear Interpolation)
Interpolating model weights on a hypersphere.
Model Breadcrumbs
Sparse task vectors for efficient model editing.

Term	Definition
torch.compile
PyTorch's JIT compilation with backends like inductor, cudagraphs, and Triton.
Inductor
PyTorch 2.0's default compiler backend generating Triton/C++ kernels.
XLA (Accelerated Linear Algebra)
Google's compiler for linear algebra, used with JAX/TensorFlow on TPUs/GPUs.
JAX
Google's composable ML framework with JIT compilation via XLA.
TVM / Apache TVM
Open-source deep learning compiler stack.
MLIR (Multi-Level Intermediate Representation)
LLVM subproject for representing and transforming ML graphs.
IREE (Intermediate Representation Execution Environment)
MLIR-based runtime for edge deployment.

| Term | Definition |

|---|---|
ReAct (Reasoning + Acting) |

Program-Aided LLMs (PAL) |
Using LLMs to generate code that solves problems programmatically. |

Term	Definition
Token Pricing
Cost model based on input (prompt) and output (completion) tokens.
Context Window Pricing
Some providers charge based on context length regardless of generation.
Batch Pricing
Discounted rates for offline/non-real-time inference.
Reserved Throughput
Guaranteed capacity provisioning at fixed cost.
Spot/Preemptible Instances
Cheaper but interruptible compute for batch inference.
Model Telemetry
Logging latency, throughput, error rates, and token counts.
Prompt Injection Detection
Guardrails to detect and block malicious prompt patterns.
Output Moderation
Real-time content filtering of generated text.
PII Redaction
Removing personally identifiable information from inputs/outputs.

| Term | Definition |

|---|---|
Tokens per Second (tok/s) |

Primary throughput metric for inference systems. | FLOPs (Floating Point Operations) | Count of arithmetic operations; used to measure computational cost. | FLOPS (Floating Point Operations Per Second) | Measure of compute throughput. | Memory Bandwidth | Rate at which data can be read from/written to GPU memory (GB/s). | Model FLOPs Utilization (MFU) | Ratio of actual throughput to theoretical peak FLOPS, accounting for memory bandwidth and overhead. | Model Memory Bandwidth Utilization (MBU) | Ratio of actual memory bandwidth used to theoretical peak. | Benchmark Suites | Standardized evaluation sets: MMLU, HellaSwag, ARC, TruthfulQA, HumanEval, GSM8K, etc. | Perplexity (PPL) | Exponential of average negative log-likelihood; measures model confidence in a text. | Long-Context Benchmarks | Needle-in-a-haystack, passkey retrieval, long-document QA for evaluating context window effectiveness. | LMBench / LLMPerf | Benchmarking frameworks for measuring serving performance. | AnyScale LLMPerf | Open-source benchmark for LLM serving systems. | Artificial Load Testing | Generating synthetic request patterns to stress-test serving infrastructure. |

Term	Definition
System Prompt
Instructions prepended to every conversation to guide model behavior.
Jailbreaking
Prompt engineering techniques designed to bypass safety guardrails.
Prompt Injection
Attacks that embed malicious instructions within user input to manipulate model behavior.
RLHF (Reinforcement Learning from Human Feedback)
Training method using human preferences to align model outputs.
DPO (Direct Preference Optimization)
Simpler alternative to RLHF that directly optimizes on preference pairs without a separate reward model.
Constitutional AI
Training models using a set of principles (constitution) for self-correction.
Inference-Time Compute Scaling

Allocating more computation at inference (e.g., Chain-of-Thought, best-of-N sampling) to improve output quality. |
Chain-of-Thought (CoT) |

Prompting the model to show intermediate reasoning steps. | Tree of Thoughts (ToT) | Generalizing CoT to explore multiple reasoning paths and backtrack. | Self-Consistency | Generating multiple reasoning paths and selecting the most frequent answer. | Best-of-N Sampling | Generating N candidates and selecting the best via a reward model or verifier. | Process Reward Model (PRM) | Model that scores individual reasoning steps rather than final outputs. | Outcome Reward Model (ORM) | Model that scores final outputs. | Verifier | Model trained to judge correctness of generated outputs. | Majority Voting | Aggregating multiple samples by selecting the most common answer. |

Monte Carlo Tree Search (MCTS) |
Using MCTS to explore the reasoning space during inference. |

| Term | Definition |

|---|---|
TMA (Tensor Memory Accelerator) |

Hardware unit in Hopper for asynchronous tensor transfers. | WGMMA (Warp Group Matrix Multiply-Accumulate) | Hopper's warp-group-level MMA instruction for higher throughput. | PTX (Parallel Thread Execution) | NVIDIA's intermediate instruction set architecture for CUDA. | SASS | NVIDIA's machine code (binary instructions) executed by the GPU. | Warp / Warp Group | A group of 32 threads executing in SIMT fashion; warp groups contain 4 warps (128 threads) on Hopper. | Thread Block / Cluster | Groups of threads/warps for cooperative execution. | Asynchronous Copy | Overlapping data transfer with computation using cp.async instructions. |

This glossary is a living document. The field of LLM inference engineering evolves weekly — new terms, techniques, and hardware primitives are constantly emerging. Last compiled: August 2026.

source & further reading

gist.github.com — original article 12 Rules for Claude.md (Example) Run a 4B LLM model locally on your PC, this has been tested on Windows 10 Efektif AGENTS.md oluşturma prompt

llm-inference-glossary.md

Run your AI side-project on zahid.host