# llm-inference-glossary.md > Source: > Published: 2026-08-06 09:32:03+00:00 A living reference for engineers building, optimizing, and serving large language models. Last Updated: August 2026|Contributions welcome [Core Architecture & Concepts](#1-core-architecture--concepts)[KV Cache & Memory Management](#2-kv-cache--memory-management)[Quantization](#3-quantization)[Batching & Scheduling](#4-batching--scheduling)[Sampling & Decoding Strategies](#5-sampling--decoding-strategies)[Speculative Decoding & Draft Models](#6-speculative-decoding--draft-models)[Distributed Inference](#7-distributed-inference)[Kernel Optimization & Hardware](#8-kernel-optimization--hardware)[Model Serving & Deployment](#9-model-serving--deployment)[Advanced Optimization Techniques](#10-advanced-optimization-techniques)[Hardware & Accelerators](#11-hardware--accelerators)[Model Formats & Serialization](#12-model-formats--serialization)[Serving Frameworks & Runtimes](#13-serving-frameworks--runtimes)[Advanced Attention Variants](#14-advanced-attention-variants)[Context Management & Long Context](#15-context-management--long-context)[Model Merging & Composition](#16-model-merging--composition)[Compilation & Graph Optimization](#17-compilation--graph-optimization)[Inference-Time Agents & Advanced Reasoning](#18-inference-time-agents--advanced-reasoning)[Observability, Cost & Operations](#19-observability-cost--operations)[Benchmarking & Evaluation](#20-benchmarking--evaluation)[Safety, Alignment & Inference-Time Techniques](#21-safety-alignment--inference-time-techniques)[Low-Level GPU/Hardware Primitives](#22-low-level-gpuhardware-primitives) | Term | Definition | |---|---| Autoregressive Generation | The process by which LLMs generate tokens one at a time, with each new token conditioned on all previously generated tokens. | Prefill Phase | The initial forward pass that processes the input prompt to compute key-value (KV) cache entries for all prompt tokens. Also called the "context phase." | Decode Phase | The generation phase where tokens are produced one at a time, each depending on the previous token and the KV cache. | Prompt | The input text provided to the model to condition its generation. | Completion | The output text generated by the model in response to a prompt. | Token | The atomic unit of text for the model, produced by a tokenizer. Can be a word, subword, or character. | Tokenizer | The component that converts between raw text and token IDs using algorithms like BPE, WordPiece, or SentencePiece. | Embedding | A dense vector representation of a token in a high-dimensional space, learned during training. | Logits | The raw, unnormalized output scores from the final linear layer of the model, one per token in the vocabulary. | Softmax | The function that converts logits into a probability distribution over the vocabulary. | Attention Mechanism | The core operation in transformers that computes weighted sums of values based on query-key similarity scores. | Self-Attention | Attention where queries, keys, and values all come from the same sequence. | Multi-Head Attention (MHA) | Running multiple attention operations in parallel with different learned projections. | Grouped Query Attention (GQA) | A variant where multiple query heads share the same key/value heads, reducing KV cache size. | Multi-Query Attention (MQA) | An extreme form of GQA where all query heads share a single key head and single value head. | Feed-Forward Network (FFN) | The position-wise fully connected sublayer in each transformer block, typically using SwiGLU, GeGLU, or GELU activations. | Layer Normalization | Normalization technique applied per layer to stabilize training and inference. | RMSNorm | Root Mean Square Layer Normalization, a simplified variant without mean-centering, commonly used in LLaMA and Mistral. | Rotary Position Embedding (RoPE) | A position encoding method that rotates query/key vectors by a position-dependent angle, enabling better length generalization. | ALiBi (Attention with Linear Biases) | Position encoding that adds linear biases to attention scores instead of explicit embeddings. | Sliding Window Attention | Attention restricted to a fixed-size local window, used in models like Mistral to handle long contexts efficiently. | Mixture of Experts (MoE) | Architecture where only a subset of parameters (experts) are activated per token, enabling massive scale without proportional compute increase. | Top-k Routing | In MoE, selecting the k highest-scoring experts for each token. | MLA (Multi-head Latent Attention) | DeepSeek's attention mechanism that compresses KV cache via low-rank key-value joint compression. | | Term | Definition | |---|---| KV Cache | Stored key and value tensors from previous tokens, eliminating redundant recomputation during autoregressive generation. | KV Cache Size | Memory footprint = `2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_size` . | PagedAttention | vLLM's memory management system that partitions KV cache into fixed-size blocks (pages), enabling efficient sharing and dynamic allocation. | Block Table | A mapping from logical token positions to physical memory blocks in PagedAttention. | Copy-on-Write (CoW) | Memory optimization where KV cache blocks are shared between sequences and only copied when modified, used for beam search and parallel sampling. | Prefix Caching | Reusing KV cache computations for shared prompt prefixes across multiple requests. | KV Cache Eviction | Strategies to remove less important tokens from cache when memory is constrained (e.g., H2O, Heavy Hitter Oracle). | KV Cache Quantization | Compressing cached keys/values to lower precision (INT8, FP8, INT4) to reduce memory usage. | KV Cache Offloading | Moving KV cache to CPU/disk memory when GPU memory is exhausted, trading latency for capacity. | FlashAttention | IO-aware exact attention algorithm that reduces HBM reads/writes by using tiling and recomputation in SRAM. | FlashAttention-2 | Improved version with better parallelism, work partitioning, and reduced non-matmul FLOPs. | FlashAttention-3 | Further optimized for Hopper architecture with FP8 support, warp-specialization, and interleaved block-wise GEMMs. | FlashDecoding | Extension of FlashAttention optimized for the decode phase with small batch sizes and long sequences. | FlashInfer | A unified library for attention kernels supporting various attention variants and sparse patterns. | RadixAttention | SGLang's attention mechanism that automatically reuses KV cache across multiple calls by treating the cache as a radix tree. | | Term | Definition | |---|---| Quantization | Reducing numerical precision of model weights/activations to decrease memory and increase throughput. | Post-Training Quantization (PTQ) | Quantizing a pre-trained model without retraining. | Quantization-Aware Training (QAT) | Training with simulated quantization to learn robust low-precision representations. | Weight-Only Quantization | Quantizing only model weights while keeping activations in higher precision. | Weight-Activation Quantization | Quantizing both weights and activations. | INT8 | 8-bit integer quantization, typically using symmetric or asymmetric schemes. | INT4/INT3/INT2 | Extreme weight quantization packing multiple weights per byte. | FP8 (E4M3/E5M2) | 8-bit floating point formats defined in the FP8 standard (E4M3 for weights/activations, E5M2 for gradients). | NF4 (Normal Float 4) | A 4-bit data type with non-uniform bins optimized for normally distributed weights, used in QLoRA. | AWQ (Activation-Aware Weight Quantization) | PTQ method that protects salient weight channels based on activation magnitudes. | GPTQ | One-shot weight quantization using approximate second-order information for layer-wise optimal quantization. | SmoothQuant | Migration of quantization difficulty from activations to weights via per-channel scaling. | LLM.int8() | Mixed-precision decomposition that keeps outlier features in FP16 while quantizing the rest to INT8. | GGUF/GGML | File formats and quantization schemes for efficient CPU inference (Q4_0, Q5_K_M, Q8_0, etc.). | QLoRA | Quantized LoRA fine-tuning where the base model is quantized to NF4 while LoRA adapters remain in FP16. | BitsAndBytes | Library implementing LLM.int8() and 4-bit quantization for easy integration. | Dynamic Quantization | Quantizing on-the-fly during inference based on runtime activation statistics. | Static Quantization | Pre-computing quantization parameters from calibration data. | Group-wise Quantization | Applying separate quantization parameters to groups of weights (e.g., 128 consecutive weights). | Channel-wise Quantization | Separate quantization parameters per output channel. | Token-wise Quantization | Separate quantization parameters per token position for activations. | | Term | Definition | |---|---| Static Batching | Grouping multiple requests into a fixed batch before inference; all requests wait for the longest one. | Dynamic Batching | Continuously adding new requests to a running batch as others complete. | Continuous Batching (In-flight Batching) | Dynamically adding/removing requests from a running batch at every iteration, maximizing GPU utilization. | Iteration-level Scheduling | Scheduling decisions made at each generation step rather than per-request. | Request Scheduling | Policies (FCFS, shortest-job-first, priority-based) for ordering incoming requests. | Preemption | Pausing a request mid-generation to free GPU resources for higher-priority requests, with state saved for resumption. | Swap-out/Swap-in | Moving KV cache of preempted requests between GPU and CPU memory. | Chunked Prefill | Breaking long prefill computations into smaller chunks and interleaving them with decode steps to prevent decode starvation. | Prompt Chunking | Processing long prompts in segments to bound prefill latency. | Microbatching | Splitting a batch into smaller sub-batches for pipeline parallelism. | Max Batch Size | The upper limit on concurrent sequences, constrained by memory and latency SLOs. | Max Sequence Length | The maximum context window supported by the model or deployment. | Token Budget | A limit on total tokens (prefill + decode) a request can consume. | | Term | Definition | |---|---| Greedy Decoding | Always selecting the token with the highest probability. Deterministic but often suboptimal. | Temperature Scaling | Dividing logits by a temperature T before softmax; T < 1 makes distribution sharper (more deterministic), T > 1 makes it flatter (more random). | Top-k Sampling | Restricting sampling to the k most likely tokens, setting others' probabilities to zero. | Top-p (Nucleus) Sampling | Sampling from the smallest set of tokens whose cumulative probability exceeds p. | Min-p Sampling | Sampling from tokens with probability ≥ min_p × max_probability, adaptive to the distribution shape. | Typical Sampling | Sampling from tokens with information content close to the conditional entropy of the distribution. | Repetition Penalty | Downscaling logits of tokens that have already appeared in the sequence to reduce repetition. | Frequency Penalty | Penalizing tokens proportionally to their frequency in the generated text. | Presence Penalty | Applying a fixed penalty to tokens that have appeared at least once. | Beam Search | Maintaining k candidate sequences and expanding the most promising ones at each step. | Diverse Beam Search | Modified beam search that encourages diversity among beam candidates. | Contrastive Search | Decoding that penalizes tokens too similar to previous context, improving coherence. | Speculative Sampling | A verification-based sampling method used in speculative decoding to accept/reject draft tokens. | Watermarked Sampling | Embedding statistical watermarks into generated text for detection/attribution. | Structured Decoding / Constrained Decoding | Enforcing output to follow a grammar, regex, or JSON schema during generation. | Guided Decoding | Frameworks like Outlines, Guidance, or lm-format-enforcer for structured generation. | Logit Bias | Adding fixed values to specific token logits to encourage/discourage their selection. | EOS (End-of-Sequence) Token | Special token signaling the model to stop generating. | Stop Sequences | User-defined strings that trigger generation termination. | Max New Tokens | Hard limit on the number of tokens to generate. | | Term | Definition | |---|---| Speculative Decoding | Using a small draft model to predict multiple future tokens, which are then verified in parallel by the target model. | Draft Model | A smaller, faster model that generates candidate token sequences for verification. | Target Model | The full-size model that verifies draft tokens. | Verification | The process of checking draft tokens against the target model's true distribution. | Acceptance Rate | The fraction of draft tokens accepted by the target model. | Lookahead Decoding | Speculative decoding using the target model itself via Jacobi iteration, eliminating the need for a separate draft model. | Medusa | Method that adds multiple decoding heads to a model for parallel prediction of future tokens. | EAGLE | Enhanced speculative decoding using auto-regressive heads at the feature level for higher acceptance rates. | Prompt Lookup Decoding | Using n-gram matches from the prompt/context as draft tokens instead of a model. | Restoration Loss | Training objective for draft models to match the target model's distribution. | Tree Attention | Attention pattern that verifies multiple speculative token sequences simultaneously in a tree structure. | | Term | Definition | |---|---| Tensor Parallelism (TP) | Splitting individual layers across multiple GPUs by partitioning weight matrices (column-wise or row-wise). | Pipeline Parallelism (PP) | Distributing consecutive layers across different GPUs, with activations passed between stages. | Data Parallelism (DP) | Replicating the model across GPUs and processing different requests on each replica. | Sequence Parallelism | Distributing the sequence dimension across GPUs, useful for long-context inference. | Expert Parallelism (EP) | In MoE models, distributing different experts across GPUs while keeping shared layers replicated. | ZeRO (Zero Redundancy Optimizer) | Memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks. | All-Reduce | Collective communication operation where tensors are summed/averaged across all GPUs. | All-Gather | Collective operation where each GPU receives the full concatenation of tensors from all GPUs. | Reduce-Scatter | Reducing and then scattering results across GPUs. | Point-to-Point Communication | Direct data transfer between specific GPU pairs, used in pipeline parallelism. | Communication Overlap | Hiding communication latency by overlapping with computation (e.g., using CUDA streams). | NCCL (NVIDIA Collective Communications Library) | Library optimized for multi-GPU collective operations. | Ray | Distributed computing framework commonly used for serving LLMs across clusters. | vLLM | High-throughput LLM serving engine featuring PagedAttention and continuous batching. | TensorRT-LLM | NVIDIA's optimized inference library with kernel fusion, quantization, and multi-GPU support. | DeepSpeed-Inference | Microsoft's inference optimization framework with ZeRO partitioning and custom kernels. | Hugging Face TGI (Text Generation Inference) | Production-ready serving stack with flash attention and safetensors support. | SGLang | Efficient serving framework with structured generation and RadixAttention. | LMDeploy | Efficient LLM serving with persistent batching and blocked KV cache. | | Term | Definition | |---|---| Kernel Fusion | Combining multiple operations into a single GPU kernel to reduce kernel launch overhead and memory traffic. | Operator Fusion | Merging mathematically equivalent operations (e.g., bias-add + activation + layernorm). | Flash-Decoding | Fused attention kernel optimized for the memory-bound decode phase. | CUTLASS | NVIDIA's CUDA template library for high-performance GEMM and convolution kernels. | cuBLAS | NVIDIA's optimized BLAS library for dense linear algebra. | Triton | Open-source Python-like language from OpenAI for writing custom GPU kernels. | Marlin | Optimized INT4/INT8 GEMM kernel for weight-only quantization on Ampere+ GPUs. | FasterTransformer | NVIDIA's transformer optimization library (now largely superseded by TensorRT-LLM). | CUDA Graphs | Recording and replaying sequences of GPU operations to eliminate CPU launch overhead. | Stream-K | Work decomposition strategy for GEMMs that improves GPU utilization for small batch sizes. | Warp-specialization | Assigning different warps to different phases of computation (e.g., GEMM + softmax in FlashAttention-3). | Tensor Cores | Specialized GPU units for mixed-precision matrix multiply-accumulate operations. | Memory-Bound vs. Compute-Bound | Classification of whether performance is limited by memory bandwidth or FLOPS. Decode is typically memory-bound; prefill is compute-bound. | Arithmetic Intensity | Ratio of FLOPs to bytes of memory traffic; determines whether a kernel is compute or memory bound. | Roofline Model | Performance analysis model plotting achievable performance against arithmetic intensity. | HBM (High Bandwidth Memory) | GPU memory technology; bandwidth is often the bottleneck for inference. | L2 Cache | On-GPU cache layer between HBM and SM registers; important for FlashAttention's tiling strategy. | Shared Memory / SRAM | Fast, programmable on-chip memory within each Streaming Multiprocessor (SM). | Register Pressure | The demand for GPU registers; high pressure can limit occupancy. | Occupancy | The ratio of active warps to maximum possible warps on an SM. | Wave Quantization | Performance loss when the problem size doesn't evenly divide the GPU's wave size. | | Term | Definition | |---|---| Throughput | Number of tokens or requests processed per unit time (tokens/sec or requests/sec). | Latency | Time from request submission to first token (TTFT) or between consecutive tokens (TBT/ITL). | Time to First Token (TTFT) | Latency from request arrival to the first generated token. Dominated by prefill. | Time Between Tokens (TBT) / Inter-Token Latency (ITL) | Time between consecutive generated tokens. Dominated by decode. | End-to-End Latency | Total time from request to complete response. | Latency SLO | Service Level Objective specifying maximum acceptable latency. | Goodput | The rate of requests that meet their latency SLOs. | Request Rate | Incoming requests per second (RPS). | Auto-scaling | Dynamically adjusting the number of serving instances based on load. | Load Balancing | Distributing requests across multiple serving instances. | Model Sharding | Splitting a model across multiple devices for serving. | Model Replication | Creating multiple copies of a model for parallel request handling. | Disaggregated Serving | Separating prefill and decode onto different GPU pools, each optimized for their respective workload. | Prefix-Aware Routing | Routing requests to instances that already have the prefix cached. | Streaming Response | Sending generated tokens to the client as they are produced, rather than waiting for completion. | OpenAI API Compatible | Serving endpoint implementing the OpenAI chat/completions API format. | gRPC | High-performance RPC framework often used for model serving. | REST API | HTTP-based API for model inference. | Model Registry | Centralized storage for model versions and artifacts. | A/B Testing | Serving different model versions to compare performance. | Shadow Mode | Running a new model in parallel without serving its outputs, for validation. | Canary Deployment | Gradually rolling out a new model version to a subset of traffic. | KV Cache Store | External storage system for KV caches (e.g., Redis, custom stores) for multi-turn conversations. | Function Calling / Tool Use | Model capability to invoke external tools/APIs during generation. | RAG (Retrieval-Augmented Generation) | Augmenting prompts with retrieved documents to ground generation in external knowledge. | | Term | Definition | |---|---| LoRA (Low-Rank Adaptation) | Fine-tuning method that trains low-rank decomposition matrices instead of full weights, enabling efficient adaptation. | QLoRA | LoRA with a quantized base model, enabling fine-tuning on consumer GPUs. | DoRA (Weight-Decomposed Low-Rank Adaptation) | Decomposing weights into magnitude and direction for more stable LoRA training. | Adapter Layers | Small bottleneck layers inserted into a frozen pretrained model for task-specific adaptation. | Prompt Tuning | Learning soft prompt embeddings while keeping the model frozen. | Prefix Tuning | Learning continuous prefix vectors prepended to keys and values in attention layers. | P-Tuning v2 | Scaling prefix tuning to deeper layers for better performance. | Distillation | Training a smaller student model to mimic a larger teacher model's behavior. | Cascade Inference | Using smaller models for easier inputs and larger models only when needed. | Early Exit | Allowing the model to exit at intermediate layers for simple inputs. | Sparse Attention | Attention patterns that skip certain token pairs (e.g., Longformer, BigBird, Ring Attention). | Ring Attention | Distributed attention for extremely long sequences by partitioning along the sequence dimension in a ring topology. | Striped Attention | Attention pattern that alternates between local and global attention. | H2O (Heavy Hitter Oracle) | KV cache eviction policy that retains tokens with high accumulated attention scores. | StreamingLLM | Enabling infinite-length generation by maintaining attention sinks (initial tokens) and a rolling KV cache. | LM-Infinite | Framework for handling infinite context via segment-based processing. | Activation Checkpointing | Recomputing activations during backward pass to save memory (relevant for training, sometimes inference). | Gradient Checkpointing | Same as activation checkpointing. | CPU Offloading | Moving model weights or KV cache to CPU memory when GPU memory is full. | Disk Offloading | Using disk storage for model weights when CPU/GPU memory is insufficient. | Model Compression | General term for techniques reducing model size (pruning, quantization, distillation). | Pruning | Removing less important weights or neurons from a model. | Structured Pruning | Removing entire structures (heads, layers, channels) for hardware efficiency. | Unstructured Pruning | Removing individual weights, requiring sparse matrix support. | SparseGPT | One-shot pruning method for GPT-style models using approximate Hessian information. | Wanda | Pruning weights based on the product of weight magnitude and activation norm. | Mamba / State Space Models (SSMs) | Alternative architectures to transformers with linear-time sequence modeling and no KV cache. | RWKV | RNN-like architecture combining transformer parallelizability with RNN-like memory efficiency. | RetNet | Architecture proposing retention mechanism as an alternative to attention for efficient decoding. | Mixture of Depths | Dynamically skipping layers for less important tokens. | LayerSkip | Training models to be robust to layer skipping for early-exit inference. | | Term | Definition | |---|---| TPU (Tensor Processing Unit) | Google's ASIC optimized for matrix operations; TPU v4/v5p/v5e used for LLM training/inference. | Groq LPU (Language Processing Unit) | Groq's deterministic tensor streaming processor with no external HBM, enabling extremely low and predictable latency. | AWS Inferentia / Trainium | Amazon's custom AI chips (Inf2, Trn1) for cost-efficient inference. | SambaNova SN40L | Reconfigurable dataflow accelerator with large on-chip memory for model weights. | Cerebras Wafer-Scale Engine | Massive wafer-scale chip designed for training and inference of extremely large models. | Qualcomm AI Stack / NPU | Mobile/edge inference on Snapdragon NPUs via ONNX/QNN. | Apple Neural Engine (ANE) | Dedicated neural processing unit in Apple Silicon for on-device inference. | NVIDIA H100/H200/Blackwell | GPU generations with Transformer Engine, FP8 support, and NVLink/NVSwitch for scale-out. | NVLink / NVSwitch | High-bandwidth interconnect for multi-GPU communication within a node. | InfiniBand / RoCE | High-speed networking for inter-node GPU clusters. | Transformer Engine | NVIDIA library and hardware feature for automatic FP8 precision management. | | Term | Definition | |---|---| Safetensors | Hugging Face's secure tensor format (no pickle, memory-mappable, faster loading). | Pickle / PyTorch .bin | Traditional PyTorch serialization format (security risk due to arbitrary code execution). | ONNX (Open Neural Network Exchange) | Cross-platform model format for portable inference. | TensorRT Engine | NVIDIA's compiled, optimized model format for deployment. | Core ML | Apple's format for on-device inference on iOS/macOS. | TFLite (TensorFlow Lite) | Lightweight format for mobile and edge devices. | OpenVINO IR | Intel's intermediate representation for optimized inference on Intel hardware. | GGUF (GPT-Generated Unified Format) | Successor to GGML; supports metadata, multiple quantization types, and CPU inference. | | Term | Definition | |---|---| llama.cpp | C/C++ implementation of LLaMA optimized for CPU and Apple Silicon inference. | Ollama | User-friendly wrapper around llama.cpp for local model running. | Triton Inference Server | NVIDIA's multi-framework serving platform with dynamic batching and model ensemble. | ONNX Runtime | Cross-platform inference engine supporting ONNX models with hardware accelerators. | OpenVINO | Intel's toolkit for optimizing and deploying deep learning models. | TorchServe / TensorFlow Serving | Framework-native serving solutions. | MLflow / BentoML | Model packaging and serving orchestration platforms. | KServe | Kubernetes-native model serving platform. | Ray Serve | Scalable model serving built on Ray distributed framework. | Modal / Beam / Replicate | Serverless GPU inference platforms. | | Term | Definition | |---|---| Linear Attention | Attention approximations reducing complexity from O(n²) to O(n) via kernel feature maps. | Performer | Uses FAVOR+ (Fast Attention Via Orthogonal Random Features) for linear-time attention. | RFA (Random Feature Attention) | Approximates softmax attention using random Fourier features. | CosFormer | Linear attention using cosine-based reweighting for locality bias. | Local Attention / Strided Attention | Restricting attention to local windows or fixed strides. | Dilated Attention | Skipping tokens at regular intervals within attention windows. | Factorized Attention | Decomposing full attention into multiple cheaper patterns. | | Term | Definition | |---|---| Context Compression | Techniques to summarize or compress long contexts into shorter representations. | Hierarchical Attention | Multi-scale attention operating on token, sentence, and paragraph levels. | Memory-Augmented Networks | External memory modules the model can read/write during generation. | MemGPT | OS-inspired virtual context management paging memory in/out of context. | LoRA-XS / LongLoRA | Efficient fine-tuning specifically for extending context windows. | YaRN (Yet another RoPE extension method) | Interpolation/extrapolation technique for extending RoPE-based models to longer contexts. | NTK-Aware Scaling | Non-linear interpolation of RoPE base frequencies for context extension. | PI (Positional Interpolation) | Linearly interpolating position indices to fit longer sequences into trained context limits. | Self-Extend | Training-free context extension by grouping positions. | | Term | Definition | |---|---| Model Soups | Averaging weights of multiple fine-tuned models. | Task Arithmetic | Adding/subtracting task vectors (fine-tuned minus pre-trained weights) to compose capabilities. | TIES-Merging | Trimming, electing sign, and disjoint merging to resolve interference between models. | DARE (Drop And REscale) | Dropping a large fraction of delta parameters before merging to reduce interference. | SLERP (Spherical Linear Interpolation) | Interpolating model weights on a hypersphere. | Model Breadcrumbs | Sparse task vectors for efficient model editing. | | Term | Definition | |---|---| torch.compile | PyTorch's JIT compilation with backends like inductor, cudagraphs, and Triton. | Inductor | PyTorch 2.0's default compiler backend generating Triton/C++ kernels. | XLA (Accelerated Linear Algebra) | Google's compiler for linear algebra, used with JAX/TensorFlow on TPUs/GPUs. | JAX | Google's composable ML framework with JIT compilation via XLA. | TVM / Apache TVM | Open-source deep learning compiler stack. | MLIR (Multi-Level Intermediate Representation) | LLVM subproject for representing and transforming ML graphs. | IREE (Intermediate Representation Execution Environment) | MLIR-based runtime for edge deployment. | | Term | Definition | |---|---| ReAct (Reasoning + Acting) | Interleaving reasoning traces with tool/actions in a loop. | Reflexion | Self-reflective agents that learn from verbal feedback. | Plan-and-Solve | Decomposing problems into sub-plans before execution. | Program-Aided LLMs (PAL) | Using LLMs to generate code that solves problems programmatically. | ToolFormer | Training LLMs to decide when and how to call external APIs. | Gorilla / APIBench | Models fine-tuned for accurate API calling. | DSPy | Framework for programming language models with declarative modules. | LangChain / LlamaIndex | Orchestration frameworks for RAG and agentic applications. | | Term | Definition | |---|---| Token Pricing | Cost model based on input (prompt) and output (completion) tokens. | Context Window Pricing | Some providers charge based on context length regardless of generation. | Batch Pricing | Discounted rates for offline/non-real-time inference. | Reserved Throughput | Guaranteed capacity provisioning at fixed cost. | Spot/Preemptible Instances | Cheaper but interruptible compute for batch inference. | Model Telemetry | Logging latency, throughput, error rates, and token counts. | Prompt Injection Detection | Guardrails to detect and block malicious prompt patterns. | Output Moderation | Real-time content filtering of generated text. | PII Redaction | Removing personally identifiable information from inputs/outputs. | | Term | Definition | |---|---| Tokens per Second (tok/s) | Primary throughput metric for inference systems. | FLOPs (Floating Point Operations) | Count of arithmetic operations; used to measure computational cost. | FLOPS (Floating Point Operations Per Second) | Measure of compute throughput. | Memory Bandwidth | Rate at which data can be read from/written to GPU memory (GB/s). | Model FLOPs Utilization (MFU) | Ratio of actual throughput to theoretical peak FLOPS, accounting for memory bandwidth and overhead. | Model Memory Bandwidth Utilization (MBU) | Ratio of actual memory bandwidth used to theoretical peak. | Benchmark Suites | Standardized evaluation sets: MMLU, HellaSwag, ARC, TruthfulQA, HumanEval, GSM8K, etc. | Perplexity (PPL) | Exponential of average negative log-likelihood; measures model confidence in a text. | Long-Context Benchmarks | Needle-in-a-haystack, passkey retrieval, long-document QA for evaluating context window effectiveness. | LMBench / LLMPerf | Benchmarking frameworks for measuring serving performance. | AnyScale LLMPerf | Open-source benchmark for LLM serving systems. | Artificial Load Testing | Generating synthetic request patterns to stress-test serving infrastructure. | | Term | Definition | |---|---| System Prompt | Instructions prepended to every conversation to guide model behavior. | Jailbreaking | Prompt engineering techniques designed to bypass safety guardrails. | Prompt Injection | Attacks that embed malicious instructions within user input to manipulate model behavior. | RLHF (Reinforcement Learning from Human Feedback) | Training method using human preferences to align model outputs. | DPO (Direct Preference Optimization) | Simpler alternative to RLHF that directly optimizes on preference pairs without a separate reward model. | Constitutional AI | Training models using a set of principles (constitution) for self-correction. | Inference-Time Compute Scaling | Allocating more computation at inference (e.g., Chain-of-Thought, best-of-N sampling) to improve output quality. | Chain-of-Thought (CoT) | Prompting the model to show intermediate reasoning steps. | Tree of Thoughts (ToT) | Generalizing CoT to explore multiple reasoning paths and backtrack. | Self-Consistency | Generating multiple reasoning paths and selecting the most frequent answer. | Best-of-N Sampling | Generating N candidates and selecting the best via a reward model or verifier. | Process Reward Model (PRM) | Model that scores individual reasoning steps rather than final outputs. | Outcome Reward Model (ORM) | Model that scores final outputs. | Verifier | Model trained to judge correctness of generated outputs. | Majority Voting | Aggregating multiple samples by selecting the most common answer. | Monte Carlo Tree Search (MCTS) | Using MCTS to explore the reasoning space during inference. | | Term | Definition | |---|---| TMA (Tensor Memory Accelerator) | Hardware unit in Hopper for asynchronous tensor transfers. | WGMMA (Warp Group Matrix Multiply-Accumulate) | Hopper's warp-group-level MMA instruction for higher throughput. | PTX (Parallel Thread Execution) | NVIDIA's intermediate instruction set architecture for CUDA. | SASS | NVIDIA's machine code (binary instructions) executed by the GPU. | Warp / Warp Group | A group of 32 threads executing in SIMT fashion; warp groups contain 4 warps (128 threads) on Hopper. | Thread Block / Cluster | Groups of threads/warps for cooperative execution. | Asynchronous Copy | Overlapping data transfer with computation using `cp.async` instructions. | *This glossary is a living document. The field of LLM inference engineering evolves weekly — new terms, techniques, and hardware primitives are constantly emerging. Last compiled: August 2026.*