llm-inference-glossary.md

An engineer has published a comprehensive living glossary covering over 200 terms related to LLM inference, including core architecture, KV cache management, quantization, batching, speculative decoding, and distributed inference. The reference is designed for engineers building, optimizing, and serving large language models, and it is open to community contributions.

A living reference for engineers building, optimizing, and serving large language models. Last Updated: August 2026|Contributions welcome Core Architecture & Concepts 1-core-architecture--concepts KV Cache & Memory Management 2-kv-cache--memory-management Quantization 3-quantization Batching & Scheduling 4-batching--scheduling Sampling & Decoding Strategies 5-sampling--decoding-strategies Speculative Decoding & Draft Models 6-speculative-decoding--draft-models Distributed Inference 7-distributed-inference Kernel Optimization & Hardware 8-kernel-optimization--hardware Model Serving & Deployment 9-model-serving--deployment Advanced Optimization Techniques 10-advanced-optimization-techniques Hardware & Accelerators 11-hardware--accelerators Model Formats & Serialization 12-model-formats--serialization Serving Frameworks & Runtimes 13-serving-frameworks--runtimes Advanced Attention Variants 14-advanced-attention-variants Context Management & Long Context 15-context-management--long-context Model Merging & Composition 16-model-merging--composition Compilation & Graph Optimization 17-compilation--graph-optimization Inference-Time Agents & Advanced Reasoning 18-inference-time-agents--advanced-reasoning Observability, Cost & Operations 19-observability-cost--operations Benchmarking & Evaluation 20-benchmarking--evaluation Safety, Alignment & Inference-Time Techniques 21-safety-alignment--inference-time-techniques Low-Level GPU/Hardware Primitives 22-low-level-gpuhardware-primitives | Term | Definition | |---|---| Autoregressive Generation | The process by which LLMs generate tokens one at a time, with each new token conditioned on all previously generated tokens. | Prefill Phase | The initial forward pass that processes the input prompt to compute key-value KV cache entries for all prompt tokens. Also called the "context phase." | Decode Phase | The generation phase where tokens are produced one at a time, each depending on the previous token and the KV cache. | Prompt | The input text provided to the model to condition its generation. | Completion | The output text generated by the model in response to a prompt. | Token | The atomic unit of text for the model, produced by a tokenizer. Can be a word, subword, or character. | Tokenizer | The component that converts between raw text and token IDs using algorithms like BPE, WordPiece, or SentencePiece. | Embedding | A dense vector representation of a token in a high-dimensional space, learned during training. | Logits | The raw, unnormalized output scores from the final linear layer of the model, one per token in the vocabulary. | Softmax | The function that converts logits into a probability distribution over the vocabulary. | Attention Mechanism | The core operation in transformers that computes weighted sums of values based on query-key similarity scores. | Self-Attention | Attention where queries, keys, and values all come from the same sequence. | Multi-Head Attention MHA | Running multiple attention operations in parallel with different learned projections. | Grouped Query Attention GQA | A variant where multiple query heads share the same key/value heads, reducing KV cache size. | Multi-Query Attention MQA | An extreme form of GQA where all query heads share a single key head and single value head. | Feed-Forward Network FFN | The position-wise fully connected sublayer in each transformer block, typically using SwiGLU, GeGLU, or GELU activations. | Layer Normalization | Normalization technique applied per layer to stabilize training and inference. | RMSNorm | Root Mean Square Layer Normalization, a simplified variant without mean-centering, commonly used in LLaMA and Mistral. | Rotary Position Embedding RoPE | A position encoding method that rotates query/key vectors by a position-dependent angle, enabling better length generalization. | ALiBi Attention with Linear Biases | Position encoding that adds linear biases to attention scores instead of explicit embeddings. | Sliding Window Attention | Attention restricted to a fixed-size local window, used in models like Mistral to handle long contexts efficiently. | Mixture of Experts MoE | Architecture where only a subset of parameters experts are activated per token, enabling massive scale without proportional compute increase. | Top-k Routing | In MoE, selecting the k highest-scoring experts for each token. | MLA Multi-head Latent Attention | DeepSeek's attention mechanism that compresses KV cache via low-rank key-value joint compression. | | Term | Definition | |---|---| KV Cache | Stored key and value tensors from previous tokens, eliminating redundant recomputation during autoregressive generation. | KV Cache Size | Memory footprint = 2 × num layers × num kv heads × head dim × seq len × batch size × dtype size . | PagedAttention | vLLM's memory management system that partitions KV cache into fixed-size blocks pages , enabling efficient sharing and dynamic allocation. | Block Table | A mapping from logical token positions to physical memory blocks in PagedAttention. | Copy-on-Write CoW | Memory optimization where KV cache blocks are shared between sequences and only copied when modified, used for beam search and parallel sampling. | Prefix Caching | Reusing KV cache computations for shared prompt prefixes across multiple requests. | KV Cache Eviction | Strategies to remove less important tokens from cache when memory is constrained e.g., H2O, Heavy Hitter Oracle . | KV Cache Quantization | Compressing cached keys/values to lower precision INT8, FP8, INT4 to reduce memory usage. | KV Cache Offloading | Moving KV cache to CPU/disk memory when GPU memory is exhausted, trading latency for capacity. | FlashAttention | IO-aware exact attention algorithm that reduces HBM reads/writes by using tiling and recomputation in SRAM. | FlashAttention-2 | Improved version with better parallelism, work partitioning, and reduced non-matmul FLOPs. | FlashAttention-3 | Further optimized for Hopper architecture with FP8 support, warp-specialization, and interleaved block-wise GEMMs. | FlashDecoding | Extension of FlashAttention optimized for the decode phase with small batch sizes and long sequences. | FlashInfer | A unified library for attention kernels supporting various attention variants and sparse patterns. | RadixAttention | SGLang's attention mechanism that automatically reuses KV cache across multiple calls by treating the cache as a radix tree. | | Term | Definition | |---|---| Quantization | Reducing numerical precision of model weights/activations to decrease memory and increase throughput. | Post-Training Quantization PTQ | Quantizing a pre-trained model without retraining. | Quantization-Aware Training QAT | Training with simulated quantization to learn robust low-precision representations. | Weight-Only Quantization | Quantizing only model weights while keeping activations in higher precision. | Weight-Activation Quantization | Quantizing both weights and activations. | INT8 | 8-bit integer quantization, typically using symmetric or asymmetric schemes. | INT4/INT3/INT2 | Extreme weight quantization packing multiple weights per byte. | FP8 E4M3/E5M2 | 8-bit floating point formats defined in the FP8 standard E4M3 for weights/activations, E5M2 for gradients . | NF4 Normal Float 4 | A 4-bit data type with non-uniform bins optimized for normally distributed weights, used in QLoRA. | AWQ Activation-Aware Weight Quantization | PTQ method that protects salient weight channels based on activation magnitudes. | GPTQ | One-shot weight quantization using approximate second-order information for layer-wise optimal quantization. | SmoothQuant | Migration of quantization difficulty from activations to weights via per-channel scaling. | LLM.int8 | Mixed-precision decomposition that keeps outlier features in FP16 while quantizing the rest to INT8. | GGUF/GGML | File formats and quantization schemes for efficient CPU inference Q4 0, Q5 K M, Q8 0, etc. . | QLoRA | Quantized LoRA fine-tuning where the base model is quantized to NF4 while LoRA adapters remain in FP16. | BitsAndBytes | Library implementing LLM.int8 and 4-bit quantization for easy integration. | Dynamic Quantization | Quantizing on-the-fly during inference based on runtime activation statistics. | Static Quantization | Pre-computing quantization parameters from calibration data. | Group-wise Quantization | Applying separate quantization parameters to groups of weights e.g., 128 consecutive weights . | Channel-wise Quantization | Separate quantization parameters per output channel. | Token-wise Quantization | Separate quantization parameters per token position for activations. | | Term | Definition | |---|---| Static Batching | Grouping multiple requests into a fixed batch before inference; all requests wait for the longest one. | Dynamic Batching | Continuously adding new requests to a running batch as others complete. | Continuous Batching In-flight Batching | Dynamically adding/removing requests from a running batch at every iteration, maximizing GPU utilization. | Iteration-level Scheduling | Scheduling decisions made at each generation step rather than per-request. | Request Scheduling | Policies FCFS, shortest-job-first, priority-based for ordering incoming requests. | Preemption | Pausing a request mid-generation to free GPU resources for higher-priority requests, with state saved for resumption. | Swap-out/Swap-in | Moving KV cache of preempted requests between GPU and CPU memory. | Chunked Prefill | Breaking long prefill computations into smaller chunks and interleaving them with decode steps to prevent decode starvation. | Prompt Chunking | Processing long prompts in segments to bound prefill latency. | Microbatching | Splitting a batch into smaller sub-batches for pipeline parallelism. | Max Batch Size | The upper limit on concurrent sequences, constrained by memory and latency SLOs. | Max Sequence Length | The maximum context window supported by the model or deployment. | Token Budget | A limit on total tokens prefill + decode a request can consume. | | Term | Definition | |---|---| Greedy Decoding | Always selecting the token with the highest probability. Deterministic but often suboptimal. | Temperature Scaling | Dividing logits by a temperature T before softmax; T < 1 makes distribution sharper more deterministic , T 1 makes it flatter more random . | Top-k Sampling | Restricting sampling to the k most likely tokens, setting others' probabilities to zero. | Top-p Nucleus Sampling | Sampling from the smallest set of tokens whose cumulative probability exceeds p. | Min-p Sampling | Sampling from tokens with probability ≥ min p × max probability, adaptive to the distribution shape. | Typical Sampling | Sampling from tokens with information content close to the conditional entropy of the distribution. | Repetition Penalty | Downscaling logits of tokens that have already appeared in the sequence to reduce repetition. | Frequency Penalty | Penalizing tokens proportionally to their frequency in the generated text. | Presence Penalty | Applying a fixed penalty to tokens that have appeared at least once. | Beam Search | Maintaining k candidate sequences and expanding the most promising ones at each step. | Diverse Beam Search | Modified beam search that encourages diversity among beam candidates. | Contrastive Search | Decoding that penalizes tokens too similar to previous context, improving coherence. | Speculative Sampling | A verification-based sampling method used in speculative decoding to accept/reject draft tokens. | Watermarked Sampling | Embedding statistical watermarks into generated text for detection/attribution. | Structured Decoding / Constrained Decoding | Enforcing output to follow a grammar, regex, or JSON schema during generation. | Guided Decoding | Frameworks like Outlines, Guidance, or lm-format-enforcer for structured generation. | Logit Bias | Adding fixed values to specific token logits to encourage/discourage their selection. | EOS End-of-Sequence Token | Special token signaling the model to stop generating. | Stop Sequences | User-defined strings that trigger generation termination. | Max New Tokens | Hard limit on the number of tokens to generate. | | Term | Definition | |---|---| Speculative Decoding | Using a small draft model to predict multiple future tokens, which are then verified in parallel by the target model. | Draft Model | A smaller, faster model that generates candidate token sequences for verification. | Target Model | The full-size model that verifies draft tokens. | Verification | The process of checking draft tokens against the target model's true distribution. | Acceptance Rate | The fraction of draft tokens accepted by the target model. | Lookahead Decoding | Speculative decoding using the target model itself via Jacobi iteration, eliminating the need for a separate draft model. | Medusa | Method that adds multiple decoding heads to a model for parallel prediction of future tokens. | EAGLE | Enhanced speculative decoding using auto-regressive heads at the feature level for higher acceptance rates. | Prompt Lookup Decoding | Using n-gram matches from the prompt/context as draft tokens instead of a model. | Restoration Loss | Training objective for draft models to match the target model's distribution. | Tree Attention | Attention pattern that verifies multiple speculative token sequences simultaneously in a tree structure. | | Term | Definition | |---|---| Tensor Parallelism TP | Splitting individual layers across multiple GPUs by partitioning weight matrices column-wise or row-wise . | Pipeline Parallelism PP | Distributing consecutive layers across different GPUs, with activations passed between stages. | Data Parallelism DP | Replicating the model across GPUs and processing different requests on each replica. | Sequence Parallelism | Distributing the sequence dimension across GPUs, useful for long-context inference. | Expert Parallelism EP | In MoE models, distributing different experts across GPUs while keeping shared layers replicated. | ZeRO Zero Redundancy Optimizer | Memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks. | All-Reduce | Collective communication operation where tensors are summed/averaged across all GPUs. | All-Gather | Collective operation where each GPU receives the full concatenation of tensors from all GPUs. | Reduce-Scatter | Reducing and then scattering results across GPUs. | Point-to-Point Communication | Direct data transfer between specific GPU pairs, used in pipeline parallelism. | Communication Overlap | Hiding communication latency by overlapping with computation e.g., using CUDA streams . | NCCL NVIDIA Collective Communications Library | Library optimized for multi-GPU collective operations. | Ray | Distributed computing framework commonly used for serving LLMs across clusters. | vLLM | High-throughput LLM serving engine featuring PagedAttention and continuous batching. | TensorRT-LLM | NVIDIA's optimized inference library with kernel fusion, quantization, and multi-GPU support. | DeepSpeed-Inference | Microsoft's inference optimization framework with ZeRO partitioning and custom kernels. | Hugging Face TGI Text Generation Inference | Production-ready serving stack with flash attention and safetensors support. | SGLang | Efficient serving framework with structured generation and RadixAttention. | LMDeploy | Efficient LLM serving with persistent batching and blocked KV cache. | | Term | Definition | |---|---| Kernel Fusion | Combining multiple operations into a single GPU kernel to reduce kernel launch overhead and memory traffic. | Operator Fusion | Merging mathematically equivalent operations e.g., bias-add + activation + layernorm . | Flash-Decoding | Fused attention kernel optimized for the memory-bound decode phase. | CUTLASS | NVIDIA's CUDA template library for high-performance GEMM and convolution kernels. | cuBLAS | NVIDIA's optimized BLAS library for dense linear algebra. | Triton | Open-source Python-like language from OpenAI for writing custom GPU kernels. | Marlin | Optimized INT4/INT8 GEMM kernel for weight-only quantization on Ampere+ GPUs. | FasterTransformer | NVIDIA's transformer optimization library now largely superseded by TensorRT-LLM . | CUDA Graphs | Recording and replaying sequences of GPU operations to eliminate CPU launch overhead. | Stream-K | Work decomposition strategy for GEMMs that improves GPU utilization for small batch sizes. | Warp-specialization | Assigning different warps to different phases of computation e.g., GEMM + softmax in FlashAttention-3 . | Tensor Cores | Specialized GPU units for mixed-precision matrix multiply-accumulate operations. | Memory-Bound vs. Compute-Bound | Classification of whether performance is limited by memory bandwidth or FLOPS. Decode is typically memory-bound; prefill is compute-bound. | Arithmetic Intensity | Ratio of FLOPs to bytes of memory traffic; determines whether a kernel is compute or memory bound. | Roofline Model | Performance analysis model plotting achievable performance against arithmetic intensity. | HBM High Bandwidth Memory | GPU memory technology; bandwidth is often the bottleneck for inference. | L2 Cache | On-GPU cache layer between HBM and SM registers; important for FlashAttention's tiling strategy. | Shared Memory / SRAM | Fast, programmable on-chip memory within each Streaming Multiprocessor SM . | Register Pressure | The demand for GPU registers; high pressure can limit occupancy. | Occupancy | The ratio of active warps to maximum possible warps on an SM. | Wave Quantization | Performance loss when the problem size doesn't evenly divide the GPU's wave size. | | Term | Definition | |---|---| Throughput | Number of tokens or requests processed per unit time tokens/sec or requests/sec . | Latency | Time from request submission to first token TTFT or between consecutive tokens TBT/ITL . | Time to First Token TTFT | Latency from request arrival to the first generated token. Dominated by prefill. | Time Between Tokens TBT / Inter-Token Latency ITL | Time between consecutive generated tokens. Dominated by decode. | End-to-End Latency | Total time from request to complete response. | Latency SLO | Service Level Objective specifying maximum acceptable latency. | Goodput | The rate of requests that meet their latency SLOs. | Request Rate | Incoming requests per second RPS . | Auto-scaling | Dynamically adjusting the number of serving instances based on load. | Load Balancing | Distributing requests across multiple serving instances. | Model Sharding | Splitting a model across multiple devices for serving. | Model Replication | Creating multiple copies of a model for parallel request handling. | Disaggregated Serving | Separating prefill and decode onto different GPU pools, each optimized for their respective workload. | Prefix-Aware Routing | Routing requests to instances that already have the prefix cached. | Streaming Response | Sending generated tokens to the client as they are produced, rather than waiting for completion. | OpenAI API Compatible | Serving endpoint implementing the OpenAI chat/completions API format. | gRPC | High-performance RPC framework often used for model serving. | REST API | HTTP-based API for model inference. | Model Registry | Centralized storage for model versions and artifacts. | A/B Testing | Serving different model versions to compare performance. | Shadow Mode | Running a new model in parallel without serving its outputs, for validation. | Canary Deployment | Gradually rolling out a new model version to a subset of traffic. | KV Cache Store | External storage system for KV caches e.g., Redis, custom stores for multi-turn conversations. | Function Calling / Tool Use | Model capability to invoke external tools/APIs during generation. | RAG Retrieval-Augmented Generation | Augmenting prompts with retrieved documents to ground generation in external knowledge. | | Term | Definition | |---|---| LoRA Low-Rank Adaptation | Fine-tuning method that trains low-rank decomposition matrices instead of full weights, enabling efficient adaptation. | QLoRA | LoRA with a quantized base model, enabling fine-tuning on consumer GPUs. | DoRA Weight-Decomposed Low-Rank Adaptation | Decomposing weights into magnitude and direction for more stable LoRA training. | Adapter Layers | Small bottleneck layers inserted into a frozen pretrained model for task-specific adaptation. | Prompt Tuning | Learning soft prompt embeddings while keeping the model frozen. | Prefix Tuning | Learning continuous prefix vectors prepended to keys and values in attention layers. | P-Tuning v2 | Scaling prefix tuning to deeper layers for better performance. | Distillation | Training a smaller student model to mimic a larger teacher model's behavior. | Cascade Inference | Using smaller models for easier inputs and larger models only when needed. | Early Exit | Allowing the model to exit at intermediate layers for simple inputs. | Sparse Attention | Attention patterns that skip certain token pairs e.g., Longformer, BigBird, Ring Attention . | Ring Attention | Distributed attention for extremely long sequences by partitioning along the sequence dimension in a ring topology. | Striped Attention | Attention pattern that alternates between local and global attention. | H2O Heavy Hitter Oracle | KV cache eviction policy that retains tokens with high accumulated attention scores. | StreamingLLM | Enabling infinite-length generation by maintaining attention sinks initial tokens and a rolling KV cache. | LM-Infinite | Framework for handling infinite context via segment-based processing. | Activation Checkpointing | Recomputing activations during backward pass to save memory relevant for training, sometimes inference . | Gradient Checkpointing | Same as activation checkpointing. | CPU Offloading | Moving model weights or KV cache to CPU memory when GPU memory is full. | Disk Offloading | Using disk storage for model weights when CPU/GPU memory is insufficient. | Model Compression | General term for techniques reducing model size pruning, quantization, distillation . | Pruning | Removing less important weights or neurons from a model. | Structured Pruning | Removing entire structures heads, layers, channels for hardware efficiency. | Unstructured Pruning | Removing individual weights, requiring sparse matrix support. | SparseGPT | One-shot pruning method for GPT-style models using approximate Hessian information. | Wanda | Pruning weights based on the product of weight magnitude and activation norm. | Mamba / State Space Models SSMs | Alternative architectures to transformers with linear-time sequence modeling and no KV cache. | RWKV | RNN-like architecture combining transformer parallelizability with RNN-like memory efficiency. | RetNet | Architecture proposing retention mechanism as an alternative to attention for efficient decoding. | Mixture of Depths | Dynamically skipping layers for less important tokens. | LayerSkip | Training models to be robust to layer skipping for early-exit inference. | | Term | Definition | |---|---| TPU Tensor Processing Unit | Google's ASIC optimized for matrix operations; TPU v4/v5p/v5e used for LLM training/inference. | Groq LPU Language Processing Unit | Groq's deterministic tensor streaming processor with no external HBM, enabling extremely low and predictable latency. | AWS Inferentia / Trainium | Amazon's custom AI chips Inf2, Trn1 for cost-efficient inference. | SambaNova SN40L | Reconfigurable dataflow accelerator with large on-chip memory for model weights. | Cerebras Wafer-Scale Engine | Massive wafer-scale chip designed for training and inference of extremely large models. | Qualcomm AI Stack / NPU | Mobile/edge inference on Snapdragon NPUs via ONNX/QNN. | Apple Neural Engine ANE | Dedicated neural processing unit in Apple Silicon for on-device inference. | NVIDIA H100/H200/Blackwell | GPU generations with Transformer Engine, FP8 support, and NVLink/NVSwitch for scale-out. | NVLink / NVSwitch | High-bandwidth interconnect for multi-GPU communication within a node. | InfiniBand / RoCE | High-speed networking for inter-node GPU clusters. | Transformer Engine | NVIDIA library and hardware feature for automatic FP8 precision management. | | Term | Definition | |---|---| Safetensors | Hugging Face's secure tensor format no pickle, memory-mappable, faster loading . | Pickle / PyTorch .bin | Traditional PyTorch serialization format security risk due to arbitrary code execution . | ONNX Open Neural Network Exchange | Cross-platform model format for portable inference. | TensorRT Engine | NVIDIA's compiled, optimized model format for deployment. | Core ML | Apple's format for on-device inference on iOS/macOS. | TFLite TensorFlow Lite | Lightweight format for mobile and edge devices. | OpenVINO IR | Intel's intermediate representation for optimized inference on Intel hardware. | GGUF GPT-Generated Unified Format | Successor to GGML; supports metadata, multiple quantization types, and CPU inference. | | Term | Definition | |---|---| llama.cpp | C/C++ implementation of LLaMA optimized for CPU and Apple Silicon inference. | Ollama | User-friendly wrapper around llama.cpp for local model running. | Triton Inference Server | NVIDIA's multi-framework serving platform with dynamic batching and model ensemble. | ONNX Runtime | Cross-platform inference engine supporting ONNX models with hardware accelerators. | OpenVINO | Intel's toolkit for optimizing and deploying deep learning models. | TorchServe / TensorFlow Serving | Framework-native serving solutions. | MLflow / BentoML | Model packaging and serving orchestration platforms. | KServe | Kubernetes-native model serving platform. | Ray Serve | Scalable model serving built on Ray distributed framework. | Modal / Beam / Replicate | Serverless GPU inference platforms. | | Term | Definition | |---|---| Linear Attention | Attention approximations reducing complexity from O n² to O n via kernel feature maps. | Performer | Uses FAVOR+ Fast Attention Via Orthogonal Random Features for linear-time attention. | RFA Random Feature Attention | Approximates softmax attention using random Fourier features. | CosFormer | Linear attention using cosine-based reweighting for locality bias. | Local Attention / Strided Attention | Restricting attention to local windows or fixed strides. | Dilated Attention | Skipping tokens at regular intervals within attention windows. | Factorized Attention | Decomposing full attention into multiple cheaper patterns. | | Term | Definition | |---|---| Context Compression | Techniques to summarize or compress long contexts into shorter representations. | Hierarchical Attention | Multi-scale attention operating on token, sentence, and paragraph levels. | Memory-Augmented Networks | External memory modules the model can read/write during generation. | MemGPT | OS-inspired virtual context management paging memory in/out of context. | LoRA-XS / LongLoRA | Efficient fine-tuning specifically for extending context windows. | YaRN Yet another RoPE extension method | Interpolation/extrapolation technique for extending RoPE-based models to longer contexts. | NTK-Aware Scaling | Non-linear interpolation of RoPE base frequencies for context extension. | PI Positional Interpolation | Linearly interpolating position indices to fit longer sequences into trained context limits. | Self-Extend | Training-free context extension by grouping positions. | | Term | Definition | |---|---| Model Soups | Averaging weights of multiple fine-tuned models. | Task Arithmetic | Adding/subtracting task vectors fine-tuned minus pre-trained weights to compose capabilities. | TIES-Merging | Trimming, electing sign, and disjoint merging to resolve interference between models. | DARE Drop And REscale | Dropping a large fraction of delta parameters before merging to reduce interference. | SLERP Spherical Linear Interpolation | Interpolating model weights on a hypersphere. | Model Breadcrumbs | Sparse task vectors for efficient model editing. | | Term | Definition | |---|---| torch.compile | PyTorch's JIT compilation with backends like inductor, cudagraphs, and Triton. | Inductor | PyTorch 2.0's default compiler backend generating Triton/C++ kernels. | XLA Accelerated Linear Algebra | Google's compiler for linear algebra, used with JAX/TensorFlow on TPUs/GPUs. | JAX | Google's composable ML framework with JIT compilation via XLA. | TVM / Apache TVM | Open-source deep learning compiler stack. | MLIR Multi-Level Intermediate Representation | LLVM subproject for representing and transforming ML graphs. | IREE Intermediate Representation Execution Environment | MLIR-based runtime for edge deployment. | | Term | Definition | |---|---| ReAct Reasoning + Acting | Interleaving reasoning traces with tool/actions in a loop. | Reflexion | Self-reflective agents that learn from verbal feedback. | Plan-and-Solve | Decomposing problems into sub-plans before execution. | Program-Aided LLMs PAL | Using LLMs to generate code that solves problems programmatically. | ToolFormer | Training LLMs to decide when and how to call external APIs. | Gorilla / APIBench | Models fine-tuned for accurate API calling. | DSPy | Framework for programming language models with declarative modules. | LangChain / LlamaIndex | Orchestration frameworks for RAG and agentic applications. | | Term | Definition | |---|---| Token Pricing | Cost model based on input prompt and output completion tokens. | Context Window Pricing | Some providers charge based on context length regardless of generation. | Batch Pricing | Discounted rates for offline/non-real-time inference. | Reserved Throughput | Guaranteed capacity provisioning at fixed cost. | Spot/Preemptible Instances | Cheaper but interruptible compute for batch inference. | Model Telemetry | Logging latency, throughput, error rates, and token counts. | Prompt Injection Detection | Guardrails to detect and block malicious prompt patterns. | Output Moderation | Real-time content filtering of generated text. | PII Redaction | Removing personally identifiable information from inputs/outputs. | | Term | Definition | |---|---| Tokens per Second tok/s | Primary throughput metric for inference systems. | FLOPs Floating Point Operations | Count of arithmetic operations; used to measure computational cost. | FLOPS Floating Point Operations Per Second | Measure of compute throughput. | Memory Bandwidth | Rate at which data can be read from/written to GPU memory GB/s . | Model FLOPs Utilization MFU | Ratio of actual throughput to theoretical peak FLOPS, accounting for memory bandwidth and overhead. | Model Memory Bandwidth Utilization MBU | Ratio of actual memory bandwidth used to theoretical peak. | Benchmark Suites | Standardized evaluation sets: MMLU, HellaSwag, ARC, TruthfulQA, HumanEval, GSM8K, etc. | Perplexity PPL | Exponential of average negative log-likelihood; measures model confidence in a text. | Long-Context Benchmarks | Needle-in-a-haystack, passkey retrieval, long-document QA for evaluating context window effectiveness. | LMBench / LLMPerf | Benchmarking frameworks for measuring serving performance. | AnyScale LLMPerf | Open-source benchmark for LLM serving systems. | Artificial Load Testing | Generating synthetic request patterns to stress-test serving infrastructure. | | Term | Definition | |---|---| System Prompt | Instructions prepended to every conversation to guide model behavior. | Jailbreaking | Prompt engineering techniques designed to bypass safety guardrails. | Prompt Injection | Attacks that embed malicious instructions within user input to manipulate model behavior. | RLHF Reinforcement Learning from Human Feedback | Training method using human preferences to align model outputs. | DPO Direct Preference Optimization | Simpler alternative to RLHF that directly optimizes on preference pairs without a separate reward model. | Constitutional AI | Training models using a set of principles constitution for self-correction. | Inference-Time Compute Scaling | Allocating more computation at inference e.g., Chain-of-Thought, best-of-N sampling to improve output quality. | Chain-of-Thought CoT | Prompting the model to show intermediate reasoning steps. | Tree of Thoughts ToT | Generalizing CoT to explore multiple reasoning paths and backtrack. | Self-Consistency | Generating multiple reasoning paths and selecting the most frequent answer. | Best-of-N Sampling | Generating N candidates and selecting the best via a reward model or verifier. | Process Reward Model PRM | Model that scores individual reasoning steps rather than final outputs. | Outcome Reward Model ORM | Model that scores final outputs. | Verifier | Model trained to judge correctness of generated outputs. | Majority Voting | Aggregating multiple samples by selecting the most common answer. | Monte Carlo Tree Search MCTS | Using MCTS to explore the reasoning space during inference. | | Term | Definition | |---|---| TMA Tensor Memory Accelerator | Hardware unit in Hopper for asynchronous tensor transfers. | WGMMA Warp Group Matrix Multiply-Accumulate | Hopper's warp-group-level MMA instruction for higher throughput. | PTX Parallel Thread Execution | NVIDIA's intermediate instruction set architecture for CUDA. | SASS | NVIDIA's machine code binary instructions executed by the GPU. | Warp / Warp Group | A group of 32 threads executing in SIMT fashion; warp groups contain 4 warps 128 threads on Hopper. | Thread Block / Cluster | Groups of threads/warps for cooperative execution. | Asynchronous Copy | Overlapping data transfer with computation using cp.async instructions. | This glossary is a living document. The field of LLM inference engineering evolves weekly — new terms, techniques, and hardware primitives are constantly emerging. Last compiled: August 2026.