{"slug": "llm-inference-glossary-md", "title": "llm-inference-glossary.md", "summary": "An engineer has published a comprehensive living glossary covering over 200 terms related to LLM inference, including core architecture, KV cache management, quantization, batching, speculative decoding, and distributed inference. The reference is designed for engineers building, optimizing, and serving large language models, and it is open to community contributions.", "body_md": "A living reference for engineers building, optimizing, and serving large language models.\n\nLast Updated: August 2026|Contributions welcome\n\n[Core Architecture & Concepts](#1-core-architecture--concepts)[KV Cache & Memory Management](#2-kv-cache--memory-management)[Quantization](#3-quantization)[Batching & Scheduling](#4-batching--scheduling)[Sampling & Decoding Strategies](#5-sampling--decoding-strategies)[Speculative Decoding & Draft Models](#6-speculative-decoding--draft-models)[Distributed Inference](#7-distributed-inference)[Kernel Optimization & Hardware](#8-kernel-optimization--hardware)[Model Serving & Deployment](#9-model-serving--deployment)[Advanced Optimization Techniques](#10-advanced-optimization-techniques)[Hardware & Accelerators](#11-hardware--accelerators)[Model Formats & Serialization](#12-model-formats--serialization)[Serving Frameworks & Runtimes](#13-serving-frameworks--runtimes)[Advanced Attention Variants](#14-advanced-attention-variants)[Context Management & Long Context](#15-context-management--long-context)[Model Merging & Composition](#16-model-merging--composition)[Compilation & Graph Optimization](#17-compilation--graph-optimization)[Inference-Time Agents & Advanced Reasoning](#18-inference-time-agents--advanced-reasoning)[Observability, Cost & Operations](#19-observability-cost--operations)[Benchmarking & Evaluation](#20-benchmarking--evaluation)[Safety, Alignment & Inference-Time Techniques](#21-safety-alignment--inference-time-techniques)[Low-Level GPU/Hardware Primitives](#22-low-level-gpuhardware-primitives)\n\n| Term | Definition |\n|---|---|\nAutoregressive Generation |\nThe process by which LLMs generate tokens one at a time, with each new token conditioned on all previously generated tokens. |\nPrefill Phase |\nThe initial forward pass that processes the input prompt to compute key-value (KV) cache entries for all prompt tokens. Also called the \"context phase.\" |\nDecode Phase |\nThe generation phase where tokens are produced one at a time, each depending on the previous token and the KV cache. |\nPrompt |\nThe input text provided to the model to condition its generation. |\nCompletion |\nThe output text generated by the model in response to a prompt. |\nToken |\nThe atomic unit of text for the model, produced by a tokenizer. Can be a word, subword, or character. |\nTokenizer |\nThe component that converts between raw text and token IDs using algorithms like BPE, WordPiece, or SentencePiece. |\nEmbedding |\nA dense vector representation of a token in a high-dimensional space, learned during training. |\nLogits |\nThe raw, unnormalized output scores from the final linear layer of the model, one per token in the vocabulary. |\nSoftmax |\nThe function that converts logits into a probability distribution over the vocabulary. |\nAttention Mechanism |\nThe core operation in transformers that computes weighted sums of values based on query-key similarity scores. |\nSelf-Attention |\nAttention where queries, keys, and values all come from the same sequence. |\nMulti-Head Attention (MHA) |\nRunning multiple attention operations in parallel with different learned projections. |\nGrouped Query Attention (GQA) |\nA variant where multiple query heads share the same key/value heads, reducing KV cache size. |\nMulti-Query Attention (MQA) |\nAn extreme form of GQA where all query heads share a single key head and single value head. |\nFeed-Forward Network (FFN) |\nThe position-wise fully connected sublayer in each transformer block, typically using SwiGLU, GeGLU, or GELU activations. |\nLayer Normalization |\nNormalization technique applied per layer to stabilize training and inference. |\nRMSNorm |\nRoot Mean Square Layer Normalization, a simplified variant without mean-centering, commonly used in LLaMA and Mistral. |\nRotary Position Embedding (RoPE) |\nA position encoding method that rotates query/key vectors by a position-dependent angle, enabling better length generalization. |\nALiBi (Attention with Linear Biases) |\nPosition encoding that adds linear biases to attention scores instead of explicit embeddings. |\nSliding Window Attention |\nAttention restricted to a fixed-size local window, used in models like Mistral to handle long contexts efficiently. |\nMixture of Experts (MoE) |\nArchitecture where only a subset of parameters (experts) are activated per token, enabling massive scale without proportional compute increase. |\nTop-k Routing |\nIn MoE, selecting the k highest-scoring experts for each token. |\nMLA (Multi-head Latent Attention) |\nDeepSeek's attention mechanism that compresses KV cache via low-rank key-value joint compression. |\n\n| Term | Definition |\n|---|---|\nKV Cache |\nStored key and value tensors from previous tokens, eliminating redundant recomputation during autoregressive generation. |\nKV Cache Size |\nMemory footprint = `2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_size` . |\nPagedAttention |\nvLLM's memory management system that partitions KV cache into fixed-size blocks (pages), enabling efficient sharing and dynamic allocation. |\nBlock Table |\nA mapping from logical token positions to physical memory blocks in PagedAttention. |\nCopy-on-Write (CoW) |\nMemory optimization where KV cache blocks are shared between sequences and only copied when modified, used for beam search and parallel sampling. |\nPrefix Caching |\nReusing KV cache computations for shared prompt prefixes across multiple requests. |\nKV Cache Eviction |\nStrategies to remove less important tokens from cache when memory is constrained (e.g., H2O, Heavy Hitter Oracle). |\nKV Cache Quantization |\nCompressing cached keys/values to lower precision (INT8, FP8, INT4) to reduce memory usage. |\nKV Cache Offloading |\nMoving KV cache to CPU/disk memory when GPU memory is exhausted, trading latency for capacity. |\nFlashAttention |\nIO-aware exact attention algorithm that reduces HBM reads/writes by using tiling and recomputation in SRAM. |\nFlashAttention-2 |\nImproved version with better parallelism, work partitioning, and reduced non-matmul FLOPs. |\nFlashAttention-3 |\nFurther optimized for Hopper architecture with FP8 support, warp-specialization, and interleaved block-wise GEMMs. |\nFlashDecoding |\nExtension of FlashAttention optimized for the decode phase with small batch sizes and long sequences. |\nFlashInfer |\nA unified library for attention kernels supporting various attention variants and sparse patterns. |\nRadixAttention |\nSGLang's attention mechanism that automatically reuses KV cache across multiple calls by treating the cache as a radix tree. |\n\n| Term | Definition |\n|---|---|\nQuantization |\nReducing numerical precision of model weights/activations to decrease memory and increase throughput. |\nPost-Training Quantization (PTQ) |\nQuantizing a pre-trained model without retraining. |\nQuantization-Aware Training (QAT) |\nTraining with simulated quantization to learn robust low-precision representations. |\nWeight-Only Quantization |\nQuantizing only model weights while keeping activations in higher precision. |\nWeight-Activation Quantization |\nQuantizing both weights and activations. |\nINT8 |\n8-bit integer quantization, typically using symmetric or asymmetric schemes. |\nINT4/INT3/INT2 |\nExtreme weight quantization packing multiple weights per byte. |\nFP8 (E4M3/E5M2) |\n8-bit floating point formats defined in the FP8 standard (E4M3 for weights/activations, E5M2 for gradients). |\nNF4 (Normal Float 4) |\nA 4-bit data type with non-uniform bins optimized for normally distributed weights, used in QLoRA. |\nAWQ (Activation-Aware Weight Quantization) |\nPTQ method that protects salient weight channels based on activation magnitudes. |\nGPTQ |\nOne-shot weight quantization using approximate second-order information for layer-wise optimal quantization. |\nSmoothQuant |\nMigration of quantization difficulty from activations to weights via per-channel scaling. |\nLLM.int8() |\nMixed-precision decomposition that keeps outlier features in FP16 while quantizing the rest to INT8. |\nGGUF/GGML |\nFile formats and quantization schemes for efficient CPU inference (Q4_0, Q5_K_M, Q8_0, etc.). |\nQLoRA |\nQuantized LoRA fine-tuning where the base model is quantized to NF4 while LoRA adapters remain in FP16. |\nBitsAndBytes |\nLibrary implementing LLM.int8() and 4-bit quantization for easy integration. |\nDynamic Quantization |\nQuantizing on-the-fly during inference based on runtime activation statistics. |\nStatic Quantization |\nPre-computing quantization parameters from calibration data. |\nGroup-wise Quantization |\nApplying separate quantization parameters to groups of weights (e.g., 128 consecutive weights). |\nChannel-wise Quantization |\nSeparate quantization parameters per output channel. |\nToken-wise Quantization |\nSeparate quantization parameters per token position for activations. |\n\n| Term | Definition |\n|---|---|\nStatic Batching |\nGrouping multiple requests into a fixed batch before inference; all requests wait for the longest one. |\nDynamic Batching |\nContinuously adding new requests to a running batch as others complete. |\nContinuous Batching (In-flight Batching) |\nDynamically adding/removing requests from a running batch at every iteration, maximizing GPU utilization. |\nIteration-level Scheduling |\nScheduling decisions made at each generation step rather than per-request. |\nRequest Scheduling |\nPolicies (FCFS, shortest-job-first, priority-based) for ordering incoming requests. |\nPreemption |\nPausing a request mid-generation to free GPU resources for higher-priority requests, with state saved for resumption. |\nSwap-out/Swap-in |\nMoving KV cache of preempted requests between GPU and CPU memory. |\nChunked Prefill |\nBreaking long prefill computations into smaller chunks and interleaving them with decode steps to prevent decode starvation. |\nPrompt Chunking |\nProcessing long prompts in segments to bound prefill latency. |\nMicrobatching |\nSplitting a batch into smaller sub-batches for pipeline parallelism. |\nMax Batch Size |\nThe upper limit on concurrent sequences, constrained by memory and latency SLOs. |\nMax Sequence Length |\nThe maximum context window supported by the model or deployment. |\nToken Budget |\nA limit on total tokens (prefill + decode) a request can consume. |\n\n| Term | Definition |\n|---|---|\nGreedy Decoding |\nAlways selecting the token with the highest probability. Deterministic but often suboptimal. |\nTemperature Scaling |\nDividing logits by a temperature T before softmax; T < 1 makes distribution sharper (more deterministic), T > 1 makes it flatter (more random). |\nTop-k Sampling |\nRestricting sampling to the k most likely tokens, setting others' probabilities to zero. |\nTop-p (Nucleus) Sampling |\nSampling from the smallest set of tokens whose cumulative probability exceeds p. |\nMin-p Sampling |\nSampling from tokens with probability ≥ min_p × max_probability, adaptive to the distribution shape. |\nTypical Sampling |\nSampling from tokens with information content close to the conditional entropy of the distribution. |\nRepetition Penalty |\nDownscaling logits of tokens that have already appeared in the sequence to reduce repetition. |\nFrequency Penalty |\nPenalizing tokens proportionally to their frequency in the generated text. |\nPresence Penalty |\nApplying a fixed penalty to tokens that have appeared at least once. |\nBeam Search |\nMaintaining k candidate sequences and expanding the most promising ones at each step. |\nDiverse Beam Search |\nModified beam search that encourages diversity among beam candidates. |\nContrastive Search |\nDecoding that penalizes tokens too similar to previous context, improving coherence. |\nSpeculative Sampling |\nA verification-based sampling method used in speculative decoding to accept/reject draft tokens. |\nWatermarked Sampling |\nEmbedding statistical watermarks into generated text for detection/attribution. |\nStructured Decoding / Constrained Decoding |\nEnforcing output to follow a grammar, regex, or JSON schema during generation. |\nGuided Decoding |\nFrameworks like Outlines, Guidance, or lm-format-enforcer for structured generation. |\nLogit Bias |\nAdding fixed values to specific token logits to encourage/discourage their selection. |\nEOS (End-of-Sequence) Token |\nSpecial token signaling the model to stop generating. |\nStop Sequences |\nUser-defined strings that trigger generation termination. |\nMax New Tokens |\nHard limit on the number of tokens to generate. |\n\n| Term | Definition |\n|---|---|\nSpeculative Decoding |\nUsing a small draft model to predict multiple future tokens, which are then verified in parallel by the target model. |\nDraft Model |\nA smaller, faster model that generates candidate token sequences for verification. |\nTarget Model |\nThe full-size model that verifies draft tokens. |\nVerification |\nThe process of checking draft tokens against the target model's true distribution. |\nAcceptance Rate |\nThe fraction of draft tokens accepted by the target model. |\nLookahead Decoding |\nSpeculative decoding using the target model itself via Jacobi iteration, eliminating the need for a separate draft model. |\nMedusa |\nMethod that adds multiple decoding heads to a model for parallel prediction of future tokens. |\nEAGLE |\nEnhanced speculative decoding using auto-regressive heads at the feature level for higher acceptance rates. |\nPrompt Lookup Decoding |\nUsing n-gram matches from the prompt/context as draft tokens instead of a model. |\nRestoration Loss |\nTraining objective for draft models to match the target model's distribution. |\nTree Attention |\nAttention pattern that verifies multiple speculative token sequences simultaneously in a tree structure. |\n\n| Term | Definition |\n|---|---|\nTensor Parallelism (TP) |\nSplitting individual layers across multiple GPUs by partitioning weight matrices (column-wise or row-wise). |\nPipeline Parallelism (PP) |\nDistributing consecutive layers across different GPUs, with activations passed between stages. |\nData Parallelism (DP) |\nReplicating the model across GPUs and processing different requests on each replica. |\nSequence Parallelism |\nDistributing the sequence dimension across GPUs, useful for long-context inference. |\nExpert Parallelism (EP) |\nIn MoE models, distributing different experts across GPUs while keeping shared layers replicated. |\nZeRO (Zero Redundancy Optimizer) |\nMemory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks. |\nAll-Reduce |\nCollective communication operation where tensors are summed/averaged across all GPUs. |\nAll-Gather |\nCollective operation where each GPU receives the full concatenation of tensors from all GPUs. |\nReduce-Scatter |\nReducing and then scattering results across GPUs. |\nPoint-to-Point Communication |\nDirect data transfer between specific GPU pairs, used in pipeline parallelism. |\nCommunication Overlap |\nHiding communication latency by overlapping with computation (e.g., using CUDA streams). |\nNCCL (NVIDIA Collective Communications Library) |\nLibrary optimized for multi-GPU collective operations. |\nRay |\nDistributed computing framework commonly used for serving LLMs across clusters. |\nvLLM |\nHigh-throughput LLM serving engine featuring PagedAttention and continuous batching. |\nTensorRT-LLM |\nNVIDIA's optimized inference library with kernel fusion, quantization, and multi-GPU support. |\nDeepSpeed-Inference |\nMicrosoft's inference optimization framework with ZeRO partitioning and custom kernels. |\nHugging Face TGI (Text Generation Inference) |\nProduction-ready serving stack with flash attention and safetensors support. |\nSGLang |\nEfficient serving framework with structured generation and RadixAttention. |\nLMDeploy |\nEfficient LLM serving with persistent batching and blocked KV cache. |\n\n| Term | Definition |\n|---|---|\nKernel Fusion |\nCombining multiple operations into a single GPU kernel to reduce kernel launch overhead and memory traffic. |\nOperator Fusion |\nMerging mathematically equivalent operations (e.g., bias-add + activation + layernorm). |\nFlash-Decoding |\nFused attention kernel optimized for the memory-bound decode phase. |\nCUTLASS |\nNVIDIA's CUDA template library for high-performance GEMM and convolution kernels. |\ncuBLAS |\nNVIDIA's optimized BLAS library for dense linear algebra. |\nTriton |\nOpen-source Python-like language from OpenAI for writing custom GPU kernels. |\nMarlin |\nOptimized INT4/INT8 GEMM kernel for weight-only quantization on Ampere+ GPUs. |\nFasterTransformer |\nNVIDIA's transformer optimization library (now largely superseded by TensorRT-LLM). |\nCUDA Graphs |\nRecording and replaying sequences of GPU operations to eliminate CPU launch overhead. |\nStream-K |\nWork decomposition strategy for GEMMs that improves GPU utilization for small batch sizes. |\nWarp-specialization |\nAssigning different warps to different phases of computation (e.g., GEMM + softmax in FlashAttention-3). |\nTensor Cores |\nSpecialized GPU units for mixed-precision matrix multiply-accumulate operations. |\nMemory-Bound vs. Compute-Bound |\nClassification of whether performance is limited by memory bandwidth or FLOPS. Decode is typically memory-bound; prefill is compute-bound. |\nArithmetic Intensity |\nRatio of FLOPs to bytes of memory traffic; determines whether a kernel is compute or memory bound. |\nRoofline Model |\nPerformance analysis model plotting achievable performance against arithmetic intensity. |\nHBM (High Bandwidth Memory) |\nGPU memory technology; bandwidth is often the bottleneck for inference. |\nL2 Cache |\nOn-GPU cache layer between HBM and SM registers; important for FlashAttention's tiling strategy. |\nShared Memory / SRAM |\nFast, programmable on-chip memory within each Streaming Multiprocessor (SM). |\nRegister Pressure |\nThe demand for GPU registers; high pressure can limit occupancy. |\nOccupancy |\nThe ratio of active warps to maximum possible warps on an SM. |\nWave Quantization |\nPerformance loss when the problem size doesn't evenly divide the GPU's wave size. |\n\n| Term | Definition |\n|---|---|\nThroughput |\nNumber of tokens or requests processed per unit time (tokens/sec or requests/sec). |\nLatency |\nTime from request submission to first token (TTFT) or between consecutive tokens (TBT/ITL). |\nTime to First Token (TTFT) |\nLatency from request arrival to the first generated token. Dominated by prefill. |\nTime Between Tokens (TBT) / Inter-Token Latency (ITL) |\nTime between consecutive generated tokens. Dominated by decode. |\nEnd-to-End Latency |\nTotal time from request to complete response. |\nLatency SLO |\nService Level Objective specifying maximum acceptable latency. |\nGoodput |\nThe rate of requests that meet their latency SLOs. |\nRequest Rate |\nIncoming requests per second (RPS). |\nAuto-scaling |\nDynamically adjusting the number of serving instances based on load. |\nLoad Balancing |\nDistributing requests across multiple serving instances. |\nModel Sharding |\nSplitting a model across multiple devices for serving. |\nModel Replication |\nCreating multiple copies of a model for parallel request handling. |\nDisaggregated Serving |\nSeparating prefill and decode onto different GPU pools, each optimized for their respective workload. |\nPrefix-Aware Routing |\nRouting requests to instances that already have the prefix cached. |\nStreaming Response |\nSending generated tokens to the client as they are produced, rather than waiting for completion. |\nOpenAI API Compatible |\nServing endpoint implementing the OpenAI chat/completions API format. |\ngRPC |\nHigh-performance RPC framework often used for model serving. |\nREST API |\nHTTP-based API for model inference. |\nModel Registry |\nCentralized storage for model versions and artifacts. |\nA/B Testing |\nServing different model versions to compare performance. |\nShadow Mode |\nRunning a new model in parallel without serving its outputs, for validation. |\nCanary Deployment |\nGradually rolling out a new model version to a subset of traffic. |\nKV Cache Store |\nExternal storage system for KV caches (e.g., Redis, custom stores) for multi-turn conversations. |\nFunction Calling / Tool Use |\nModel capability to invoke external tools/APIs during generation. |\nRAG (Retrieval-Augmented Generation) |\nAugmenting prompts with retrieved documents to ground generation in external knowledge. |\n\n| Term | Definition |\n|---|---|\nLoRA (Low-Rank Adaptation) |\nFine-tuning method that trains low-rank decomposition matrices instead of full weights, enabling efficient adaptation. |\nQLoRA |\nLoRA with a quantized base model, enabling fine-tuning on consumer GPUs. |\nDoRA (Weight-Decomposed Low-Rank Adaptation) |\nDecomposing weights into magnitude and direction for more stable LoRA training. |\nAdapter Layers |\nSmall bottleneck layers inserted into a frozen pretrained model for task-specific adaptation. |\nPrompt Tuning |\nLearning soft prompt embeddings while keeping the model frozen. |\nPrefix Tuning |\nLearning continuous prefix vectors prepended to keys and values in attention layers. |\nP-Tuning v2 |\nScaling prefix tuning to deeper layers for better performance. |\nDistillation |\nTraining a smaller student model to mimic a larger teacher model's behavior. |\nCascade Inference |\nUsing smaller models for easier inputs and larger models only when needed. |\nEarly Exit |\nAllowing the model to exit at intermediate layers for simple inputs. |\nSparse Attention |\nAttention patterns that skip certain token pairs (e.g., Longformer, BigBird, Ring Attention). |\nRing Attention |\nDistributed attention for extremely long sequences by partitioning along the sequence dimension in a ring topology. |\nStriped Attention |\nAttention pattern that alternates between local and global attention. |\nH2O (Heavy Hitter Oracle) |\nKV cache eviction policy that retains tokens with high accumulated attention scores. |\nStreamingLLM |\nEnabling infinite-length generation by maintaining attention sinks (initial tokens) and a rolling KV cache. |\nLM-Infinite |\nFramework for handling infinite context via segment-based processing. |\nActivation Checkpointing |\nRecomputing activations during backward pass to save memory (relevant for training, sometimes inference). |\nGradient Checkpointing |\nSame as activation checkpointing. |\nCPU Offloading |\nMoving model weights or KV cache to CPU memory when GPU memory is full. |\nDisk Offloading |\nUsing disk storage for model weights when CPU/GPU memory is insufficient. |\nModel Compression |\nGeneral term for techniques reducing model size (pruning, quantization, distillation). |\nPruning |\nRemoving less important weights or neurons from a model. |\nStructured Pruning |\nRemoving entire structures (heads, layers, channels) for hardware efficiency. |\nUnstructured Pruning |\nRemoving individual weights, requiring sparse matrix support. |\nSparseGPT |\nOne-shot pruning method for GPT-style models using approximate Hessian information. |\nWanda |\nPruning weights based on the product of weight magnitude and activation norm. |\nMamba / State Space Models (SSMs) |\nAlternative architectures to transformers with linear-time sequence modeling and no KV cache. |\nRWKV |\nRNN-like architecture combining transformer parallelizability with RNN-like memory efficiency. |\nRetNet |\nArchitecture proposing retention mechanism as an alternative to attention for efficient decoding. |\nMixture of Depths |\nDynamically skipping layers for less important tokens. |\nLayerSkip |\nTraining models to be robust to layer skipping for early-exit inference. |\n\n| Term | Definition |\n|---|---|\nTPU (Tensor Processing Unit) |\nGoogle's ASIC optimized for matrix operations; TPU v4/v5p/v5e used for LLM training/inference. |\nGroq LPU (Language Processing Unit) |\nGroq's deterministic tensor streaming processor with no external HBM, enabling extremely low and predictable latency. |\nAWS Inferentia / Trainium |\nAmazon's custom AI chips (Inf2, Trn1) for cost-efficient inference. |\nSambaNova SN40L |\nReconfigurable dataflow accelerator with large on-chip memory for model weights. |\nCerebras Wafer-Scale Engine |\nMassive wafer-scale chip designed for training and inference of extremely large models. |\nQualcomm AI Stack / NPU |\nMobile/edge inference on Snapdragon NPUs via ONNX/QNN. |\nApple Neural Engine (ANE) |\nDedicated neural processing unit in Apple Silicon for on-device inference. |\nNVIDIA H100/H200/Blackwell |\nGPU generations with Transformer Engine, FP8 support, and NVLink/NVSwitch for scale-out. |\nNVLink / NVSwitch |\nHigh-bandwidth interconnect for multi-GPU communication within a node. |\nInfiniBand / RoCE |\nHigh-speed networking for inter-node GPU clusters. |\nTransformer Engine |\nNVIDIA library and hardware feature for automatic FP8 precision management. |\n\n| Term | Definition |\n|---|---|\nSafetensors |\nHugging Face's secure tensor format (no pickle, memory-mappable, faster loading). |\nPickle / PyTorch .bin |\nTraditional PyTorch serialization format (security risk due to arbitrary code execution). |\nONNX (Open Neural Network Exchange) |\nCross-platform model format for portable inference. |\nTensorRT Engine |\nNVIDIA's compiled, optimized model format for deployment. |\nCore ML |\nApple's format for on-device inference on iOS/macOS. |\nTFLite (TensorFlow Lite) |\nLightweight format for mobile and edge devices. |\nOpenVINO IR |\nIntel's intermediate representation for optimized inference on Intel hardware. |\nGGUF (GPT-Generated Unified Format) |\nSuccessor to GGML; supports metadata, multiple quantization types, and CPU inference. |\n\n| Term | Definition |\n|---|---|\nllama.cpp |\nC/C++ implementation of LLaMA optimized for CPU and Apple Silicon inference. |\nOllama |\nUser-friendly wrapper around llama.cpp for local model running. |\nTriton Inference Server |\nNVIDIA's multi-framework serving platform with dynamic batching and model ensemble. |\nONNX Runtime |\nCross-platform inference engine supporting ONNX models with hardware accelerators. |\nOpenVINO |\nIntel's toolkit for optimizing and deploying deep learning models. |\nTorchServe / TensorFlow Serving |\nFramework-native serving solutions. |\nMLflow / BentoML |\nModel packaging and serving orchestration platforms. |\nKServe |\nKubernetes-native model serving platform. |\nRay Serve |\nScalable model serving built on Ray distributed framework. |\nModal / Beam / Replicate |\nServerless GPU inference platforms. |\n\n| Term | Definition |\n|---|---|\nLinear Attention |\nAttention approximations reducing complexity from O(n²) to O(n) via kernel feature maps. |\nPerformer |\nUses FAVOR+ (Fast Attention Via Orthogonal Random Features) for linear-time attention. |\nRFA (Random Feature Attention) |\nApproximates softmax attention using random Fourier features. |\nCosFormer |\nLinear attention using cosine-based reweighting for locality bias. |\nLocal Attention / Strided Attention |\nRestricting attention to local windows or fixed strides. |\nDilated Attention |\nSkipping tokens at regular intervals within attention windows. |\nFactorized Attention |\nDecomposing full attention into multiple cheaper patterns. |\n\n| Term | Definition |\n|---|---|\nContext Compression |\nTechniques to summarize or compress long contexts into shorter representations. |\nHierarchical Attention |\nMulti-scale attention operating on token, sentence, and paragraph levels. |\nMemory-Augmented Networks |\nExternal memory modules the model can read/write during generation. |\nMemGPT |\nOS-inspired virtual context management paging memory in/out of context. |\nLoRA-XS / LongLoRA |\nEfficient fine-tuning specifically for extending context windows. |\nYaRN (Yet another RoPE extension method) |\nInterpolation/extrapolation technique for extending RoPE-based models to longer contexts. |\nNTK-Aware Scaling |\nNon-linear interpolation of RoPE base frequencies for context extension. |\nPI (Positional Interpolation) |\nLinearly interpolating position indices to fit longer sequences into trained context limits. |\nSelf-Extend |\nTraining-free context extension by grouping positions. |\n\n| Term | Definition |\n|---|---|\nModel Soups |\nAveraging weights of multiple fine-tuned models. |\nTask Arithmetic |\nAdding/subtracting task vectors (fine-tuned minus pre-trained weights) to compose capabilities. |\nTIES-Merging |\nTrimming, electing sign, and disjoint merging to resolve interference between models. |\nDARE (Drop And REscale) |\nDropping a large fraction of delta parameters before merging to reduce interference. |\nSLERP (Spherical Linear Interpolation) |\nInterpolating model weights on a hypersphere. |\nModel Breadcrumbs |\nSparse task vectors for efficient model editing. |\n\n| Term | Definition |\n|---|---|\ntorch.compile |\nPyTorch's JIT compilation with backends like inductor, cudagraphs, and Triton. |\nInductor |\nPyTorch 2.0's default compiler backend generating Triton/C++ kernels. |\nXLA (Accelerated Linear Algebra) |\nGoogle's compiler for linear algebra, used with JAX/TensorFlow on TPUs/GPUs. |\nJAX |\nGoogle's composable ML framework with JIT compilation via XLA. |\nTVM / Apache TVM |\nOpen-source deep learning compiler stack. |\nMLIR (Multi-Level Intermediate Representation) |\nLLVM subproject for representing and transforming ML graphs. |\nIREE (Intermediate Representation Execution Environment) |\nMLIR-based runtime for edge deployment. |\n\n| Term | Definition |\n|---|---|\nReAct (Reasoning + Acting) |\nInterleaving reasoning traces with tool/actions in a loop. |\nReflexion |\nSelf-reflective agents that learn from verbal feedback. |\nPlan-and-Solve |\nDecomposing problems into sub-plans before execution. |\nProgram-Aided LLMs (PAL) |\nUsing LLMs to generate code that solves problems programmatically. |\nToolFormer |\nTraining LLMs to decide when and how to call external APIs. |\nGorilla / APIBench |\nModels fine-tuned for accurate API calling. |\nDSPy |\nFramework for programming language models with declarative modules. |\nLangChain / LlamaIndex |\nOrchestration frameworks for RAG and agentic applications. |\n\n| Term | Definition |\n|---|---|\nToken Pricing |\nCost model based on input (prompt) and output (completion) tokens. |\nContext Window Pricing |\nSome providers charge based on context length regardless of generation. |\nBatch Pricing |\nDiscounted rates for offline/non-real-time inference. |\nReserved Throughput |\nGuaranteed capacity provisioning at fixed cost. |\nSpot/Preemptible Instances |\nCheaper but interruptible compute for batch inference. |\nModel Telemetry |\nLogging latency, throughput, error rates, and token counts. |\nPrompt Injection Detection |\nGuardrails to detect and block malicious prompt patterns. |\nOutput Moderation |\nReal-time content filtering of generated text. |\nPII Redaction |\nRemoving personally identifiable information from inputs/outputs. |\n\n| Term | Definition |\n|---|---|\nTokens per Second (tok/s) |\nPrimary throughput metric for inference systems. |\nFLOPs (Floating Point Operations) |\nCount of arithmetic operations; used to measure computational cost. |\nFLOPS (Floating Point Operations Per Second) |\nMeasure of compute throughput. |\nMemory Bandwidth |\nRate at which data can be read from/written to GPU memory (GB/s). |\nModel FLOPs Utilization (MFU) |\nRatio of actual throughput to theoretical peak FLOPS, accounting for memory bandwidth and overhead. |\nModel Memory Bandwidth Utilization (MBU) |\nRatio of actual memory bandwidth used to theoretical peak. |\nBenchmark Suites |\nStandardized evaluation sets: MMLU, HellaSwag, ARC, TruthfulQA, HumanEval, GSM8K, etc. |\nPerplexity (PPL) |\nExponential of average negative log-likelihood; measures model confidence in a text. |\nLong-Context Benchmarks |\nNeedle-in-a-haystack, passkey retrieval, long-document QA for evaluating context window effectiveness. |\nLMBench / LLMPerf |\nBenchmarking frameworks for measuring serving performance. |\nAnyScale LLMPerf |\nOpen-source benchmark for LLM serving systems. |\nArtificial Load Testing |\nGenerating synthetic request patterns to stress-test serving infrastructure. |\n\n| Term | Definition |\n|---|---|\nSystem Prompt |\nInstructions prepended to every conversation to guide model behavior. |\nJailbreaking |\nPrompt engineering techniques designed to bypass safety guardrails. |\nPrompt Injection |\nAttacks that embed malicious instructions within user input to manipulate model behavior. |\nRLHF (Reinforcement Learning from Human Feedback) |\nTraining method using human preferences to align model outputs. |\nDPO (Direct Preference Optimization) |\nSimpler alternative to RLHF that directly optimizes on preference pairs without a separate reward model. |\nConstitutional AI |\nTraining models using a set of principles (constitution) for self-correction. |\nInference-Time Compute Scaling |\nAllocating more computation at inference (e.g., Chain-of-Thought, best-of-N sampling) to improve output quality. |\nChain-of-Thought (CoT) |\nPrompting the model to show intermediate reasoning steps. |\nTree of Thoughts (ToT) |\nGeneralizing CoT to explore multiple reasoning paths and backtrack. |\nSelf-Consistency |\nGenerating multiple reasoning paths and selecting the most frequent answer. |\nBest-of-N Sampling |\nGenerating N candidates and selecting the best via a reward model or verifier. |\nProcess Reward Model (PRM) |\nModel that scores individual reasoning steps rather than final outputs. |\nOutcome Reward Model (ORM) |\nModel that scores final outputs. |\nVerifier |\nModel trained to judge correctness of generated outputs. |\nMajority Voting |\nAggregating multiple samples by selecting the most common answer. |\nMonte Carlo Tree Search (MCTS) |\nUsing MCTS to explore the reasoning space during inference. |\n\n| Term | Definition |\n|---|---|\nTMA (Tensor Memory Accelerator) |\nHardware unit in Hopper for asynchronous tensor transfers. |\nWGMMA (Warp Group Matrix Multiply-Accumulate) |\nHopper's warp-group-level MMA instruction for higher throughput. |\nPTX (Parallel Thread Execution) |\nNVIDIA's intermediate instruction set architecture for CUDA. |\nSASS |\nNVIDIA's machine code (binary instructions) executed by the GPU. |\nWarp / Warp Group |\nA group of 32 threads executing in SIMT fashion; warp groups contain 4 warps (128 threads) on Hopper. |\nThread Block / Cluster |\nGroups of threads/warps for cooperative execution. |\nAsynchronous Copy |\nOverlapping data transfer with computation using `cp.async` instructions. |\n\n*This glossary is a living document. The field of LLM inference engineering evolves weekly — new terms, techniques, and hardware primitives are constantly emerging. Last compiled: August 2026.*", "url": "https://wpnews.pro/news/llm-inference-glossary-md", "canonical_source": "https://gist.github.com/abhigyan631/2208e8e6d2dfb2249b76a1afcdb63fe8", "published_at": "2026-08-06 09:32:03+00:00", "updated_at": "2026-08-13 05:17:26.145353+00:00", "lang": "en", "topics": ["large-language-models", "mlops", "ai-infrastructure", "developer-tools"], "entities": ["LLaMA", "Mistral"], "alternates": {"html": "https://wpnews.pro/news/llm-inference-glossary-md", "markdown": "https://wpnews.pro/news/llm-inference-glossary-md.md", "text": "https://wpnews.pro/news/llm-inference-glossary-md.txt", "jsonld": "https://wpnews.pro/news/llm-inference-glossary-md.jsonld"}}