# Ace the AI Engineer Interview: LLM Fundamentals > Source: > Published: 2026-08-23 22:01:01+00:00 A Practical Cheat Sheet for Core LLM Concepts, Architecture, Training, and Evaluation for AI Engineer Interviews 0. Prep Principals Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime. The AI era is evolving furiously. I will do my best to keep this blog updated, but it is impossible to cover every potential interview question for every company. So, I hope this note serves as a source of inspiration rather than a bible to follow mechanically. Prepare as if you were a real engineer. Go through a toy AI project, such as a chatbot, and think deeply about the following: Break down each step and identify the practical problems that may arise. Explore what techniques are commonly used to solve those problems. Think through the trade-offs between different techniques, and how you would make decisions under different scenarios. I believe that, by doing this, YOU CAN eventually build a strong, transferable skill set, shine, and ACE the AI Engineer Interviews. 1. NLP Data Augmentation A common problem in LLM training is data insufficiency. Thus data augmentation is a frequented tested point. The data augmentation in NLP area typically can be classified into two categories: rule-based (more classic), and generative (more modern). The details are shown below: 1.1. Rule-based Revise a small portion of the document using explicit rules; in many cases, simple functions with dictionaries or a small LLM are sufficient. Paraphrasing/Extension Synonym replacement(random swap) : E.g. A good person -> A nice person.) Format change: E.g. datetime yyyy-mm-dd -> mm/dd/yyyy, money USD 1.5k -> $1,500) Random insertion/deletion: E.g. A dog -> An old dog. Noise injection： World/lexical level: E.g. spelling errors ,including keyboard proximity, ocr error. Sentence/syntactic level: E.g.grammar/syntax errors like word order, tense, singular/plural, upper/lower case. Document level: E.g. adding irrelevant sentences. Comparison: Paraphrasing vs Noise Injection Paraphrasing/Extension — increase generalization (LLM can handle broader valid cases) Noise injection — increase robustness (LLM can handle erroneous inputs.) 1.2. Generative Generate a large portion of the document, typically relying on the strong general-purpose intelligence of a large LLM. Paraphrasing From given textual data, produce alternative surface forms that preserve the same semantic information. For example: Rephrasing/rewriting: express the same sentence in a different way. Back-translation: translate into another language and then translate back. Partial rewriting: E.g. for a (Q, A) pair, create multiple paraphrased versions of A for the same Q. Text Generation: Introduce new semantic information based on existing textual or non-textual inputs. For example: From-scratch: generate text from non-text data. E.g., given relational DB query result like “capital (America)-> Washington”, generate “The capital of America is Washington.” Expansion: turn a short document into a longer one. E.g. Summary Expansion — Expanding a summary into a detailed passage. Partial generation: fill in missing parts of a document. E.g. Pseudo User Query — generate Q based on a real fact A; Data labeling — generate A for an existing Q. 1.3. Comparison: Rule-based vs Generative Rule-based: Pro: cheaper and more controllable in terms of quality and correctness. Con: limited coverage and weaker generalization. Best for: small datasets and cold-start (initial training/data augmentation) phases. Generative Con: less controllable; quality depends on a strong model, with higher cost and more complex validation. Pro: stronger at improving generalization. Best for: scenarios where a powerful model is already available. Trade-off summary: think in terms of cold-start vs mature stage, the required level of controllability, cost and validation complexity. 1.4. Risk/Cons of Data Augmentation Overfitting: the model may memorize specific augmented samples instead of learning general patterns. Data quality degradation: the model may pick up noise and errors from augmented data and ultimately generate lower-quality outputs. 2. LLM Architecture 2.0. Why LLM? (Comparison: Transformer vs RNN) Notation: Assume the sequence length is O(T). Training LLM’s pro vs RNN Long‑term dependency: Transformers model global token‑to‑token dependencies, whereas in RNNs, even with a memory state, long‑term information typically decays over time. Faster training (and inference prefilling) speed: processing existing input tokens can be parallelized in Transformers, while it is sequential in RNNs. I.e., sequential wall-time complexity: Transformer O(1) vs RNN O(T), assuming enough parallel hardware. LLM’s con vs RNN High memory usage: For RNNs, the model parameters are shared across tokens, but during training, we need to store per-token hidden states/activations for backpropagation (BP), so the memory consumption over the sequence is O(T). For Transformers, storing per-token activations is also O(T), but the peak memory often comes from sequence-level pairwise attention computation, such as the QKᵀ attention matrix, which takes O(T²) memory. In short, for a sequence of length T, memory complexity: Transformer O(T²) vs RNN O(T). Higher FLOPs complexity: Standard Transformer self-attention computes pairwise interactions between tokens. For each token, attending to all other tokens costs O(T), so for a sequence of length T, the total compute complexity is O(T²). In contrast, an RNN processes one token at a time together with a hidden state that compresses information from previous tokens, leading to O(1) compute per token and O(T) compute for the full sequence. In short, for a sequence of length T, FLOPs complexity: Transformer O(T²) vs RNN O(T). Inference LLM’s pro vs RNN Long-term dependency: Same explanation as training. LLM’s tie vs RNN Wall-time complexity: Both vanilla Transformers and RNNs follow the autoregressive generation pattern, decoding one token at a time. Therefore, sequence generation wall-time complexity for both models: O(T). LLM’s con vs RNN High memory usage: For RNNs, decoding each token only requires keeping the current hidden state, leading to O(1) memory complexity with respect to sequence length. For Transformers, memory usage depends on whether KV cache is used. Without KV cache, Transformer decoding recomputes attention over the full prefix, and the attention matrix QKᵀ can take O(T²) memory at sequence length T. With KV cache, Transformers store K/V states for all previous tokens, leading to O(T) resident memory during decoding. In short, inference memory complexity: Transformer O(T) (w/ KV cache, O(T²) w/o KV cache) vs RNN O(1). Higher FLOPs complexity: For RNNs, each token requires O(1) compute to process the current token and update the hidden state, leading to O(T) compute for a sequence of length T. For Transformers, even with KV cache, each newly decoded token still attends to all previous tokens, which costs O(T) at sequence length T. Therefore, sequence-level FLOPs complexity: Transformer O(T²) vs RNN O(T). 2.1. Transformer Structure General components in Transformer: Embedding layer / lookup table, positional encoding, Transformer blocks, linear layer, and softmax head. Similarity between encoder and decoder models? Both encoder and decoder models mostly use token-level cross-entropy loss in pre-training. (The difference is just in the prediction objective: encoders often predict masked tokens in MLM, while decoders predict the next token in CLM / autoregressive language modeling.) Difference between encoder and decoder model? Encoder Training method / mask: MLM Prediction target: representation / embedding of the input token or sequence. Multi-head self-attention: No intentional causal mask for valid tokens; only uses an attention mask for padding tokens. Training method / mask: CLM / Autoregression. Prediction target: probability distribution of the next token after the input tokens. Masked multi-head self-attention: Self-attention with a causal mask, so each token can only attend to previous tokens and itself. Cross-attention: Cross-attention means Q comes from the model’s current sequence, while K/V come from another sequence, modality, or external memory. By definition, an encoder is not forbidden from using cross-attention. However, in a common seq2seq setting, the encoder is used to interpret sequence A, while the decoder uses cross-attention to acquire information from sequence A and generate sequence B. To enrich the representation and increase model expressivity. This is similar in spirit to the SVM kernel trick: a structure that is hard to separate in a lower-dimensional space may become easier to represent or separate in a higher-dimensional space. 2.2. Attention Mechanism Describe the (self) attention mechanism: The output is a weighted sum of all tokens’ value vectors (V), where the weights are computed by applying softmax to the dot products between the current token’s query (Q) and other tokens’ keys (K), scaled by sqrt(dim). Note: The “self” in self-attention means that Q, K, and V are all generated from the same sequence. In contrast, in cross-attention, Q comes from one sequence, while K and V come from another. Describe the self-attention formula: QKV generation: The Q, K, and V matrices are generated from the input token embeddings through linear transformations. QK product: Each token’s query vector (Q) takes a dot product with all tokens’ key vectors (K) to produce the raw attention logits. Scaling: The QK logits are scaled by sqrt(d_k). Causal mask: Add -inf to the logits corresponding to future tokens relative to the current position, so they receive zero attention weight after softmax. Softmax: Apply softmax to the logits to obtain normalized attention weights. Weighted sum: The attention output is a weighted sum of all tokens’ value vectors (V), using the attention weights. Why scale the dot product by sqrt(d_k)? The dot product is scaled by sqrt(d_k) to keep its variance roughly constant, reducing the chance of extremely large attention logits. Extremely large logits can saturate the softmax, making the output probabilities close to 0 or 1. In these saturated regions, gradients become very small, which can lead to vanishing gradients and slower or less stable training. Why does the attention mechanism work? (i.e. what are its advantages over traditional language models?) Autoregressive task: Attention vs. RNN Long-term dependency: RNNs compress previous information into a recurrent hidden state, which makes long-range dependencies harder to preserve. Transformers can directly attend to tokens across the full available context. Parallelism: RNNs process tokens sequentially, while Transformer training can process tokens in parallel across sequence positions. Explainability: The hidden state or memory cell in an RNN/LSTM is difficult to interpret directly, while a Transformer’s attention weights can be visualized to inspect which tokens attend to which others. Embedding task: Transformer vs. Static embedding techniques Contextualized representation: Classical embeddings such as Word2Vec and GloVe are static, meaning a word has the same embedding regardless of context; While modern transformer-based embeddingsare contextualized, meaning the representation of a token changes depending on the surrounding context. Note: the Transformer’s input embedding lookup table is also static. However, in most downstream applications, we use the contextualized hidden states produced by the Transformer rather than the raw lookup-table embeddings. What is Multi-Head Attention (MHA)? What are its advantages over single-head attention? Definition The input embeddings are sent to multiple attention heads. Each head has its own Q, K, and V projection matrices and computes attention independently. The outputs from all heads are then concatenated and passed through a final output projection. Advantages: Richer representation space: Different heads can attend to different representation subspaces and different token positions, potentially capturing different types of information, such as semantic relationships, syntactic patterns, and short- or long-range dependencies. Multiple attention patterns: Compared with a single head, MHA allows the model to represent multiple attention patterns in parallel. Typical number of heads: Common choices include 8, 12, 16, 32, or more, depending on the model size. Very large models may use substantially more heads. 2.3. Tokenization What is tokenization? Tokenization is the process of splitting a text string into smaller units called tokens, such as words, subwords, or characters. Advantage of subword tokenization over word-level tokenization? Fewer OOV tokens Smaller vocabulary size Trade-off of using a larger vocabulary in subword tokenization? Pro: Shorter sequences and more direct representation of common words/subwords Con: Higher memory and computation cost; rare tokens require more data to learn well What are common subword tokenization algorithms, and how do they differ? Byte-Pair Encoding (BPE) Algorithm: Start with a base vocabulary of small units, typically characters or bytes. Count adjacent token-pair frequencies in the training corpus. Iteratively merge the most frequent adjacent pair into a new token. Stop when the target vocabulary size or number of merges is reached. Key idea: Frequency-based bottom-up merging. Frequent adjacent token pairs are gradually combined into larger subword tokens. Merge criterion: Typically select the most frequent adjacent pair: Score(w1, w2) = Count(w1, w2) BPE was adapted for subword tokenization by iteratively merging frequent symbol pairs. WordPiece Algorithm: Similar to BPE, WordPiece builds a subword vocabulary by combining smaller units into larger units. However, instead of selecting merges purely by pair frequency, it prefers merges that better improve the language-model likelihood / vocabulary objective. Key idea: Likelihood-oriented merging rather than pure frequency-based merging. Common intuition for merge score:Score(w1, w2) ≈ Count(w1, w2) / (Count(w1) * Count(w2)) This favors pairs that occur together more strongly relative to the individual frequencies of their components. Common usage: WordPiece is strongly associated with Google NLP systems and is used by BERT. Unigram Language Model Algorithm: Unlike BPE and WordPiece, Unigram does not build the vocabulary primarily through iterative merging. Start with a large candidate subword vocabulary. Assign a probability to each candidate token. Model the probability of a tokenized sequence as the product of its token probabilities. Iteratively remove tokens whose removal causes the smallest degradation in the corpus likelihood. Stop when the target vocabulary size is reached. Key idea: Top-down vocabulary pruning based on a probabilistic language-model objective. Difference from BPE: BPE: start small and merge tokens. Unigram: start large and prune tokens. The Unigram LM method explicitly defines a probabilistic segmentation model and iteratively prunes candidate pieces while preserving corpus likelihood as much as possible. SentencePiece SentencePiece is not a tokenization algorithm itself. It is a tokenization framework / library. It can train and apply multiple tokenization models, especially: BPE Unigram Language Model A key feature is that it can train directly from raw text rather than requiring language-specific pre-tokenization. SentencePiece’s official implementation supports both BPE and Unigram segmentation models. 2.4. Positional Encoding Why do we need positional encoding (PE)? PE helps the model distinguish tokens at different positions. Without PE, the same token at different positions would have the same input embedding, and self-attention itself has no inherent notion of token order. How does positional encoding work (i.e., help distinguish same tokens at different positions)? For different positions, it incorporates different positional information into the token representations, either directly or indirectly, allowing the model to distinguish the same token appearing at different positions. What are the main types of positional encoding? Explain their key ideas and compare their differences. Absolute Positional Encoding (APE): Each absolute position is assigned a specific positional representation. Con: Learned APE is typically tied to a predefined maximum position range and does not naturally provide translation invariance. Relative Positional Encoding (RPE): Encodes the relative position or distance between token pairs rather than only their absolute positions. E.g. A common approach is to add a relative-position term or bias to the attention score: softmax(QK^T / sqrt(d_k) + B_relative) Pro: Translation invariant: shifting both tokens by the same offset preserves their relative position. Con: The positional signal is relatively simple and depends on the chosen distance-bias slopes. Rotary Positional Embedding (RoPE): Applies position-dependent rotations to Q and K based on their absolute positions. The resulting Q-K dot product naturally depends on their relative position. Pro: Combines absolute-position-based transformations with relative-position-aware attention, providing a richer positional signal than a simple scalar distance bias. When should we choose which positional encoding? APE: Does not have translation invariance, so it is less suitable for long text. E.g. Learnable APE: Simple, suitable for short text with clear positional patterns. Pro: Trainable, potentially more accurate. Con: Limited context window. Example: Short-text encoder. RPE (e.g., ALiBi / T5): Suitable for long context with relatively simple positional patterns. Con: The positional effect is often mainly distance-based, which may be too simple. Example: Sentence-level translation. RoPE: Combines long-range decay with local oscillation, making it suitable for more complex positional patterns. Example: Agents, long documents. 2.5. Normalization Why do we need normalization layers? DNNs, including Transformers, have inherent training stability problems: Internal covariate shift: Parameter updates change the distribution (e.g., mean and variance) of intermediate activations. Such distribution shifts can be amplified through deeper layers. This may lead to unstable gradients, including gradient explosion or vanishing. Batch randomness: Different batches may have different data distributions, making training unstable, slower, or even causing divergence. How does normalization help? Stabilizes the input distribution of each layer (e.g., around zero mean and unit variance). Makes optimization more stable and gradient flow smoother, leading to more efficient training. What normalization does the Transformer use? Explain it. LayerNorm: Normalizes across the feature / embedding dimensions within each token position independently. Why is BatchNorm (BN) not suitable for autoregressive (AR) tasks? BN requires: 1. i.i.d. samples, and 2. a large batch size. I.i.d.: Tokens within the same sequence are highly correlated, so they are not i.i.d. The same token positions across different sequences may also have different distributions due to variable sequence lengths and [PAD] tokens. Batch size: Long sequences and large autoregressive models consume significant memory, often limiting batch size. A smaller batch provides fewer samples for estimating mean and variance, making BN statistics noisier and training less stable. Why does LayerNorm (LN)not have the above problems for AR tasks? LayerNorm normalizes each token independently across its feature / embedding dimensions, so it does not depend on other tokens in the sequence or other samples in the batch. What is the core difference between Pre-LN and Post-LN? Post-LN: LayerNorm is applied to both the transformation output and the skip connection. Pre-LN: LayerNorm is applied only to the transformation output, leaving the skip connection unaffected. This helps preserve a cleaner gradient path through the skip connection, leading to smoother gradient flow and better scalability for deeper Transformer architectures. What is the difference between RMSNorm andLayerNorm? LayerNorm: Normalizes using both the mean and variance: mu = mean(emb) sigma = sqrt(mean((emb - mu)^2)) emb' = gamma * (emb - mu) / sigma + beta RMSNorm: A simplified variant of LayerNorm that does not subtract the mean: RMS(emb) = sqrt(mean(emb^2)) emb' = gamma * emb / RMS(emb) Pros of RMSNorm: Lower computation cost because no mean subtraction is required. Slightly fewer learnable parameters if the LayerNorm bias term is omitted. Often achieves performance comparable to LayerNorm. 3. Pre-training 3.1. Pre-training Configuration What regularization techniques are commonly used in LLM training? Weight decay (L2 regularization) Dropout Early stopping Data augmentation Other related training-stability techniques? Gradient clipping Layer Normalization How to train a new LLM from scratch? What are the key hyperparameters / configurations? Initialization Xavier initialization: Keeps input and output variance at a similar scale. More suitable for sigmoid / tanh-like activations. Kaiming (He) initialization: Uses a larger variance to account for ReLU truncating negative activations. Suitable for ReLU-like activations. Note: For Transformers, initialization is often architecture-specific rather than simply choosing Xavier or Kaiming. Optimizer Adam: Adaptive learning rate + momentum. AdamW: Adam with decoupled weight decay; commonly used for LLM training. Learning Rate (LR) Model- and scale-dependent; commonly around 1e-3 to 1e-4 for smaller models, and often lower for larger models. LR Scheduler Warmup: Gradually increase the global LR at the beginning to avoid early training divergence. Cosine decay: Gradually decrease the LR after warmup to reduce oscillation and improve convergence. Batch Size For LLM pretraining, global batch size is often measured by the number of tokens per optimization step, rather than only the number of sequences. Large-scale training may use millions of tokens per step; the exact value depends strongly on model size and compute scale. Regularization and training-stability techniques See questions above Quantization Commonly BF16 mixed-precision training. Lower-precision formats such as FP8 may also be used on supported large-scale training systems. What is stored in the optimizer state, e.g., AdamW? First moment (exp_avg): Exponential moving average of gradients; acts like a momentum-smoothed gradient or “velocity.” Second moment (exp_avg_sq): Exponential moving average of squared gradients; used to adapt the effective learning rate for each parameter. Step count: Number of optimization steps; used for bias correction of the first and second moment estimates. 3.2. Scaling Law Explain the scaling law? The scaling law states that test loss decreases approximately as a power law when increasing: Compute budget Dataset size Model size (number of parameters) Equivalently, in log-log space, test loss decreases approximately linearly with these scaling factors. Why do we need scaling laws? LLM training is expensive. Scaling laws help us understand and predict the trade-off between model performance (e.g., loss) and cost factors such as data size, model size, and compute budget. 3.3. Memory-Bound vs. Compute-Bound What is the difference between memory-bound and compute-bound? Memory-bound: More time is spent moving / accessing data than performing computation; compute units may wait for data. Compute-bound: More time is spent performing computation; performance is limited mainly by data throughput. What are common indicators of memory-bound vs. compute-bound? GPU utilization: High GPU utilization → more likely compute-bound. Low GPU utilization → more likely memory-bound. Explanation: Lower GPU utilization may indicate that more time is spent waiting for data movement for the same amount of work. Amount of data generated: More data generated → more likely memory / I/O-bound. Less data generated → more likely compute-bound. Example: If 10 numbers generate 100 numbers, more output data needs to be moved, making the operation more likely memory-bound. If 10 numbers generate 1 number, less time is spent on data movement, making it more likely compute-bound. When is a Transformer memory-bound vs. compute-bound? Transformer-block wise: FFN → Compute-bound: Input is O(T × d), output is also O(T × d), while substantial matrix multiplication is performed. Attention → Memory-bound: Input is O(T × d), while the attention matrix is O(T²), causing significantly more data movement / I/O. Sequence-length wise: Long sequence → More memory-bound: QK^T scales as O(T²), causing more attention data movement. Short sequence → More compute-bound: Less time is spent on attention I/O relative to computation. Batch-size wise: Large batch size → More compute-bound: Higher GPU utilization and better compute saturation. Small batch size → More memory-bound: Lower GPU utilization, with relatively more time spent waiting on data movement. Methods for compute-bound and memory / IO-bound problems? KV Cache: Compute-memory trade-off Reduces FLOPs by storing and reusing previous K and V states instead of recomputing them. Trade-off: Increases memory usage and KV-cache reads. Note: KV cache does not make decoding I/O-bound simply because “cache is expensive.” It significantly reduces recomputation, so compute cost drops; at long context, reading the growing KV cache can then become the bottleneck. Gradient Checkpointing: Memory-compute trade-off (Used only during training, not inference.) Stores only selected activations during forward propagation and recomputes the missing activations during backward propagation. Reduces memory usage by increasing FLOPs. Conceptually, somewhat opposite to KV cache: recompute instead of store. FlashAttention: I/O optimization Reduces HBM I/O by tiling attention computation and avoiding materialization of the full attention matrix. May use some recomputation / extra on-chip computation to reduce expensive memory traffic. Trade-off: More local computation for significantly less HBM I/O. Model compression (e.g., quantization, pruning, distillation): Can reduce both computation and memory / I/O cost. Exact benefit depends on the method and hardware support. Distributes memory and computation across multiple devices. Reduces per-device memory and compute workload. Does not necessarily reduce total FLOPs or total I/O; it mainly improves wall-clock time at the cost of more hardware and communication overhead. 3.4. Flash Attention What is FlashAttention? FlashAttention splits large Q, K, and V tensors into blocks and computes attention block by block in fast on-chip memory (SRAM). It fuses attention operations and avoids materializing the full O(L²) attention matrix in HBM, significantly reducing memory I/O. Why do we need FlashAttention, and how does it work? Problem in attention calculation process: Standard attention generates large intermediate matrices: 1. QK^T 2. Softmax(QK^T) 3. Softmax(QK^T) @ V These intermediate results may require expensive reads / writes between HBM and on-chip memory, causing an I/O bottleneck. How does FlashAttention solve the problem: FlashAttention can be viewed as a fused, I/O-aware attention algorithm. Splits large Q, K, and V tensors into blocks so that blocks of Q_i, K_j, and V_j fit into fast on-chip SRAM. Computes each block’s attention contribution locally without writing the intermediate Q_i K_j^T or attention-probability matrix back to HBM. Uses online softmax to maintain running normalization statistics and correctly combine contributions from different K/V blocks. Therefore, it computes the exact attention result while significantly reducing HBM I/O. Why can it compute the same result without storing the full L × L attention matrix? Because the intermediate QK^T and softmax attention matrix do not need to be explicitly materialized in HBM. FlashAttention processes them block by block and uses online softmax to incrementally update the normalized output. 4. SFT 4.1 PEFT Approaches Overview Why is PEFT needed? Full fine-tuning of LLMs is computationally expensive and requires large amounts of GPU memory and training data. PEFT adapts the model by updating only a small fraction of parameters, making fine-tuning practical on limited hardware (often a single GPU). Compared with full fine-tuning, PEFT generally reduces the risk of catastrophic forgetting. Major PEFT approaches? Selective: Fine-tune only selected existing layers; the chosen parameters are updated directly. Reparameterization: Fine-tune selected layers through a low-dimensional parameterization (e.g., low-rank updates), greatly reducing the number of trainable parameters. Additive: Keep the original model frozen and add trainable modules or virtual parameters. Example techniques for different PEFT approaches? Selective fine-tuning: Shallow-layer fine-tuning for data-distribution adaptation Deeper-layer fine-tuning for task adaptation Reparameterization: LoRA Additive methods: Adapters: Add small trainable layers after the attention or FFN sublayers. Prompt tuning / soft prompting: Prepend trainable virtual tokens to the input embeddings. What are the application scenarios for different PEFT approaches? Additive vs. Selective / Reparameterization Additive methods: Better preserve the pretrained model because the original parameters remain frozen. Suitable when one shared base model needs to support multiple customized versions, such as an enterprise chatbot personalized for different customers. Selective / Reparameterization methods: Better when stronger adaptation to a specialized task or domain is required, potentially at the cost of some generalization ability. Selective vs. Reparameterization Selective fine-tuning: Suitable when only one specialized model is needed, such as on-device deployment or infrequently updated models. However, each task may require storing and transferring a larger modified checkpoint. Reparameterization (e.g., LoRA): Suitable when one server needs to support multiple tasks or domains. Multiple lightweight adapters can share the same frozen base model and be loaded or switched efficiently. 4.2 LoRA Explain the LoRA method. Freeze the original weight matrices. For each selected weight matrix, add a trainable low-rank update. Instead of learning a full weight matrix, the update is parameterized as the product of two low-rank matrices: ΔW = αAB The effective weight becomes: W = W_frozen + αAB How does LoRA reduce the number of trainable parameters? This is easier to illustrate with a demo. Suppose the original weight matrix has shape: 100 × 100 Full fine-tuning: 100 × 100 = 10,000 trainable parameters. LoRA with rank = 10: A: 100 × 10 B: 10 × 100 Total trainable parameters = 2,000 80% fewer trainable parameters. What are the trade-offs of LoRA compared with full fine-tuning? Pro: fewer trainable parameters, therefore: requires less training data, lower risk of overfitting, lower GPU memory consumption, faster training and less catastrophic forgetting. Con: lower performance ceiling Variants of LoRA? QLoRA QLoRA = LoRA + quantization of the frozen base model. The frozen model is stored in low precision (typically 4-bit) to reduce GPU memory. Computation and LoRA adapters are still performed in higher precision (e.g., BF16). Additional dequantization is performed during computation. LoRA+ LoRA+ = different learning rates for A and B. Typically trains faster and achieves better convergence. AdaLoRA AdaLoRA = LoRA + adaptive rank allocation. The total rank budget remains approximately constant. During training, ranks are dynamically reallocated across layers according to their estimated importance. More important layers receive higher ranks, while less important layers receive lower ranks. 5. RL 5.1. RL Overview Choosing between RL and SFT? SFT: Better for learning fixed output patterns with relatively rigid inter-token dependencies, such as QA and tool-use tasks. RL: Better for optimizing a high-level objective that can be achieved through multiple valid routes, where the optimal path to the best answer is ambiguous. Common sources of RL reward signals? RLHF (Human Feedback): The source of the supervision signal is human annotators. RLAIF (AI Feedback): The source of the supervision signal is an AI judge, typically another LLM. RLVR (Verifiable Rewards): The source of the supervision signal is an automated evaluation system, such as unit tests, code execution, mathematical answer checking, or rule-based verifiers. 5.2. Reward Design Possible cause of RL training divergence or failure to learn? Sparse reward problem: Reward is received only at the end (or very infrequently), making it difficult for the agent to identify which actions contributed to success. Solutions to the sparse reward problem? Level reward: Assign reward scores to intermediate states or milestones. (E.g., fully achieving the goal -> 1.0, while partially achieving it -> 0.3.) Process Reward Model (PRM): Train a model to evaluate whether each intermediate action contributes to a successful final outcome. (E.g. after a planning step, if the agent generates a correct, incorrect, or no tool-call query, the planning step may receive rewards of +0.5, -0.1, or 0, respectively. ) What is reward hacking? The agent learns to exploit flaws in the reward function, achieving a high reward through undesired behaviors instead of accomplishing the intended objective. Causes and solutions for reward hacking? Cause 1: Reward-objective misalignment — The reward does not fully or accurately represent the true objective. E.g. A recommendation model maximizes clicks by generating clickbait. Multi-objective reward design: 1. Add guardrail and safety-related reward terms. 2. Add explicit penalties for undesired behaviors. Cause 2: Reward-model overfitting — The reward model overfits its training data, and the policy exploits erroneous regions of the reward landscape. Reduce reward-model overfitting: Use reward-model ensembles and add more diverse or adversarial training data. Regularize the policy during training: Mix in real / supervised data or add a KL penalty to constrain deviation from the reference policy. Control actions during inference: Restrict the allowed action space through explicit instructions or infrastructure-level tool-use constraints. 5.3. RL Algorithms Describe the process of RLHF with the PPO algorithm. Generate multiple responses (completions) for the same query (prompt) and obtain feedback from human annotators. Prepare the samples in a pairwise format and train the reward model, typically using a pairwise ranking loss. During policy training, a prompt is passed to the LLM to generate a completion. The full (prompt, completion) pair is then sent to the reward model to obtain a reward score. The reward score is consumed by the PPO algorithm to update the LLM. What are the limitations of PPO? Relies on a reward model: Training is more complex and may introduce reward hacking. Relies on a value function/model: Training is more computationally expensive and may amplify reward hacking, since the value function estimates the discounted cumulative future reward. How do alternative algorithms address these limitations? Reward model is expensive to train → DPO Removes the reward model. Reformulates the two-stage reward modeling + RL pipeline into a one-stage preference classification / optimization problem. Value function is expensive to train → GRPO Removes the value function. Instead of estimating the advantage using a learned value function, GRPO computes the advantage from the relative rewards within a group of sampled responses. When should we use PPO, DPO, or GRPO? Algorithm choice largely depends on the reward-signal format and available computational resources. PPO Suitable when a large volume of preference data is available, so the reward model can be trained accurately. Also suitable when there are only one or a few simple reward functions, resulting in a relatively simple reward landscape and making the value function easier to train for accurate advantage estimation. Requires abundant computational resources for training both the reward model and the value function. DPO Suitable when only a relatively small amount of preference data is available and training a reliable reward model for PPO is difficult. Requires fewer computational resources because it avoids separate reward-model training and directly optimizes preference pairs. GRPO Suitable when there are multiple reward functions, making the value function difficult to train reliably with PPO. Avoids training a value function, making it easier to train and more computationally efficient than PPO. 5.4. RL Metrics Model Training Monitoring Actor loss: Measures how well the policy is being optimized. It is used to monitor whether the policy is learning and whether policy optimization is stable. Critic loss: Measures the prediction error of the value function (critic), i.e., how accurately it estimates the expected cumulative reward. A lower critic loss generally indicates more accurate advantage estimation. Policy Model Comparison Cumulative Reward: The total reward obtained by the agent when completing a task or episode. Variants include average episodic reward, average reward per step, and discounted cumulative reward. Success Rate: The percentage of tasks or episodes completed successfully. Stability: Measures whether performance is consistent across different tasks or episodes, often represented by the variance of cumulative rewards. RL Algorithm Comparison Sample Efficiency: The number of samples or environment interactions required to reach a given performance level. Convergence Rate: How quickly the algorithm approaches a stable and effective policy during training. 6. Evaluation 6.1. Evaluation Methods Human Evaluation Forms: Binary pairwise comparison (preference) and absolute scoring. Pros: Accurate and flexible for evaluating open-ended tasks. Cons: Expensive, slow, difficult to scale, and may suffer from annotation inconsistency. LLM-as-a-Judge: Use a powerful LLM to evaluate another model. Metrics: Binary pairwise comparison and absolute scoring (typically based on explicit evaluation criteria). Pros: Inexpensive, scalable, and flexible for open-ended tasks. Cons: Less reliable than human evaluation or reference-based evaluation. Public Benchmarks / Datasets Metrics: Reference-based or reference-free metrics, depending on the benchmark. Pros: Accurate, inexpensive, and standardized. Cons: Static, limited in scope, and may not fully align with the target application. Use-case capability: Whether the model supports the target task, such as chat, summarization, reasoning, or tool use. Cost: Inference cost, model size, and available compression options such as quantization. Latency: Whether the model satisfies the application’s response-time requirements. Benchmark performance: Performance on relevant public benchmarks and, more importantly, task-specific evaluations. How can we handle context-window limits for long-text generation? Positional-encoding scaling: Rule-based relative PE (e.g., ALiBi): More naturally scalable to longer contexts. RoPE scaling: Extends the context window by modifying or rescaling RoPE frequencies. Absolute PE: Uses positional interpolation or extrapolation. Context-window operations: Truncation: Remove part of the text so the input fits within the context window. Sliding window: Process overlapping text segments while moving the window by a fixed stride. Prompt engineering / summarization: Contextual summarization: Sequentially summarize earlier chunks and combine the running summary with the next chunk. Hierarchical summarization: Summarize chunks in parallel, then recursively combine and summarize adjacent summaries. Prompt chaining: Break a large prompt into smaller sub-prompts and process them sequentially. RAG: Chunk and index the document, then retrieve only the relevant chunks when needed. How can we evaluate LLM applications with insufficient human annotations? Data augmentation for human annotations: Generate additional evaluation samples using generative AI or rule-based transformations, then use the augmented samples as evaluation data. LLM-as-a-judge calibrated with human annotations: Run the LLM judge on human-annotated samples and use human–LLM disagreement signals to calibrate the evaluation. One example is Prediction-Powered Inference (PPI). Proxy labels: Use user behavior or product signals as indirect labels, such as clicks, retention, task completion, or user corrections. Label-free evaluation: Evaluate properties that do not require ground-truth labels. For example, measure whether the model’s decisions are self-consistent or stable across repeated runs or prompt variations. This article may not be reproduced, distributed, republished, or adapted, in whole or in part, without the author’s prior written permission, except as permitted by applicable copyright law. Sharing the original Medium link is welcome.