Cross-Validation From Scratch and a Surprise at n=100 Ken Koon Wong, in a blog post published August 5, 2026, coded K-Fold cross-validation from scratch in R and found that at n=1000, leave-one-out cross-validation (LOOCV) has lower bias but higher variance than 10-fold and 5-fold, as textbooks claim, but at n=100 this pattern does not hold. The post simulates data with a known data-generating process and compares candidate models using CV RMSE, verifying the best model on a held-out test set. Cross-Validation From Scratch and a Surprise at n=100 By Ken Koon Wong in r https://www.kenkoonwong.com/categories/r R https://www.kenkoonwong.com/categories/r crossvalidation https://www.kenkoonwong.com/categories/crossvalidation resampling https://www.kenkoonwong.com/categories/resampling bias https://www.kenkoonwong.com/categories/bias variance https://www.kenkoonwong.com/categories/variance August 5, 2026 Textbooks say LOOCV has the lowest bias but highest variance compared to 10 and 5-fold. Coded a K-Fold CV from scratch for learning to test that on simulated data 🔍📊 — and at n=1000 it holds up. At n=100? Not so much. 🤔 The above image was generated via chatGPT. Uploaded all the text of this blog post and asked it to generate a cartoon. Very impressive It used to be spelling error and gibberish of text in the past, but now cohesive words on image. Just wow. Motivations Crossvalidation is such a crucial step in Machine Learning and traditional methods that nowadays is incorporated in easy to use sklearn or tidymodels without us needing to build one from scratch. As with my other learning experience, the best way to learn the concept other than learning the concept 🤣 is to code it from the ground up and see how it works In K-Fold CV, the training data is split into K chunks; the model is trained K times, each time holding out a different chunk. Performance is averaged across all K folds, giving a more stable estimate. A special case is Leave-One-Out CV LOOCV , where each individual observation serves as its own validation set. It’s thorough but computationally expensive. I was told that, bias LOOCV < 10-fold < 5-fold; whereas variance LOOCV 10-fold 5-fold. Is that true? Also, what’s with the repeats, does that really reduce variance? Let’s check them out. Objectives: - Simulate data with a known data-generating process simulate - Implement K-Fold cross-validation from scratch kfold - Assessing RMSE rmse - Compare candidate models using CV RMSE compare - Verify the best model on a held-out test set verify - Opportunities For Improvement opportunities - Lessons Learnt lessons Simulate Data library tidyverse set.seed 1 n <- 1000 x <- rnorm n w <- rnorm n y <- 0.5 x^2 + -0.5 w + 0.3 w x + rnorm n df <- tibble x,y,w idx <- sample 1:n, size=0.8 n train <- df idx, test <- df -idx, The above code simulates a dataset with 1000 observations, where the response variable y is generated based on a known data-generating process involving predictors x and w . The dataset is then split into a training set 80% and a test set 20% . Let’s visualize. df | mutate w cut = cut interval w, n=5 | ggplot aes x=x, y=y, color=w cut, group=w cut + geom point alpha=0.5 + theme bw + geom smooth method = "gam", se=F Wow, very interesting visualization where the relationships are definitely not linear here. It’s some form of interaction between x and w . Let’s see if we can recover the underlying data-generating process using K-Fold Cross-Validation. K-Fold Cross-Validation From Scratch folds <- 5 segment portion <- nrow train /folds formula list <- list as.formula "y~x" ,as.formula "y~I x^2 " ,as.formula "y~I x^2 +w+w:x" ,as.formula "y~I x^3 +w+w:x" , as.formula "y~w:x" ,as.formula "y~w" ,as.formula "y~x+w+x:w" ,as.formula "y~I x^2 +w:x" , as.formula "y~I x^2 +w" cv log <- tibble for formula in formula list { print formula predict log <- y log <- vector mode="numeric",length=segment portion folds start <- 1 end <- segment portion for fold in 1:folds { val i <- train start:end, train i <- train -c start:end , model i <- lm formula,train i predict i <- predict model i, val i predict log start:end <- predict i y log start:end <- val i$y start <- end + 1 end <- start + segment portion - 1 } val df <- tibble predict=predict log,y=y log | mutate formula=deparse formula cv log <- cv log | bind rows val df } y ~ x y ~ I x^2 y ~ I x^2 + w + w:x y ~ I x^3 + w + w:x y ~ w:x y ~ w y ~ x + w + x:w y ~ I x^2 + w:x y ~ I x^2 + w Alright, what we’ve done above is a manual implementation of K-Fold Cross-Validation. We loop through each formula in our list, and for each formula, we split the training data into 5 folds. For each fold, we train the model on the other 4 folds and validate it on the current fold. We store the predictions and actual values for later evaluation. We basically want to see which formula has the lowest RMSE across the folds. Let’s calculate that next. From the DGP formula, we know that the best model should be y~I x^2 +w+w:x . Let’s see if we can recover that using K-Fold CV. Assessing RMSE cv log | group by formula | summarize rmse = sqrt mean y-predict ^2 | arrange rmse | mutate rmse = format rmse, digits = 8 A tibble: 9 × 2 formula rmse