I Have 10 Minutes to Train an AI Model. Here’s Exactly What Happened.

A developer trained a DistilBERT model on the AG News dataset in 10 minutes using a rented GPU and the Crunr CLI, demonstrating a cold-start fine-tuning pipeline. The experiment highlights the importance of pinning dependencies to the GPU architecture to avoid CUDA mismatches, and includes a baseline evaluation to measure training improvement.

A few days ago I sat down, opened a terminal, and gave myself a stupid little constraint: train a real model, on a real dataset, with real eval numbers, in the time it takes to make and drink a cup of chai. No pre-warmed environment. No “oh I already have the CUDA drivers set up.” Cold start, on a rented GPU I’d never touched before, using Crunr https://crunr.com/ the CLI I’ve been building to get from pip install to a finished, evaluated model. I recorded the whole thing. If you want the unscripted version, it’s here: 10 Minutes to Train an AI Model https://youtu.be/xdqmuLJzZJk?si=sbcrXj0-sZwk8XaK . This post is the write-up: the code, the numbers, and the two or three things that actually matter if you want to repeat this yourself. Nothing exotic. I fine-tuned distilbert-base-uncased on AG News, a 4-class topic classification dataset World, Sports, Business, Sci/Tech . It's a well-worn benchmark on purpose. When you're testing infrastructure, the last thing you want is a dataset that's also fighting you. The script does five things: That baseline step matters more than people give it credit for. It’s easy to fine-tune a model, see 94% accuracy, and feel great about it. It’s a different thing to see the untrained model sitting at 23.5% first and watch the actual delta your training run bought you. torch =2.0.0transformers =4.38.0datasets =2.18.0scikit-learn =1.3.0numpy =1.24.0tqdm =4.66.0accelerate =0.27.0 Looks harmless. It isn’t. This is the part I want to actually call out, because it’s the difference between a 10-minute job and a 45-minute job spent staring at a CUDA error. Pin your dependencies to your GPU, not just your framework. torch =2.0.0 will happily resolve to a wheel that either doesn't have a CUDA build matching your driver, or pulls a CUDA version your GPU architecture doesn't support well. An RTX Pro 6000 Blackwell-class wants a recent CUDA build behind PyTorch, something that actually ships kernels for its compute capability. Throw that same loose torch =2.0.0 at an older L40S or a V100 and you can end up with a mismatched build, silent CPU fallback, or a job that OOMs in a way that has nothing to do with your batch size. So before you fire off a job: This is exactly the kind of thing that eats people’s time when they’re moving fast, and it’s the reason I built the container/environment layer in Crunr the way I did: controlled, versioned images instead of “pip install and pray” on a fresh box every time. Here’s the full script, train.py. Nothing fancy: standard HuggingFace training loop, tokenization, a base-model baseline pass, per-epoch evaluation, best-checkpoint saving, and a formatted comparison report at the end. """Fine-tune DistilBERT on AG News dataset for topic classification. php AG News label mapping: 0 - World 1 - Sports 2 - Business 3 - Sci/Tech Usage: python train.py full fine-tune + eval python train.py --eval-only only run evaluation needs saved model python train.py --epochs 5 custom epochs""" python import osimport timeimport argparseimport warningswarnings.filterwarnings "ignore" python import numpy as npimport torchfrom torch.utils.data import DataLoaderfrom torch.optim import AdamW python from transformers import DistilBertTokenizerFast, DistilBertForSequenceClassification, get linear schedule with warmup, from datasets import load datasetfrom sklearn.metrics import accuracy score, f1 score, precision score, recall score, classification report, from tqdm import tqdm ConfigBASE MODEL = "distilbert-base-uncased"OUTPUT DIR = "./outputs/finetuned model"RESULTS FILE = "./outputs/evaluation results.txt"MAX LEN = 128BATCH SIZE = 32EPOCHS = 3LR = 2e-5WARMUP RATIO = 0.1SEED = 42NUM LABELS = 4 LABEL NAMES = "World", "Sports", "Business", "Sci/Tech" python def set seed seed: int : torch.manual seed seed np.random.seed seed if torch.cuda.is available : torch.cuda.manual seed all seed python def load ag news tokenizer, max len: int, batch size: int : """Download AG News and return train loader, test loader .""" print "\n 1/5 Loading AG News dataset ..." dataset = load dataset "fancyzhx/ag news" python def tokenize batch : return tokenizer batch "text" , padding="max length", truncation=True, max length=max len, tokenized = dataset.map tokenize, batched=True, batch size=1000 tokenized.set format type="torch", columns= "input ids", "attention mask", "label" , train loader = DataLoader tokenized "train" , batch size=batch size, shuffle=True, num workers=2, pin memory=True test loader = DataLoader tokenized "test" , batch size=batch size, shuffle=False, num workers=2, pin memory=True print f" Train samples : {len dataset 'train' :,}" print f" Test samples : {len dataset 'test' :,}" return train loader, test loader php def evaluate model, loader, device - dict: """Run inference and return metric dict.""" model.eval all preds, all labels = , with torch.no grad : for batch in tqdm loader, desc=" Evaluating", leave=False : input ids = batch "input ids" .to device attention mask = batch "attention mask" .to device labels = batch "label" .to device outputs = model input ids=input ids, attention mask=attention mask preds = torch.argmax outputs.logits, dim=-1 all preds.extend preds.cpu .numpy all labels.extend labels.cpu .numpy metrics = { "accuracy" : accuracy score all labels, all preds , "f1 macro" : f1 score all labels, all preds, average="macro" , "f1 weighted": f1 score all labels, all preds, average="weighted" , "precision macro": precision score all labels, all preds, average="macro" , "recall macro" : recall score all labels, all preds, average="macro" , "report" : classification report all labels, all preds, target names=LABEL NAMES, digits=4 , } return metrics php def evaluate base model tokenizer, test loader, device - dict: """ Evaluate the raw pretrained DistilBERT randomly initialized classification head to establish a baseline. """ print "\n 2/5 Evaluating BASE model no fine-tuning ..." base model = DistilBertForSequenceClassification.from pretrained BASE MODEL, num labels=NUM LABELS .to device metrics = evaluate base model, test loader, device del base model if torch.cuda.is available : torch.cuda.empty cache return metrics php def train model, train loader, test loader, device, epochs: int - dict: """Fine-tune and return best test metrics.""" print f"\n 3/5 Fine-tuning for {epochs} epoch s ..." total steps = len train loader epochs warmup steps = int total steps WARMUP RATIO optimizer = AdamW model.parameters , lr=LR, weight decay=0.01 scheduler = get linear schedule with warmup optimizer, num warmup steps=warmup steps, num training steps=total steps, best f1 = 0.0 best metrics = {} train start = time.time for epoch in range 1, epochs + 1 : model.train epoch loss = 0.0 epoch start = time.time pbar = tqdm train loader, desc=f" Epoch {epoch}/{epochs}", leave=True for step, batch in enumerate pbar, 1 : input ids = batch "input ids" .to device attention mask = batch "attention mask" .to device labels = batch "label" .to device optimizer.zero grad outputs = model input ids=input ids, attention mask=attention mask, labels=labels, loss = outputs.loss loss.backward torch.nn.utils.clip grad norm model.parameters , 1.0 optimizer.step scheduler.step epoch loss += loss.item pbar.set postfix loss=f"{epoch loss / step:.4f}" epoch time = time.time - epoch start avg loss = epoch loss / len train loader print f" Epoch {epoch}, avg loss: {avg loss:.4f} {epoch time:.1f}s " per-epoch evaluation metrics = evaluate model, test loader, device print f" Val acc={metrics 'accuracy' :.4f} " f"f1 macro={metrics 'f1 macro' :.4f} " f"precision={metrics 'precision macro' :.4f} " f"recall={metrics 'recall macro' :.4f}" if metrics "f1 macro" best f1: best f1 = metrics "f1 macro" best metrics = metrics model.save pretrained OUTPUT DIR print f" New best model saved f1={best f1:.4f} " total time = time.time - train start print f"\n Training complete in {total time / 60:.1f} min" return best metrics python def write results base metrics: dict, ft metrics: dict, path: str : """Write side-by-side comparison to a .txt file.""" sep = "=" 70 sep2 = "-" 70 python def delta ft val, base val : d = ft val - base val sign = "+" if d = 0 else "" return f"{sign}{d:.4f}" lines = sep, " DistilBERT - AG News Topic Classification - Evaluation Report", sep, "", f" Base model : {BASE MODEL}", f" Fine-tuned epochs: {EPOCHS}", f" Max sequence len : {MAX LEN}", f" Batch size : {BATCH SIZE}", f" Learning rate : {LR}", "", sep2, " SUMMARY - Test-set metrics", sep2, f" {'Metric':<25} {'Base': 10} {'Fine-tuned': 12} {'Delta': 10}", sep2, f" {'Accuracy':<25} {base metrics 'accuracy' : 10.4f} {ft metrics 'accuracy' : 12.4f} {delta ft metrics 'accuracy' , base metrics 'accuracy' : 10}", f" {'F1 macro ':<25} {base metrics 'f1 macro' : 10.4f} {ft metrics 'f1 macro' : 12.4f} {delta ft metrics 'f1 macro' , base metrics 'f1 macro' : 10}", f" {'F1 weighted ':<25} {base metrics 'f1 weighted' : 10.4f} {ft metrics 'f1 weighted' : 12.4f} {delta ft metrics 'f1 weighted' , base metrics 'f1 weighted' : 10}", f" {'Precision macro ':<25} {base metrics 'precision macro' : 10.4f} {ft metrics 'precision macro' : 12.4f} {delta ft metrics 'precision macro' , base metrics 'precision macro' : 10}", f" {'Recall macro ':<25} {base metrics 'recall macro' : 10.4f} {ft metrics 'recall macro' : 12.4f} {delta ft metrics 'recall macro' , base metrics 'recall macro' : 10}", sep2, "", sep2, " BASE MODEL - Per-class Classification Report", sep2, base metrics "report" , "", sep2, " FINE-TUNED MODEL - Per-class Classification Report", sep2, ft metrics "report" , "", sep, " End of report", sep, os.makedirs os.path.dirname os.path.abspath path or ".", exist ok=True with open path, "w" as f: f.write "\n".join lines php print f"\n 5/5 Results saved - {path}" python def parse args : p = argparse.ArgumentParser description="Fine-tune DistilBERT on AG News" p.add argument "--epochs", type=int, default=EPOCHS, help="Training epochs" p.add argument "--batch-size",type=int, default=BATCH SIZE, help="Batch size" p.add argument "--lr", type=float, default=LR, help="Learning rate" p.add argument "--max-len", type=int, default=MAX LEN, help="Max token length" p.add argument "--output-dir",type=str, default=OUTPUT DIR, help="Where to save model" p.add argument "--results", type=str, default=RESULTS FILE,help="Results .txt path" p.add argument "--eval-only", action="store true", help="Skip training; load saved model and re-evaluate" return p.parse args python def main : args = parse args global EPOCHS, BATCH SIZE, LR, MAX LEN, OUTPUT DIR, RESULTS FILE EPOCHS = args.epochs BATCH SIZE = args.batch size LR = args.lr MAX LEN = args.max len OUTPUT DIR = args.output dir RESULTS FILE= args.results set seed SEED device = torch.device "cuda" if torch.cuda.is available else "cpu" print f"\n Device : {device}" if torch.cuda.is available : print f" GPU : {torch.cuda.get device name 0 }" print f"\n 0/5 Loading tokenizer {BASE MODEL} ..." tokenizer = DistilBertTokenizerFast.from pretrained BASE MODEL train loader, test loader = load ag news tokenizer, MAX LEN, BATCH SIZE base metrics = evaluate base model tokenizer, test loader, device print f" Base acc={base metrics 'accuracy' :.4f} " f"f1 macro={base metrics 'f1 macro' :.4f} " f"precision={base metrics 'precision macro' :.4f} " f"recall={base metrics 'recall macro' :.4f}" if args.eval only: print f"\n 3/5 --eval-only: loading fine-tuned model from {OUTPUT DIR} ..." if not os.path.isdir OUTPUT DIR : raise FileNotFoundError f"No saved model found at '{OUTPUT DIR}'. Run without --eval-only first." ft model = DistilBertForSequenceClassification.from pretrained OUTPUT DIR .to device print "\n 4/5 Evaluating fine-tuned model ..." ft metrics = evaluate ft model, test loader, device else: ft model = DistilBertForSequenceClassification.from pretrained BASE MODEL, num labels=NUM LABELS .to device ft metrics = train ft model, train loader, test loader, device, EPOCHS print "\n 4/5 Final evaluation with best checkpoint ..." ft model = DistilBertForSequenceClassification.from pretrained OUTPUT DIR .to device ft metrics = evaluate ft model, test loader, device print f" Fine-tuned acc={ft metrics 'accuracy' :.4f} " f"f1 macro={ft metrics 'f1 macro' :.4f} " f"precision={ft metrics 'precision macro' :.4f} " f"recall={ft metrics 'recall macro' :.4f}" write results base metrics, ft metrics, RESULTS FILE print "\n" + "=" 70 print " FINAL COMPARISON" print "=" 70 header = f" {'Metric':<25} {'Base': 10} {'Fine-tuned': 12} {'Delta': 10}" print header print "-" 70 for name, key in "Accuracy", "accuracy" , "F1 macro ", "f1 macro" , "F1 weighted ", "f1 weighted" , "Precision macro ", "precision macro" , "Recall macro ", "recall macro" , : b = base metrics key f = ft metrics key d = f - b sign = "+" if d = 0 else "" print f" {name:<25} {b: 10.4f} {f: 12.4f} {sign}{d: 9.4f}" print "=" 70 if name == " main ": main Once the environment’s sorted, this is the entire “infra” step: pip install crunr Grab an API key from cloud.crunr.com https://cloud.crunr.com/ , then: crunr run train.py --instance rtxpro6000 That’s it. One line. Crunr provisions the instance, ships the code, runs it, streams the logs back, and tears the instance down when it’s done, so you’re not paying for idle GPU time while you’re reading the eval report. The RTX Pro 6000 is genuinely overkill for a DistilBERT job this size, you could run this on something far cheaper, but the point of the exercise was showing the ceiling, not the floor. Pick your instance size to match your model and dataset, not your ego. Base model no fine-tuning, just DistilBERT with a randomly initialized classification head sitting on top : Metric Base Fine-tuned Delta Accuracy 0.2350 0.9455 +0.7105 F1 macro 0.1303 0.9456 +0.8153 F1 weighted 0.1303 0.9456 +0.8153 Precision macro 0.3266 0.9456 +0.6190 Recall macro 0.2350 0.9455 +0.7105 The base model’s behavior is a good little lesson on its own. It wasn’t randomly guessing across classes, it was collapsing almost everything into “Sci/Tech” 85.8% recall on that class alone, near-zero on the others . That’s what an untrained head on top of a good encoder looks like: the representations are fine, the classifier on top is noise, and it finds whatever local minimum requires the least commitment. After 3 epochs: Class Precision Recall F1 World 0.9602 0.9532 0.9567 Sports 0.9873 0.9853 0.9863 Business 0.9176 0.9142 0.9159 Sci/Tech 0.9174 0.9295 0.9234 Sports is the easy class here, it has the most distinctive vocabulary, and Business/Sci-Tech are the ones that bleed into each other a little, which tracks with how those two categories actually overlap in real news writing a chip earnings report is both . Not to flex a 94.5% F1 on a benchmark that’s been solved a hundred times over. The point was the loop: cold GPU, environment that just works, training, honest evaluation, report, all in single-digit minutes, without babysitting a provisioning dashboard or debugging a CUDA mismatch at minute 25 of your 10. That loop is the actual product. The model is just proof it ran. If you want to poke at the script yourself, or try it against a different GPU tier, the whole thing’s in the video, and Crunr’s free to try at cloud.crunr.com https://cloud.crunr.com/ . I’d genuinely like to hear where it breaks for you, that’s usually more useful to me than someone telling me it worked. I Have 10 Minutes to Train an AI Model. Here’s Exactly What Happened. https://blog.stackademic.com/i-have-10-minutes-to-train-an-ai-model-heres-exactly-what-happened-9c1fa24ac78b was originally published in Stackademic https://blog.stackademic.com on Medium, where people are continuing the conversation by highlighting and responding to this story.