The setup #
- Base model: Mistral-7B-v0.1 (4-bit quantized via bitsandbytes)
- Dataset: 12k Alpaca-format examples (~400MB tokenized)
- Method: LoRA rank 16, alpha 32, targeting q_proj/v_proj
- Colab: Free tier, T4 GPU (16GB VRAM), 12GB RAM, 2-hour disconnect
from transformers import TrainingArguments
training_args = TrainingArguments(
output_dir="./mistral-7b-lora",
per_device_train_batch_size=1,
gradient_accumulation_steps=16,
num_train_epochs=3,
learning_rate=2e-4,
fp16=True,
optim="paged_adamw_8bit",
logging_steps=10,
save_steps=500,
max_steps=1500, # ~2 hours at this rate
)
What broke #
First run: OOM at step 47. The gradient accumulation buffer + optimizer states + model weights pushed past 16GB. T4 has 16GB but Colab's overhead eats ~2GB before you start.
Second run: Switched to gradient_checkpointing=True
, dropped batch to 1, accumulation to 32. Made it to step 200 before the runtime disconnected. Colab free kills at exactly 2 hours — no warning, no checkpoint recovery.
Third run: Added save_steps=100
and save_total_limit=3
. Got to step 800. Then the 12GB system RAM filled up (dataset caching + tokenizer + intermediate tensors) and the kernel died with MemoryError: Unable to allocate 1.2 GiB
.
The actual limits I hit #
| Resource | Free tier | What I needed |
|----------|-----------|---------------|
| VRAM | 16GB (T4) | ~14GB usable for 7B 4-bit + LoRA |
| System RAM | 12GB | 16GB+ for dataset + overhead |
| Max session | 2 hours | 4-6 hours for 3 epochs |
| Disk | ~70GB | Fine, not the bottleneck |
What does work on free Colab #
1.5B-3B models full fine-tune (Phi-2, TinyLlama, Gemma-2B)7B LoRA with aggressive quantization (4-bit +max_memory={0: "13GiB"}
) + dataset streamingInference only— 7B-13B 4-bit runs fine for generation
from datasets import load_dataset
dataset = load_dataset("json", data_files="train.jsonl", split="train", streaming=True)
dataset = dataset.shuffle(buffer_size=1000, seed=42)
My workaround (not pretty) #
Ended up renting a RunPod A100 40GB for $1.10/hr. Same code, finished 3 epochs in 47 minutes. Cost: ~$0.85 total.
Colab Pro ($10/mo) gets you T4/V100 priority and 24hr sessions — might work for 7B LoRA if you're patient. But free tier? Save yourself the grief.
TL;DR: Free Colab cannot reliably fine-tune 7B models. The 2-hour hard limit + 12GB RAM ceiling + 16GB VRAM (shared) is a triple constraint. Use it for inference, small models, or prototyping — not production fine-tunes.
Next Debugging the timeout cascade that killed our UPI integration →