I wanted to check something I usually take for granted: if effective batch is same, does the per-device batch still make much difference to training time?
I ran Qwen3-1.7B with TRL and LoRA on one T4. I kept the model, data, sequence length, FP16, seed and 100 optimizer steps fixed.
| Per-device batch | Accumulation | Effective batch | Runtime | Step time | Peak reserved |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 4 | 287.6s | 2871ms | 5.45 GiB |
| 2 | 2 | 4 | 258.8s | 2585ms | 5.58 GiB |
| 4 | 1 | 4 | 238.2s | 2379ms | 5.81 GiB |
The 4 × 1 run finished about 17% faster than 1 × 4, while using around 0.36 GiB more reserved memory. Most of the measured difference was in forward and backward.
This is not a new rule. The [Hugging Face docs](https://huggingface.co/docs/transformers/grad_accumulation) already recommend using gradient accumulation when a larger physical batch does not fit. But I found the size of difference on this T4 useful to see.
I maintain TraceML and used it here to see where the step time was going. I also put the experiment in a Colab, no Hugging Face login needed:
- Colab: https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb
- Notebook source: https://github.com/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb
If anyone tries it on L4, A10 or A100, I would be interested to compare the numbers.
source & further reading
discuss.huggingface.co — original article
Rakarrack-0.6.1 port making progress! ( AI assisted )
Cloud Storage Poll
Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)