# Same effective batch, different LoRA training time: a small TRL diagnostic

> Source: <https://discuss.huggingface.co/t/same-effective-batch-different-lora-training-time-a-small-trl-diagnostic/178803#post_1>
> Published: 2026-08-18 12:02:02+00:00

```
I wanted to check something I usually take for granted: if effective batch is same, does the per-device batch still make much difference to training time?

I ran Qwen3-1.7B with TRL and LoRA on one T4. I kept the model, data, sequence length, FP16, seed and 100 optimizer steps fixed.

| Per-device batch | Accumulation | Effective batch | Runtime | Step time | Peak reserved |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 4 | 287.6s | 2871ms | 5.45 GiB |
| 2 | 2 | 4 | 258.8s | 2585ms | 5.58 GiB |
| 4 | 1 | 4 | 238.2s | 2379ms | 5.81 GiB |

The 4 × 1 run finished about 17% faster than 1 × 4, while using around 0.36 GiB more reserved memory. Most of the measured difference was in forward and backward.

This is not a new rule. The [Hugging Face docs](https://huggingface.co/docs/transformers/grad_accumulation) already recommend using gradient accumulation when a larger physical batch does not fit. But I found the size of difference on this T4 useful to see.

I maintain TraceML and used it here to see where the step time was going. I also put the experiment in a Colab, no Hugging Face login needed:

- Colab: https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb
- Notebook source: https://github.com/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb

If anyone tries it on L4, A10 or A100, I would be interested to compare the numbers.
```