cd /news/machine-learning/why-your-fine-tuning-job-takes-longe… · home › topics › machine-learning › article
[ARTICLE · art-172042] src=huggingface.co ↗ pub= topic=machine-learning verified=true sentiment=↑ positive

Why your fine-tuning job takes longer than the FLOPs say

A CatBoost regressor trained on 272 production supervised fine-tuning jobs on NVIDIA H200 GPUs predicted end-to-end job duration with R² = 0.98 and a mean absolute error of 1.71 hours (median 0.51 hours) in 5-fold cross-validation, the team behind the upcoming arXiv paper "Predicting End-to-End Fine-Tuning Time for Generative Language Models" reports. The standard FLOP-based estimate, T_theoretical = (6·P·D)/(N_GPU·FLOPS_peak·MFU), explained little of the observed runtime across the roughly 7,382 GPU-hours studied, reaching only R² = 0.31 with a 12.41-hour MAE. The gap comes from setup, evaluation, checkpointing and I/O overheads that the compute-only formula omits, and the learned predictor uses only information available before dispatch, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime.

recommendation

Replace FLOP-only duration estimates with a learned regressor over pre-dispatch features like evaluation cadence, checkpoint policy and per-GPU load; it cut MAE from 12.41 to 1.71 hours on 272 H200 jobs.

more in playbook: Inference and self-hosting
by read9 min views1 publishedNov 18, 2026
Why your fine-tuning job takes longer than the FLOPs say
Image: Hugging Face Blog

#

	Why your fine-tuning job takes longer than the FLOPs say

Team Article

Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.

A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.

GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.

Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.

We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched. So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total, and compared analytical estimates against learned predictors.

The short version: the usual FLOP-based estimate explains little of what we observed (R² = 0.31, MAE 12.41 h). A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The gap comes from two things a closed-form estimate cannot do. The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

#

	The estimate everyone uses is compute-only

For a dense Transformer, training costs roughly 6·P·D floating-point operations, where P is the parameter count and D the number of training tokens. Divide that by what your GPUs can deliver and you get a training-loop time:

T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU)

Here N_GPU is the number of GPUs, FLOPS_peak is the peak throughput of one GPU from its spec sheet, and MFU (Model FLOPs Utilization) is the fraction of that peak the training actually achieves. Real jobs never run at peak because of memory traffic, communication and kernel inefficiencies, so MFU typically sits somewhere between 30% and 60%, and the right value for a given job is not known in advance.

This is a good description of the optimization loop. A job is more than its loop, though. Before the first step there is environment setup, model download and , and tokenization. During training there are periodic evaluations and checkpoint uploads. At the end, the final model goes to object storage. On a multi-tenant cluster, all of it competes with other jobs for network and storage I/O.

These overheads depend on configuration, and they do not scale neatly with workload or GPU count. Two jobs with the same nominal compute can finish hours apart.

To keep the comparison fair, the FLOP baseline we test counts the scheduled evaluation tokens as well as the training tokens, and we sweep MFU over a wide range.

#

	What we built

We treat end-to-end duration as a supervised regression problem. Given a feature vector built from the job configuration and model metadata, predict the wall-clock time between the moment the job gets its GPUs and the moment it is done. Queue time is excluded.

Two design choices shaped the work:

  • Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress. That is what makes the prediction usable for scheduling.
  • Features that expose the overheads. On top of the raw configuration we derive quantities such asnum_eval_data (samples processed across all evaluation cycles),num_saves andnum_params_saved (checkpoint I/O), and per-GPU token volumes. The FLOP estimate itself, with MFU fixed at 0.45, goes in as one more feature (baseline_time ), so the model starts from the compute-only number and learns the corrections.

We compared Linear Regression, Ridge, XGBoost and CatBoost. Feature selection combined correlation filtering, mutual information and permutation importance across 18 candidate feature sets, followed by a grid search with MAE as the criterion. The final CatBoost model uses 24 features.

#

	The data
  • 272 SFT runs from the operational logs of a corporate production environment, covering several model families up to 15 billion parameters, both LoRA and full fine-tuning, and a wide range of dataset sizes and hyperparameters.
  • Shared infrastructure. Jobs ran on 1, 2 or 4 GPUs drawn from a pool of 8 to 32 H200s across 1 to 4 nodes, alongside other training and inference workloads. The measured times include real contention.
  • Durations from 1.6 to 172.4 hours , with a mean of 23.43 h and a median of 15.29 h.
  • Stack: Hugging Face Transformers and Unsloth. We model the stack as a categorical feature and make no assumption about a fixed speedup between the two.

Distribution of end-to-end duration across the 272 runs. Mean 23.43 h, median 15.29 h.

#

	Results

All numbers come from stratified 5-fold cross-validation. The first four rows are analytical and involve no fitting.

Model R² RMSE (h) MAE (h)
FLOP baseline (MFU 30%) 0.31 20.45 12.41
FLOP baseline (MFU 45%) 0.18 22.38 13.01
FLOP baseline (MFU 60%) 0.01 24.56 14.62
Calculon −0.51 30.40 20.36
Linear Regression 0.69 ± 0.14 13.52 ± 4.23 7.52 ± 0.49
Ridge 0.73 ± 0.09 12.72 ± 3.70 7.23 ± 0.68
XGBoost 0.97 ± 0.01 4.18 ± 1.17 1.95 ± 0.32
CatBoost 0.98 ± 0.01 3.52 ± 0.78 1.71 ± 0.33

A few things stand out.

No single MFU value fixes the FLOP baseline. We swept MFU from 0.25 to 0.65. The best setting, 30%, still leaves a mean error above 12 hours on jobs that last 23 hours on average.

A more detailed analytical model does not close the gap. As a second baseline we took Calculon (Isaev et al., SC '23), an analytical performance model for LLM training from prior work. It builds an explicit layer graph and accounts for compute, memory traffic and inter-device communication. We extended the public implementation to cover LoRA, grouped-query attention, fused attention, bfloat16 and SwiGLU. Its estimates still correspond to a median of only 11% of the observed wall-clock time. That says nothing against Calculon as a step-level tool, which is what it was designed for. It shows how much of a production job happens outside the optimization step.

Linear models help, and then stall. Ridge brings MAE down to 7.23 h. The relationship between configuration and duration is not well captured by a purely linear model.

Gradient boosting does the heavy lifting. CatBoost and XGBoost land within one standard deviation of each other on every metric. We carry CatBoost forward because it has the lowest mean error across the board.

#

	What the predictor relies on

CatBoost's feature importance concentrates in a handful of features. Model and workload variables are there, as expected, but so are evaluation and checkpoint variables. num_eval_data, the total number of samples processed in evaluation over the life of the job, ranks first, followed by num_trainable_params, batch_eval_size_percent, num_all_params and num_gpus*batch_train_percent, with num_params_saved close behind. SHAP values tell the same story: high evaluation volume consistently pushes the predicted time up, while trainable parameter count has a nonlinear effect that only becomes large for the biggest models.

Mean feature importance of the CatBoost model across folds (PredictionValuesChange).

Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent. How often you evaluate, how large the evaluation set is and how many checkpoints you persist all shape the wall-clock time, alongside model and dataset scale. These are choices people often make by habit when writing a training config.

One caveat on reading the plots: larger GPU allocations show positive contributions to predicted time. In our data, bigger allocations go with bigger jobs, so num_gpus acts partly as a proxy for job scale. It does not mean adding GPUs makes a job slower.

#

	How wrong is it, and where

Looking at all 272 out-of-fold predictions, the median absolute error is 0.51 h. 80% of predictions are within 2.20 h and 90% are within 4.31 h. Broken down by job duration:

Job duration (h) Runs Median abs. error (h) P90 (h)
1.56 to 5 60 0.30 1.55
5 to 10 63 0.18 1.27
10 to 30 69 0.73 4.17
30 to 60 67 1.06 5.50
60 to 175 13 5.19 18.51

Up to 60 hours, the median error stays at about one hour or less. The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.

Out-of-fold absolute error against real duration, colored by duration range. Errors stay low up to 60 h and widen in the sparse tail.

More data should still help. On a fixed 20% holdout, MAE drops from 1.63 h to 1.50 h when going from 85% to 100% of the training partition, with no clear plateau.

#

	What it is good for

In practice, the predictor gives a scheduler a duration it can plan around before a job starts: which jobs to run together, when GPUs will free up, and whether a run will finish in time for its deadline.

#

	Limitations
  • SFT only. We did not include other training and alignment strategies such as RLHF, DPO or RLVR.
  • One GPU type. Every run used H200s. The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining. A predictor for mixed pools would also need hardware descriptors such as compute throughput, memory capacity and bandwidth, and interconnect topology.
  • Few very long runs. Errors widen above 60 hours.

What does transfer is the method: overhead-aware features, multicollinearity-aware selection and the comparison protocol. The full paper will report the feature formulas, the selection procedure, the search spaces and the final hyperparameters in full, so the pipeline can be rebuilt on any execution-log dataset with a similar structure.

Next on our list: hardware-specific descriptors and validation on other GPU types, transfer learning for configurations with few observations, other training objectives, and more data on long runs.

#

	What's next

The full paper is coming to arXiv soon. Follow the Dharma-AI organization on Hugging Face to be notified when it is out, and leave your questions in the comments below. We would like to hear how you estimate job duration on your own clusters.

#

	Further reading

Dharma-AI R&D · We Make AI Your Model

── more in #machine-learning 4 stories · sorted by recency
── more on @catboost 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/why-your-fine-tuning…] indexed:0 read:9min 2026-11-18 · —