# Engineering Journey: Fine-Tuning LLMs from Laptop to Production

> Source: <https://pub.towardsai.net/engineering-journey-fine-tuning-llms-from-laptop-to-production-b1852f8ada39?source=rss----98111c9905da---4>
> Published: 2026-09-08 21:01:01+00:00

The project started simply. I needed a fine-tuned math reasoning model and wanted to prove it was possible.

I reached for **MLX** — Apple’s ML framework for Silicon — because my M4 Mac Mini was the fastest thing I had. mlx_lm Made LoRA fine-tuning straightforward: load the model, freeze everything except the adapter layers, run a training loop. I had a Qwen2.5-1.5B adapter with val loss dropping from 1.8 to 0.9 on MathInstruct.

The next problem was immediate: I ran six experiments and lost track of which adapter came from which hyperparameters. So I added **MLflow** for experiment tracking and **DVC** for data versioning.

DVC turned out to be the most important early decision. The core idea is simple — instead of tracking dataset files in git, DVC stores a content hash in dvc.lock and keeps the actual bytes in a remote cache (S3). That hash, combined with the git commit SHA, becomes a permanent fingerprint for every training run:

```
git_sha + dvc.lock_MD5  →  "e90324a9-dvc3f2a1b7c"
```

This string became the dvc.commit tag on every MLflow run. From it, I can always reconstruct the exact state: git checkout <sha> && dvc checkout. That guarantee — not just "I think I used the same data" but provably the same data — carried forward into every pipeline iteration.

**What worked:** Rapid iteration, zero infrastructure cost, and a lineage model that scaled to the cloud pipelines that followed.

**What broke:** Training on the full MathInstruct dataset (260k examples) took 14 hours on the M2. That was the ceiling. And there was no path to multi-GPU training or team collaboration from a local MLX setup.

**Repo**: [lora-finetune-mlx](https://github.com/ArkaSanka/lora-finetune-mlx)

The move to the cloud was forced by three converging pressures: training time, environment reproducibility, and the need for a proper pipeline instead of a collection of scripts.

I chose **SageMaker Pipelines V2** because it gave me exactly what was missing: a managed DAG with caching, conditional branching, and a visual execution graph. I designed a five-step pipeline:

```
PrepareData → TrainLoRA → Evaluate → ReadEvalReport → CheckTestLoss → RegisterModel
```

The ConditionStep at the end is the key production-grade feature — the model only gets promoted to the MLflow registry if test loss actually improved. No manual approval gate, no accidentally shipping a regression.

Training on ml.g4dn.xlarge spot reduced a 14-hour MLX run to **3.3 hours**, with spot savings of roughly 60% over on-demand. The checkpoint-to-S3 mechanism meant a spot interruption cost five minutes of lost progress, not the entire run.

The bugs I hit were real and expensive to diagnose:

**processing_class vs** **tokenizer in SFTTrainer.** The HuggingFace DLC for PyTorch 2.3 ships with trl < 0.12. My script crashed 30 minutes into the first run at a TypeError I didn't expect. Fix: use tokenizer= for backward compatibility.

**KeyError: 'ModelArtifacts' after Ctrl+C.** When you interrupt a SageMaker job, estimator.model_data raises KeyError because the model hasn't been packaged yet. I wrote an explicit _wait_training() polling loop instead of relying on the SDK.

**PropertyFile only works with** **ProcessingStep.** My evaluation ran as a TrainingStep (to use the GPU), but ConditionStep requires a PropertyFile which only attaches to ProcessingStep outputs. I had to add a tiny relay step (ReadEvalReport) just to bridge the metric. The SageMaker docs don't cover this combination clearly.

Each of these bugs cost 30–60 minutes of compute time to reproduce and diagnose. I documented every fix so the next engineer doesn’t pay the same tax.

The DVC lineage pattern from Chapter 1 carried forward unchanged. The git+DVC hash became a DVCCommitHash SageMaker Pipeline parameter — indexed in the SageMaker Lineage Graph — and logged as a dvc.commit MLflow tag inside the container. Two sources of truth for the same fact.

**What it bought:** Reproducible, auditable, cost-optimized single-node training with conditional model promotion. The right pipeline for production when one GPU is enough.

**What it couldn’t do:** Scale to multiple instances. SFTTrainer is a single-process trainer. Getting to multi-GPU required a different training stack.

**Repo**: [lora-finetune-sagemaker](https://github.com/ArkaSanka/lora-finetune-sagemaker)

Two things pushed me to build the Ray pipeline as a parallel track:

First, we started exploring 7B models. A single ml.g4dn.xlarge (16GB VRAM) isn't enough for a 7B model at full precision. I needed data parallelism across instances without rewriting the entire training loop.

Second, I wanted better observability during training. The V2 pipeline gives you CloudWatch system metrics after the fact. Ray adds a live Dashboard showing per-worker task graphs, OOM events, and gradient sync timing — and TensorBoard events stream to S3 during training via TensorBoardOutputConfig. Watching loss curves update in real-time during a 3-hour job changes how you debug.

The architectural difference from the V2 pipeline is intentional:

```
SM V2 Pipeline:  SageMaker orchestrates the DAG → SFTTrainer runs (single process)Ray pipeline:    Python chains the jobs → SageMaker provisions → Ray coordinates workers
```

SageMaker handles instance provisioning and spot recovery. Ray handles worker coordination inside those instances. The training scripts have zero SageMaker SDK imports — pure Ray + HuggingFace — which makes them testable locally and portable to other compute backends.

The bug that cost the most time: **Ray Train worker** **cwd is not** **/opt/ml/code/**. Ray spawns workers in subprocesses with their own working directory. A relative params.yaml path worked on the head node and failed silently on every worker. I traced it through CloudWatch logs to a FileNotFoundError on worker 1, 45 minutes into the first run. Fix: Path(__file__).parent / args.config Resolved to absolute before passing to train_loop_config.

Scaling data parallelism is a single flag:

```
python -m ray_pipeline.launch --num-instances 2   # 2 workers, LR auto-scaled 2×python -m ray_pipeline.launch --num-instances 4   # 4 workers, LR auto-scaled 4×
```

The learning rate linear scaling rule (Goyal et al. 2017) is wired into the training script — doubling workers doubles the effective batch size, so LR doubles to maintain training dynamics.

The DVC lineage pattern is identical to the V2 pipeline. _dvc_commit_hash() in launch.py computes the same git+DVC fingerprint, passes it as a SageMaker hyperparameter, and the container logs it as a dvc.commit MLflow tag. Any model from any of the three pipelines can be traced back to its exact code and data version the same way.

**What it bought:** True multi-node data parallelism, live TensorBoard monitoring, and a simpler orchestration model that’s easier to extend.

**What it trades off:** No conditional DAG, no built-in model registry gating, no visual pipeline graph in SageMaker Studio.

**Repo**: [lora-finetune-ray](https://github.com/ArkaSanka/lora-finetune-ray)

The same lineage fingerprint appears in every repo:

```
git commit SHA ─┐                ├─→  "e90324a9-dvc3f2a1b7c"dvc.lock MD5  ─┘         │                          ├─→  MLflow tag: dvc.commit                          ├─→  SageMaker parameter: DVCCommitHash                          └─→  mlflow.log_input() dataset record
Reproduce any run from any pipeline:  git checkout e90324a9 && dvc checkout
```

This isn’t accidental. I built it in from Chapter 1 because the cost of adding it later — retrofitting lineage to existing runs — is much higher than starting with it. Every interview question about reproducibility or compliance now has a concrete, working answer.

[Engineering Journey: Fine-Tuning LLMs from Laptop to Production](https://pub.towardsai.net/engineering-journey-fine-tuning-llms-from-laptop-to-production-b1852f8ada39) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.