# How a Few YAML Files Saved Me Hours of Work and Wasted GPU Money During Fine-Tuning > Source: > Published: 2026-10-26 10:28:38+00:00 When I first started working on machine learning projects, I did what many beginners naturally do. I put configuration values directly inside the code. Learning rate? Hardcoded. Batch size? Hardcoded. Checkpoint settings? Somewhere inside another Python file. Hugging Face repository information? Somewhere else. Logging settings? Another file. It worked. And because it worked, I didn’t immediately see the problem. The problem only became obvious when the project became bigger. Suddenly, changing one parameter wasn’t really changing one parameter anymore. I had to ask myself: *Where else is this value being used?* In an industry-grade codebase with hundreds or thousands of lines of code, that question becomes painful very quickly. A small experiment could mean searching through several files, checking that I hadn’t missed another dependency, modifying the value, running the pipeline, and then discovering that something else was still using the old configuration. Sometimes a change that should have taken a minute could consume around 10 minutes. When you have hundreds of parameters spread across model configuration, logging, monitoring, data processing, checkpoints, guardrails, and infrastructure, the total configuration work can easily become hours. In my case, I was dealing with more than **250 configuration-related values** across the fine-tuning system. Manually preparing or changing everything could easily cost me around **two hours** before the actual training even became the interesting part. Then I started treating YAML differently. Not as a place to dump some parameters. But as the **control layer of the entire fine-tuning pipeline**. And that changed the way I worked. Imagine this simplified situation: ``` learning_rate = 0.00002batch_size = 4max_epochs = 10save_steps = 500 ``` At first, this feels convenient. Everything is right there. But eventually another file needs the same value. Then another. Maybe your training code assumes one checkpoint strategy while your evaluation script assumes another. Maybe your local test needs a small model while the AWS training job should use a much larger one. Maybe your monitoring configuration changes between environments. The code starts knowing too much about the environment in which it is running. That was the important lesson for me. **The pipeline logic and the pipeline configuration are not the same thing.** The code should know *how to train*. The configuration should tell it *how this particular run should behave*. Once I understood that separation, YAML started becoming much more valuable. Instead of spreading important values throughout the codebase, I organized them inside a configuration folder. Conceptually, my setup had six main areas: ``` configs/│├── guardrails.yaml├── logging.yaml├── model_input.yaml├── model_output.yaml├── model_parameters.yaml└── monitoring.yaml ``` Each file had one responsibility. That sounds like a small architectural decision. For me, it became one of the most useful decisions in the entire fine-tuning pipeline. The first configuration layer is the one I now appreciate the most. **Guardrails.** Cloud training changes how you should think about failure. If something breaks after two seconds on your laptop, it is annoying. If something breaks after hours on a GPU instance, it is annoying **and expensive**. So before allowing the expensive part of the pipeline to begin, I want to validate as much as possible. For example: The principle became: ***Fail early when failure is cheap.*** If my learning rate is invalid, I don’t want AWS to discover that for me after starting an expensive job. If my dataset path is broken, I don’t want the GPU waiting while the pipeline crashes. If a required repository configuration is missing, training should never start. The guardrails become the gatekeeper. Fine-tuning without good logging becomes surprisingly difficult. You remember that “one run performed better.” But then you ask:**Why?** Without structured logging, experiments become memories instead of evidence. So I separated logging configuration as well. I prefer structured output such as JSON containing information like: ``` timestamprun_idmodelhardwarecheckpointguardrail resultstraining stateevaluation state ``` Secrets obviously need to be protected rather than dumped into logs. But the broader idea is important: *Every experiment should be reconstructable from evidence.* This becomes especially valuable after running many experiments. Then comes the input side. This answers a very simple question: *What exactly am I feeding into the model?* The input configuration can describe things such as: Separating this from the training logic helps enormously when experimenting. I can change the dataset configuration without rewriting the training pipeline itself. The output side handles what training produces. For my workflow, Hugging Face is an important part of this. The output configuration can describe things such as: Checkpoint configuration is especially important. Training can run for hours. I don’t want one interruption to mean starting from zero again. The output layer defines how that progress is preserved. Then comes the section everyone normally thinks about first: **training parameters.** Things such as: ``` learning_rate:batch_size:epochs:warmup:gradient_accumulation:quantization:early_stopping: ``` The exact values depend on the model, dataset, hardware, and experiment. But moving them outside the training code changed my workflow. Suppose monitoring tells me the learning rate is too low. The model is barely learning. Previously, I might need to find where that value was defined in the code. Then check whether another part of the pipeline overrides it. Then modify it carefully. With the configuration layer, I already know where to look. I change the configuration. The implementation stays untouched. That sounds almost trivial until you’re doing many experiments. Then the saved time compounds quickly. Monitoring became another important part of the system. During training, I want to understand questions such as: This is where configuration starts translating directly into money. One of the clearest examples is **early stopping**. Earlier, imagine I planned a training run for: 10 epochs Without proper stopping behavior, the job could continue running even when the model had stopped meaningfully improving. That means GPU time is still being consumed. But the model isn’t giving me meaningful additional value. So I added early stopping behavior. Conceptually: ``` early_stopping: enabled: true patience: 3 metric: eval_loss ``` The exact configuration can vary, but the principle is simple. If the model has not meaningfully improved for several evaluations, stop. In my case, I think about something like **three consecutive non-improving stages** before deciding the run is no longer worth continuing. Instead of blindly running for around 10 hours, some runs could finish around **6–8 hours**, depending on the data and experiment. That is potentially **2–4 hours of compute avoided on a single experiment**. And that changed how I thought about early stopping. It isn’t only an ML technique. It is also a **cost-control mechanism**. *If the model has stopped learning, the GPU should probably stop earning.* But configuration alone wasn’t enough. The next improvement was changing **where I test the pipeline**. I now think about fine-tuning in three stages. I don’t immediately throw the full workload onto AWS. First, I want to know whether the pipeline fundamentally works. On my lower-resource local system, I can use something much smaller — for example, a small model such as a **0.5B parameter model**. I am not trying to get the final model here. I am testing things like: ``` Can the code start?Can the dataset load?Does tokenization work?Do the configurations resolve?Does training execute?Can checkpoints be created?Does evaluation run?Does the pipeline finish? ``` If something is broken, I want to discover it here. Because local failure is cheap. Passing locally doesn’t guarantee everything will behave correctly in the cloud. So instead of jumping directly into the complete training job, I run a **pilot test**. Maybe only: 10–20 samples or another deliberately small subset. Now I’m verifying the actual AWS environment: This stage catches environment-specific failures without committing the full budget. Only after both checks pass do I move to the expensive run. The workflow becomes: That simple progression changed the economics of experimentation for me. Instead of debugging with an expensive GPU, I debug **before the expensive GPU becomes necessary**. This is where the configuration architecture becomes powerful. I don’t want three completely different codebases for: I want the same pipeline behaving differently based on configuration. ``` profile: local_smoke ``` can become: ``` profile: cloud_pilot ``` and finally: ``` profile: cloud_training ``` The underlying code doesn’t need to be constantly rewritten. The behavior changes through configuration. That is a much safer mental model. There is another benefit I didn’t fully expect. Today, coding agents can modify a lot of code very quickly. That’s useful. But I don’t necessarily want an agent freely deciding my fine-tuning configuration. Parameters such as: ``` learning ratebatch sizecheckpoint policydatasetquantizationmonitoringoutput repositoryearly stopping ``` can materially change an experiment. So I prefer having a clearly defined configuration layer. The code can evolve. The agent can help implement features. But the experiment-critical values live somewhere explicit and reviewable. That gives me a boundary. I know where the decisions are. And I can review them before spending compute. Before this structure, changing a parameter could sometimes take around **10 minutes** because I needed to locate it and understand what else depended on it. Across hundreds of configuration values, properly preparing an experiment could take roughly **two hours**. With centralized YAML configuration, much of that hunting disappears. Instead of asking: *Where in the codebase is this value hiding?* I ask: *Which configuration file owns this value?* That is a much easier question. And when a training run that might have continued for around 10 hours can be stopped around 6–8 hours because monitoring says it is no longer improving, the saving isn’t only developer time anymore. It is compute time too. After all of this, I don’t think the biggest lesson is: *“Use YAML.”* You could use another configuration format. The deeper lesson is: ***Separate decisions from implementation.*** Your Python code contains the mechanism. Your configuration contains the decisions for a particular run. Your guardrails validate those decisions. Your monitoring tells you whether those decisions are working. Your early stopping protects you when they aren’t. And your testing stages make sure you fail at the cheapest possible point. That’s the architecture that saved me time. YAML simply made that architecture easy to see and operate. If I were rebuilding a fine-tuning pipeline from zero today, I would introduce configuration management much earlier. I would not wait until the project became large enough to hurt. I would separate at least: ``` guardrailslogginginputoutputtraining parametersmonitoring ``` from the beginning. I would also design three execution profiles immediately: ``` local smokecloud pilotfull training ``` And I would treat early stopping as both an ML decision and an infrastructure decision. Because cloud GPUs don’t care whether your model is learning. They continue running until **you tell them to stop**. I started using YAML because I wanted an easier way to change parameters. I ended up learning something much bigger. Good configuration management isn’t about avoiding a few lines of Python. It is about creating a system where experiments are easier to understand, reproduce, control, debug, and stop. For me, the progression looked like this: ***Hardcoded values → centralized configuration → guardrails → cheap testing → monitored training → early stopping.*** And somewhere along that path, YAML stopped being “just a configuration file.” It became one of the simplest ways I found to protect both my **time** and my **GPU budget**. [How a Few YAML Files Saved Me Hours of Work and Wasted GPU Money During Fine-Tuning](https://blog.stackademic.com/how-a-few-yaml-files-saved-me-hours-of-work-and-wasted-gpu-money-during-fine-tuning-833f0548e640) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.