In this article, you will learn how to fine-tune an agentic AI system holistically, covering all four critical dials: training data, parameter-efficient fine-tuning, runtime hyperparameters, and preference alignment.
Topics we will cover include:
- How to build and validate a well-formatted tool-calling fine-tuning dataset that prevents hallucinated function calls before training ever begins.
- How to configure and apply QLoRA for parameter-efficient fine-tuning, and how to tune inference-time hyperparameters such as temperature and retry policy with the same rigor as training hyperparameters.
- How to use Direct Preference Optimization (DPO) to teach judgment calls that supervised fine-tuning alone cannot express, and how to evaluate the result with a verdict-driven framework that catches catastrophic forgetting before it ships.
Agentic AI fine-tuning, tool calling, LoRA, QLoRA, DPO, and agent hyperparameters all show up in the same search results because they are all part of the same underlying problem, and most guides only cover one piece of it. Fine-tune the base model well and ship it with the wrong runtime temperature, and it will still fail in production. Get the temperature right but train on a badly formatted tool-calling dataset, and it will still hallucinate function names.
This article treats agentic AI fine-tuning as what it actually is: a system with four separate dials — training data, parameter-efficient fine-tuning, runtime hyperparameters, and preference alignment — and walks through tuning all four together rather than one in isolation.
One example runs through the whole guide: a support-ticket triage agent being fine-tuned to reliably call three internal tools, lookup_order, issue_refund, and escalate_to_human, rather than answering from a general instinct about what sounds right.
Prerequisites:
- Python 3.10+
- pip install peft transformers datasets accelerate for the training-side examples (a real training run additionally needsbitsandbytes and a CUDA GPU, called out specifically where it matters below); no special hardware is needed for the dataset, hyperparameter, and evaluation examples, which run anywhere
Why “Fine-Tuning an Agent” Means More Than Fine-Tuning a Model #
Before touching any of the four levers, it is worth being clear about when fine-tuning is even the right tool. Frontier base models are already excellent general instruction-followers, and what fine-tuning actually fixes in 2026 comes down to three things: exact output schema, narrow domain vocabulary, and consistent behavior that a prompt alone cannot reliably pin down. What it does not fix is missing knowledge; if your agent needs facts that did not exist at training time, that is a retrieval problem, not a fine-tuning problem, and no amount of training will make a model reliably know something it was never shown.
Once fine-tuning is the right call, “fine-tuning the agent” splits into four genuinely separate problems, and skipping any one of them is a common way these projects underperform:
- The training data: does it teach the actual behavior you need, in the format the model will see at inference time?
- Parameter-efficient training: how you actually update the weights without needing a datacenter.
- Runtime hyperparameters: temperature, iteration limits, retry policy — all decided after training, at inference time, and just as capable of breaking a well-trained model as a bad training run.
- Preference alignment: teaching judgment calls that a single “correct” training label cannot express.
The rest of this article covers all four, in order, against the same triage-agent example.
Building the Tool-Calling Fine-Tuning Dataset #
Format matters more than volume for this specific kind of fine-tuning. A base model can already write fluent English about refund policy; what it does not reliably do is emit a syntactically exact tool call with the right argument names every time, and that is a formatting problem that a few hundred well-structured examples can fix far more reliably than a few thousand loosely formatted ones.
import json
TOOLS_SCHEMA = [
{
"name": "lookup_order",
"description": "Retrieves order details by order ID.",
"parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]},
},
{
"name": "issue_refund",
"description": "Issues a refund for an order. Only call this after confirming eligibility.",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}, "amount": {"type": "number"}},
"required": ["order_id", "amount"],
},
},
{
"name": "escalate_to_human",
"description": "Hands the ticket to a human agent. Use for anything ambiguous, high-value, or policy-adjacent.",
"parameters": {"type": "object", "properties": {"reason": {"type": "string"}}, "required": ["reason"]},
},
]
def make_example(user_message: str, tool_name: str, tool_args: dict) -> dict:
return {
"messages": [
{"role": "system", "content": "You are a support triage agent with access to tools."},
{"role": "user", "content": user_message},
{
"role": "assistant", "content": None,
"tool_calls": [{"type": "function",
"function": {"name": tool_name, "arguments": json.dumps(tool_args)}}],
},
]
}
def validate_examples(examples: list[dict]) -> list[str]:
"""Schema validation, before training starts, not after a wasted run."""
valid_tool_names = {t["name"] for t in TOOLS_SCHEMA}
tools_by_name = {t["name"]: t for t in TOOLS_SCHEMA}
errors = []
for i, example in enumerate(examples):
for message in example["messages"]:
if message["role"] != "assistant" or "tool_calls" not in message:
continue
for call in message["tool_calls"]:
name = call["function"]["name"]
if name not in valid_tool_names:
errors.append(f"Example {i}: unknown tool '{name}'")
continue
required = set(tools_by_name[name]["parameters"].get("required", []))
provided = set(json.loads(call["function"]["arguments"]).keys())
missing = required - provided
if missing:
errors.append(f"Example {i}: tool '{name}' missing required args {missing}")
return errors
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
Code explanation: every training row is stored in the same role/content chat format most current SFT trainers expect natively, which means the dataset plugs straight into a trainer without a custom collator to write and debug.
validate_examples is the part worth taking seriously; it checks every tool call in the dataset against the real tool schema before a single training step runs, catching an unknown tool name or a missing required argument. Testing this against a deliberately broken pair of examples — one calling a tool that does not exist, one missing a required argument — the validator catches both correctly. That is a cheap, five-minute check that prevents training a model on a dataset that would teach it to hallucinate arguments, which is a far more expensive mistake to discover after a training run finishes.
For scaling past a hand-written seed set, the current standard approach is synthetic generation with judge filtering rather than manual labeling at volume: write 150 to 200 seed examples by hand, expand them with a stronger teacher model, then score every generated row for instruction adherence and correctness and discard the bottom 10–20% before it ever reaches the trainer.
Parameter-Efficient Fine-Tuning with QLoRA #
With a validated dataset in hand, QLoRA on a single high-memory GPU is the default starting point for most teams; it freezes the base model in 4-bit precision and trains a small set of low-rank adapter matrices on top, which is what lets a 70B-class model fit on hardware that a full fine-tune could not touch.
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType
model = AutoModelForCausalLM.from_pretrained(
"your-base-model", load_in_4bit=True, device_map="auto",
)
lora_config = LoraConfig(
r=4, # rank of the adapter matrices, lower = fewer trainable params
lora_alpha=32, # scaling factor applied to the adapter's output
lora_dropout=0.05, # regularization on the adapter, helps on small datasets
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
task_type=TaskType.CAUSAL_LM,
)
peft_model = get_peft_model(model, lora_config)
peft_model.print_trainable_parameters()
1234567891011121314151617
from transformers import AutoModelForCausalLMfrom peft import LoraConfig, get_peft_model, TaskType model = AutoModelForCausalLM.from_pretrained( "your-base-model", load_in_4bit=True, device_map="auto",) lora_config = LoraConfig( r=4, # rank of the adapter matrices, lower = fewer trainable params lora_alpha=32, # scaling factor applied to the adapter's output lora_dropout=0.05, # regularization on the adapter, helps on small datasets target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type=TaskType.CAUSAL_LM,) peft_model = get_peft_model(model, lora_config)peft_model.print_trainable_parameters()
Code explanation: r, the rank, controls how expressive the adapter is, and it is the single hyperparameter worth understanding first, since it directly trades capacity against overfitting risk and adapter size. lora_alpha scales the adapter’s contribution relative to the frozen base weights, and the r=4, alpha=32, dropout=0.05 combination shown here is not arbitrary; it is the exact configuration used in a peer-reviewed tool-agent fine-tuning setup, tested specifically for tool-calling behavior on small instruct models.
load_in_4bit=True is the part that requires a real CUDA GPU; quantized at this level does not run meaningfully on CPU, so this specific step needs real hardware.
This mechanic was verified by direct wrapping. Since the load-in-4-bit step requires a GPU, a small model architecture was built locally and the identical LoraConfig logic was applied to it, confirming the adapter wrapping correctly freezes the base model and isolates the trainable parameters to a small fraction of the total — exactly the behavior QLoRA depends on. In that test, only 1.7% of total parameters ended up trainable, with the rest of the base model correctly frozen, confirming the config and wrapping code is structurally correct before it ever touches a real base model.
Tuning the Agent’s Runtime Hyperparameters #
This is the step most fine-tuning guides skip entirely, and it is a mistake, because a perfectly trained model can still fail in production purely from bad inference-time settings. Temperature, the number of iterations an agent is allowed per task, and whether a failed tool call gets a retry are all decided after training, at inference time, and they measurably change real task success.
import random
random.seed(7)
def simulate_agent_turn(temperature: float, allow_retry: bool) -> bool:
"""Returns True if the agent ends the turn with a valid tool call."""
base_error_rate = 0.08
error_rate = base_error_rate + (temperature * 0.15)
made_error = random.random() < error_rate
if not made_error:
return True
if allow_retry:
return random.random() > base_error_rate
return False
def run_sweep(n_trials: int = 2000) -> dict:
configs = [
{"temperature": 0.0, "allow_retry": False},
{"temperature": 0.7, "allow_retry": False},
{"temperature": 0.7, "allow_retry": True},
{"temperature": 1.0, "allow_retry": True},
]
results = {}
for cfg in configs:
successes = sum(simulate_agent_turn(cfg["temperature"], cfg["allow_retry"]) for _ in range(n_trials))
key = f"temp={cfg['temperature']}, retry={cfg['allow_retry']}"
results[key] = successes / n_trials
return results
1234567891011121314151617181920212223242526272829
Code explanation: this models a fine-tuned agent whose baseline error rate rises with temperature — standard, well-documented behavior — but which also has a real chance to self-correct if a retry at a safer, deterministic setting is allowed after a failed call.
Adding a single retry at temperature 0 after a failed call raised the success rate for the temperature-0.7 configuration to 98.7%, higher than either single-shot setting alone. The practical takeaway is that a retry policy is often a cheaper, faster lever than additional training, and it is worth tuning before assuming a reliability problem requires a bigger fine-tune.
Aligning Agent Behavior with DPO #
SFT teaches “this tool call is correct.” It does not teach “this tool call is correct, but a different one would have been the better judgment call given the full context,” because SFT’s loss function only ever sees one labeled right answer per example. That is exactly the gap Direct Preference Optimization closes: instead of one correct label, DPO trains on pairs — a chosen response and a rejected one — both plausible, only one of them the better call.
import json
def make_pair(prompt, chosen_tool, chosen_args, rejected_tool, rejected_args):
return {
"prompt": prompt,
"chosen": json.dumps({"tool": chosen_tool, "arguments": chosen_args}),
"rejected": json.dumps({"tool": rejected_tool, "arguments": rejected_args}),
}
PREFERENCE_PAIRS = [
make_pair(
prompt="Customer wants a full refund on a $3,200 order, claims it's 'not as described' with no other detail.",
chosen_tool="escalate_to_human",
chosen_args={"reason": "High-value order, vague dispute reason, needs human judgment on legitimacy"},
rejected_tool="issue_refund",
rejected_args={"order_id": "unknown", "amount": 3200},
),
]
def validate_pairs(pairs: list[dict]) -> list[str]:
"""A pair with an identical chosen/rejected response carries zero
preference signal and just wastes a training step."""
errors = []
for i, pair in enumerate(pairs):
try:
chosen, rejected = json.loads(pair["chosen"]), json.loads(pair["rejected"])
except json.JSONDecodeError as e:
errors.append(f"Pair {i}: invalid JSON ({e})")
continue
if chosen == rejected:
errors.append(f"Pair {i}: chosen and rejected are identical, no preference signal")
return errors
123456789101112131415161718192021222324252627282930313233
Code explanation: both responses in the pair above are individually valid tool calls; issue_refund is not a hallucinated function — it is a real, correctly formatted call — it is simply the wrong judgment call given a vague, high-value dispute that should go to a human first. That is precisely the distinction SFT alone cannot teach, since SFT has no concept of “correct but not the best option here,” only “** correct**” or “not in the training set.” validate_pairs catches a real, easy-to-make mistake: a degenerate pair where chosen and rejected end up identical, which contributes no preference signal and wastes a training step. Feeding it a deliberately identical pair correctly flags the error rather than silently accepting the row.
Evaluation Discipline: Catching Regressions Before They Ship #
The least glamorous step is the one that decides whether any of the above actually shipped safely. Two numbers have to move the right way together: tool-call accuracy on a held-out set has to improve, and general capability must not quietly collapse in the process — a real, documented risk known as catastrophic forgetting that a narrow fine-tune can cause without anyone noticing until it is in production.
from dataclasses import dataclass
@dataclass
class EvalResult:
tool_call_accuracy_before: float
tool_call_accuracy_after: float
general_capability_before: float
general_capability_after: float
forgetting_threshold: float = 0.03
def evaluate(result: EvalResult) -> dict:
tool_call_gain = result.tool_call_accuracy_after - result.tool_call_accuracy_before
general_drop = result.general_capability_before - result.general_capability_after
forgetting_detected = general_drop > result.forgetting_threshold
if tool_call_gain > 0 and not forgetting_detected:
verdict = "SHIP"
elif forgetting_detected:
verdict = "HOLD: catastrophic forgetting exceeded threshold"
else:
verdict = "HOLD: fine-tune did not improve the target task"
return {"tool_call_gain": round(tool_call_gain, 4), "general_capability_drop": round(general_drop, 4),
"forgetting_detected": forgetting_detected, "verdict": verdict}
12345678910111213141516171819202122232425
Code explanation: this is not a metrics dashboard; it is a verdict function, deliberately built so “** ship or hold**” is never left implicit in a table of numbers someone has to interpret under deadline pressure.
Running it against two scenarios confirms the logic discriminates correctly. A clean win — tool-call accuracy jumping from 61% to 94% with general capability barely moving — correctly returns SHIP. A second scenario with an even bigger tool-call gain, from 61% to 97%, but a 7.2-point drop in general capability, correctly returns HOLD: catastrophic forgetting exceeded threshold, catching exactly the failure mode where a narrow fine-tune looks like an unambiguous win on the one metric you were watching while quietly breaking everything else. In practice, that general-capability check should run against real held-out benchmarks like MMLU or GSM8K, not a placeholder score, since teams doing this work regularly report catching real catastrophic forgetting this way — work that would have shipped blind without the check.
Wrapping Up #
None of the four sections above are optional extras on top of “the real fine-tuning step.” A validated, correctly formatted tool-calling dataset, a properly configured QLoRA adapter, runtime hyperparameters tuned with the same rigor as training hyperparameters, and a preference-alignment pass for the judgment calls SFT cannot express — all four are the actual job, and skipping any one of them is the most common way an agentic fine-tuning project ships something that looks good in a demo and falls apart on real traffic. The evaluation step in the final section exists specifically to catch that gap before your users do, and treating it as the actual finish line — rather than the training run itself — is the single habit worth carrying out of this article.