Good Results when training Qwen 3 4B to learn a new domain

A developer successfully used Unsloth to perform continued pretraining on Qwen 3 4B, teaching the small local LLM to act as a travel advisor for a fictional city by targeting only 66 million parameters (1.6% of the total) via LoRA adapters. The experiment highlights the feasibility of domain-specific training on consumer hardware and the challenges of synthesizing a high-quality training corpus with AI agents.

Teaching a Local LLM a New Domain As an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining CPT on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city. Continued Pretraining CPT As the name suggests, CPT is a continuation of the model’s initial training, typically on a specific domain to allow the model to specialize on top of what it already knows. Normally, full CPT on a model of any size would be impractical on consumer hardware due to high VRAM requirements. However, there is a clever workaround called LoRA Low-Rank Adaption . The basic idea behind LoRA is that the original model weights are kept frozen while small trainable LoRA adapters are attached to some of the layers of the model. During training, we only update the parameters in the LoRA adapter. In practice this means we only have to touch a fraction of the full set of parameters. Example: I am working with qwen 3 4B, which has 4 billion trainable parameters. However with my current LoRA configuration, I am only targeting 66 million parameters - 1.6% of the total number of parameters To do the actual LoRA based CPT training I am using a framework called Unsloth. I have included the full source here https://github.com/thelgevold/continued-pretraining in case you are interested in checking it out. Domain The domain can be anything, but I decided to try to teach the model to act as a travel advisor for a fictional city called Awesomeville in the country of Greatness. Everything about the city is made up of course, but the town has its own subway with multiple historical sites located near the subway stations. The goal of this exercise is to teach the model to reason about the subway map and belonging historical sites. As an example I want the model to reliably answer questions like: What is the subway route from Museum of Greatness History to Founder's Square? As an illustration, I have added a snapshot of the subway map below: As you can see there are three subway lines in the city Green, Blue and Gold with various stations located near historical sites in town. The map also shows that the lines connect through central station, so a travel advisor needs to be able to reason and recommend multi-line routes. Some of the key scenarios are listed below: - Same line travel e.g. Blue line stations only - Single transfer through Central station e.g. Starting on Blue line and transferring to Green line - Connect trips with multiple transfers e.g. Blue - Green - Gold Corpus Create Data As expected, the most time-consuming part of this project by far was defining the corpus, the collection of data used to train the model. Since the entire universe of Awesomeville is fictional, all data had to be synthesized. Using an agent for this is of course the most practical solution these days, but it’s not as easy as just asking an agent for a perfect training corpus. I discovered a few pitfalls when synthesizing data using agents. One thing to keep in mind is that agents often create text generators, which may lead to templated language with lots of repetition from shared intros and fragments. Too much common phrasing around a few variables like station names may make it harder for the model to learn the subtle differences. Another issue is that agents generate data so fast that it’s easy to lose track, and before you know it, your corpus has grown to 10k entries of questionable quality. One of the key goals of this exercise was to come up with a set of training data that would enable the model to reason. I wanted to avoid a situation where the data consists of a large amount of specific route examples since this tends to lead to route memorization and likely poor generalization. Structuring the Data Experimental Phase The first draft of my training data was sort of a disaster. Initially, I felt like I was moving in the right direction but before I knew it, I had generated a 10k bloated dataset that relied far too much on memorization of specific route scenarios. Performance would often be poor when unseen examples were introduced. The large dataset did slow down training, but I learned that when working with large datasets, you may not need to train from scratch after every change. Instead, you can split your training into multiple phases. I would start with pre-training using the full dataset followed by post training on the pre-trained model. Post training would be the same Unsloth process, but using a smaller, much more targeted dataset. Re-Design Luckily, I realized quickly that I had to start over and build up the training data incrementally and put more thought into the design. I would first start with training data for simple single line travel, then move to one-line-transfers and two-line-transfers. Finally, I added bindings from subway stations to historic sites. As I was building out the model’s training set, I also built a comprehensive test suite for doing evals to gauge how well training was progressing. Out of a full test suite of 105 tests, 85 of the scenarios were scenarios that were not covered directly during training. See table below with summary: | Test scenario | Tests | Share | What it tests | Truly held out | Partially represented | Directly represented | |---|---|---|---|---|---|---| | Green same-line journey | 9 | 8.6% | Correct Green Line selection; no transfer | 5 | 0 | 4 | | Blue same-line journey | 16 | 15.2% | Correct Blue Line selection; no transfer | 12 | 0 | 4 | | Gold same-line journey | 9 | 8.6% | Correct Gold Line selection; no transfer | 5 | 0 | 4 | | One-transfer journey | 33 | 31.4% | Origin line → central station → destination line | 31 | 0 | 2 | | Two-journey / errand scenario | 33 | 31.4% | Solve two routes independently; reset state between journeys | 27 | 6 | 0 | | Historic-site → historic-site | 5 | 4.8% | Resolve both sites to stations, then route between them | 5 | 0 | 0 | | Total | 105 | 100% | 85 | 6 | 14 | One of the things that surprised me is how much I was able to shrink the original corpus and still have decent performance. However, I did notice that the small qwen model would often struggle to reliably transfer between lines in single-transfer and two-transfer scenarios. Based on this observation I decided to try to map the original subway map onto an internal representation using synthetic names as seen in the graphic below. The main benefit of this is that the synthetic names make it easier on the small qwen model since line membership is encoded in the name. You also get some help with ordinals from the numeric suffix. I did a similar thing to the historic site mappings. Awesomeville Subway Map Original human-readable station names mapped to the locked synthetic naming convention. Line names remain unchanged; Central Station is the shared interchange represented by central station . Blue Line blue station one blue station two blue station three blue station four central station blue station six Green Line green station one green station two central station green station four green station five Gold Line gold station one gold station two central station gold station four gold station five Based on eval performance, moving to synthetic names in the internal map representation resulted in a performance gain of 24% in accuracy. Most of the gains came in scenarios covering line transfers. Complete Corpus After the redesign I ended up with a total of 700 entries across 30 categories in my full training dataset. I have included a link to the dataset here https://github.com/thelgevold/continued-pretraining/blob/main/city training/data/city lines.jsonl . I have also included a grouping of the data categories below: | Category | Count | Share | Section | Actual example from corpus | |---|---|---|---|---| historic direct binding | 208 | 29.7% | city training historic sites | historic binding blue historic site three 001 blue historic site three blue station three | station line membership | 84 | 12.0% | city training membership | membership blue station one 001 STATION: blue station one LINE: Blue Line | routing rule | 30 | 4.3% | city training transfers | routing rule 001 ROUTING RULE: If origin and destination are on the same line, remain on that line and set TRANSFER COUNT: 0. | multi journey rule | 28 | 4.0% | city training transfers | multi journey rule 001 MULTI JOURNEY RULE: Solve each journey independently. | cross line output rule | 24 | 3.4% | city training output contract | cross line output rule v11 001 CROSS LINE OUTPUT RULE CONDITION: ORIGIN LINE = DESTINATION LINE OUTPUT: - origin station - origin line - central station as the transfer - destination line - destination station DO NOT OUTPUT: - non-transfer intermediate stations | positive output demo | 24 | 3.4% | city training output examples | positive same line output v12 001 QUESTION: What is the subway route from blue station two to blue station four? ANSWER: Take Blue Line from blue station two to blue station four. No transfer is required. | routing decision rule | 24 | 3.4% | city training route rules | routing decision rule v11 001 ROUTING DECISION RULE 1. Resolve origin to a subway station if needed. 2. Resolve destination to a subway station if needed. 3. Determine ORIGIN LINE. 4. Determine DESTINATION LINE. 5. If ORIGIN LINE == DESTINATION LINE: TRANSFER COUNT = 0. 6. If ORIGIN LINE = DESTINATION LINE: TRANSFER COUNT = 1 at central station. | same line output rule | 24 | 3.4% | city training output contract | same line output rule v11 001 SAME LINE OUTPUT RULE CONDITION: ORIGIN LINE == DESTINATION LINE OUTPUT: - origin station - line name - destination station - no transfer DO NOT OUTPUT: - central station unless it is origin or destination - intermediate stations | nontransfer output invariant | 20 | 2.9% | city training output contract | nontransfer output invariant v10 001 NON TRANSFER OUTPUT INVARIANT CONDITION: origin and destination use the same subway line. TRANSFER COUNT: 0 FINAL OUTPUT: origin + line + destination + no transfer OMIT: all intermediate stations OMIT: central station unless central station is itself the origin or destination | routing summary rule | 20 | 2.9% | city training output contract | routing summary rule v10 001 ROUTE OUTPUT SELECTION RULE If TRANSFER COUNT = 0: - output origin station - output subway line - output destination station - state no transfer - do not output intermediate stations - do not mention central station unless it is origin or destination If TRANSFER COUNT = 1: - output origin station and origin line - output central station as the transfer station - output destination line and destination station - do not output other intermediate stations | central balanced decision | 18 | 2.6% | city training transfers | central balanced continue v4 001 CENTRAL ROUTE DECISION CURRENT LINE: Blue Line DESTINATION LINE: Blue Line AT: central station ACTION: CONTINUE TRANSFER COUNT: 0 | central transfer decision | 18 | 2.6% | city training transfers | central decision 001 CURRENT LINE: Blue Line REQUIRED LINE: Blue Line AT: central station ACTION: STAY TRANSFER COUNT: 0 | historic routing rule | 18 | 2.6% | city training historic sites | historic routing rule v11 001 php HISTORIC ROUTING RULE 1. Resolve historic site - access station. 2. Replace the historic-site token with the station. 3. Use that station's line membership for routing. 4. Final route output uses station identifiers only. | multi journey decision rule | 18 | 2.6% | city training transfers | multi journey decision rule v11 001 MULTI JOURNEY DECISION RULE For each journey independently: 1. resolve origin and destination 2. determine origin line and destination line 3. decide transfer count 4. output only routing-relevant endpoints, lines, and transfer 5. reset before the next journey | transfer output invariant | 16 | 2.3% | city training output contract | transfer output invariant v10 001 TRANSFER OUTPUT INVARIANT CONDITION: origin and destination require different subway lines. TRANSFER COUNT: 1 FINAL OUTPUT: origin + origin line + central station transfer + destination line + destination OMIT: all non-transfer intermediate stations | historic resolution invariant | 12 | 1.7% | city training historic sites | historic resolution invariant v9 001 php ENTITY RESOLUTION INVARIANT historic site - access station After resolution, the historic-site identifier is no longer a routing node. FINAL ROUTE may contain station identifiers only. | multi journey demo | 12 | 1.7% | city training transfers | multi journey demo 001 JOURNEY 1: Take Green Line from green station five to central station; transfer at central station to Blue Line; continue to blue station two. END JOURNEY 1 RESET ROUTE STATE JOURNEY 2: Take Blue Line from blue station two to central station; transfer at central station to Gold Line; continue to gold station five. | transfer invariant | 12 | 1.7% | city training transfers | transfer invariant v3 001 TRANSFER INVARIANT: A transfer occurs only when the subway line changes. | multi journey line roles | 9 | 1.3% | city training transfers | multi journey line roles v3 001 TWO JOURNEY LINE ROLES JOURNEY 1 ORIGIN LINE: Blue Line JOURNEY 1 DESTINATION LINE: Green Line JOURNEY 1 ACTION: ride Blue Line to central station; transfer to Green Line; finish journey 1. RESET ROUTE STATE JOURNEY 2 ORIGIN LINE: Green Line JOURNEY 2 DESTINATION LINE: Gold Line JOURNEY 2 ACTION: ride Green Line to central station; transfer to Gold Line; finish journey 2. | same line central invariant | 9 | 1.3% | city training transfers | same line central invariant v4 001 SAME LINE CENTRAL CASE ORIGIN: blue station two DESTINATION: blue station six LINE: Blue Line RULE: If the route passes through central station, remain on Blue Line. ACTION AT CENTRAL: CONTINUE TRANSFER COUNT: 0 | same line route demo | 9 | 1.3% | city training route demos | same line demo 001 ORIGIN: blue station one DESTINATION: blue station three LINE: Blue Line STATIONS: blue station one - blue station two - blue station three TRANSFER COUNT: 0 | single transfer route demo | 9 | 1.3% | city training route demos | transfer demo 001 Take Blue Line from blue station one to central station; transfer at central station to Green Line; continue to green station five. | destination invariant | 8 | 1.1% | city training route rules | destination invariant v9 001 DESTINATION INVARIANT: The requested destination is fixed and must not change during route construction. | historic graph rule | 8 | 1.1% | city training historic sites | historic graph rule v8 001 HISTORIC GRAPH RULE 1. Resolve the historic-site identifier to its access station. 2. Replace the historic-site identifier with that station. 3. Route between stations using canonical graph rules. 4. Use station identifiers, not historic-site identifiers, in the route output. | line output invariant | 8 | 1.1% | city training output contract | line output invariant v9 001 LINE OUTPUT INVARIANT: Every final route must explicitly name each subway line used. | multi journey graph rule | 8 | 1.1% | city training transfers | multi journey graph rule v8 001 MULTI JOURNEY GRAPH RULE For each journey independently: 1. resolve origin/destination, 2. read required canonical graph s , 3. construct the route, 4. finish the journey, 5. discard active route state before the next journey. | multi journey output rule | 8 | 1.1% | city training output contract | multi journey output rule v10 001 MULTI JOURNEY OUTPUT RULE: Summarize each journey independently. Output only its origin, required line or lines, any actual transfer at central station, and its destination. Do not carry intermediate stations or transfer state into the next journey. | historic route composition demo | 6 | 0.9% | city training historic sites | historic route demo 001 php Resolve blue historic site three - blue station three. Resolve gold historic site two - gold station two. Then route only between station identifiers. Take Blue Line from blue station three to central station; transfer at central station to Gold Line; continue to gold station two. | route output invariant | 6 | 0.9% | city training output contract | route output invariant v5 001 ROUTE OUTPUT INVARIANT Every route answer must explicitly name the subway line used. | destination central invariant | 2 | 0.3% | city training transfers | destination central invariant v5 001 DESTINATION CENTRAL INVARIANT If central station is the destination, stop when central station is reached. Do not transfer after reaching the destination. | The most stubborn scenarios to stabilize were related to bindings between historic sites and subway stations. I used the same trick of mapping human readable names to internal names, but ~30% of the training data had to be dedicated to enforcing these mappings. Another stubborn case was determining transfer vs. no transfer for a subway journey. The main challenge is that all transfers occur through Central Station, but the model still needs to distinguish same line journeys from cross line journeys when passing through Central Station. One of the key points to enforce is that just passing through Central Station should not automatically trigger a transfer. It must occur in combination with the origin and destination being on different lines. Another side effect of enforcing this point was that the model would often incorrectly tack on Central Station to a single line journey. Almost as if it was trying to prove the point that passing through Central Station wouldn’t trigger a transfer for same line travel. To counteract this, I introduced a few targeted rules in the same line central invariant category. See one example below: Overall Performance I had to go through multiple iterations to arrive at a high performing dataset. However, at this point I would say model performance is very good. I think it’s fair to say that the model can generalize well and reason across historic sites, lines and stations. Out of 105 eval cases, only two tests fail now. The two failing tests both fail because of incorrectly including Central Station on a journey on the same line. The targeted enforcement described above helped with several of these cases, but the underlying issue isn’t 100% solved. Another area of improvement is ordinals and enumerating the full sequence of stations along a complete journey. The model is very good at mapping full journeys from origin to destination but is not trained to correctly list the full list of stations along the way. The model does get some help with ordinals from the numeric suffix in the station name, but more training scenarios are needed to enforce this point. It’s not a critical skill for route recommendations though. I have included the full repo here https://github.com/thelgevold/continued-pretraining in case you are interested having a look.