{"slug": "good-results-when-training-qwen-3-4b-to-learn-a-new-domain", "title": "Good Results when training Qwen 3 4B to learn a new domain", "summary": "A developer successfully used Unsloth to perform continued pretraining on Qwen 3 4B, teaching the small local LLM to act as a travel advisor for a fictional city by targeting only 66 million parameters (1.6% of the total) via LoRA adapters. The experiment highlights the feasibility of domain-specific training on consumer hardware and the challenges of synthesizing a high-quality training corpus with AI agents.", "body_md": "# Teaching a Local LLM a New Domain\n\nAs an interesting experiment I wanted to learn how to teach a tiny local llm a new domain by doing Continued Pretraining (CPT) on domain specific data. This article is a write-up of my experiences from using Unsloth to train qwen 3 4B to act as a travel advisor for a fictional city.\n\n#### Continued Pretraining (CPT)\n\nAs the name suggests, CPT is a continuation of the model’s initial training, typically on a specific domain to allow the model to specialize on top of what it already knows.\n\nNormally, full CPT on a model of any size would be impractical on consumer hardware due to high VRAM requirements. However, there is a clever workaround called LoRA (Low-Rank Adaption). The basic idea behind LoRA is that the original model weights are kept frozen while small trainable LoRA adapters are attached to some of the layers of the model. During training, we only update the parameters in the LoRA adapter. In practice this means we only have to touch a fraction of the full set of parameters.\n\nExample: I am working with qwen 3 4B, which has 4 billion trainable parameters. However with my current LoRA configuration, I am only targeting 66 million parameters - 1.6% of the total number of parameters!\n\nTo do the actual LoRA based CPT training I am using a framework called Unsloth. I have included the full source [here](https://github.com/thelgevold/continued-pretraining) in case you are interested in checking it out.\n\n#### Domain\n\nThe domain can be anything, but I decided to try to teach the model to act as a travel advisor for a fictional city called Awesomeville in the country of Greatness. Everything about the city is made up of course, but the town has its own subway with multiple historical sites located near the subway stations.\n\nThe goal of this exercise is to teach the model to reason about the subway map and belonging historical sites.\n\nAs an example I want the model to reliably answer questions like: What is the subway route from Museum of Greatness History to Founder's Square?\n\nAs an illustration, I have added a snapshot of the subway map below:\n\nAs you can see there are three subway lines in the city (Green, Blue and Gold) with various stations located near historical sites in town.\n\nThe map also shows that the lines connect through central station, so a travel advisor needs to be able to reason and recommend multi-line routes. Some of the key scenarios are listed below:\n\n- Same line travel (e.g. Blue line stations only)\n- Single transfer through Central station (e.g. Starting on Blue line and transferring to Green line)\n- Connect trips with multiple transfers (e.g. Blue -> Green -> Gold)\n\n#### Corpus\n\n##### Create Data\n\nAs expected, the most time-consuming part of this project by far was defining the corpus, the collection of data used to train the model. Since the entire universe of Awesomeville is fictional, all data had to be synthesized. Using an agent for this is of course the most practical solution these days, but it’s not as easy as just asking an agent for a perfect training corpus.\n\nI discovered a few pitfalls when synthesizing data using agents.\n\nOne thing to keep in mind is that agents often create text generators, which may lead to templated language with lots of repetition from shared intros and fragments. Too much common phrasing around a few variables like station names may make it harder for the model to learn the subtle differences.\n\nAnother issue is that agents generate data so fast that it’s easy to lose track, and before you know it, your corpus has grown to 10k entries of questionable quality.\n\nOne of the key goals of this exercise was to come up with a set of training data that would enable the model to reason. I wanted to avoid a situation where the data consists of a large amount of specific route examples since this tends to lead to route memorization and likely poor generalization.\n\n#### Structuring the Data\n\n##### Experimental Phase\n\nThe first draft of my training data was sort of a disaster. Initially, I felt like I was moving in the right direction but before I knew it, I had generated a 10k bloated dataset that relied far too much on memorization of specific route scenarios. Performance would often be poor when unseen examples were introduced. The large dataset did slow down training, but I learned that when working with large datasets, you may not need to train from scratch after every change. Instead, you can split your training into multiple phases. I would start with pre-training using the full dataset followed by post training on the pre-trained model. Post training would be the same Unsloth process, but using a smaller, much more targeted dataset.\n\n##### Re-Design\n\nLuckily, I realized quickly that I had to start over and build up the training data incrementally and put more thought into the design. I would first start with training data for simple single line travel, then move to one-line-transfers and two-line-transfers. Finally, I added bindings from subway stations to historic sites.\n\nAs I was building out the model’s training set, I also built a comprehensive test suite for doing evals to gauge how well training was progressing. Out of a full test suite of 105 tests, 85 of the scenarios were scenarios that were not covered directly during training. See table below with summary:\n\n| Test scenario | Tests | Share | What it tests | Truly held out | Partially represented | Directly represented |\n|---|---|---|---|---|---|---|\n| Green same-line journey | 9 | 8.6% | Correct Green Line selection; no transfer | 5 | 0 | 4 |\n| Blue same-line journey | 16 | 15.2% | Correct Blue Line selection; no transfer | 12 | 0 | 4 |\n| Gold same-line journey | 9 | 8.6% | Correct Gold Line selection; no transfer | 5 | 0 | 4 |\n| One-transfer journey | 33 | 31.4% | Origin line → central_station → destination line | 31 | 0 | 2 |\n| Two-journey / errand scenario | 33 | 31.4% | Solve two routes independently; reset state between journeys | 27 | 6 | 0 |\n| Historic-site → historic-site | 5 | 4.8% | Resolve both sites to stations, then route between them | 5 | 0 | 0 |\n| Total | 105 | 100% | 85 | 6 | 14 |\n\nOne of the things that surprised me is how much I was able to shrink the original corpus and still have decent performance. However, I did notice that the small qwen model would often struggle to reliably transfer between lines in single-transfer and two-transfer scenarios.\n\nBased on this observation I decided to try to map the original subway map onto an internal representation using synthetic names as seen in the graphic below. The main benefit of this is that the synthetic names make it easier on the small qwen model since line membership is encoded in the name. You also get some help with ordinals from the numeric suffix. I did a similar thing to the historic site mappings.\n\n# Awesomeville Subway Map\n\nOriginal human-readable station names mapped to the locked synthetic naming convention. Line names remain unchanged; **Central Station** is the shared interchange represented by `central_station`\n\n.\n\n## Blue Line\n\n`blue_station_one`\n\n`blue_station_two`\n\n`blue_station_three`\n\n`blue_station_four`\n\n`central_station`\n\n`blue_station_six`\n\n## Green Line\n\n`green_station_one`\n\n`green_station_two`\n\n`central_station`\n\n`green_station_four`\n\n`green_station_five`\n\n## Gold Line\n\n`gold_station_one`\n\n`gold_station_two`\n\n`central_station`\n\n`gold_station_four`\n\n`gold_station_five`\n\nBased on eval performance, moving to synthetic names in the internal map representation resulted in a performance gain of 24% in accuracy. Most of the gains came in scenarios covering line transfers.\n\n#### Complete Corpus\n\nAfter the redesign I ended up with a total of 700 entries across 30 categories in my full training dataset. I have included a link to the dataset [here]( https://github.com/thelgevold/continued-pretraining/blob/main/city_training/data/city_lines.jsonl).\n\nI have also included a grouping of the data categories below:\n\n| Category | Count | Share | Section | Actual example from corpus |\n|---|---|---|---|---|\n`historic_direct_binding` |\n208 | 29.7% | `city_training_historic_sites` |\nhistoric_binding_blue_historic_site_three_001\n\n```\nblue_historic_site_three\nblue_station_three\n```\n\n |\n`station_line_membership` |\n84 | 12.0% | `city_training_membership` |\nmembership_blue_station_one_001\n\n```\nSTATION: blue_station_one\nLINE: Blue Line\n```\n\n |\n`routing_rule` |\n30 | 4.3% | `city_training_transfers` |\nrouting_rule_001\n\n```\nROUTING_RULE: If origin and destination are on the same line, remain on that line and set TRANSFER_COUNT: 0.\n```\n\n |\n`multi_journey_rule` |\n28 | 4.0% | `city_training_transfers` |\nmulti_journey_rule_001\n\n```\nMULTI_JOURNEY_RULE: Solve each journey independently.\n```\n\n |\n`cross_line_output_rule` |\n24 | 3.4% | `city_training_output_contract` |\ncross_line_output_rule_v11_001\n\n```\nCROSS_LINE_OUTPUT_RULE\nCONDITION: ORIGIN_LINE != DESTINATION_LINE\nOUTPUT:\n- origin station\n- origin line\n- central_station as the transfer\n- destination line\n- destination station\nDO NOT OUTPUT:\n- non-transfer intermediate stations\n```\n\n |\n`positive_output_demo` |\n24 | 3.4% | `city_training_output_examples` |\npositive_same_line_output_v12_001\n\n```\nQUESTION:\nWhat is the subway route from blue_station_two to blue_station_four?\n\nANSWER:\nTake Blue Line from blue_station_two to blue_station_four. No transfer is required.\n```\n\n |\n`routing_decision_rule` |\n24 | 3.4% | `city_training_route_rules` |\nrouting_decision_rule_v11_001\n\n```\nROUTING_DECISION_RULE\n1. Resolve origin to a subway station if needed.\n2. Resolve destination to a subway station if needed.\n3. Determine ORIGIN_LINE.\n4. Determine DESTINATION_LINE.\n5. If ORIGIN_LINE == DESTINATION_LINE: TRANSFER_COUNT = 0.\n6. If ORIGIN_LINE != DESTINATION_LINE: TRANSFER_COUNT = 1 at central_station.\n```\n\n |\n`same_line_output_rule` |\n24 | 3.4% | `city_training_output_contract` |\nsame_line_output_rule_v11_001\n\n```\nSAME_LINE_OUTPUT_RULE\nCONDITION: ORIGIN_LINE == DESTINATION_LINE\nOUTPUT:\n- origin station\n- line name\n- destination station\n- no transfer\nDO NOT OUTPUT:\n- central_station unless it is origin or destination\n- intermediate stations\n```\n\n |\n`nontransfer_output_invariant` |\n20 | 2.9% | `city_training_output_contract` |\nnontransfer_output_invariant_v10_001\n\n```\nNON_TRANSFER_OUTPUT_INVARIANT\nCONDITION: origin and destination use the same subway line.\nTRANSFER_COUNT: 0\nFINAL_OUTPUT: origin + line + destination + no transfer\nOMIT: all intermediate stations\nOMIT: central_station unless central_station is itself the origin or destination\n```\n\n |\n`routing_summary_rule` |\n20 | 2.9% | `city_training_output_contract` |\nrouting_summary_rule_v10_001\n\n```\nROUTE_OUTPUT_SELECTION_RULE\nIf TRANSFER_COUNT = 0:\n- output origin station\n- output subway line\n- output destination station\n- state no transfer\n- do not output intermediate stations\n- do not mention central_station unless it is origin or destination\nIf TRANSFER_COUNT = 1:\n- output origin station and origin line\n- output central_station as the transfer station\n- output destination line and destination station\n- do not output other intermediate stations\n```\n\n |\n`central_balanced_decision` |\n18 | 2.6% | `city_training_transfers` |\ncentral_balanced_continue_v4_001\n\n```\nCENTRAL_ROUTE_DECISION\nCURRENT_LINE: Blue Line\nDESTINATION_LINE: Blue Line\nAT: central_station\nACTION: CONTINUE\nTRANSFER_COUNT: 0\n```\n\n |\n`central_transfer_decision` |\n18 | 2.6% | `city_training_transfers` |\ncentral_decision_001\n\n```\nCURRENT_LINE: Blue Line\nREQUIRED_LINE: Blue Line\nAT: central_station\nACTION: STAY\nTRANSFER_COUNT: 0\n```\n\n |\n`historic_routing_rule` |\n18 | 2.6% | `city_training_historic_sites` |\nhistoric_routing_rule_v11_001\n\n``` php\nHISTORIC_ROUTING_RULE\n1. Resolve historic_site -> access_station.\n2. Replace the historic-site token with the station.\n3. Use that station's line membership for routing.\n4. Final route output uses station identifiers only.\n```\n\n |\n`multi_journey_decision_rule` |\n18 | 2.6% | `city_training_transfers` |\nmulti_journey_decision_rule_v11_001\n\n```\nMULTI_JOURNEY_DECISION_RULE\nFor each journey independently:\n1. resolve origin and destination\n2. determine origin line and destination line\n3. decide transfer count\n4. output only routing-relevant endpoints, lines, and transfer\n5. reset before the next journey\n```\n\n |\n`transfer_output_invariant` |\n16 | 2.3% | `city_training_output_contract` |\ntransfer_output_invariant_v10_001\n\n```\nTRANSFER_OUTPUT_INVARIANT\nCONDITION: origin and destination require different subway lines.\nTRANSFER_COUNT: 1\nFINAL_OUTPUT: origin + origin line + central_station transfer + destination line + destination\nOMIT: all non-transfer intermediate stations\n```\n\n |\n`historic_resolution_invariant` |\n12 | 1.7% | `city_training_historic_sites` |\nhistoric_resolution_invariant_v9_001\n\n``` php\nENTITY_RESOLUTION_INVARIANT\nhistoric_site -> access_station\nAfter resolution, the historic-site identifier is no longer a routing node.\nFINAL_ROUTE may contain station identifiers only.\n```\n\n |\n`multi_journey_demo` |\n12 | 1.7% | `city_training_transfers` |\nmulti_journey_demo_001\n\n```\nJOURNEY_1: Take Green Line from green_station_five to central_station; transfer at central_station to Blue Line; continue to blue_station_two.\nEND_JOURNEY_1\nRESET_ROUTE_STATE\nJOURNEY_2: Take Blue Line from blue_station_two to central_station; transfer at central_station to Gold Line; continue to gold_station_five.\n```\n\n |\n`transfer_invariant` |\n12 | 1.7% | `city_training_transfers` |\ntransfer_invariant_v3_001\n\n```\nTRANSFER_INVARIANT: A transfer occurs only when the subway line changes.\n```\n\n |\n`multi_journey_line_roles` |\n9 | 1.3% | `city_training_transfers` |\nmulti_journey_line_roles_v3_001\n\n```\nTWO_JOURNEY_LINE_ROLES\nJOURNEY_1_ORIGIN_LINE: Blue Line\nJOURNEY_1_DESTINATION_LINE: Green Line\nJOURNEY_1_ACTION: ride Blue Line to central_station; transfer to Green Line; finish journey 1.\nRESET_ROUTE_STATE\nJOURNEY_2_ORIGIN_LINE: Green Line\nJOURNEY_2_DESTINATION_LINE: Gold Line\nJOURNEY_2_ACTION: ride Green Line to central_station; transfer to Gold Line; finish journey 2.\n```\n\n |\n`same_line_central_invariant` |\n9 | 1.3% | `city_training_transfers` |\nsame_line_central_invariant_v4_001\n\n```\nSAME_LINE_CENTRAL_CASE\nORIGIN: blue_station_two\nDESTINATION: blue_station_six\nLINE: Blue Line\nRULE: If the route passes through central_station, remain on Blue Line.\nACTION_AT_CENTRAL: CONTINUE\nTRANSFER_COUNT: 0\n```\n\n |\n`same_line_route_demo` |\n9 | 1.3% | `city_training_route_demos` |\nsame_line_demo_001\n\n```\nORIGIN: blue_station_one\nDESTINATION: blue_station_three\nLINE: Blue Line\nSTATIONS: blue_station_one -> blue_station_two -> blue_station_three\nTRANSFER_COUNT: 0\n```\n\n |\n`single_transfer_route_demo` |\n9 | 1.3% | `city_training_route_demos` |\ntransfer_demo_001\n\n```\nTake Blue Line from blue_station_one to central_station; transfer at central_station to Green Line; continue to green_station_five.\n```\n\n |\n`destination_invariant` |\n8 | 1.1% | `city_training_route_rules` |\ndestination_invariant_v9_001\n\n```\nDESTINATION_INVARIANT: The requested destination is fixed and must not change during route construction.\n```\n\n |\n`historic_graph_rule` |\n8 | 1.1% | `city_training_historic_sites` |\nhistoric_graph_rule_v8_001\n\n```\nHISTORIC_GRAPH_RULE\n1. Resolve the historic-site identifier to its access station.\n2. Replace the historic-site identifier with that station.\n3. Route between stations using canonical graph rules.\n4. Use station identifiers, not historic-site identifiers, in the route output.\n```\n\n |\n`line_output_invariant` |\n8 | 1.1% | `city_training_output_contract` |\nline_output_invariant_v9_001\n\n```\nLINE_OUTPUT_INVARIANT: Every final route must explicitly name each subway line used.\n```\n\n |\n`multi_journey_graph_rule` |\n8 | 1.1% | `city_training_transfers` |\nmulti_journey_graph_rule_v8_001\n\n```\nMULTI_JOURNEY_GRAPH_RULE\nFor each journey independently:\n1. resolve origin/destination,\n2. read required canonical graph(s),\n3. construct the route,\n4. finish the journey,\n5. discard active route state before the next journey.\n```\n\n |\n`multi_journey_output_rule` |\n8 | 1.1% | `city_training_output_contract` |\nmulti_journey_output_rule_v10_001\n\n```\nMULTI_JOURNEY_OUTPUT_RULE: Summarize each journey independently. Output only its origin, required line or lines, any actual transfer at central_station, and its destination. Do not carry intermediate stations or transfer state into the next journey.\n```\n\n |\n`historic_route_composition_demo` |\n6 | 0.9% | `city_training_historic_sites` |\nhistoric_route_demo_001\n\n``` php\nResolve blue_historic_site_three -> blue_station_three. Resolve gold_historic_site_two -> gold_station_two. Then route only between station identifiers. Take Blue Line from blue_station_three to central_station; transfer at central_station to Gold Line; continue to gold_station_two.\n```\n\n |\n`route_output_invariant` |\n6 | 0.9% | `city_training_output_contract` |\nroute_output_invariant_v5_001\n\n```\nROUTE_OUTPUT_INVARIANT\nEvery route answer must explicitly name the subway line used.\n```\n\n |\n`destination_central_invariant` |\n2 | 0.3% | `city_training_transfers` |\ndestination_central_invariant_v5_001\n\n```\nDESTINATION_CENTRAL_INVARIANT\nIf central_station is the destination, stop when central_station is reached. Do not transfer after reaching the destination.\n```\n\n |\n\nThe most stubborn scenarios to stabilize were related to bindings between historic sites and subway stations. I used the same trick of mapping human readable names to internal names, but ~30% of the training data had to be dedicated to enforcing these mappings.\n\nAnother stubborn case was determining transfer vs. no transfer for a subway journey. The main challenge is that all transfers occur through Central Station, but the model still needs to distinguish same line journeys from cross line journeys when passing through Central Station.\n\nOne of the key points to enforce is that just passing through Central Station should not automatically trigger a transfer. It must occur in combination with the origin and destination being on different lines.\n\nAnother side effect of enforcing this point was that the model would often incorrectly tack on Central Station to a single line journey. Almost as if it was trying to prove the point that passing through Central Station wouldn’t trigger a transfer for same line travel.\n\nTo counteract this, I introduced a few targeted rules in the same_line_central_invariant category. See one example below:\n\n#### Overall Performance\n\nI had to go through multiple iterations to arrive at a high performing dataset. However, at this point I would say model performance is very good. I think it’s fair to say that the model can generalize well and reason across historic sites, lines and stations.\n\nOut of 105 eval cases, only two tests fail now. The two failing tests both fail because of incorrectly including Central Station on a journey on the same line. The targeted enforcement described above helped with several of these cases, but the underlying issue isn’t 100% solved.\n\nAnother area of improvement is ordinals and enumerating the full sequence of stations along a complete journey. The model is very good at mapping full journeys from origin to destination but is not trained to correctly list the full list of stations along the way. The model does get some help with ordinals from the numeric suffix in the station name, but more training scenarios are needed to enforce this point. It’s not a critical skill for route recommendations though.\n\nI have included the full repo [here](https://github.com/thelgevold/continued-pretraining) in case you are interested having a look.", "url": "https://wpnews.pro/news/good-results-when-training-qwen-3-4b-to-learn-a-new-domain", "canonical_source": "https://www.teachmecoolstuff.com/viewarticle/teaching-a-local-llm-a-new-domain", "published_at": "2026-08-21 13:21:11+00:00", "updated_at": "2026-08-21 13:44:26.198176+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "ai-tools"], "entities": ["Unsloth", "Qwen 3 4B", "LoRA", "Awesomeville", "Greatness"], "alternates": {"html": "https://wpnews.pro/news/good-results-when-training-qwen-3-4b-to-learn-a-new-domain", "markdown": "https://wpnews.pro/news/good-results-when-training-qwen-3-4b-to-learn-a-new-domain.md", "text": "https://wpnews.pro/news/good-results-when-training-qwen-3-4b-to-learn-a-new-domain.txt", "jsonld": "https://wpnews.pro/news/good-results-when-training-qwen-3-4b-to-learn-a-new-domain.jsonld"}}