{"slug": "we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu", "title": "We distilled a job posting extractor into a 400M model that runs on an 8 GB GPU", "summary": "The solo operator behind the Job Opportunities API released joa-job-extractor-400m-gguf, an Apache 2.0 model distilled from IBM's Granite 3.1 1B-A400M and fine-tuned with LoRA to extract 13 structured fields from job postings, running on an 8 GB GTX 1070. Labels came from roughly 47,300 evidence bundles annotated by GLM-5.3-Flash and filtered for grounding, with training and evaluation done on the EuroHPC Discoverer+ supercomputer. The developer says the salary parser found pay in only 59% of 886 hand-checked postings that stated it, motivating a small local model over hosted inference.", "body_md": "I run [Job Opportunities API](https://jobopportunitiesapi.org) (JOA) on my own: a jobs data API with one rule, every posting comes straight from the employer's own systems. Today that is about 2.69 million live postings, each one confirmed at its source within the last 18 hours.\n\nThis week we published the model we built to read those postings: [joa-job-extractor-400m-gguf](https://huggingface.co/JobOpportunitiesAPI/joa-job-extractor-400m-gguf), Apache 2.0, free to use. This post is about why it exists, how we made it, where it is good, and where it is not.\n\nMost of our cleaning is rules: parsers, mappings, regular expressions. Rules are cheap, fast and auditable, and for most fields they are excellent. Salary is where they break down.\n\nEmployers write pay in prose. \"From 52,000 rising to 58,000 after probation.\" \"Up to 4,200 euros gross a month\", written in German. \"Competitive, plus a 15% bonus.\" Our salary parser is precise when it fires, but in an audit of 886 postings checked by hand against the live pages, it found the pay in only 59% of the postings that actually stated one.\n\nA large language model reads that prose easily. The problem is scale and money. Running a big hosted model over millions of postings, again every time a posting changes, is not something a one person company can afford. And we did not want our data quality to depend on someone else's API being up and cheap.\n\nSo the goal was a model small enough to run on hardware we already own (one GTX 1070 with 8 GB), good enough to catch the pay our rules miss, and honest enough that we could check every number it returns.\n\nBefore training anything, we ran a long bake-off: the same 100 postings, many models, scored field by field against reference answers. We expected the biggest, newest models to win. They did not, or not by much.\n\nThat taught us the idea this whole project rests on: a bigger model is often not the answer. The right input (a clean evidence bundle) and narrow training on one job matter more than size. It was the start of our theory, and the 400M model is the proof.\n\n**1. Evidence bundles.** For each posting we build one text input with everything we know, in a fixed order: the rendered page a candidate sees, the JSON-LD JobPosting if there is one, the record from the employer's applicant tracking system, simple liveness signals, and our own stored text as the lowest priority. Typical bundles are 4,000 to 5,000 tokens.\n\n**2. A teacher writes the labels.** A larger model, [GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (MIT licence), read about 47,300 bundles and answered in one flat JSON schema with 13 fields: is the job still open, title, company, location, salary minimum, maximum, currency and period, posted date, agency or not, remote type and seniority. A grounding filter then removed or corrected values the posting text did not support.\n\n**3. A student learns them.** We fine-tuned IBM's [Granite 3.1 1B-A400M](https://huggingface.co/ibm-granite/granite-3.1-1b-a400m-instruct), a mixture of experts model with 1 billion parameters in total and about 400 million active per token. LoRA on the attention projections only, rank 64, one epoch. Then we merged the adapter and converted it to GGUF for llama.cpp, in three sizes: Q4_K_M (0.82 GB), Q8_0 (1.42 GB) and f16 (2.67 GB).\n\nNone of this would have happened on our own hardware. The teacher runs, the training and the evaluations ran on **Discoverer+** in Sofia, a EuroHPC supercomputer with NVIDIA H200 GPUs, through the [EuroHPC AI Factories Playground access](https://www.eurohpc-ju.europa.eu/playground-access-ai-factories_en), which is made for startups and SMEs.\n\nThe honest version, because I wish someone had told me: we were accepted within two days, but real access came almost a month later, and the \"onboarding\" was one email. The clusters are shared and busy, so a job can wait in the queue for a day, and you learn Slurm the hard way (always set --time, and never ask for more memory than you need, because you are billed for what you allocate). Still, I know of no other way for a small European company to get that many H200 hours. If you are building something in Europe that needs real compute, apply.\n\nOn our test set of 100 held out postings, scored strictly field by field:\n\n| model | strict F1 | \n|---|---|\n| trained adapter on a GPU | 0.926 | \n| GGUF Q4_K_M on a GTX 1070 | 0.923 | \n| GGUF Q8_0 on a GTX 1070 | 0.922 | \n\nThe quantised files lose almost nothing, so the 0.82 GB Q4_K_M is the practical choice.\n\nOn the wider audit (1,084 postings from 74 job sources, 886 checked against the source):\n\n| salary, on postings that state one | recall | precision | F1 | \n|---|---|---|---|\n| this model | 86 | 94 | 90 | \n| our rule based parser | 59 | 97 | 74 | \n\nIt also produced valid JSON for all 1,084 postings, including German, French and Dutch ones. Title, company and city land between 95 and 99.\n\nSpeed: about 2 seconds per posting on the GTX 1070, one at a time. On one H200 with 32 requests in parallel it did about 62,900 postings an hour. On a CPU it works but is slow (tens of seconds per posting), so treat that as a batch tool.\n\nNot as the main extractor. Rules go first, because they are precise and explainable. The model runs second, only where the rules found no pay, and every value it returns must pass the \"digits are in the text\" check before we store it. In our API every field also says where it came from: published by the employer, inferred by us, or missing.\n\n```\n# download the 0.82 GB file and the prompt\nhuggingface-cli download JobOpportunitiesAPI/joa-job-extractor-400m-gguf \\\n  joa-job-extractor-400m-Q4_K_M.gguf system_prompt.txt --local-dir .\n\n# serve it with llama.cpp (24K context covers long postings)\nllama-server -m joa-job-extractor-400m-Q4_K_M.gguf -c 24576 -ngl 99\n```\n\nThen send the system prompt (with {today} replaced by the fetch date) and one evidence bundle to the OpenAI compatible chat endpoint, at temperature 0. The model card has the bundle layout and a real worked example.\n\nBecause a model like this is only useful to people who already have the job posting text, and that is exactly the hard part of the problem we sell. If it helps your scraper, your research or your job board, good. If you need the postings themselves, already cleaned, deduplicated and confirmed at their source, that is what [Job Opportunities API](https://jobopportunitiesapi.org) does, with a free key and an MCP server for AI agents.\n\nQuestions, results on your own data, or bugs: [hello@jobopportunitiesapi.org](mailto:hello@jobopportunitiesapi.org).\n\n*Loukas Tzekos, founder, Job Opportunities API*", "url": "https://wpnews.pro/news/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu", "canonical_source": "https://dev.to/lucagiftzek/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu-1bnk", "published_at": "2026-11-06 15:46:15+00:00", "updated_at": "2026-11-06 15:59:43.000608+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "machine-learning", "ai-infrastructure"], "entities": ["Job Opportunities API", "IBM", "Granite 3.1 1B-A400M", "GLM-5.3-Flash", "EuroHPC", "Discoverer+", "NVIDIA", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu", "markdown": "https://wpnews.pro/news/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu.md", "text": "https://wpnews.pro/news/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu.txt", "jsonld": "https://wpnews.pro/news/we-distilled-a-job-posting-extractor-into-a-400m-model-that-runs-on-an-8-gb-gpu.jsonld"}}