{"slug": "why-does-my-local-llm-keep-hallucinating-api-endpoints-that", "title": "Why does my local LLM keep hallucinating API endpoints that", "summary": "A developer reports that local 7B LLMs hallucinate invalid FastAPI code, such as deprecated imports and missing typing imports, and that a post-generation lint loop with ruff achieves ~85% first-try compilability but adds 12-18 seconds latency per snippet. The developer is considering switching to a 13B quantized model (q4_k_m) for better reasoning despite slower inference.", "body_md": "# Why does my local LLM keep hallucinating API endpoints that\n\n`RequestValidationError`\n\nimport from `fastapi.exceptions`\n\nthat was deprecated two versions ago, or suggests `response_model=List[User]`\n\nwithout importing `List`\n\nfrom `typing`\n\n. The model confidently outputs code that looks syntactically correct but fails at import time.Tried a few approaches:\n\n1. **Added a system prompt with version-pinned docs** — pasted the FastAPI 0.110 reference into the context window. Helped with imports but the model still invents parameter names like `request_body`\n\ninstead of `body`\n\nfor `Body(...)`\n\n.\n\n2. **Few-shot with 5 corrected examples** — better, but now it overfits to the pattern and repeats the same CRUD structure even when I ask for a webhook handler.\n\n3. ** RAG with the actual codebase** — indexed my project with\n\n`langchain`\n\n+ `chroma`\n\n, retrieval works but the context window fills fast. 7B model only has 4k context (8k if I push `num_ctx`\n\n), and the retrieved chunks eat 2k tokens before the prompt.\n\n``` python\n# Current workaround: post-generation lint loop\nimport subprocess\nimport ast\n\ndef validate_python(code: str) -> tuple[bool, str]:\n    try:\n        ast.parse(code)\n        result = subprocess.run(\n            [\"ruff\", \"check\", \"--select=F401,F821\", \"-\"],\n            input=code.encode(),\n            capture_output=True,\n            timeout=5\n        )\n        return result.returncode == 0, result.stderr.decode()\n    except SyntaxError as e:\n        return False, str(e)\n```\n\nRun the generated code through this, feed errors back as a follow-up prompt, max 3 iterations. Gets me to ~85% compilable on first try, but the latency adds up — 12-18 seconds per usable snippet.\n\nQuestions for anyone doing this in production:\n\n- Are you fine-tuning a small model on your framework's patterns, or just accepting the retry loop?\n- Has anyone tried\n`guidance`\n\n/`lmql`\n\nstyle constrained generation to force valid imports? - For local models, is 7B just too small for reliable codegen, or am I prompting wrong?\n\nThe\n\n`num_ctx`\n\nbump to 8192 helps retrieval but slows inference noticeably on my 24GB VRAM. Considering switching to a 13B quant (q4_k_m) and accepting slower tokens for better reasoning.[Google Earth's AI Fabricates Satellite Images 21d ago](/en/news/4559/)\n\n[Next Agentforce partner program feels like a bait-and-switch →](/en/news/7219/)\n\n[an AI side-hustle playbook](https://tanyan888.com/), with plenty of directly applicable cases.\n\n## All Replies （0）\n\nNo replies yet — be the first!", "url": "https://wpnews.pro/news/why-does-my-local-llm-keep-hallucinating-api-endpoints-that", "canonical_source": "https://promptcube3.com/en/news/7221/", "published_at": "2026-08-21 22:08:29+00:00", "updated_at": "2026-08-21 22:12:44.505566+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["FastAPI", "langchain", "chroma", "ruff", "Google Earth"], "alternates": {"html": "https://wpnews.pro/news/why-does-my-local-llm-keep-hallucinating-api-endpoints-that", "markdown": "https://wpnews.pro/news/why-does-my-local-llm-keep-hallucinating-api-endpoints-that.md", "text": "https://wpnews.pro/news/why-does-my-local-llm-keep-hallucinating-api-endpoints-that.txt", "jsonld": "https://wpnews.pro/news/why-does-my-local-llm-keep-hallucinating-api-endpoints-that.jsonld"}}