Why does my local LLM keep hallucinating API endpoints that

A developer reports that local 7B LLMs hallucinate invalid FastAPI code, such as deprecated imports and missing typing imports, and that a post-generation lint loop with ruff achieves ~85% first-try compilability but adds 12-18 seconds latency per snippet. The developer is considering switching to a 13B quantized model (q4_k_m) for better reasoning despite slower inference.

Why does my local LLM keep hallucinating API endpoints that RequestValidationError import from fastapi.exceptions that was deprecated two versions ago, or suggests response model=List User without importing List from typing . The model confidently outputs code that looks syntactically correct but fails at import time.Tried a few approaches: 1. Added a system prompt with version-pinned docs — pasted the FastAPI 0.110 reference into the context window. Helped with imports but the model still invents parameter names like request body instead of body for Body ... . 2. Few-shot with 5 corrected examples — better, but now it overfits to the pattern and repeats the same CRUD structure even when I ask for a webhook handler. 3. RAG with the actual codebase — indexed my project with langchain + chroma , retrieval works but the context window fills fast. 7B model only has 4k context 8k if I push num ctx , and the retrieved chunks eat 2k tokens before the prompt. python Current workaround: post-generation lint loop import subprocess import ast def validate python code: str - tuple bool, str : try: ast.parse code result = subprocess.run "ruff", "check", "--select=F401,F821", "-" , input=code.encode , capture output=True, timeout=5 return result.returncode == 0, result.stderr.decode except SyntaxError as e: return False, str e Run the generated code through this, feed errors back as a follow-up prompt, max 3 iterations. Gets me to ~85% compilable on first try, but the latency adds up — 12-18 seconds per usable snippet. Questions for anyone doing this in production: - Are you fine-tuning a small model on your framework's patterns, or just accepting the retry loop? - Has anyone tried guidance / lmql style constrained generation to force valid imports? - For local models, is 7B just too small for reliable codegen, or am I prompting wrong? The num ctx bump to 8192 helps retrieval but slows inference noticeably on my 24GB VRAM. Considering switching to a 13B quant q4 k m and accepting slower tokens for better reasoning. Google Earth's AI Fabricates Satellite Images 21d ago /en/news/4559/ Next Agentforce partner program feels like a bait-and-switch → /en/news/7219/ an AI side-hustle playbook https://tanyan888.com/ , with plenty of directly applicable cases. All Replies （0） No replies yet — be the first