Why does my local LLM keep hallucinating API endpoints that

RequestValidationError

import from fastapi.exceptions

that was deprecated two versions ago, or suggests response_model=List[User]

without importing List

from typing

. The model confidently outputs code that looks syntactically correct but fails at import time.Tried a few approaches:

1. Added a system prompt with version-pinned docs — pasted the FastAPI 0.110 reference into the context window. Helped with imports but the model still invents parameter names like request_body

instead of body

for Body(...)

.

2. Few-shot with 5 corrected examples — better, but now it overfits to the pattern and repeats the same CRUD structure even when I ask for a webhook handler.

3. ** RAG with the actual codebase** — indexed my project with

langchain

• chroma

, retrieval works but the context window fills fast. 7B model only has 4k context (8k if I push num_ctx

), and the retrieved chunks eat 2k tokens before the prompt.

import subprocess
import ast

def validate_python(code: str) -> tuple[bool, str]:
    try:
        ast.parse(code)
        result = subprocess.run(
            ["ruff", "check", "--select=F401,F821", "-"],
            input=code.encode(),
            capture_output=True,
            timeout=5
        )
        return result.returncode == 0, result.stderr.decode()
    except SyntaxError as e:
        return False, str(e)

Run the generated code through this, feed errors back as a follow-up prompt, max 3 iterations. Gets me to ~85% compilable on first try, but the latency adds up — 12-18 seconds per usable snippet.

Questions for anyone doing this in production:

• Are you fine-tuning a small model on your framework's patterns, or just accepting the retry loop?
• Has anyone tried guidance

/lmql

style constrained generation to force valid imports? - For local models, is 7B just too small for reliable codegen, or am I prompting wrong?

The

num_ctx

bump to 8192 helps retrieval but slows inference noticeably on my 24GB VRAM. Considering switching to a 13B quant (q4_k_m) and accepting slower tokens for better reasoning.Google Earth's AI Fabricates Satellite Images 21d ago

Next Agentforce partner program feels like a bait-and-switch →

an AI side-hustle playbook, with plenty of directly applicable cases.

All Replies （0） #

No replies yet — be the first!