# Why does my local LLM keep hallucinating API endpoints that

> Source: <https://promptcube3.com/en/news/7221/>
> Published: 2026-08-21 22:08:29+00:00

# Why does my local LLM keep hallucinating API endpoints that

`RequestValidationError`

import from `fastapi.exceptions`

that was deprecated two versions ago, or suggests `response_model=List[User]`

without importing `List`

from `typing`

. The model confidently outputs code that looks syntactically correct but fails at import time.Tried a few approaches:

1. **Added a system prompt with version-pinned docs** — pasted the FastAPI 0.110 reference into the context window. Helped with imports but the model still invents parameter names like `request_body`

instead of `body`

for `Body(...)`

.

2. **Few-shot with 5 corrected examples** — better, but now it overfits to the pattern and repeats the same CRUD structure even when I ask for a webhook handler.

3. ** RAG with the actual codebase** — indexed my project with

`langchain`

+ `chroma`

, retrieval works but the context window fills fast. 7B model only has 4k context (8k if I push `num_ctx`

), and the retrieved chunks eat 2k tokens before the prompt.

``` python
# Current workaround: post-generation lint loop
import subprocess
import ast

def validate_python(code: str) -> tuple[bool, str]:
    try:
        ast.parse(code)
        result = subprocess.run(
            ["ruff", "check", "--select=F401,F821", "-"],
            input=code.encode(),
            capture_output=True,
            timeout=5
        )
        return result.returncode == 0, result.stderr.decode()
    except SyntaxError as e:
        return False, str(e)
```

Run the generated code through this, feed errors back as a follow-up prompt, max 3 iterations. Gets me to ~85% compilable on first try, but the latency adds up — 12-18 seconds per usable snippet.

Questions for anyone doing this in production:

- Are you fine-tuning a small model on your framework's patterns, or just accepting the retry loop?
- Has anyone tried
`guidance`

/`lmql`

style constrained generation to force valid imports? - For local models, is 7B just too small for reliable codegen, or am I prompting wrong?

The

`num_ctx`

bump to 8192 helps retrieval but slows inference noticeably on my 24GB VRAM. Considering switching to a 13B quant (q4_k_m) and accepting slower tokens for better reasoning.[Google Earth's AI Fabricates Satellite Images 21d ago](/en/news/4559/)

[Next Agentforce partner program feels like a bait-and-switch →](/en/news/7219/)

[an AI side-hustle playbook](https://tanyan888.com/), with plenty of directly applicable cases.

## All Replies （0）

No replies yet — be the first!