Yesterday tried out a small tool called SIMURG. It sits in front of an OpenAI-compatible endpoint and monitors the generation. If it detects that the model is going off the rails, it stops the request and retries it. Hallucinations mostly, and right now i have really consumer GPU RTX 3060
I tested it with Qwen3.8 27B running a 2-bit quant on an RTX 3060. The model is surprisingly capable for its size, but at 2-bit I was getting quite a few hallucinations and made-up facts and actually bad, i mean not bad, and not so good due to hallucinations because of Chinese symbols and so on.
So just used SIMURG, the responses have been noticeably more consistent so far. It's still early and I'm testing how well it holds up across different prompts, but the difference is pretty noticeable on this particular setup and actually not bad i trained this tool on my type of hallucination when model starts to do shitty tool calls and leaks its own tool call style so yep and simurg just aborting it and end user never sees hallucination and btw this is machine learning model that contains 15 parameters.
You can install it if you want:
pip install simurg
Or link it's open-sourced: GitHub