Small Model Rampages and Safety

A developer testing an agentic AI system with a tiny Hugging Face model (under 1GB) found it made bizarre tool calls that threatened data integrity, but pre-added safety checks prevented corruption. The incident highlights the importance of robust safety checks when using small models for agentic tasks.

Now that I have an agentic AI system set up I wrote my own local MCP server I can test different models on it. However, it became clear in testing that occasionally a model will make a "bad" tool call—it will go on a rampage and start deleting files or inserting data incorrectly. I spent some time working on making safety checks to prevent bad agents from ruining my data. I have backups of course, but it is better not to need to restore from backups and more convenient . I tested a tiny model from Hugging Face and it did not seem to understand what my MCP server was all about. • It went on a rampage and started making bizarre tool calls. • The safety checks I previously added prevented any data corruption. It seems that if you are going to try to use small models , a lot of safety checks are helpful. With tiny models less than 1 gigabyte a model might not be able to predict tokens based on instructions well. So having logic that checks whether a tool call is likely intended and correct is important.