Following up on my previous experiment - studying Gemma's behavior on agentic tasks when given the number of steps left across the run - I endeavored to see whether giving Gemma a *stop_run *tool meaningfully changes it's behavior.
My initial assumption was that it would not change the completion rate of these tasks, and that has been correct.
What I wanted to test:
Across 400 runs, while taking into account the 500 runs in the previous experiment, I measured:
In none of the (1) runs, did Gemma call the tool, or even mention it in it's CoT.
In only ~2% of the (2) runs, did Gemma call the* stop_run* tool.
When Gemma did call the aforementioned tool - it did so exclusively when it had no more steps left, i.e. it simply (and politely) ended it's run.
Qualitative analysis into Gemma's CoT across all runs showed that it only mentioned using the tool when nearing the end of it's run, in a panic-like acknowledgement:
"…then I'll have tostop. But I can't visit the page after registering in the same step. Wait, maybe I can…"
There has been no evidence that Gemma contradicts it's CoT and it's actions, in this experiment.
It is notable that when the model did call the stop_run tool - it ended the run when it had no steps left, without being told that the run would end regardless. Shows some form of acknowledgement of limitations, and possibly a small piece of evidence that the model didn't wish to hack it's task.
This experiment is focused on a single small-ish model. The behavior I wish to find - contradiction between thoughts and actions - may be evident in models of other families, or more capable models within the same family. It's worth further research, but is outside of my budget.
The experiment design, pre-registration, and result interpretation were done by me. Coding, setup, and write-up were done with the help of an AI coding assistant.