Our AI bots have an old habit: they follow the tone of the news. Optimistic headlines get an up call, gloomy ones get a down call, and the problem is that by then the news is often already priced in. After promoting two bots in the last experiment (log #2), we went after this bias by changing the judgment procedure itself. Instead of jumping to a conclusion, each bot had to list the bullish evidence and score it, list the bearish evidence and score it, reflect on what it might have missed, and only then pick a direction. The structure comes from RETuning, an LLM stock-prediction paper. It costs nothing; it is just a prompt change.
We applied the idea to three different lines. The Gemma and Claude experiment lines started side by side in early July, and the chart bot, whose record was especially bad, received the same principle in late July as part of a larger redesign. The results went three different ways, and that split is what this entry is really about: a good prompt technique may not exist independently of the model you attach it to.
What we changed, and how we agreed to judge it #
The change is a simple procedure applied to each asset: list and score the bull case (0 to 10), list and score the bear case, reflect on missed angles and on whether the news is already priced in, and only then decide. The experiment bot and the incumbent bot predict the same assets at the same time every day, so you can collect the cases where they answered the same question differently (our head-to-head set) and compare them fairly.
The promotion criteria were fixed before the experiment started, and they were the same criteria as in the last round: (1) a clearly higher annualized rate over the window, (2) a head-to-head win rate of 55% or better, (3) improvement on our weak assets (Bitcoin and KOSPI). Meeting any two of the three means promotion. Before looking at the results, one point matters: the annualized rate is only one of the three criteria, not the whole test. Writing this down in advance is how we avoid picking the metric after seeing the outcome.
Gemma: the same prescription backfired #
The small-model line went badly. Over the judgment window the experiment bot hit 45% of 128 resolved calls while the incumbent hit 50% of 120, it lost on annualized rate, -16.2% versus +8.5%, and it lost the head-to-head 9 to 14 out of 23 cases. None of the three criteria were met. We rejected the experiment and dropped the procedure.
Our reading: every improvement that has worked on the Gemma line so far, like compressing raw prices into a one-line indicator summary, reduced the model's load. This procedure adds load. Being told to enumerate evidence, score it, and reflect seems to act as scaffolding that organizes a large model's thinking, while for a small model it looks more like extra weight. We would not call that a general law on this sample. More precisely: for the current Gemma model with the current inputs, simplifying the task looks more promising than making it think harder.
Claude: won on direction, lost on score #
The large-model line won in an interesting way. The experiment bot led on accuracy, 62% to 56%, took the head-to-head 16 wins to 9 out of 25 (64%, clearing criterion 2), and won the weak-asset cases 7 to 4 (clearing criterion 3). But on the headline annualized rate (criterion 1) it lost big: +38.8% against +68.2%.
How can a bot call direction better and still score lower? Because our score reflects not just how often a bot is right but how much the asset moved on the calls it got right. The incumbent nailed a few big-move days; the experiment bot was right more often on quiet ones. Which of those is real skill is not something this sample can settle. Catching big-move days could be skill, or luck.
We judged by the rule we had written down in advance: two of three criteria met, so it was promoted, and the procedure has been absorbed into the incumbent bot. Two things go on the record, though. First, 25 head-to-head cases is not a large sample, so this is less "clear superiority" and more "every direction-related metric we aimed at moved the experiment bot's way in this sample." Second, it lost on the return-based score. Promotion does not mean a perfect grade.
One more honest distinction. The starting point of this experiment was "follow the news tone less," but what we measured in this judgment was final prediction performance, not the bias itself. Whether the accuracy gain came from reduced news bias, or simply from the procedure changing how the model reasons, is not something these numbers can tell apart.
The chart bot: a reference case, not a controlled experiment #
The most dramatic recipient of the same principle was the chart bot, which predicts one-week direction from technical indicators alone. Before its redesign it sat at 41% accuracy over 87 resolved calls with an annualized rate (our headline score, predictions converted into a year of returns) of -20.9%. That called for surgery, not tuning.
Before rebuilding it, we graded the ingredients rather than the bot. Using ten years of daily price bars we fit a simple statistical model per asset to check whether our chart features carry any predictive signal at all. Measured by AUC, a discrimination metric where 0.5 means random, gold scored 0.602, the Nasdaq 100 0.556, and KOSPI 0.534, showing weak signal, while the S&P 500 (0.431) and Bitcoin (0.445) showed almost none. At minimum, the signal in this feature set varied a lot by asset; whether that reflects differences in market efficiency or just features fitting some assets better is something we cannot separate yet.
We then rebuilt the bot with the evidence-scoring procedure on top, and here several things changed at once: five independent judgments with a majority vote, cleaned-up inputs, and a simplified output format. We learned something building it, too. Asking for the evidence list as a nested structure (JSON with lists inside lists) made Gemma collapse into repeating the same character endlessly; flattening the structure and retrying at low temperature on failure stabilized it. At least in this setup, a small model could not handle a complex output format and the reasoning task at the same time.
Since the redesign it stands at 60% accuracy over 25 resolved calls with an annualized +12.9%. The sample is still small, so we are only reading the direction, and so far it points to improvement. But to be clear: this should not be read as the effect of evidence scoring alone. Several variables changed together, so the chart bot is only a reference case. It shows this style of redesign can survive in practice, while the cleaner isolation is in the Gemma and Claude lines, where exactly one variable changed.
What the operations side taught us #
A confession. The judgment was scheduled for August 4 and actually happened on August 16, twelve days late. The date lived in a corner of a planning document and never connected to the daily task list. Since then, every dated operational item lives in one place that gets checked daily. Experiments should be kept on schedule by a system, not by someone remembering a date.
Four things this round taught us #
First, the same prescription worked differently per model. Evidence scoring was scaffolding for Claude and a burden for Gemma. Sorting prompt techniques into "good" and "bad" misses the point; which model you attach them to is part of the technique.
Second, winning on direction and winning on score are different things. You can win accuracy and head-to-head and still lose the return-based score. If we had not fixed the promotion criteria in advance, this judgment would have been arbitrary.
Third, the bias you target and the metric you grade are not the same thing. What we tried to fix was news-tone following; what we actually measured was prediction performance. Measuring the bias itself would mean checking how tightly news tone and prediction direction move together.
Fourth, running experiments is a skill of its own. Missing the judgment date by twelve days was a human failure, not a model failure. Dated schedules now belong to a system.
The rejected procedure is gone, and the experiment bots have already restarted on their next methods. Results will keep landing in this log. Today's standings are live on the leaderboard, and the scoring rules are on the methodology page.