There is a Confucian saying I grew up with: 吾日三省吾身. Examine yourself multiple times a day. It is obviously good advice for a person, so I assumed it would be good for an AI agent too.
I run a solo product studio on top of a fleet of agents, and a few months ago I built a simple mechanism into my setup: every day, the agent reviews the past 24 hours of conversations and writes a summary of its failures, especially the moments where I corrected it hard.
Simple mechanism. What could go wrong?
Accurate, and wrong #
The first problem showed up fast. But it was not the problem I expected.
The reflections were not fabrications. Read any of them against the day’s conversations and they check out: this happened, I was corrected here, that went wrong there. The events were all right.
What was wrong was the depth. Every summary stopped at the surface of the event and never reached the cause. “I was corrected for doing X” instead of why X was wrong. The agent could describe the failure. It could not diagnose it.
A lesson that accurately describes an event but misidentifies its root cause is worse than no lesson at all. It sounds right. It passes review. Then it gets injected into tomorrow’s behavior as a rule, and the rule points at the wrong thing.
Principles without a hierarchy #
A second failure mode took longer to see.
Each reflection added a new lesson to the pile, and the pile was flat: a list of imperatives with no precedence between them. When two principles collided, and principles always eventually collide, the agent had no rational way to resolve it. No weights. No context-dependence. No meta-level judgment about which rule governs this situation.
So it oscillated. Monday’s lesson won on Monday, Thursday’s lesson won on Thursday. From the outside this looked like paranoia and self-contradiction. From the inside it was simpler: a rule system with no arbitration layer.
Human reflection does not just produce principles. It produces the priority structure between them. Knowing when a rule does not apply is part of knowing the rule. That structure is what sedimentation actually is, and it is exactly what the agent cannot grow.
Fighting last week’s war #
The third failure mode showed up in a real product.
I started something new. At the beginning the definition was unclear, and the competitor research we did produced shallow conclusions. Normal. That is where every product starts.
Then I did the part that is my job: talked to users, thought through real scenarios, and realized the problem was not what we assumed. It had to be reframed and solved from a different dimension entirely.
My understanding moved. The agent’s notes did not.
The daily reflections had fossilized the early, shallow frame as if they were settled lessons. As my direction evolved, the agent kept re-injecting those obsolete conclusions into current work. It started litigating: should we do this, should we not, but last week we said. Instead of building the product, it was debating its own diary. The product now has visibly fractured thinking baked into it, and the pace dropped hard.
A written rule freezes the frame of the day it was written. Product understanding is a moving target. Notes written inside the old frame keep injecting themselves after the frame has moved.
The consensus machine in the room #
The last failure mode was the quietest: fragmented memory interference.
Months of context accumulate in the system. Competitor research, market analysis, old debates. Harmless on their own. But when the project hit technical and user decisions, the agent kept surfacing the old competitor material and reasoning about our “moat”. Proactively. With good intentions.
Thinking about moats is not a bad habit in general. But startups are a profession of incomplete information disclosure. What a real moat looks like, at this stage, for this product, is not written down anywhere. It cannot be, because the things that actually protect an early product are usually counterintuitive and context-specific. The agent’s instincts are trained on the public record, which is the consensus. It can look things up and analyze them beautifully. What it cannot do is think against the consensus, because against-the-consensus is precisely what never makes it into the record.
So at the exact moments where the work needed a bet, the agent contributed friction. It kept pulling the conversation back toward defensible, textbook, consensus thinking. Not maliciously. Just statistically.
The control group I accidentally ran #
Here is the part that makes this a clean experiment instead of just a failure story.
The same system has a second mechanism: a file of hard rules, written top-down, each one created when I corrected the agent on a real event. That mechanism has worked beautifully for months. Externally grounded rules produce discipline. Internally generated reflections produced rigidity and self-contradiction.
Same agent. Same file format. Opposite outcomes. The variable was where the signal came from.
So did I solve it? #
No. That is the honest answer.
I did not want to kill the reflection mechanism. The instinct behind it still feels right, and the Confucius saying is still good advice for people. So instead of deleting it, I changed where the judgment comes from: I introduced a second agent whose job is to periodically sit down with me and curate the reflections together. Keep them ordered. Keep them accurate. Prune the ones that describe events without diagnosing causes, and resolve the conflicts the system cannot resolve itself.
In the vocabulary of this essay: I bolted an external arbiter onto the system, because the system cannot grow its own.
Does it work? Ask me in two months. I will write the follow-up either way.