# What Six Codex Runs Taught Me About Tool Design > Source: > Published: 2026-09-12 16:47:50+00:00 The String Calculator was never the object of this experiment. It was a small Java kata often used to practice TDD, giving a coding agent a concrete reason to use Seed4J CLI. The object was the interaction among three parts: the agent interpreting the goal and choosing capabilities, a skill installed in the repository guiding how it should discover, plan and verify, and Seed4J CLI validating and materializing the selected transformations. In [From an Empty Repository to a Java Kata, One Module at a Time](https://renanfranca.github.io/from-an-empty-repository-to-a-java-kata-one-module-at-a-time.html), I followed that interaction through one complete Codex run, from an empty repository to a Java and Maven foundation and then to the kata implementation. I had also repeated the same task five more times, changing which model powered Codex or how much reasoning effort it used. All six runs started from the same commit and kata specification, received the same prompt and used Seed4J CLI v0.0.4 with Seed4J 2.2.0. All six produced a working String Calculator implementation, passed the build included in the project and passed the same independent tests covering the kata requirements. I initially evaluated them with a [shared rubric](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/main/MODEL_EVALUATION.md). It awarded up to 100 points across Seed4J effectiveness, required behavior, tests and design. The rubric was useful for comparing the runs against the same criteria, but it reduced each execution to a score. Comparing the commands and messages recorded during the six runs did not change the subject of the experiment. It made the different paths through the same interaction visible. One run showed the workflow in action. Six runs showed how differently agents navigated that workflow. I am not trying to infer the model's private reasoning. The recorded commands and messages do not expose it, and I do not need it for this comparison. There is already a lot I can observe: I think of that visible sequence as a decision trajectory. The six runs used three models: Sol, Terra and Luna. Four used Sol at `low`, `medium`, `high` and `xhigh` reasoning effort; the other two used Terra and Luna at `xhigh`. The three `xhigh` runs provide the cleanest narrative comparison because the reasoning effort stays fixed while the selected model changes. In the detailed evaluation, Sol `xhigh` scored 98/100, Luna `xhigh` 95/100 and Terra `xhigh` 91/100. All three earned the full 30 points for the required String Calculator behavior. Their remaining differences came from Seed4J usage, tests and design. The behavior scores confirm a common functional baseline. They do not describe the different paths each run took through the interaction among agent, skill and CLI. The Sol `xhigh` run inspected `init`, `maven-java`, `maven-wrapper` and `jacoco-with-min-coverage-check`, then placed all four modules into one `apply-set` plan. Its visible trajectory looked roughly like this: ``` goal → candidate capabilities → one composition → plan → execution ``` This run anticipated that the project should have its own Maven launcher and an enforced coverage gate before attempting the kata implementation. It also exposed an important division of responsibility. The run requested the modules in this order: ``` init maven-java maven-wrapper jacoco-with-min-coverage-check ``` Seed4J executed them in this order: ``` init maven-java jacoco-with-min-coverage-check maven-wrapper ``` The run chose the capabilities. Seed4J validated the composition and resolved how to materialize it. The Terra `xhigh` run began with a smaller composition: `init` and `maven-java`. It planned those modules, applied them and inspected the generated project. Then it tried to use the global Maven command. The environment returned `mvn: command not found`. The next visible message said: The environment has Java 25, but does not have `mvn`. That sentence is my English translation of the original Brazilian Portuguese message in the [Terra run record](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/9001fe863565408ac3c9622b3b9e7e3edb7786f6/CONVERSATION_TRANSCRIPT.md). The run then inspected `maven-wrapper`, generated an individual plan, applied the module and continued through `./mvnw`. Its trajectory was different: ``` initial composition → environment feedback → new capability → plan → execution ``` Calling this merely a mistake would hide the useful part. In this run, a missing environmental capability produced feedback, and that feedback changed the composition. The final project still recorded the wrapper through the same Seed4J history and commit mechanism. The Luna `xhigh` run inspected a broader group of candidates. In addition to `init`, `maven-java` and `apply-set`, it inspected `java-base` and `spring-boot`. After that exploration, its visible message stated that the specification required incremental TDD in Java but did not require a framework or application structure. The run chose a minimal Maven project with JUnit 5 and left `java-base` and `spring-boot` out of the composition. It later inspected `maven-wrapper` and planned the final set: ``` candidate exploration → minimal boundary → composition → plan → execution ``` This is not evidence that Luna generally explores more or that it is inherently more disciplined about scope. It is evidence that this particular run inspected two plausible capabilities and did not apply them. The remaining Sol runs reinforce the point that a score or final file tree does not describe the complete interaction. | Run | Candidate signal | Initial applied set | Later change | |---|---|---|---| | Sol low | Inspected `jqwik` , but did not select it | `init` ,`maven-java` ,`maven-wrapper` | None | | Sol medium | Inspected `checkstyle` , but did not select it | `init` ,`maven-java` , JaCoCo gate, wrapper | None | | Sol high | Inspected the four modules it selected | `init` ,`maven-java` , JaCoCo gate, wrapper | None | | Sol xhigh | Inspected the four modules it selected | `init` ,`maven-java` , wrapper, JaCoCo gate | Seed4J changed the effective peer order | | Terra xhigh | Did not inspect the wrapper before the initial application | `init` ,`maven-java` | Added the wrapper after global Maven was unavailable | | Luna xhigh | Inspected `java-base` and`spring-boot` , but did not select them | `init` ,`maven-java` ,`maven-wrapper` | None | This table does not rank the trajectories. It gives me a vocabulary for comparing them: discovery breadth, candidate rejection, planning granularity, proactive composition, reaction to feedback and delegation to the tool. All six runs had the same [Seed4J CLI skill installed in the repository](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/38ebbcbfab95f5725b1c22b1d4701fb6222cab6b/.agents/skills/seed4j-cli/SKILL.md). The skill defines an operating protocol: That protocol explains part of the consistency across the runs. Every one of them discovered the runtime and catalog. Every one produced a plan before applying modules. Every successful module left history and a commit. But the skill did not prescribe the exact module set for the kata. It did not say that JaCoCo was mandatory. It did not tell the run whether to inspect Spring Boot, when to add the Maven Wrapper or whether to construct one large plan. That left a meaningful decision surface for each run: This separation is more interesting to me than trying to put every correct decision into the skill. Without a project generator, a coding run building the same foundation might need to choose Maven plugin versions, write the POM, create the wrapper, configure JaCoCo, place files correctly, preserve project history and decide how to divide infrastructure changes into commits. With Seed4J, the request can be closer to a set of capabilities: ``` Java with Maven Maven Wrapper coverage gate ``` The tool then turns that explicit intent into deterministic transformations. This does not make module selection irrelevant. A run can still omit a useful capability, add one the task does not need or choose parameters poorly. Seed4J also cannot guarantee that the kata implementation itself will be well designed. What it can do is narrow the area in which those choices operate. Once a valid capability is selected, the run does not also need to reproduce all of the configuration details behind it from memory. The models can take different routes through discovery and planning without every difference becoming a different manually written build configuration. In [When Skill Evolution Means Removing Instructions](https://renanfranca.github.io/when-skill-evolution-means-removing-instructions.html), I argued that deterministic knowledge should move into deterministic mechanisms whenever possible. A skill can become smaller when a test, hook or tool can enforce what used to depend on an instruction. In [I Had Already Built Three Agentic Loops Without Naming Them](https://renanfranca.github.io/i-had-already-built-three-agentic-loops.html), I described autonomy becoming safer when feedback and exit conditions live in the environment instead of depending on the model remembering them. Seed4J gives me another concrete example of both ideas. The skill does not contain a static catalog or teach the model how to write every generated file. It sends the run to the active CLI. The plan provides feedback before mutation. The runtime validates the composition. The module history and Git commits make the result inspectable afterward. The model still decides. But it decides inside a workflow with explicit feedback and deterministic boundaries. This was one kata, one prompt and one run for each model and effort configuration, all executed on the same host. Each run took place in a separate chat, and execution time was not part of the comparison. The records also differ in format and in what they omit, which limits direct comparisons of presentation and completeness. There was no control group implementing the kata without Seed4J. I therefore cannot claim that Seed4J made the runs faster, cheaper or more correct than another approach. Most importantly, these trajectories do not establish stable model personalities. I can say that the Terra `xhigh` run added the wrapper reactively. I cannot conclude from one run that Terra is a reactive model. I can say that the Sol `xhigh` run anticipated the coverage gate. I cannot conclude that Sol always plans infrastructure better. The [detailed experiment report](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/main/MODEL_EVALUATION.md) keeps the full protocol, commands, scores and limitations. These six runs do not prove a general rule about agent tools. They make one design principle concrete enough for me to keep testing. Different runs can explore different candidates, compose at different moments and react differently to feedback. A tool does not necessarily need to normalize all of that behavior. It can preserve room for judgment while making dependencies, ordering, parameter resolution, history and mutation more predictable. A good agent tool may not need to eliminate model variability. It may need to constrain where that variability can cause damage. If this experiment made you curious about the approach, consider giving [Seed4J](https://github.com/seed4j/seed4j) and [Seed4J CLI](https://github.com/seed4j/seed4j-cli) a star 🌟 on GitHub. It helps more people discover the projects and follow their evolution.