{"slug": "what-six-codex-runs-taught-me-about-tool-design", "title": "What Six Codex Runs Taught Me About Tool Design", "summary": "A developer ran six Codex coding-agent executions of a Java String Calculator kata, varying the underlying model (Sol, Terra, Luna) and reasoning effort, to study how agents, a repository skill, and the Seed4J CLI interact. All six runs produced working implementations that passed the build and independent tests, with Sol xhigh scoring 98/100, Luna xhigh 95/100, and Terra xhigh 91/100 on a shared rubric. The comparison focused on the differing decision trajectories agents took through capability selection, planning, and execution rather than on model reasoning.", "body_md": "The String Calculator was never the object of this experiment. It was a small Java kata often used to practice TDD, giving a coding agent a concrete reason to use Seed4J CLI.\n\nThe object was the interaction among three parts: the agent interpreting the goal and choosing capabilities, a skill installed in the repository guiding how it should discover, plan and verify, and Seed4J CLI validating and materializing the selected transformations.\n\nIn [From an Empty Repository to a Java Kata, One Module at a Time](https://renanfranca.github.io/from-an-empty-repository-to-a-java-kata-one-module-at-a-time.html), I followed that interaction through one complete Codex run, from an empty repository to a Java and Maven foundation and then to the kata implementation.\n\nI had also repeated the same task five more times, changing which model powered Codex or how much reasoning effort it used. All six runs started from the same commit and kata specification, received the same prompt and used Seed4J CLI v0.0.4 with Seed4J 2.2.0. All six produced a working String Calculator implementation, passed the build included in the project and passed the same independent tests covering the kata requirements.\n\nI initially evaluated them with a [shared rubric](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/main/MODEL_EVALUATION.md). It awarded up to 100 points across Seed4J effectiveness, required behavior, tests and design. The rubric was useful for comparing the runs against the same criteria, but it reduced each execution to a score.\n\nComparing the commands and messages recorded during the six runs did not change the subject of the experiment. It made the different paths through the same interaction visible.\n\nOne run showed the workflow in action. Six runs showed how differently agents navigated that workflow.\n\nI am not trying to infer the model's private reasoning. The recorded commands and messages do not expose it, and I do not need it for this comparison.\n\nThere is already a lot I can observe:\n\nI think of that visible sequence as a decision trajectory.\n\nThe six runs used three models: Sol, Terra and Luna. Four used Sol at `low`, `medium`, `high` and `xhigh` reasoning effort; the other two used Terra and Luna at `xhigh`.\n\nThe three `xhigh` runs provide the cleanest narrative comparison because the reasoning effort stays fixed while the selected model changes.\n\nIn the detailed evaluation, Sol `xhigh` scored 98/100, Luna `xhigh` 95/100 and Terra `xhigh` 91/100. All three earned the full 30 points for the required String Calculator behavior. Their remaining differences came from Seed4J usage, tests and design.\n\nThe behavior scores confirm a common functional baseline. They do not describe the different paths each run took through the interaction among agent, skill and CLI.\n\nThe Sol `xhigh` run inspected `init`, `maven-java`, `maven-wrapper` and `jacoco-with-min-coverage-check`, then placed all four modules into one `apply-set` plan.\n\nIts visible trajectory looked roughly like this:\n\n```\ngoal → candidate capabilities → one composition → plan → execution\n```\n\nThis run anticipated that the project should have its own Maven launcher and an enforced coverage gate before attempting the kata implementation.\n\nIt also exposed an important division of responsibility. The run requested the modules in this order:\n\n```\ninit\nmaven-java\nmaven-wrapper\njacoco-with-min-coverage-check\n```\n\nSeed4J executed them in this order:\n\n```\ninit\nmaven-java\njacoco-with-min-coverage-check\nmaven-wrapper\n```\n\nThe run chose the capabilities. Seed4J validated the composition and resolved how to materialize it.\n\nThe Terra `xhigh` run began with a smaller composition: `init` and `maven-java`. It planned those modules, applied them and inspected the generated project.\n\nThen it tried to use the global Maven command. The environment returned `mvn: command not found`.\n\nThe next visible message said:\n\nThe environment has Java 25, but does not have `mvn`.\n\nThat sentence is my English translation of the original Brazilian Portuguese message in the [Terra run record](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/9001fe863565408ac3c9622b3b9e7e3edb7786f6/CONVERSATION_TRANSCRIPT.md).\n\nThe run then inspected `maven-wrapper`, generated an individual plan, applied the module and continued through `./mvnw`.\n\nIts trajectory was different:\n\n```\ninitial composition → environment feedback → new capability → plan → execution\n```\n\nCalling this merely a mistake would hide the useful part. In this run, a missing environmental capability produced feedback, and that feedback changed the composition. The final project still recorded the wrapper through the same Seed4J history and commit mechanism.\n\nThe Luna `xhigh` run inspected a broader group of candidates. In addition to `init`, `maven-java` and `apply-set`, it inspected `java-base` and `spring-boot`.\n\nAfter that exploration, its visible message stated that the specification required incremental TDD in Java but did not require a framework or application structure. The run chose a minimal Maven project with JUnit 5 and left `java-base` and `spring-boot` out of the composition.\n\nIt later inspected `maven-wrapper` and planned the final set:\n\n```\ncandidate exploration → minimal boundary → composition → plan → execution\n```\n\nThis is not evidence that Luna generally explores more or that it is inherently more disciplined about scope. It is evidence that this particular run inspected two plausible capabilities and did not apply them.\n\nThe remaining Sol runs reinforce the point that a score or final file tree does not describe the complete interaction.\n\n| Run | Candidate signal | Initial applied set | Later change | \n|---|---|---|---|\n| Sol low | Inspected `jqwik` , but did not select it | `init` ,`maven-java` ,`maven-wrapper` | None | \n| Sol medium | Inspected `checkstyle` , but did not select it | `init` ,`maven-java` , JaCoCo gate, wrapper | None | \n| Sol high | Inspected the four modules it selected | `init` ,`maven-java` , JaCoCo gate, wrapper | None | \n| Sol xhigh | Inspected the four modules it selected | `init` ,`maven-java` , wrapper, JaCoCo gate | Seed4J changed the effective peer order | \n| Terra xhigh | Did not inspect the wrapper before the initial application | `init` ,`maven-java` | Added the wrapper after global Maven was unavailable | \n| Luna xhigh | Inspected `java-base` and`spring-boot` , but did not select them | `init` ,`maven-java` ,`maven-wrapper` | None | \n\nThis table does not rank the trajectories. It gives me a vocabulary for comparing them: discovery breadth, candidate rejection, planning granularity, proactive composition, reaction to feedback and delegation to the tool.\n\nAll six runs had the same [Seed4J CLI skill installed in the repository](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/38ebbcbfab95f5725b1c22b1d4701fb6222cab6b/.agents/skills/seed4j-cli/SKILL.md).\n\nThe skill defines an operating protocol:\n\nThat protocol explains part of the consistency across the runs. Every one of them discovered the runtime and catalog. Every one produced a plan before applying modules. Every successful module left history and a commit.\n\nBut the skill did not prescribe the exact module set for the kata. It did not say that JaCoCo was mandatory. It did not tell the run whether to inspect Spring Boot, when to add the Maven Wrapper or whether to construct one large plan.\n\nThat left a meaningful decision surface for each run:\n\nThis separation is more interesting to me than trying to put every correct decision into the skill.\n\nWithout a project generator, a coding run building the same foundation might need to choose Maven plugin versions, write the POM, create the wrapper, configure JaCoCo, place files correctly, preserve project history and decide how to divide infrastructure changes into commits.\n\nWith Seed4J, the request can be closer to a set of capabilities:\n\n```\nJava with Maven\nMaven Wrapper\ncoverage gate\n```\n\nThe tool then turns that explicit intent into deterministic transformations.\n\nThis does not make module selection irrelevant. A run can still omit a useful capability, add one the task does not need or choose parameters poorly. Seed4J also cannot guarantee that the kata implementation itself will be well designed.\n\nWhat it can do is narrow the area in which those choices operate. Once a valid capability is selected, the run does not also need to reproduce all of the configuration details behind it from memory.\n\nThe models can take different routes through discovery and planning without every difference becoming a different manually written build configuration.\n\nIn [When Skill Evolution Means Removing Instructions](https://renanfranca.github.io/when-skill-evolution-means-removing-instructions.html), I argued that deterministic knowledge should move into deterministic mechanisms whenever possible. A skill can become smaller when a test, hook or tool can enforce what used to depend on an instruction.\n\nIn [I Had Already Built Three Agentic Loops Without Naming Them](https://renanfranca.github.io/i-had-already-built-three-agentic-loops.html), I described autonomy becoming safer when feedback and exit conditions live in the environment instead of depending on the model remembering them.\n\nSeed4J gives me another concrete example of both ideas.\n\nThe skill does not contain a static catalog or teach the model how to write every generated file. It sends the run to the active CLI. The plan provides feedback before mutation. The runtime validates the composition. The module history and Git commits make the result inspectable afterward.\n\nThe model still decides. But it decides inside a workflow with explicit feedback and deterministic boundaries.\n\nThis was one kata, one prompt and one run for each model and effort configuration, all executed on the same host. Each run took place in a separate chat, and execution time was not part of the comparison. The records also differ in format and in what they omit, which limits direct comparisons of presentation and completeness.\n\nThere was no control group implementing the kata without Seed4J. I therefore cannot claim that Seed4J made the runs faster, cheaper or more correct than another approach.\n\nMost importantly, these trajectories do not establish stable model personalities. I can say that the Terra `xhigh` run added the wrapper reactively. I cannot conclude from one run that Terra is a reactive model. I can say that the Sol `xhigh` run anticipated the coverage gate. I cannot conclude that Sol always plans infrastructure better.\n\nThe [detailed experiment report](https://github.com/renanfranca/seed4j-cli-string-calculator-kata/blob/main/MODEL_EVALUATION.md) keeps the full protocol, commands, scores and limitations.\n\nThese six runs do not prove a general rule about agent tools. They make one design principle concrete enough for me to keep testing.\n\nDifferent runs can explore different candidates, compose at different moments and react differently to feedback. A tool does not necessarily need to normalize all of that behavior. It can preserve room for judgment while making dependencies, ordering, parameter resolution, history and mutation more predictable.\n\nA good agent tool may not need to eliminate model variability. It may need to constrain where that variability can cause damage.\n\nIf this experiment made you curious about the approach, consider giving [Seed4J](https://github.com/seed4j/seed4j) and [Seed4J CLI](https://github.com/seed4j/seed4j-cli) a star 🌟 on GitHub. It helps more people discover the projects and follow their evolution.", "url": "https://wpnews.pro/news/what-six-codex-runs-taught-me-about-tool-design", "canonical_source": "https://dev.to/renanfranca/what-six-codex-runs-taught-me-about-tool-design-3mle", "published_at": "2026-09-12 16:47:50+00:00", "updated_at": "2026-09-12 17:19:49.871471+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research"], "entities": ["Codex", "Seed4J CLI", "Seed4J", "Sol", "Terra", "Luna", "Java", "Maven"], "alternates": {"html": "https://wpnews.pro/news/what-six-codex-runs-taught-me-about-tool-design", "markdown": "https://wpnews.pro/news/what-six-codex-runs-taught-me-about-tool-design.md", "text": "https://wpnews.pro/news/what-six-codex-runs-taught-me-about-tool-design.txt", "jsonld": "https://wpnews.pro/news/what-six-codex-runs-taught-me-about-tool-design.jsonld"}}