I'd like to share my experience managing LLMs on my project.
I started the project by asking ChatGPT to prototype a language design idea. A month and a half later, it evolved into a process with specifications, a conformance matrix, a detailed implementation plan, a set of executable examples, GitHub project cards, automated CI/CD with released artifacts, automated tests and instructions for coding agents. This is the story of this evolution.
The project is a prototype of an experimental functional language called Caret (you can try it on GitHub). I had been pondering some language design ideas for quite some time, and recently started prototyping the language using Codex. The way I interact with Codex has evolved over time, and will probably evolve more. The solution to AI drift turned out to be not better prompting, but surrounding the project with specific artifacts that tracked its progress and kept the consistency of design in check. But let's look at it as the project evolved.
It's quite impressive how well the AI can understand your ideas, but it's never guaranteed to understand them exactly as you meant them. It can be extremely verbose. The amount of text it produces can get tiresome, but it goes into great detail just to paraphrase what you just told it.
And when it comes to documentation, this verbosity is pretty much what you want. I don't like to write so much text myself. What it cranks out is not perfect for a human reader, but it's detailed and precise enough as an instruction for implementation.
I first bounced the language ideas with ChatGPT. It apparently understands the ideas in compact form when you're being precise enough, but the solutions it offers in response are quite surface-level and situational. It can point at some flaws, ambiguities or underspecified areas in your design pretty well though. So it provokes more thought, more ideas, and it provokes you to continue the design, which is useful when you work alone.
After a number of ideas became defined enough, I asked the AI if it could provide me with a prototype of the language, and instead of answering yes or no, it just produced a tiny Java project with a parser, an AST, and a few more classes. I asked it to add a language spec, and it assembled a LANGUAGE.md with a very brief description of what is already in the prototype and what is not yet implemented.
That was quick, so I went on to install a Codex plugin in the IDE, and started instructing it to code this prototype further.
ChatGPT told me I can instruct the Codex agents on how to do certain things, and what are the rules on this project, and I added an initial AGENTS.md with such instructions to the repository.
In the first iteration of my process, the specification and implementation were separate flows.
The documentation flow looked like this:
I discuss my idea with ChatGPT (until the design is clear, and I see that ChatGPT understands the idea well)
↓
I ask ChatGPT to write a piece of spec (and check that it matches my design)
I manually add it to the language specification
I added a few features to the specification this way, and it became clear that Codex wouldn't implement all of them in one go. The scope of the project was already too big.
The implementation flow looked like this:
I ask Codex to propose the next step of implementation
Codex implements the proposed step
From the first iteration of the process, it was clear that the result of such development is not transparent at all. How do I even use this prototype? I introduced a REPL loop and a rudimentary test suite in Caret. With the REPL, you could now input Caret code and get it interpreted and executed immediately. With the test suite, you could write unit tests in Caret that could compare the function results with expected values.
Of course, the REPL didn't work right away. It took a couple of iterations to get it to work inside the IDE's terminal.
At this point I was regularly asking Codex to update the language specification and README.md to reflect the current development status. In this sense, Codex was now the owner of documentation. Just pasting in the sections written by ChatGPT made less sense now, because ChatGPT was unaware of what Codex changed.
So I switched to a second iteration of my documentation flow:
I discuss the idea with ChatGPT
I ask ChatGPT to write a prompt for Codex
I feed the prompt to Codex
Codex adds a section to the spec
Asking Codex for each next implementation step proposal became a bit repetitive, so I asked Codex to break the whole language spec into implementation steps and document the resulting plan. Codex added a PLAN.md. It broke the whole project into 10 implementation phases.
Soon I concluded that since both ChatGPT and Codex use similar reasoning and are based on closely related models, asking ChatGPT to make a prompt for Codex is actually redundant. I could directly ask Codex for the changes instead. The third iteration of the document flow was both shorter and more detailed:
I tell my idea to Codex and ask it to add it to the spec
Codex asks me specific architectural questions that need clarification
Codex updates the language spec
I ask Codex to update all the specs, including the plan and readme
The implementation flow at this point became more defined:
In plan mode, I ask Codex to propose the implementation of the next step according to the documented plan
I correct the plan or accept it
Codex implements the planned step
I commit the changes
I ask Codex to run a code review, and it finds some issues
I ask Codex to fix the findings, and then I commit the fix
Soon, phase 0 was complete, and phases 1 and 2 were in progress. I wanted to see which specific steps of the plan were completed, which were ready for implementation, and which had unresolved dependencies, and Codex added the conformance matrix CONFORMANCE.md with every feature listed in a table with an ID and a status.
After adding some more feature designs, the plan now had 13 phases. I haven't added more big ideas to the language since then – I want to get the prototype of the existing ones completed first. 13 phases is already a big project.
Then I asked to create example Caret programs for every implemented feature, and for every error that the interpreter produces. After each implementation step, Codex had to run the whole test suite, including the Java unit tests and all the Caret programs for all the implemented language features and errors.
I looked at the test programs periodically, and if the language drifted from my design, I asked Codex to fix that, first in the specification, then in the code and in the tests.
The verbose sections of the language specification concatenated into one long sausage became hard to read and to follow, so I asked Codex to break the spec into separate files. It produced 14 files, roughly in accordance with the project phases. Much more readable and easier to maintain.
Then there was a relatively short architectural decision marathon in order to resolve all open questions for phase 2.
Looking at the conformance matrix after each change is cumbersome, so I added a GitHub project with cards to better visualize the progress. I asked Codex to create a project card for every planned implementation step remaining in phase 2.
I went through a cycle:
I ask Codex to plan, then implement the next step
I ask Codex to run a code review
I ask Codex to fix all the findings
I ask Codex to update the specs (so the plan, the conformance matrix, and the language spec match the actual implementation)
I ask Codex to update the test suite (so all new functionality and errors are covered)
I ask Codex to run all the tests
Rinse and repeat ↟
The GitHub project allowed me to list new ideas without having to immediately add them to the project scope. I listed the ideas with just enough detail to make them clear. Then I asked Codex to add details to the cards, so I could inspect how well it understood the idea. In most cases, it understood the ideas very well. In some cases, I had to edit some details or ask Codex to change what it got wrong.
I noticed that I'm asking the same things using detailed prompts over and over again, so I asked Codex to add the repeated actions to AGENTS.md with specific instructions on how to perform them. The actions were:
| Instruction | Meaning |
|---|---|
next step |
Plan and implement the next card in the ToDo column of the GitHub project, then run the full test suite, including the Java unit tests and Caret test programs |
code review |
Run a thorough code review on the changes made since the last merge to main branch |
new PR |
Create a new pull request with all the commits since the last merge to main . Also update the version number for the artifact that will be published on the next merge tomain . |
spec sync |
Make sure there are no discrepancies in the specs, the plan and the conformance matrix are up to date with the current implementation. |
test coverage |
Update the test suite, both Java and Caret, to match the actual implemented features and the current specs. Add the missing tests, so all the features and errors are covered. |
card details |
Take all the cards in the Underspecified column of the GitHub project and add details to them. After that, move them to theToDo column |
Now, when I gave these short instructions to Codex, it followed the longer instructions specified in the AGENTS.md
Soon, I noticed that running code reviews after each implementation step drains the tokens very fast, so I asked Codex to run the next step instruction repeatedly, until the current phase is complete or there is a question that requires my intervention.
It still burns through the token allowance a few times during a phase implementation as well, but the progress became noticeably faster.
At the end of the phase, I asked Codex to run the code review a couple of times in a row, sync the specs, and update the test coverage. I heard from other developers that LLMs perform better code reviews in a new chat. When implementing, a chat tends to accumulate context that kind of clouds its judgment and prevents it from seeing all the issues. This is similar to actual developers: you want somebody else to review the code you wrote. So I adopted this technique: for the code review, I opened a new chat every time.
In this manner, I ran the already established cycle through phase 3, and now it's complete. You can savor its fruit in the repository.
Phase 4 started with a huge architectural decision marathon, but that's another story that I may tell some other time.
So the main takeaway for me is this: the remedy to AI drift is not better prompts, but better process transparency. My current process, with a set of specifications, with a plan, and a conformance matrix, with a broad set of tests for each feature and error, and with regular code reviews, lets me be sure enough that the language implementation doesn't drift away from my design, as long as I keep an eye on the set of test Caret programs. When there are several observable artifacts that can be cross-checked against each other, contradictions become noticeable, and the drift is more likely to be stopped before it propagates into the documentation and implementation.
I hope my experience is useful to those of you who use or plan to use agents in development. Please share your thoughts, feedback, or how your experience is different.