People of all ages and experience levels who know a little bit about LLMs and want to do data analysis without coding are turning to agentic tools. These tools are numerous by now and include tools from model builders such as Anthropic (Claude Code) and OpenAI (GPT Codex), as well as many other open-source alternatives (OpenCode, etc.).
What is an agent in this context? An agent is usually a language model most of the time with vision capability (can see images and convert them to text). It can inspect data, install libraries, write scripts, troubleshoot failures, and produce an analysis. This can all be done using only the user’s instructions. However, every request to the agent will result in a new batch of text generation and potential web searches, and a new batch of code will be created and executed. This new code will not be identical to the previous runs even if you provided the same task. This stochasticity is inherent in large language models, and we have to live with it. But this hurts reproducibility, reusability, and racks up the costs. For example, if you want to perform a routine analysis of transcriptome data, there is not much variation in the code you will use. You or your agent don’t have to reinvent the wheel every time you want to analyze data. [SPOILER: if you don’t want to read further here is how to make agents more reproducible: https://github.com/rolv-io/recipe]
Luckily, there are mechanisms to provide pre-determined instructions to the agents so that they don’t have to reinvent things every time we give a task that requires specialized knowledge or skills.
SKILL.md files and their accompanying scripts, references, and other files are the prevailing way to provide these instructions. They allow us to package instructions and working code so that an agent does not need to reinvent an entire workflow every time.
However, reusable instructions provided with the skills approach are not the same thing as a reproducible and continuously improving workflow. Here are the current problems with the agentic coding and skills approach.
A good skill can tell an agent how a task should be performed and provide scripts that implement established parts of the process. This already saves tokens and reduces reinvention.
However, whenever the skill leaves a decision open, the agent has to reason about it again. It may choose a different order of operations, a different library, a different intermediate representation, or a different response to an error.
A SKILL.md file can say:
Load the data, validate it, transform it, run the analysis, and generate the output.
It can also provide scripts for doing each of these things.
But there is still a difference between describing a workflow and explicitly representing its execution graph.
This becomes particularly important when something fails. An agent trying to solve a problem may modify several scripts, change a dependency, alter an earlier transformation, or rewrite an important part of the workflow simply because doing so seems plausible. I have experienced this many times, even with frontier models. The agents were trying to fix problems at the wrong steps.
Skills can specify dependencies and include environment files, but they do not necessarily have to. A skill may work because the required libraries, tools, or system dependencies happen to already be available in the agent’s environment. Dependency management is critical for reproducibility and should not be optional.
The initial skill may describe the correct workflow. However, the more interesting knowledge often appears later. Perhaps a particular CSV export occasionally changes its encoding. An API starts returning a slightly different schema. A package version introduces a bug. Packages have different installation procedures in windows vs linux. A model produces malformed output under a particular condition. An apparently harmless transformation turns out to break one class of datasets.
Agents solve these problems during execution, but those discoveries should not disappear with the session. The problems they encounter are usually not one-off problems.
Repeated successful execution can create false confidence. A workflow might run without errors while quietly producing incorrect results. What is worse, it may be creating imaginary datasets to make certain parts work. An expert who is lazy not to check the details or a novice who has no idea how thigns actually work will miss these problems easily. No mechanism is forcing skill builders to provide tests with their skills.
A shared skill lets many agents start from the same knowledge.
But there is another step: allowing the shared workflow to improve from what those agents discover while using it. This is a completely missed opportunity. Agents are again solving problems over and over while using faulty skills, and their learnings are never transfered to other agents. There is no hive-mind, although maybe there should be.
Once a skill contains scripts, tools, dependencies, instructions, and contributions from many agents or people, it becomes executable infrastructure.
That creates a supply-chain problem. The skills can include suspicious dependencies, malicious commands, unexpected network access, dangerous tools, credential access, prompt-injection-like instructions, or changes that unexpectedly expand what the skill is allowed to do previously. There is no central check on what security risk each skill can pose.
As workflows evolve, some steps, dependency pins, or troubleshooting rules may begin to look unnecessary. But they may exist because of failures discovered in previous executions.
If the reason for a constraint is not preserved, future agents may remove or change it without understanding why it was introduced, causing previously solved problems to reappear.
The idea is to extend the current skill approach with a more structured artifact: a recipe.md file and accompanying files that define not only how a task should be approached, but how an established workflow should be executed, maintained, tested, and improved over time. In hindsight, we can call this skill files or skill.md as well, but the divergence from a regular skill file set is substantial and we don’t want people to think this is same as the skill files.
A recipe would add several things that are not mandatory in today’s skill approach.
The recipe should explicitly define the steps of the workflow, their order, their inputs and outputs, and the conditions under which different branches can be taken.
Instead of repeatedly asking the agent to decide how to perform an established task, the recipe defines the default execution path:
Step A → validate → Step B → Step C → validate output
The agent can still reason where judgment is required, but it should not redesign parts of the workflow that have already been established.
This should reduce unnecessary token use and make repeated executions more consistent. This idea is of course very similar to workflow managers such as snakemake, nextflow, apache airflow etc. The fundamental difference is that AI runs, troubleshoots and intreprent the steps. The artifacts are human readable rather than a programming language.
When something fails, agents should not have unlimited freedom to modify the entire workflow.
Each step or component can carry a troubleshooting policy such as:
This allows knowledge from previous executions to constrain future troubleshooting.
A recipe also specifies via the workflow graph which upstream and downstream steps depend on the component being modified, so the agent can understand the likely consequences of a change before making it.
Preserve why important rules exist
Recipes should record not only important constraints but also a short explanation of why they exist. If a step is marked core, the recipe should say why. If a validation step was introduced because of a previous failure, that relationship should remain visible.
The detailed evidence can remain in the troubleshooting record (which will be explained below), while the current recipe contains a short rationale and a reference to the relevant incident.
For example:
step: normalize_columnstroubleshooting_policy: corereason: Previous changes to this step caused downstream schema mismatches.
This prevents future agents from removing safeguards simply because they no longer know why those safeguards were introduced.
Unlike skills, where dependency information may or may not be included, a recipe should not be considered complete unless its dependencies are explicitly declared.
This can include libraries and versions, command-line tools, external services, APIs, models, system dependencies, and dependencies between individual workflow steps.
The goal is not only to document what is needed, but to make it possible to reproduce a known-working environment and quickly identify dependency changes when something stops working.
Every execution that encounters a meaningful problem should be able to add structured information to the recipe’s troubleshooting history.
That record might include:
Future agents should consult this record before beginning their own troubleshooting.
The aim is simple: if one agent has already spent time diagnosing and solving a recurring problem, another agent should not have to rediscover the same solution from scratch.
A recipe that executes successfully is not necessarily producing the correct result.
Recipes should therefore be able to include automated tests, including small toy datasets for which the expected outputs are already known.
After an agent changes a script, dependency, or workflow step, the recipe can be run against these examples to verify that previously correct behavior has not been broken.
This also provides a safer way to incorporate improvements discovered during troubleshooting.
A shared recipe should improve from the experience of the agents using it.
After an execution, an agent can submit what it learned: a newly discovered failure mode, a better troubleshooting procedure, an important dependency constraint, or an improvement to an existing step.
These contributions should not necessarily be incorporated automatically. They can first be tested, compared with the existing recipe, scanned for security issues, and, when necessary, approved by a human.
Once accepted, however, the improvement becomes available to every subsequent execution of the recipe.
The important difference is that knowledge flows back into the shared workflow rather than remaining inside an individual agent session.
Recipes and their accompanying files are executable infrastructure and should be treated accordingly.
A central repository can scan recipes for malicious or suspicious behavior before they are distributed or updated. This could include unexpected network access, suspicious package dependencies, credential access, dangerous shell commands, malicious tools, prompt-injection-like instructions, or changes that substantially expand the permissions of an existing recipe.
The same checks can be applied whenever an agent proposes an update to a shared recipe.
Recipes should be versioned so that changes to workflow logic, dependencies, troubleshooting rules, and tests are visible and reversible.
The initial recipe should be reviewed by a human, and important changes can go through the same process. The purpose of human oversight is to approve the reusable procedure and significant changes to it, rather than supervise every individual run.
A version change can then capture both what changed and why, while the troubleshooting record preserves the detailed history behind that change.
The resulting process would look something like this:
In this model, recipe.md adds a stricter layer for workflows that are already established and where determinism, reproducibility, testing, and accumulated operational knowledge matter.
The goal is not to eliminate agent reasoning. It is to avoid spending agent reasoning on decisions that have already been made and problems that have already been solved. Most of these ideas are implemented in our app rolv.io (a local-first agentic data analysis app), and the formal definition is here: https://github.com/rolv-io/recipe.
Towards Deterministic and Reproducible Agentic Data Analysis was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.