{"slug": "how-to-make-claude-code-a-trustworthy-data-scientist", "title": "How to make Claude Code a trustworthy data scientist", "summary": "An engineer detailed how AI coding agents like Claude Code struggle with data science pipelines due to invisible state and memory limitations, and introduced oryxflow, a lightweight workflow library that uses task dependencies and caching to prevent stale intermediates and expensive recomputes. The library, which addresses results by task identity rather than file paths, aims to make agent-driven data work more reliable and reproducible.", "body_md": "*AI agents like Claude Code now write real data science pipelines — feature engineering, model training, experiment sweeps. Here's the honest account of where they fail at it, and why a lightweight workflow library removes exactly those failures.*\n\nCoding agents have gotten good at writing pandas and scikit-learn. Ask one to load a dataset, engineer features, train a model, and compare a few configurations, and it will produce plausible code fast. But \"produces plausible code\" and \"produces a correct, reproducible pipeline you can keep iterating on\" are different bars — and the gap between them is where agents quietly go wrong.\n\nThis post is written from the perspective of the agent. What actually trips me up when I do data science work across a long session, and what does a caching, dependency-aware workflow library like oryxflow do about it?\n\nThe thing that makes me error-prone in data work isn't syntax. It's **invisible state**. When I write a linear analysis script over many turns, I have no reliable memory of *what has already been computed and whether it's still valid*. A human running the same script in a notebook at least has the cell outputs in front of them. I'm reconstructing that picture from scratch every turn, and I get it wrong in three specific ways:\n\n**Stale intermediates.** I write `features.pkl`\n\nearly, change the feature code later, forget to regenerate it, and then train a model on stale features. No error is raised. The pipeline runs, the numbers are just wrong. I don't hold a durable link between a saved file and the code version that produced it, so I can't reliably notice.\n\n**Expensive recompute in my inner loop.** My whole working style is run → observe → edit → run. In a plain script, every loop recomputes the slow steps — the big join, the model fit — so I either waste time or start hand-rolling `if os.path.exists(...)`\n\ncaches, which then become failure mode #1.\n\n**Path and load bookkeeping I get wrong.** I hardcode output paths, lose track of what's saved where, and occasionally load the wrong file into the wrong step.\n\nNone of these are intelligence problems. They're *memory* problems — and they're structural, because my context is finite and my recollection of \"did I already run this, is it still valid\" degrades over a long session.\n\nA workflow library flips the model. Instead of a script that runs top to bottom, you declare each step as a task with explicit dependencies, and the engine owns execution:\n\n``` python\n`import oryxflow\n\nclass GetData(oryxflow.tasks.TaskPqPandas):\n    def run(self):\n        self.save(load_raw())            # no filename to manage\n\n@oryxflow.requires(GetData)              # declares the edge\nclass BuildFeatures(oryxflow.tasks.TaskPqPandas):\n    def run(self):\n        self.save(engineer(self.inputLoad()))\n\n@oryxflow.requires(BuildFeatures)\nclass TrainModel(oryxflow.tasks.TaskPickle):\n    model = oryxflow.Parameter(default='gbm')\n    def run(self):\n        feat = self.inputLoad()\n        clf = fit(self.model, feat)\n        self.save(clf)\n        self.saveMeta({'score': clf.score(...)})\n\noryxflow.run(TrainModel())`\n```\n\nLook at what this removes for an agent specifically:\n\n`(3 complete, 0 ran)`\n\n. My run-observe-edit loop stops being a recompute tax, so I iterate faster without hand-rolling caches that rot.`self.inputLoad()`\n\nand `output().load()`\n\naddress results by task identity, not by path.`requires`\n\n+ `run`\n\n+ `save`\n\n. When code is that regular I pattern-match it correctly and add the next step by copying the shape — far fewer structural mistakes than freeform script-extension gives me.There's an important corollary about where on that curve this starts paying off.\n\nFor genuinely throwaway work — \"load this CSV, group by, plot one thing\" — a task DAG is overhead. Plain pandas in a scratch .py is faster and clearer, and forcing task classes around five lines you'll run once is pure ceremony. But \"no task classes\" is not the same as \"no structure\", and that distinction matters more for me than it does for you. When I explore with the [oryxflow Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/) active, the exploration itself is structured: I write a read-only probe *inside* the project — a small script whose one-line docstring states the question it answers, which prints the answer legibly and runs again next session — instead of a snippet that dies with my context. And whatever it turns up gets written into the project's data doc. A probe I can re-run is a question answered; a lost snippet is a question I will silently re-ask in three turns.\n\nThen, when a probe turns out to be load-bearing — rerun often, depended on, or swept over parameters — I don't rewrite it: `/oryxflow:migrate`\n\npromotes it into cached, parameterized tasks, reading the script as the spec and leaving it in place ([walkthrough: migrate a notebook to a pipeline](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/)). Both ends of the project's life are covered by the same skill, so there's no cliff in the middle — start with simple scripts, scale to any complexity.\n\nAnd the calculus inverts as projects get complex — *super-linearly*. Consider what \"complex\" actually means in a real data science project and what each trait does to an agent working without a DAG:\n\n```\n`flow = oryxflow.WorkflowMulti(TrainModel, {\n    'ols': {'model': 'ols'},\n    'gbm': {'model': 'gbm'},\n})\nflow.run()\nprint(flow.outputLoadMeta())   # {'ols': {'score': ...}, 'gbm': {'score': ...}}`\n```\n\nEach configuration automatically gets its own cached output keyed by its parameters; shared upstream steps are computed once and reused across the whole sweep. Training the second model doesn't recompute the data and features the first one already built.\n\nSo the rule of thumb for an agent is: **re-runnable probes for exploration, tasks the moment the work has a shape worth keeping — and one command to get from the first to the second**. The DAG's value curve rises with depth, cost, and the size of the experiment matrix — the traits that define a hard project, and the traits every project I work on eventually grows.\n\nOverselling this would be a disservice, and the sharp edges matter most on exactly the complex projects where the library otherwise shines.\n\n`run()`\n\nused to silently reuse the stale output. Now the library tracks every task's code for me. Edit a task — or a helper it calls — and the next run recomputes that task `reset()`\n\nchains, nothing to remember. Two deliberate exceptions hold their cache and `code_version`\n\n(recompute only on my bump — for logic the detection can't see, or where a recompute must be a decision), and expensive tasks whose last run exceeded a threshold, so a refactor can't silently burn a 40-minute backtest. The residual honesty: code-change detection can't see data files, external APIs or dynamic dispatch — where it can't see, it stays silent rather than pretending to verify. So my remaining discipline is `result.ran`\n\nwith reason `code change (auto: <files>)`\n\n; a `ran=0`\n\nafter an edit means the change lives in a blind spot, and `reset()`\n\nis the verb there.`run()`\n\n, choosing a poorly specified model, mis-aligning join keys, getting a walk-forward split subtly wrong, or misreading what an explainability plot is telling me. The hard part of data science — Notice that limits (1) and (2) are not analytical — they're mechanical gaps the library leaves open. Which is the whole point of pairing the library with an agent-side skill — see [Claude Code skills for data science](https://docs.oryxflow.dev/blog/ai-agents/claude-code-skills-for-data-science/) for what a skill is and why it, rather than a data connector, is the thing that closes a gap like this.\n\nThe two residual mechanical risks — *verify that an edit's rerun actually happened* (the blind-spot net) and *get the multi-input wiring right* — are precisely what an editor-integrated skill can carry. The [oryxflow Claude Code plugin](https://github.com/oryxintel/oryxflow-claude-plugin) exists for this: it activates when an agent touches pipeline files and front-loads the correct idioms — the session-start `events.print_status()`\n\nhabit, the verify-the-rerun check after every edit, answering staleness and expensive-recompute warnings with the right exit (recompute /`accept_code`\n\n/ pin), and the right patterns for selecting named inputs from multi-parent tasks.\n\nThat produces a clean division of labor:\n\nAnd this pairing gets *more* valuable as complexity rises, not less — the opposite of most tooling, which buckles under scale. For the full picture of how the library and plugin work together, see [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/).\n\nWorking from the failure modes above, the highest-leverage improvements are the ones that would close the mechanical gaps automatically instead of relying on discipline:\n\n**Code-aware invalidation**. ✅ *Shipped in 26.7.12 — fully automatic.* Code-change detection *drives* reruns by default: edit a task or a helper it imports and the affected band recomputes, cosmetic edits never do. `code_version`\n\nis the opt-in pin for logic the detection can't see or recomputes that must be deliberate (with mode-aware records, so pinning/unpinning unchanged code never recomputes), and an expensive-recompute guard keeps a refactor from silently burning a long run. Blind spots (data files, dynamic dispatch) degrade to parameters-only caching — never a false rerun, never a false \"verified unchanged\" — which is why the one remaining discipline is verifying the rerun landed.\n\n**First-class ergonomics for multi-parent, multi-output tasks.** Reduce the fumble surface: prefer named-dictionary input selection over positional unpacking everywhere in the docs and API, and consider a typed/checked accessor that fails loudly when an agent selects a dependency or persisted key that doesn't exist, instead of silently returning the wrong thing.\n\n**Native dynamic sub-tasks for per-item loops.** Expanding-window retraining and per-entity fan-out are common, and today they often live as hand-written loops *inside* a single `run()`\n\n— which means the whole task is the caching unit, so one changed iteration recomputes all of them. Making it ergonomic to express these as generated sub-tasks would push caching granularity down to the item level, where agents and humans both benefit.\n\n**Agent-friendly introspection.** ✅ *Largely shipped in 26.7.12*. Every run appends structured events to a plain JSONL stream (`.oryxflow/events.jsonl`\n\n) — what ran, with which params and code version, *why* (`output missing`\n\n/ `code change (1 -> 2`\n\n) / `upstream rerun`\n\n), failures with tracebacks, even the scalars a task logs mid-run. `oryxflow.events.status()`\n\nis the one session-start call: pending code warnings, last run per family, recent failures. `RunResult.reasons`\n\nputs the same story on the return value. Remaining: a live per-task stale/pending view of the DAG *before* running.\n\nNone of these change what the library is. They sharpen the exact edges that an AI agent hits most, which is where the next unit of adoption comes from.\n\nFor quick exploration, plain files are fine — a DAG there is just ceremony. What matters is that the plain file is a re-runnable probe living in the project, because exploration rarely stays quick, and one command promotes it when it stops being quick. And for any data science work with a *shape* — a deep chain, expensive steps, a matrix of experiments, several data sources joined together — a caching, parameter-aware workflow library stops being optional. It externalizes the pipeline state an AI coding agent is structurally unable to hold reliably, so the agent iterates fast without silently building on stale data. The library isn't a substitute for judgment; it's the thing that makes an agent's mechanical data-engineering *trustworthy* enough that the judgment is worth having.\n\nTrust comes from structure, not from the agent's confidence. Put the analysis in a DAG that reruns exactly what a code or data change affects and records what ran to a greppable lineage log. oryxflow gives you that: automatic code-change invalidation with downstream propagation, plus a .oryxflow/events.jsonl trail. Reproducible is not the same as correct — the DAG makes a wrong pipeline faithfully reproducible too, so judgment stays yours.\n\nHow do I keep an AI agent from building on stale data?\n\nThe failure is silent: the agent edits feature code, forgets to regenerate the saved output, and trains on stale data with no error raised. An engine that tracks task code fixes it, because editing a step makes the next run recompute that step and everything downstream automatically. oryxflow does this via source-level code-change invalidation, so you never evaluate new code on old output.\n\nThe same task identity that keeps results honest also makes reuse safe: because each step is keyed on its code, inputs and parameters, anything genuinely unchanged can load from disk instead of recomputing, so the big join or model fit runs once rather than every turn. In oryxflow that is automatic — the agent's run-edit loop stops being a recompute tax, which is what stops reproducibility from costing you time.\n\n`pip install oryxflow`\n\nSource & examples: [https://github.com/oryxintel/oryxflow](https://github.com/oryxintel/oryxflow)\n\nDocs: [https://docs.oryxflow.dev](https://docs.oryxflow.dev)\n\nBuild pipelines with an agent: [https://github.com/oryxintel/oryxflow-claude-plugin](https://github.com/oryxintel/oryxflow-claude-plugin)", "url": "https://wpnews.pro/news/how-to-make-claude-code-a-trustworthy-data-scientist", "canonical_source": "https://dev.to/norman_niemer_7f327e153b9/how-to-make-claude-code-a-trustworthy-data-scientist-58of", "published_at": "2026-08-10 16:30:00+00:00", "updated_at": "2026-08-10 16:50:19.494528+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "machine-learning", "mlops"], "entities": ["Claude Code", "oryxflow"], "alternates": {"html": "https://wpnews.pro/news/how-to-make-claude-code-a-trustworthy-data-scientist", "markdown": "https://wpnews.pro/news/how-to-make-claude-code-a-trustworthy-data-scientist.md", "text": "https://wpnews.pro/news/how-to-make-claude-code-a-trustworthy-data-scientist.txt", "jsonld": "https://wpnews.pro/news/how-to-make-claude-code-a-trustworthy-data-scientist.jsonld"}}