Beyond Just Jupyter Notebook: How to Ship AI Code That Survives Production

A University of Washington study of 1.16 million Jupyter notebooks from 264,000 GitHub repositories found that only 24% of 863,878 executed without errors and only 4% produced identical results on a second run, highlighting the format's unreliability for production AI/ML systems. The article argues that notebooks mix state, logic, and presentation, leading to hidden state, non-deterministic execution, lack of testability, and noisy version control, and recommends shifting to modular Python with explicit data flow, automated tests, and CI/CD pipelines.

Your model worked beautifully in your notebook. Then production broke it. You handed the notebook to a colleague and it crashed on cell 3, because cell 12 had to run first. You reran it on fresh data and got different numbers from the same code. The version that produced the numbers everyone is quoting lives in your kernel’s memory, not in any file, and when the stakeholder asks you to regenerate them next week, you cannot. Nothing about the model changed. The notebook just does not survive contact with anyone but you. Jupyter notebooks are the gateway drug of data science. They are perfect for exploration: you can slice data, eyeball plots, iterate on ideas, and keep the narrative right next to the code, they are also a great tool to use if you just learn data science. But the moment you want an AI/ML system to run reliably, at scale, without you in the room, the notebook stops being a tool and starts being a liability. This is not a matter of opinion. Researchers at the University of Washington studied 1.16 million notebooks from 264,000 GitHub repositories and attempted to execute 863,878 of them from scratch. Only 24% ran without errors. Only 4% produced the same results on a second run. If your pipeline exists only as a notebook, you are almost certainly in the 96%, whether you know it or not. The problems are not cosmetic. They are structural, and they all come from the same root: a notebook is one file that mixes state, logic, and presentation, and it pretends that all three are the same thing. Cells execute in any order, and the file cannot stop you. A notebook can look completely correct because all its outputs are visible and plausible, while being broken if run top to bottom. A classic pattern: cell 5 deletes a column, cell 3 uses that column, and the author ran them in the order 3–4–5. The cached output in cell 3 still shows a valid chart, so the notebook displays fine. Anyone running cells in order 1–2–3 hits a crash on cell 3, or worse, silently different results. Hidden state lives between cells. The Python kernel remembers variables from cells you deleted hours ago. A notebook can depend on a variable that no longer exists in any visible cell. You cannot inspect this state from the file; you can only suspect it when the notebook behaves differently on another machine. The testing ecosystem does not apply. You cannot run pytest on a notebook. The entire discipline of unit tests, the mechanism that catches regressions before they reach users, simply does not exist in this format. In the GitHub study, only 1.54% of Python notebooks even imported a known test module. Version control becomes noise. Git stores a notebook as raw JSON, including outputs: base64-encoded plots, HTML tables, execution timestamps. Rerun the notebook and the diff explodes with thousands of changed lines that contain zero meaningful code changes. Code review on notebooks is either skipped or performed on a wall of JSON. Every one of these costs compounds when a second person enters the picture. One author’s notebook is an instrument. Two authors make it a collaboration, and the file format was never designed for collaboration. Production-grade AI engineering is the practice of building machine learning systems as engineered software: modular Python with explicit data flow, deterministic execution, automated tests, clean version control, and CI/CD pipelines that ship without the author’s laptop in the loop. The transition is not about the tool you type into. It is about the properties that tool gives you. A notebook gives you interactivity and hides everything else. A script-based project gives you the reverse: less instant feedback, and in exchange, testability, reviewability, and repeatability. None of these rows is about talent. They are about properties of the format. The best data scientist in the world cannot write a reliable test suite inside a notebook, cannot give a reviewer a readable diff, and cannot run a notebook in a headless CI runner. The format decides these battles before you start. Production code lives in .py files for a reason. Functions and modules create clear boundaries: each unit can be tested in isolation, reasoned about on its own, and composed into a pipeline where data flows in one direction. The notebook’s greatest weakness is exactly what modules fix, because a function cannot accidentally depend on a cell you deleted an hour ago. Its inputs are its inputs. There is a simple rule of thumb used by teams that make this transition well: if a function grows past ten lines, or you find yourself copying the same logic between cells, it belongs in a module. This is not an aesthetic preference. It is the smallest possible step that makes your code testable, importable, and diffable. The first promotion step is deceptively small. Move the pipeline’s steps into functions inside a single .py file next to the notebook, then run the pipeline end to end from that file. Keep the notebook for exploration, and let the script become the source of truth. """modular pipeline.pyThe core refactoring demo: notebook-style code vs modular productioncode. The BAD pattern is shown in the docstring below. The functionsthat follow are the GOOD pattern: explicit inputs, explicit outputs,logging instead of prints, and no reliance on hidden state.BAD notebook style : implicit state and ordering, debugging by print. df = pd.read csv "customer features.csv" churned = df "churned" .map {"yes": 1, "no": 0} cell 4 df = df df "monthly spend" .notna cell 9, depends on cell 4 print df.head squinting at output model.fit X, y works here, breaks in productionGOOD modular : every step is a pure function with explicit inputs andoutputs, so each unit can be tested in isolation and composed safely."""import loggingimport numpy as npimport pandas as pdfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.metrics import roc auc scorefrom sklearn.model selection import train test splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScalerlogging.basicConfig level=logging.INFO, format="% levelname s % name s: % message s" logger = logging.getLogger "pipeline" FEATURES = "tenure months", "monthly spend", "support tickets", "login frequency", "days since last login", "plan tier", def load data n customers: int = 2000, seed: int = 42 - pd.DataFrame: """Generate a synthetic customer table with churn labels. Also injects a few missing values so the cleaning step has something real to do. """ rng = np.random.default rng seed churned = rng.random n customers < 0.2 df = pd.DataFrame { "tenure months": np.where churned, rng.integers 1, 18, n customers , rng.integers 6, 48, n customers , "monthly spend": np.where churned, rng.normal 45, 15, n customers , rng.normal 70, 20, n customers , "support tickets": np.where churned, rng.poisson 3.5, n customers , rng.poisson 1.0, n customers , "login frequency": np.where churned, rng.normal 1.5, 1, n customers , rng.normal 5, 2, n customers , "days since last login": np.where churned, rng.integers 10, 45, n customers , rng.integers 0, 14, n customers , "plan tier": rng.choice "basic", "pro", "enterprise" , n customers, p= 0.5, 0.35, 0.15 , "churned": churned.astype int , } df.loc rng.random len df < 0.02, "monthly spend" = np.nan return dfdef clean data df: pd.DataFrame - pd.DataFrame: """Drop rows with missing values and coerce the target dtype.""" out = df.dropna .copy out "churned" = out "churned" .astype int return outdef engineer features df: pd.DataFrame - pd.DataFrame: """Add a composite engagement feature derived from existing ones.""" out = df.copy out "engagement score" = out "login frequency" 0.6 - out "days since last login" 0.05 return outdef prepare xy df: pd.DataFrame - tuple pd.DataFrame, pd.Series : """Build the model matrix and target from a cleaned, engineered frame.""" X = pd.get dummies df FEATURES , columns= "plan tier" return X, df "churned" def train model df: pd.DataFrame, seed: int = 42 - tuple Pipeline, float : """Train a gradient boosting pipeline and return model, test AUC .""" X, y = prepare xy df X train, X test, y train, y test = train test split X, y, test size=0.2, stratify=y, random state=seed pipeline = Pipeline "scaler", StandardScaler , "clf", GradientBoostingClassifier n estimators=200, max depth=4, learning rate=0.05, random state=seed , pipeline.fit X train, y train auc = roc auc score y test, pipeline.predict proba X test :, 1 logger.info "Test ROC-AUC: %.4f", auc return pipeline, aucdef main n customers: int = 2000, seed: int = 42 - float: """Run the full pipeline end to end. Deterministic given the seed.""" df = clean data engineer features load data n customers, seed , auc = train model df, seed logger.info "Pipeline finished: AUC %.4f on %d customers", auc, len df return aucif name == " main ": main The snippet shows both patterns side by side in the comments: the notebook-style version, which leans on global variables and print statements and dies the moment you change the order of operations, and the modular version, where every step is a function with explicit inputs, explicit outputs, and logging instead of prints. The modular version runs the same way on your laptop and on a server, because it never depends on state you forgot to define. Notebooks trap you in a debugging loop that has no exit: re-run the cell, print a variable, squint at the output, edit, re-run again. Every iteration takes seconds of execution plus minutes of attention, and the print statements you scatter to debug today become permanent clutter that drowns the actual output tomorrow. A proper IDE changes the game. You set a breakpoint on the line where the data is misbehaving, and execution pauses there. You inspect the variables in scope at that exact moment, watch expressions change as you step line by line, and read the call stack to see how the program got here. Bugs that take an hour of print-based archaeology in a notebook get found in minutes, because you are looking at the actual state of the program instead of guessing what it might be. The call stack deserves special attention in AI code. Machine learning bugs rarely break where they are born: a wrong data shape flows through five functions and crashes at the model fit, or worse, silently trains a model on misaligned labels. The stack trace tells you where it crashed. Stepping through the debugger tells you where it went wrong, and those are different questions. When you move to scripts, also move from print to logging. Logging survives in containers, goes to files, and carries timestamps and levels. Print statements scroll past in a production log and might as well not exist. This is a habit that only becomes available once you leave the notebook, and it is one of the first upgrades you will feel. Show a reviewer a notebook diff and watch their eyes glaze over. A single rerun of a notebook produces a diff of thousands of lines: base64-encoded plots, HTML tables, execution counts, metadata. The actual code change, the one meaningful line, is buried somewhere in the middle of a JSON blob. Review becomes impossible, so reviews stop happening, and that is how broken code reaches production. Plain Python files diff the way diffing was meant to work. One function changed, one line in the diff, one focused conversation in the pull request. Merge conflicts become solvable because the conflict is about code, not about which cell’s output blob wins. The moment you convert a project to scripts, version control stops being a chore and becomes an actual tool again. If you cannot leave notebooks behind entirely, and many teams cannot, there are guardrails that shrink the damage. nbstripout installs a Git filter that removes cell outputs before they are committed, so your diffs contain code, not base64. A companion CI action runs it in verify mode and fails the build if someone pushes an unstripped notebook. jupytext pairs a notebook with a plain text .py file that Git tracks, keeping the notebook for interactive work and the text file for review. nbdime renders notebook diffs visually when you genuinely must review notebook changes. And the cheapest discipline of all: restart the kernel and run every cell top to bottom before you commit, then commit only if it survives. A result that does not survive a fresh kernel is not a result. It is an artifact of your session. You cannot run a notebook in a CI runner. Not repeatably, not headlessly, not with the deterministic fresh-kernel execution that automated testing demands. This single constraint is the hard line between notebook projects and production systems: if your pipeline cannot run in a blank environment on a schedule, it is not deployable, it is merely present. CI/CD is what scripts unlock. With a script-based project, every pull request can trigger the same sequence: check out the code, install pinned dependencies, run the linter, run the test suite, execute a smoke run of the pipeline on small data, and build the container that will serve the model. Regressions surface in minutes instead of months, and deployments stop requiring a human who knows the secret incantation. The workflow file below is the standard shape of that promise: a test job that runs on every push and pull request, installs the project, lints it, runs the test suite, executes the pipeline entry point as a smoke test, and uploads the trained artifact. Let’s check all 3 of them. """test pipeline.pyUnit tests for modular pipeline. This is the file the CI runnerexecutes on every pull request: small, fast, deterministic teststhat fail loudly when a refactor breaks behavior.Run with: pytest"""import numpy as npimport pytestfrom modular pipeline import clean data, engineer features, load data, train model, @pytest.fixturedef raw df : return load data n customers=300, seed=7 def test clean data removes missing values raw df : cleaned = clean data raw df assert cleaned "monthly spend" .isna .sum == 0 assert len cleaned <= len raw df def test clean data coerces target to integer raw df : cleaned = clean data raw df assert np.issubdtype cleaned "churned" .dtype, np.integer assert set cleaned "churned" .unique <= {0, 1}def test engineer features adds engagement score raw df : engineered = engineer features clean data raw df assert "engagement score" in engineered.columns assert engineered "engagement score" .notna .all def test engineer features is deterministic raw df : cleaned = clean data raw df first = engineer features cleaned second = engineer features cleaned assert first "engagement score" .equals second "engagement score" def test train model returns sane auc raw df : cleaned = engineer features clean data raw df pipeline, auc = train model cleaned, seed=7 assert 0.5 < auc <= 1.0 assert pipeline is not None """cli pipeline.pyThe single entry point for the modular pipeline. This is what CI andproduction both run: no notebook, no hidden state, every parameterexplicit on the command line.Usage examples: python cli pipeline.py --n-customers 2000 --model-path model.joblib python cli pipeline.py --evaluate --model-path model.joblib"""import argparseimport loggingimport osimport joblibfrom sklearn.metrics import roc auc scorefrom modular pipeline import clean data, engineer features, load data, prepare xy, train model, logging.basicConfig level=logging.INFO, format="% levelname s % name s: % message s" logger = logging.getLogger "cli" def parse args - argparse.Namespace: parser = argparse.ArgumentParser description="Train or evaluate the churn model." parser.add argument "--n-customers", type=int, default=2000, help="Size of the synthetic customer table" parser.add argument "--seed", type=int, default=42, help="Random seed for deterministic runs" parser.add argument "--model-path", type=str, default="model.joblib", help="Where to save or load the model" parser.add argument "--evaluate", action="store true", help="Load the saved model and score it on fresh data" return parser.parse args def evaluate model model, df: "pd.DataFrame" - float: """Score a saved model on a fresh, fully prepared frame.""" X, y = prepare xy df return roc auc score y, model.predict proba X :, 1 def main - None: args = parse args if args.evaluate: if not os.path.exists args.model path : raise SystemExit f"Model not found at {args.model path}. Train it first." model = joblib.load args.model path df = engineer features clean data load data args.n customers, args.seed auc = evaluate model model, df logger.info "Evaluated AUC on fresh data: %.4f", auc return df = engineer features clean data load data args.n customers, args.seed model, auc = train model df, args.seed joblib.dump model, args.model path logger.info "Saved model to %s AUC %.4f ", args.model path, auc if name == " main ": main ci workflow.yml GitHub Actions workflow for the modular ML pipeline. Every push and pull request is linted and tested; every merge to main builds the model and stores it as a downloadable artifact. This is the file that turns "it works on my laptop" into "it works in a blank environment, automatically".name: ml-pipeline-cion: push: branches: main pull request:jobs: test: runs-on: ubuntu-latest steps: - name: Check out the repository uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt pip install pytest ruff - name: Lint run: ruff check . - name: Run the test suite run: pytest --tb=short -q - name: Smoke run of the pipeline run: python cli pipeline.py --n-customers 200 --model-path /tmp/model.joblib build: needs: test runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' steps: - name: Check out the repository uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Train the model run: python cli pipeline.py --n-customers 5000 --model-path model.joblib - name: Upload the trained model uses: actions/upload-artifact@v4 with: name: trained-model path: model.joblib The test file is what the CI runner executes: unit tests against the modular pipeline, each one small enough to run in milliseconds. The CLI file is what CI and production share: a single entry point that trains or evaluates the pipeline with explicit arguments, no notebook state required. The workflow file wires them together: every pull request gets linted and tested, every merge to the main branch builds and stores the artifact. You cannot do any of this with a notebook, not because of a missing feature, but because the format lacks the properties the automation depends on. None of this means you should abandon notebooks. Exploration is real work, and a notebook is the best tool for it ever built. The skill is knowing when to promote an experiment into an engineered system, and the moment is usually easy to spot: the first time somebody other than you needs the result, or the first time someone asks “can we automate this?” When the answer to that question is yes, the notebook has served its exploratory purpose. The promotion itself follows a reliable sequence: There is a modern version of this lesson that AI engineers in particular should hear: prompts are code. A prompt that lives in cell 34 of an untracked notebook is a prompt that will silently drift from the one your service actually uses, because the notebook and the service cannot both be the source of truth. Move prompts into modules, where they become diffable, greppable, importable, and testable, exactly like the rest of your pipeline. The same discipline that saves your model code saves your prompt engineering. The transition from notebook to production is not a rejection of exploration. It is a boundary: notebooks for thinking, scripts for shipping. The organizations that deploy AI reliably did not hire smarter people. They made their code reviewable, their execution deterministic, and their deployment automated, and every one of those properties required leaving the .ipynb behind for anything that would run more than once. Ship code that outlasts your laptop. Keep the notebook for the exploration that deserves it, then promote the work into modules, tests, and pipelines, and let the machine do the running. Here are several key takeaways from this article: Thank you for reading this article I hope you found it helpful. If you have any questions or feedback, please feel free to reach out to me. Beyond Just Jupyter Notebook: How to Ship AI Code That Survives Production https://pub.towardsai.net/beyond-just-jupyter-notebook-how-to-ship-ai-code-that-survives-production-19a2a87ecdbb was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.