Your model worked beautifully in your notebook. Then production broke it.
You handed the notebook to a colleague and it crashed on cell 3, because cell 12 had to run first. You reran it on fresh data and got different numbers from the same code. The version that produced the numbers everyone is quoting lives in your kernel’s memory, not in any file, and when the stakeholder asks you to regenerate them next week, you cannot. Nothing about the model changed. The notebook just does not survive contact with anyone but you.
Jupyter notebooks are the gateway drug of data science. They are perfect for exploration: you can slice data, eyeball plots, iterate on ideas, and keep the narrative right next to the code, they are also a great tool to use if you just learn data science. But the moment you want an AI/ML system to run reliably, at scale, without you in the room, the notebook stops being a tool and starts being a liability.
This is not a matter of opinion. Researchers at the University of Washington studied 1.16 million notebooks from 264,000 GitHub repositories and attempted to execute 863,878 of them from scratch. Only 24% ran without errors. Only 4% produced the same results on a second run. If your pipeline exists only as a notebook, you are almost certainly in the 96%, whether you know it or not.
The problems are not cosmetic. They are structural, and they all come from the same root: a notebook is one file that mixes state, logic, and presentation, and it pretends that all three are the same thing.
Cells execute in any order, and the file cannot stop you. A notebook can look completely correct because all its outputs are visible and plausible, while being broken if run top to bottom. A classic pattern: cell 5 deletes a column, cell 3 uses that column, and the author ran them in the order 3–4–5. The cached output in cell 3 still shows a valid chart, so the notebook displays fine. Anyone running cells in order 1–2–3 hits a crash on cell 3, or worse, silently different results.
Hidden state lives between cells. The Python kernel remembers variables from cells you deleted hours ago. A notebook can depend on a variable that no longer exists in any visible cell. You cannot inspect this state from the file; you can only suspect it when the notebook behaves differently on another machine.
The testing ecosystem does not apply. You cannot run pytest on a notebook. The entire discipline of unit tests, the mechanism that catches regressions before they reach users, simply does not exist in this format. In the GitHub study, only 1.54% of Python notebooks even imported a known test module.
Version control becomes noise. Git stores a notebook as raw JSON, including outputs: base64-encoded plots, HTML tables, execution timestamps. Rerun the notebook and the diff explodes with thousands of changed lines that contain zero meaningful code changes. Code review on notebooks is either skipped or performed on a wall of JSON.
Every one of these costs compounds when a second person enters the picture. One author’s notebook is an instrument. Two authors make it a collaboration, and the file format was never designed for collaboration.
Production-grade AI engineering is the practice of building machine learning systems as engineered software: modular Python with explicit data flow, deterministic execution, automated tests, clean version control, and CI/CD pipelines that ship without the author’s laptop in the loop.
The transition is not about the tool you type into. It is about the properties that tool gives you. A notebook gives you interactivity and hides everything else. A script-based project gives you the reverse: less instant feedback, and in exchange, testability, reviewability, and repeatability.
None of these rows is about talent. They are about properties of the format. The best data scientist in the world cannot write a reliable test suite inside a notebook, cannot give a reviewer a readable diff, and cannot run a notebook in a headless CI runner. The format decides these battles before you start.
Production code lives in .py files for a reason. Functions and modules create clear boundaries: each unit can be tested in isolation, reasoned about on its own, and composed into a pipeline where data flows in one direction. The notebook’s greatest weakness is exactly what modules fix, because a function cannot accidentally depend on a cell you deleted an hour ago. Its inputs are its inputs.
There is a simple rule of thumb used by teams that make this transition well: if a function grows past ten lines, or you find yourself copying the same logic between cells, it belongs in a module. This is not an aesthetic preference. It is the smallest possible step that makes your code testable, importable, and diffable.
The first promotion step is deceptively small. Move the pipeline’s steps into functions inside a single .py file next to the notebook, then run the pipeline end to end from that file. Keep the notebook for exploration, and let the script become the source of truth.
"""modular_pipeline.pyThe core refactoring demo: notebook-style code vs modular productioncode. The BAD pattern is shown in the docstring below. The functionsthat follow are the GOOD pattern: explicit inputs, explicit outputs,logging instead of prints, and no reliance on hidden state.BAD (notebook style): implicit state and ordering, debugging by print. # df = pd.read_csv("customer_features.csv") # churned = df["churned"].map({"yes": 1, "no": 0}) # cell 4 # df = df[df["monthly_spend"].notna()] # cell 9, depends on cell 4 # print(df.head()) # squinting at output # model.fit(X, y) # works here, breaks in productionGOOD (modular): every step is a pure function with explicit inputs andoutputs, so each unit can be tested in isolation and composed safely."""import loggingimport numpy as npimport pandas as pdfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.metrics import roc_auc_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScalerlogging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")logger = logging.getLogger("pipeline")FEATURES = [ "tenure_months", "monthly_spend", "support_tickets", "login_frequency", "days_since_last_login", "plan_tier",]def load_data(n_customers: int = 2000, seed: int = 42) -> pd.DataFrame: """Generate a synthetic customer table with churn labels. Also injects a few missing values so the cleaning step has something real to do. """ rng = np.random.default_rng(seed) churned = rng.random(n_customers) < 0.2 df = pd.DataFrame({ "tenure_months": np.where(churned, rng.integers(1, 18, n_customers), rng.integers(6, 48, n_customers)), "monthly_spend": np.where(churned, rng.normal(45, 15, n_customers), rng.normal(70, 20, n_customers)), "support_tickets": np.where(churned, rng.poisson(3.5, n_customers), rng.poisson(1.0, n_customers)), "login_frequency": np.where(churned, rng.normal(1.5, 1, n_customers), rng.normal(5, 2, n_customers)), "days_since_last_login": np.where(churned, rng.integers(10, 45, n_customers), rng.integers(0, 14, n_customers)), "plan_tier": rng.choice(["basic", "pro", "enterprise"], n_customers, p=[0.5, 0.35, 0.15]), "churned": churned.astype(int), }) df.loc[rng.random(len(df)) < 0.02, "monthly_spend"] = np.nan return dfdef clean_data(df: pd.DataFrame) -> pd.DataFrame: """Drop rows with missing values and coerce the target dtype.""" out = df.dropna().copy() out["churned"] = out["churned"].astype(int) return outdef engineer_features(df: pd.DataFrame) -> pd.DataFrame: """Add a composite engagement feature derived from existing ones.""" out = df.copy() out["engagement_score"] = ( out["login_frequency"] * 0.6 - out["days_since_last_login"] * 0.05 ) return outdef prepare_xy(df: pd.DataFrame) -> tuple[pd.DataFrame, pd.Series]: """Build the model matrix and target from a cleaned, engineered frame.""" X = pd.get_dummies(df[FEATURES], columns=["plan_tier"]) return X, df["churned"]def train_model(df: pd.DataFrame, seed: int = 42) -> tuple[Pipeline, float]: """Train a gradient boosting pipeline and return (model, test AUC).""" X, y = prepare_xy(df) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, stratify=y, random_state=seed ) pipeline = Pipeline([ ("scaler", StandardScaler()), ("clf", GradientBoostingClassifier( n_estimators=200, max_depth=4, learning_rate=0.05, random_state=seed )), ]) pipeline.fit(X_train, y_train) auc = roc_auc_score(y_test, pipeline.predict_proba(X_test)[:, 1]) logger.info("Test ROC-AUC: %.4f", auc) return pipeline, aucdef main(n_customers: int = 2000, seed: int = 42) -> float: """Run the full pipeline end to end. Deterministic given the seed.""" df = clean_data(engineer_features(load_data(n_customers, seed))) _, auc = train_model(df, seed) logger.info("Pipeline finished: AUC %.4f on %d customers", auc, len(df)) return aucif __name__ == "__main__": main()
The snippet shows both patterns side by side in the comments: the notebook-style version, which leans on global variables and print statements and dies the moment you change the order of operations, and the modular version, where every step is a function with explicit inputs, explicit outputs, and logging instead of prints. The modular version runs the same way on your laptop and on a server, because it never depends on state you forgot to define.
Notebooks trap you in a debugging loop that has no exit: re-run the cell, print a variable, squint at the output, edit, re-run again. Every iteration takes seconds of execution plus minutes of attention, and the print statements you scatter to debug today become permanent clutter that drowns the actual output tomorrow.
A proper IDE changes the game. You set a breakpoint on the line where the data is misbehaving, and execution s there. You inspect the variables in scope at that exact moment, watch expressions change as you step line by line, and read the call stack to see how the program got here. Bugs that take an hour of print-based archaeology in a notebook get found in minutes, because you are looking at the actual state of the program instead of guessing what it might be.
The call stack deserves special attention in AI code. Machine learning bugs rarely break where they are born: a wrong data shape flows through five functions and crashes at the model fit, or worse, silently trains a model on misaligned labels. The stack trace tells you where it crashed. Stepping through the debugger tells you where it went wrong, and those are different questions.
When you move to scripts, also move from print to logging. Logging survives in containers, goes to files, and carries timestamps and levels. Print statements scroll past in a production log and might as well not exist. This is a habit that only becomes available once you leave the notebook, and it is one of the first upgrades you will feel.
Show a reviewer a notebook diff and watch their eyes glaze over. A single rerun of a notebook produces a diff of thousands of lines: base64-encoded plots, HTML tables, execution counts, metadata. The actual code change, the one meaningful line, is buried somewhere in the middle of a JSON blob. Review becomes impossible, so reviews stop happening, and that is how broken code reaches production.
Plain Python files diff the way diffing was meant to work. One function changed, one line in the diff, one focused conversation in the pull request. Merge conflicts become solvable because the conflict is about code, not about which cell’s output blob wins. The moment you convert a project to scripts, version control stops being a chore and becomes an actual tool again.
If you cannot leave notebooks behind entirely, and many teams cannot, there are guardrails that shrink the damage. nbstripout installs a Git filter that removes cell outputs before they are committed, so your diffs contain code, not base64. A companion CI action runs it in verify mode and fails the build if someone pushes an unstripped notebook. jupytext pairs a notebook with a plain text .py file that Git tracks, keeping the notebook for interactive work and the text file for review. nbdime renders notebook diffs visually when you genuinely must review notebook changes. And the cheapest discipline of all: restart the kernel and run every cell top to bottom before you commit, then commit only if it survives. A result that does not survive a fresh kernel is not a result. It is an artifact of your session.
You cannot run a notebook in a CI runner. Not repeatably, not headlessly, not with the deterministic fresh-kernel execution that automated testing demands. This single constraint is the hard line between notebook projects and production systems: if your pipeline cannot run in a blank environment on a schedule, it is not deployable, it is merely present.
CI/CD is what scripts unlock. With a script-based project, every pull request can trigger the same sequence: check out the code, install pinned dependencies, run the linter, run the test suite, execute a smoke run of the pipeline on small data, and build the container that will serve the model. Regressions surface in minutes instead of months, and deployments stop requiring a human who knows the secret incantation.
The workflow file below is the standard shape of that promise: a test job that runs on every push and pull request, installs the project, lints it, runs the test suite, executes the pipeline entry point as a smoke test, and uploads the trained artifact. Let’s check all 3 of them.
"""test_pipeline.pyUnit tests for modular_pipeline. This is the file the CI runnerexecutes on every pull request: small, fast, deterministic teststhat fail loudly when a refactor breaks behavior.Run with: pytest"""import numpy as npimport pytestfrom modular_pipeline import ( clean_data, engineer_features, load_data, train_model,)@pytest.fixturedef raw_df(): return load_data(n_customers=300, seed=7)def test_clean_data_removes_missing_values(raw_df): cleaned = clean_data(raw_df) assert cleaned["monthly_spend"].isna().sum() == 0 assert len(cleaned) <= len(raw_df)def test_clean_data_coerces_target_to_integer(raw_df): cleaned = clean_data(raw_df) assert np.issubdtype(cleaned["churned"].dtype, np.integer) assert set(cleaned["churned"].unique()) <= {0, 1}def test_engineer_features_adds_engagement_score(raw_df): engineered = engineer_features(clean_data(raw_df)) assert "engagement_score" in engineered.columns assert engineered["engagement_score"].notna().all()def test_engineer_features_is_deterministic(raw_df): cleaned = clean_data(raw_df) first = engineer_features(cleaned) second = engineer_features(cleaned) assert first["engagement_score"].equals(second["engagement_score"])def test_train_model_returns_sane_auc(raw_df): cleaned = engineer_features(clean_data(raw_df)) pipeline, auc = train_model(cleaned, seed=7) assert 0.5 < auc <= 1.0 assert pipeline is not None
"""cli_pipeline.pyThe single entry point for the modular pipeline. This is what CI andproduction both run: no notebook, no hidden state, every parameterexplicit on the command line.Usage examples: python cli_pipeline.py --n-customers 2000 --model-path model.joblib python cli_pipeline.py --evaluate --model-path model.joblib"""import argparseimport loggingimport osimport joblibfrom sklearn.metrics import roc_auc_scorefrom modular_pipeline import ( clean_data, engineer_features, load_data, prepare_xy, train_model,)logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")logger = logging.getLogger("cli")def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description="Train or evaluate the churn model.") parser.add_argument("--n-customers", type=int, default=2000, help="Size of the synthetic customer table") parser.add_argument("--seed", type=int, default=42, help="Random seed for deterministic runs") parser.add_argument("--model-path", type=str, default="model.joblib", help="Where to save or load the model") parser.add_argument("--evaluate", action="store_true", help="Load the saved model and score it on fresh data") return parser.parse_args()def evaluate_model(model, df: "pd.DataFrame") -> float: """Score a saved model on a fresh, fully prepared frame.""" X, y = prepare_xy(df) return roc_auc_score(y, model.predict_proba(X)[:, 1])def main() -> None: args = parse_args() if args.evaluate: if not os.path.exists(args.model_path): raise SystemExit(f"Model not found at {args.model_path}. Train it first.") model = joblib.load(args.model_path) df = engineer_features(clean_data(load_data(args.n_customers, args.seed))) auc = evaluate_model(model, df) logger.info("Evaluated AUC on fresh data: %.4f", auc) return df = engineer_features(clean_data(load_data(args.n_customers, args.seed))) model, auc = train_model(df, args.seed) joblib.dump(model, args.model_path) logger.info("Saved model to %s (AUC %.4f)", args.model_path, auc)if __name__ == "__main__": main()
The test file is what the CI runner executes: unit tests against the modular pipeline, each one small enough to run in milliseconds. The CLI file is what CI and production share: a single entry point that trains or evaluates the pipeline with explicit arguments, no notebook state required. The workflow file wires them together: every pull request gets linted and tested, every merge to the main branch builds and stores the artifact. You cannot do any of this with a notebook, not because of a missing feature, but because the format lacks the properties the automation depends on.
None of this means you should abandon notebooks. Exploration is real work, and a notebook is the best tool for it ever built. The skill is knowing when to promote an experiment into an engineered system, and the moment is usually easy to spot: the first time somebody other than you needs the result, or the first time someone asks “can we automate this?” When the answer to that question is yes, the notebook has served its exploratory purpose.
The promotion itself follows a reliable sequence:
There is a modern version of this lesson that AI engineers in particular should hear: prompts are code. A prompt that lives in cell 34 of an untracked notebook is a prompt that will silently drift from the one your service actually uses, because the notebook and the service cannot both be the source of truth. Move prompts into modules, where they become diffable, greppable, importable, and testable, exactly like the rest of your pipeline. The same discipline that saves your model code saves your prompt engineering.
The transition from notebook to production is not a rejection of exploration. It is a boundary: notebooks for thinking, scripts for shipping. The organizations that deploy AI reliably did not hire smarter people. They made their code reviewable, their execution deterministic, and their deployment automated, and every one of those properties required leaving the .ipynb behind for anything that would run more than once.
Ship code that outlasts your laptop. Keep the notebook for the exploration that deserves it, then promote the work into modules, tests, and pipelines, and let the machine do the running.
Here are several key takeaways from this article:
Thank you for reading this article! I hope you found it helpful. If you have any questions or feedback, please feel free to reach out to me.
Beyond Just Jupyter Notebook: How to Ship AI Code That Survives Production was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.