# My Machine Learning Project Taught Me More About Software Engineering Than Machine Learning > Source: > Published: 2026-07-31 06:05:31+00:00 When I started building my ad click prediction project, I thought the hard part would be training a machine learning model. I was wrong. Before I even got to preprocessing the data, I found myself dealing with project architecture, testing, imports, packaging, and debugging environment issues. Looking back, those lessons were probably more valuable than the model itself. Here's what happened. Like many data science projects, everything began inside a Jupyter notebook. Loading the dataset was as simple as: ``` df = pd.read_csv("data/raw/Advertising.csv") ``` It worked perfectly. At first glance, there wasn't much reason to change it. The dataset loaded correctly, there were no missing values, no duplicates, and I could immediately start exploring the data. But there was one problem. The notebook knew how to load the data. Nothing else did. Instead of leaving everything inside the notebook, I moved the loading logic into a dedicated module. ``` src/ └── data/ └── load_data.py ``` The notebook changed from doing the work itself to simply calling a function: ``` python from src.data.load_data import load_raw_data df = load_raw_data("data/raw/Advertising.csv") ``` It doesn't feel like a huge improvement until you think about the future. Now every notebook, script, or training pipeline can reuse the same code instead of copying and pasting it. Small change. Big difference. The next improvement wasn't about functionality. It was about failure. Instead of relying on `pandas.read_csv()` to throw an error if the dataset disappeared, I added an explicit check that raises a meaningful `FileNotFoundError` . That seemed unnecessary at first. But then I realized something. Good software doesn't just work when everything is correct. It also fails in ways that are easy to understand. A descriptive error today can save a lot of debugging tomorrow. The next step was introducing automated testing. The first test was intentionally simple. It verified that: Nothing fancy. Still, it was the first time the project had a safety net. Or so I thought. Instead, it failed with this: ``` ModuleNotFoundError: No module named 'src' ``` At first I assumed I had made a mistake. I hadn't. The notebook worked because I had manually modified `sys.path` . `pytest` , however, starts a fresh Python process. It had no idea where `src` lived. The interesting part wasn't the error. It was what the error revealed. My project structure wasn't as solid as I thought. The first idea was to package the project properly using `pyproject.toml` and install it in editable mode. That's the modern Python approach and, in many environments, it's absolutely the right solution. Except... My development environment had other ideas. Running: ``` python -m pip install -e . ``` didn't work because of the packaging tools available in the hosted Anaconda environment. For a moment, it felt like I'd gone down the wrong path. But that turned into another lesson. Sometimes the technically "best" solution isn't the practical one for your environment. Instead of spending hours fighting the environment, I switched to a simpler approach. Adding a small `tests/conftest.py` file allowed `pytest` to locate the project correctly. One command later: ``` pytest ``` The result: ``` 1 passed ``` Success. More importantly, I learned something that applies far beyond Python. Engineering isn't about blindly following best practices. It's about choosing the right level of complexity for the problem you're solving. By the end of this stage, I hadn't trained a model. I hadn't engineered features. I hadn't even started preprocessing. Yet the project had already changed dramatically. Instead of a single notebook, I now had: That feels much closer to a production codebase than a collection of experiments. The next stage is building a reusable preprocessing pipeline. Rather than cleaning data directly inside the notebook, I'll move that logic into `src/data/preprocess.py` , where every transformation can be tested and reused. The notebook will become a client of the pipeline instead of the pipeline itself. I'm beginning to appreciate that building a machine learning project isn't just about choosing algorithms. It's about building software that happens to use machine learning. And honestly, that's turning out to be the most valuable lesson so far.