{"slug": "my-machine-learning-project-taught-me-more-about-software-engineering-than", "title": "My Machine Learning Project Taught Me More About Software Engineering Than Machine Learning", "summary": "A developer building an ad click prediction project discovered that software engineering challenges, such as project architecture, testing, and environment issues, proved more valuable than the machine learning model itself. The project evolved from a Jupyter notebook into a structured codebase with reusable modules and automated tests, highlighting the importance of practical engineering over best practices.", "body_md": "When I started building my ad click prediction project, I thought the hard part would be training a machine learning model.\n\nI was wrong.\n\nBefore I even got to preprocessing the data, I found myself dealing with project architecture, testing, imports, packaging, and debugging environment issues. Looking back, those lessons were probably more valuable than the model itself.\n\nHere's what happened.\n\nLike many data science projects, everything began inside a Jupyter notebook.\n\nLoading the dataset was as simple as:\n\n```\ndf = pd.read_csv(\"data/raw/Advertising.csv\")\n```\n\nIt worked perfectly.\n\nAt first glance, there wasn't much reason to change it. The dataset loaded correctly, there were no missing values, no duplicates, and I could immediately start exploring the data.\n\nBut there was one problem.\n\nThe notebook knew how to load the data.\n\nNothing else did.\n\nInstead of leaving everything inside the notebook, I moved the loading logic into a dedicated module.\n\n```\nsrc/\n└── data/\n    └── load_data.py\n```\n\nThe notebook changed from doing the work itself to simply calling a function:\n\n``` python\nfrom src.data.load_data import load_raw_data\n\ndf = load_raw_data(\"data/raw/Advertising.csv\")\n```\n\nIt doesn't feel like a huge improvement until you think about the future.\n\nNow every notebook, script, or training pipeline can reuse the same code instead of copying and pasting it.\n\nSmall change.\n\nBig difference.\n\nThe next improvement wasn't about functionality.\n\nIt was about failure.\n\nInstead of relying on `pandas.read_csv()`\n\nto throw an error if the dataset disappeared, I added an explicit check that raises a meaningful `FileNotFoundError`\n\n.\n\nThat seemed unnecessary at first.\n\nBut then I realized something.\n\nGood software doesn't just work when everything is correct.\n\nIt also fails in ways that are easy to understand.\n\nA descriptive error today can save a lot of debugging tomorrow.\n\nThe next step was introducing automated testing.\n\nThe first test was intentionally simple.\n\nIt verified that:\n\nNothing fancy.\n\nStill, it was the first time the project had a safety net.\n\nOr so I thought.\n\nInstead, it failed with this:\n\n```\nModuleNotFoundError: No module named 'src'\n```\n\nAt first I assumed I had made a mistake.\n\nI hadn't.\n\nThe notebook worked because I had manually modified `sys.path`\n\n.\n\n`pytest`\n\n, however, starts a fresh Python process.\n\nIt had no idea where `src`\n\nlived.\n\nThe interesting part wasn't the error.\n\nIt was what the error revealed.\n\nMy project structure wasn't as solid as I thought.\n\nThe first idea was to package the project properly using `pyproject.toml`\n\nand install it in editable mode.\n\nThat's the modern Python approach and, in many environments, it's absolutely the right solution.\n\nExcept...\n\nMy development environment had other ideas.\n\nRunning:\n\n```\npython -m pip install -e .\n```\n\ndidn't work because of the packaging tools available in the hosted Anaconda environment.\n\nFor a moment, it felt like I'd gone down the wrong path.\n\nBut that turned into another lesson.\n\nSometimes the technically \"best\" solution isn't the practical one for your environment.\n\nInstead of spending hours fighting the environment, I switched to a simpler approach.\n\nAdding a small `tests/conftest.py`\n\nfile allowed `pytest`\n\nto locate the project correctly.\n\nOne command later:\n\n```\npytest\n```\n\nThe result:\n\n```\n1 passed\n```\n\nSuccess.\n\nMore importantly, I learned something that applies far beyond Python.\n\nEngineering isn't about blindly following best practices.\n\nIt's about choosing the right level of complexity for the problem you're solving.\n\nBy the end of this stage, I hadn't trained a model.\n\nI hadn't engineered features.\n\nI hadn't even started preprocessing.\n\nYet the project had already changed dramatically.\n\nInstead of a single notebook, I now had:\n\nThat feels much closer to a production codebase than a collection of experiments.\n\nThe next stage is building a reusable preprocessing pipeline.\n\nRather than cleaning data directly inside the notebook, I'll move that logic into `src/data/preprocess.py`\n\n, where every transformation can be tested and reused.\n\nThe notebook will become a client of the pipeline instead of the pipeline itself.\n\nI'm beginning to appreciate that building a machine learning project isn't just about choosing algorithms.\n\nIt's about building software that happens to use machine learning.\n\nAnd honestly, that's turning out to be the most valuable lesson so far.", "url": "https://wpnews.pro/news/my-machine-learning-project-taught-me-more-about-software-engineering-than", "canonical_source": "https://dev.to/mnyawade/my-machine-learning-project-taught-me-more-about-software-engineering-than-machine-learning-4895", "published_at": "2026-07-31 06:05:31+00:00", "updated_at": "2026-07-31 06:34:16.234050+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["Jupyter", "pandas", "pytest", "Anaconda"], "alternates": {"html": "https://wpnews.pro/news/my-machine-learning-project-taught-me-more-about-software-engineering-than", "markdown": "https://wpnews.pro/news/my-machine-learning-project-taught-me-more-about-software-engineering-than.md", "text": "https://wpnews.pro/news/my-machine-learning-project-taught-me-more-about-software-engineering-than.txt", "jsonld": "https://wpnews.pro/news/my-machine-learning-project-taught-me-more-about-software-engineering-than.jsonld"}}