{"slug": "phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide", "title": "Phishing Website Detection with Machine Learning: A Step-by-Step PySpark Guide", "summary": "A step-by-step PySpark tutorial demonstrates that a machine-learning model can distinguish phishing websites from legitimate ones using URL and page-signal features from the Kaggle Web Page Phishing Detection dataset, which is based on research by Hannousse and Yahiouche (2021). The guide cites roughly 1.4 million phishing sites going live each month and about 100 million phishing emails blocked daily by Google, and walks through exploratory data analysis, correlation checks, and MLlib feature preparation with StringIndexer and VectorAssembler on PySpark 3.4.1 and Python 3.10. The author argues phishers can copy a site's look but not its history, so a long URL on a young, unranked, low-traffic domain is a signal worth flagging.", "body_md": "You get an email from your bank. The link says www.standardcharterbank-login.com. Looks fine, right?\n\nIt isn’t. And that’s exactly the problem.\n\nAbout **1.4 million phishing sites go live every month**, and Google alone blocks roughly **100 million phishing emails a day** ([AAG IT](https://aag-it.com/the-latest-phishing-statistics/)). Most of these scams work because the fake URL looks *almost* right.\n\nSo I asked a simple question: **can a machine learn to tell a fake website from a real one, just by looking at its URL and a few page signals?**\n\nTurns out it can. In this post I’ll walk you through exactly how I built it, step by step, so you can build your own.\n\n**TL;DR**\n\nPhishers copy the *look* of a real site. But they can’t easily copy its *history*. A brand-new domain, a weird long URL, zero search ranking: these leave fingerprints.\n\nBefore building anything, I explored the data to see which fingerprints actually matter. Here’s what stood out:\n\n***Rule of thumb:*** *a long URL on a young, unranked, low-traffic domain deserves a second look.*\n\nI used the [Web Page Phishing Detection dataset](https://www.kaggle.com/datasets/shashwatwork/web-page-phishing-detection-dataset) on Kaggle. It’s based on [research by Hannousse and Yahiouche (2021)](https://data.mendeley.com/datasets/c2gw7fy2j4/3), so it’s a solid, citable source.\n\nThe features fall into three groups:\n\nA balanced dataset is a gift. It means accuracy is a fair metric, and you don’t need tricks like oversampling.\n\nI ran everything on a MacBook Air M1 (8 GB RAM) with Python 3.10, PySpark 3.4.1 and Jupyter. If your laptop can run Chrome, it can run this.\n\nSpark runs on the JVM, so install Java 8 first, then check it:\n\n```\njava -version\n```\n\nNext, find where Java lives and point JAVA_HOME at it. On newer Macs the default shell is zsh, so the setting goes in ~/.zprofile (not .bash_profile):\n\n```\n/usr/libexec/java_home\n# add this line to ~/.zprofileexport JAVA_HOME=\"/Library/Java/JavaVirtualMachines/jdk-1.8.jdk/Contents/Home\"\n```\n\nI used Conda, which keeps things tidy:\n\n```\nconda install -c conda-forge pysparkpyspark --version\n```\n\nIf you see the Spark logo and a version number, you’re good.\n\n``` python\nfrom pyspark.sql import SparkSession\nspark = SparkSession.builder.appName(\"URLClassification\").getOrCreate()\ndata = spark.read.csv(\"dataset_phishing.csv\", header=True, inferSchema=True)data.printSchema()\n```\n\ninferSchema=True lets Spark guess each column's type, so numbers come in as numbers, not strings.\n\nDon’t skip this. EDA is where you find out what the model *should* learn. I checked three things first:\n\n```\n# 1. Duplicates?print(data.count(), data.dropDuplicates().count())\n# 2. Is the target balanced?data.groupBy(\"status\").count().show()\n# 3. Quick stats on a featuredata.select(\"length_url\").describe().show()\n```\n\nThen I looked for features that move together, using a correlation heatmap:\n\n``` python\nimport seaborn as snsimport matplotlib.pyplot as plt\ncorr = data.drop(\"url\", \"status\").toPandas().corr()plt.figure(figsize=(32, 20))sns.heatmap(corr, cmap=\"coolwarm\", center=0)plt.show()\n```\n\n**Tip:** toPandas() pulls everything into memory. Fine for 6 MB, risky for 6 GB. On big data, sample first.\n\nThe strongest pair was longest_word_path with longest_words_raw, which makes sense since one is part of the other.\n\nNotebook charts are fine for you. For anyone else, I rebuilt them in Tableau.\n\nIn **Tableau Prep**, I swapped the 0/1 codes for human words: 0 became \"Legit\" and 1 became \"Phishing\", and other flags became \"Yes\"/\"No\". Small change, much easier dashboards. You can explore the result on [Tableau Public](https://public.tableau.com/app/profile/maladeep/viz/WebsitePhishing/WebsitePhishingAnalysis).\n\nMLlib has two rules: labels must be numbers, and all features must sit in **one vector column**. Two tools handle that:\n\n``` python\nfrom pyspark.ml.feature import StringIndexer, VectorAssembler\nfeature_cols = [c for c in data.columns if c not in (\"url\", \"status\")]\ntarget_indexer = StringIndexer(inputCol=\"status\", outputCol=\"label\")assembler = VectorAssembler(inputCols=feature_cols, outputCol=\"features\")\n```\n\nWhy guess which algorithm is best when you can test them all? Put them in a dictionary:\n\n```\nfrom pyspark.ml.classification import (    DecisionTreeClassifier, RandomForestClassifier,    NaiveBayes, GBTClassifier, LinearSVC)\nclassifiers = {    \"DecisionTree\": DecisionTreeClassifier(labelCol=\"label\", featuresCol=\"features\"),    \"RandomForest\": RandomForestClassifier(labelCol=\"label\", featuresCol=\"features\",                                           numTrees=100, maxDepth=10),    \"NaiveBayes\":   NaiveBayes(smoothing=1.0, modelType=\"gaussian\"),    \"GBT\":          GBTClassifier(labelCol=\"label\", featuresCol=\"features\", maxIter=10),    \"LinearSVC\":    LinearSVC(labelCol=\"label\", featuresCol=\"features\"),}\n```\n\nThen wrap each one in a Pipeline, split 70/30, train and score:\n\n``` python\nfrom pyspark.ml import Pipelinefrom pyspark.ml.evaluation import (    BinaryClassificationEvaluator, MulticlassClassificationEvaluator)\ntrain, test = data.randomSplit([0.7, 0.3], seed=123)results = []\nfor name, clf in classifiers.items():    model = Pipeline(stages=[target_indexer, assembler, clf]).fit(train)    preds = model.transform(test)\nauc   = BinaryClassificationEvaluator(labelCol=\"label\").evaluate(preds)    auprc = BinaryClassificationEvaluator(labelCol=\"label\",                metricName=\"areaUnderPR\").evaluate(preds)    acc   = MulticlassClassificationEvaluator(labelCol=\"label\",                metricName=\"accuracy\").evaluate(preds)    f1    = MulticlassClassificationEvaluator(labelCol=\"label\",                metricName=\"f1\").evaluate(preds)\nresults.append((name, round(auc, 2), round(auprc, 2), round(acc, 2), round(f1, 2)))\n```\n\n**Tip:** fix the seed so every model sees the exact same train/test split. Otherwise you're comparing luck, not algorithms.\n\n89 features is a lot. Could the top 10 do just as well? Chi-Square (χ²) scores each feature by how strongly it relates to the label. It needs non-negative inputs, which suits this mostly 0/1 data.\n\n``` python\nfrom pyspark.ml.feature import ChiSqSelector\nselector = ChiSqSelector(numTopFeatures=10,                         featuresCol=\"features\",                         outputCol=\"selected_features\")\n```\n\nI ran the same loop again with the selector added to each pipeline. Keep this step in mind. It comes back in the results.\n\nHere’s how all five models scored on the 30% test set:\n\n```\nModel                    AUC    AUPRC   Accuracy   F1Random Forest            0.99   0.99    0.96       0.96Linear SVC               0.98   0.98    0.94       0.94Gradient Boosted Trees   0.98   0.98    0.93       0.93Decision Tree            0.90   0.93    0.92       0.92Naive Bayes              0.65   0.68    0.76       0.76\n```\n\nRandom Forest labeled a URL correctly **96 times out of 100**. That matches earlier research too ([Subasi et al., 2017](https://doi.org/10.1109/ICECTA.2017.8252051); [Hutchinson et al., 2018](https://doi.org/10.1007/978-3-030-00557-3_46)).\n\nWhy does it work so well here? Phishing signals combine in messy ways: a long URL alone is fine, but a long URL *plus* a young domain *plus* zero traffic is not. Hundreds of trees voting together are great at catching those combinations.\n\nNaive Bayes struggled because it assumes every feature is independent. Here they clearly aren’t (URL length and word length rise together, for example).\n\nWith Chi-Square feature selection added, **every model scored exactly the same**, to two decimal places. My first conclusion was that feature selection simply didn’t help.\n\nBut identical numbers across five different models is suspicious. Look at the code again:\n\n```\nselector = ChiSqSelector(..., outputCol=\"selected_features\")RandomForestClassifier(labelCol=\"label\", featuresCol=\"features\")  # ← still reads all 89\n```\n\nThe selector wrote its top 10 features into selected_features. But every classifier was still reading from features. So the selection ran, and then got ignored.\n\nThe fix is one argument:\n\n```\nRandomForestClassifier(labelCol=\"label\", featuresCol=\"selected_features\")\n```\n\n**Lesson:** when a change produces *exactly* the same result, don’t celebrate or shrug. Check that the change was actually wired in.\n\nIf I take this further, I’ll fix the Chi-Square wiring and compare 10 vs 89 features properly, add real cross-validation with CrossValidator, and wrap the Random Forest in a small API that checks any URL you paste in.\n\nIf this helped, give it a clap and follow for more hands-on data science. And next time a link looks *almost* right, check the URL twice.\n\n[Phishing Website Detection with Machine Learning: A Step-by-Step PySpark Guide](https://pub.towardsai.net/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide-adf7f7945184) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide", "canonical_source": "https://pub.towardsai.net/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide-adf7f7945184?source=rss----98111c9905da---4", "published_at": "2026-11-09 12:01:02+00:00", "updated_at": "2026-11-09 12:24:11.125871+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-tools"], "entities": ["PySpark", "Kaggle", "Google", "Hannousse", "Yahiouche", "Tableau", "MLlib", "MacBook Air M1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide", "markdown": "https://wpnews.pro/news/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide.md", "text": "https://wpnews.pro/news/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide.txt", "jsonld": "https://wpnews.pro/news/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide.jsonld"}}