Phishing Website Detection with Machine Learning: A Step-by-Step PySpark Guide

A step-by-step PySpark tutorial demonstrates that a machine-learning model can distinguish phishing websites from legitimate ones using URL and page-signal features from the Kaggle Web Page Phishing Detection dataset, which is based on research by Hannousse and Yahiouche (2021). The guide cites roughly 1.4 million phishing sites going live each month and about 100 million phishing emails blocked daily by Google, and walks through exploratory data analysis, correlation checks, and MLlib feature preparation with StringIndexer and VectorAssembler on PySpark 3.4.1 and Python 3.10. The author argues phishers can copy a site's look but not its history, so a long URL on a young, unranked, low-traffic domain is a signal worth flagging.

You get an email from your bank. The link says www.standardcharterbank-login.com. Looks fine, right? It isn’t. And that’s exactly the problem. About 1.4 million phishing sites go live every month , and Google alone blocks roughly 100 million phishing emails a day AAG IT https://aag-it.com/the-latest-phishing-statistics/ . Most of these scams work because the fake URL looks almost right. So I asked a simple question: can a machine learn to tell a fake website from a real one, just by looking at its URL and a few page signals? Turns out it can. In this post I’ll walk you through exactly how I built it, step by step, so you can build your own. TL;DR Phishers copy the look of a real site. But they can’t easily copy its history . A brand-new domain, a weird long URL, zero search ranking: these leave fingerprints. Before building anything, I explored the data to see which fingerprints actually matter. Here’s what stood out: Rule of thumb: a long URL on a young, unranked, low-traffic domain deserves a second look. I used the Web Page Phishing Detection dataset https://www.kaggle.com/datasets/shashwatwork/web-page-phishing-detection-dataset on Kaggle. It’s based on research by Hannousse and Yahiouche 2021 https://data.mendeley.com/datasets/c2gw7fy2j4/3 , so it’s a solid, citable source. The features fall into three groups: A balanced dataset is a gift. It means accuracy is a fair metric, and you don’t need tricks like oversampling. I ran everything on a MacBook Air M1 8 GB RAM with Python 3.10, PySpark 3.4.1 and Jupyter. If your laptop can run Chrome, it can run this. Spark runs on the JVM, so install Java 8 first, then check it: java -version Next, find where Java lives and point JAVA HOME at it. On newer Macs the default shell is zsh, so the setting goes in ~/.zprofile not .bash profile : /usr/libexec/java home add this line to ~/.zprofileexport JAVA HOME="/Library/Java/JavaVirtualMachines/jdk-1.8.jdk/Contents/Home" I used Conda, which keeps things tidy: conda install -c conda-forge pysparkpyspark --version If you see the Spark logo and a version number, you’re good. python from pyspark.sql import SparkSession spark = SparkSession.builder.appName "URLClassification" .getOrCreate data = spark.read.csv "dataset phishing.csv", header=True, inferSchema=True data.printSchema inferSchema=True lets Spark guess each column's type, so numbers come in as numbers, not strings. Don’t skip this. EDA is where you find out what the model should learn. I checked three things first: 1. Duplicates?print data.count , data.dropDuplicates .count 2. Is the target balanced?data.groupBy "status" .count .show 3. Quick stats on a featuredata.select "length url" .describe .show Then I looked for features that move together, using a correlation heatmap: python import seaborn as snsimport matplotlib.pyplot as plt corr = data.drop "url", "status" .toPandas .corr plt.figure figsize= 32, 20 sns.heatmap corr, cmap="coolwarm", center=0 plt.show Tip: toPandas pulls everything into memory. Fine for 6 MB, risky for 6 GB. On big data, sample first. The strongest pair was longest word path with longest words raw, which makes sense since one is part of the other. Notebook charts are fine for you. For anyone else, I rebuilt them in Tableau. In Tableau Prep , I swapped the 0/1 codes for human words: 0 became "Legit" and 1 became "Phishing", and other flags became "Yes"/"No". Small change, much easier dashboards. You can explore the result on Tableau Public https://public.tableau.com/app/profile/maladeep/viz/WebsitePhishing/WebsitePhishingAnalysis . MLlib has two rules: labels must be numbers, and all features must sit in one vector column . Two tools handle that: python from pyspark.ml.feature import StringIndexer, VectorAssembler feature cols = c for c in data.columns if c not in "url", "status" target indexer = StringIndexer inputCol="status", outputCol="label" assembler = VectorAssembler inputCols=feature cols, outputCol="features" Why guess which algorithm is best when you can test them all? Put them in a dictionary: from pyspark.ml.classification import DecisionTreeClassifier, RandomForestClassifier, NaiveBayes, GBTClassifier, LinearSVC classifiers = { "DecisionTree": DecisionTreeClassifier labelCol="label", featuresCol="features" , "RandomForest": RandomForestClassifier labelCol="label", featuresCol="features", numTrees=100, maxDepth=10 , "NaiveBayes": NaiveBayes smoothing=1.0, modelType="gaussian" , "GBT": GBTClassifier labelCol="label", featuresCol="features", maxIter=10 , "LinearSVC": LinearSVC labelCol="label", featuresCol="features" ,} Then wrap each one in a Pipeline, split 70/30, train and score: python from pyspark.ml import Pipelinefrom pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator train, test = data.randomSplit 0.7, 0.3 , seed=123 results = for name, clf in classifiers.items : model = Pipeline stages= target indexer, assembler, clf .fit train preds = model.transform test auc = BinaryClassificationEvaluator labelCol="label" .evaluate preds auprc = BinaryClassificationEvaluator labelCol="label", metricName="areaUnderPR" .evaluate preds acc = MulticlassClassificationEvaluator labelCol="label", metricName="accuracy" .evaluate preds f1 = MulticlassClassificationEvaluator labelCol="label", metricName="f1" .evaluate preds results.append name, round auc, 2 , round auprc, 2 , round acc, 2 , round f1, 2 Tip: fix the seed so every model sees the exact same train/test split. Otherwise you're comparing luck, not algorithms. 89 features is a lot. Could the top 10 do just as well? Chi-Square χ² scores each feature by how strongly it relates to the label. It needs non-negative inputs, which suits this mostly 0/1 data. python from pyspark.ml.feature import ChiSqSelector selector = ChiSqSelector numTopFeatures=10, featuresCol="features", outputCol="selected features" I ran the same loop again with the selector added to each pipeline. Keep this step in mind. It comes back in the results. Here’s how all five models scored on the 30% test set: Model AUC AUPRC Accuracy F1Random Forest 0.99 0.99 0.96 0.96Linear SVC 0.98 0.98 0.94 0.94Gradient Boosted Trees 0.98 0.98 0.93 0.93Decision Tree 0.90 0.93 0.92 0.92Naive Bayes 0.65 0.68 0.76 0.76 Random Forest labeled a URL correctly 96 times out of 100 . That matches earlier research too Subasi et al., 2017 https://doi.org/10.1109/ICECTA.2017.8252051 ; Hutchinson et al., 2018 https://doi.org/10.1007/978-3-030-00557-3 46 . Why does it work so well here? Phishing signals combine in messy ways: a long URL alone is fine, but a long URL plus a young domain plus zero traffic is not. Hundreds of trees voting together are great at catching those combinations. Naive Bayes struggled because it assumes every feature is independent. Here they clearly aren’t URL length and word length rise together, for example . With Chi-Square feature selection added, every model scored exactly the same , to two decimal places. My first conclusion was that feature selection simply didn’t help. But identical numbers across five different models is suspicious. Look at the code again: selector = ChiSqSelector ..., outputCol="selected features" RandomForestClassifier labelCol="label", featuresCol="features" ← still reads all 89 The selector wrote its top 10 features into selected features. But every classifier was still reading from features. So the selection ran, and then got ignored. The fix is one argument: RandomForestClassifier labelCol="label", featuresCol="selected features" Lesson: when a change produces exactly the same result, don’t celebrate or shrug. Check that the change was actually wired in. If I take this further, I’ll fix the Chi-Square wiring and compare 10 vs 89 features properly, add real cross-validation with CrossValidator, and wrap the Random Forest in a small API that checks any URL you paste in. If this helped, give it a clap and follow for more hands-on data science. And next time a link looks almost right, check the URL twice. Phishing Website Detection with Machine Learning: A Step-by-Step PySpark Guide https://pub.towardsai.net/phishing-website-detection-with-machine-learning-a-step-by-step-pyspark-guide-adf7f7945184 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.