# Practical IP-Level Unsupervised Classification Using HDBSCAN and K-Means > Source: > Published: 2026-08-24 07:27:28+00:00 Last Updated on August 24, 2026 by Editorial Team Author(s): Bassem Essameldin Omar Originally published on Towards AI. Practical IP-Level Unsupervised Classification Using HDBSCAN and K-Means A recurring challenge in network analytics arises when two distinct consumer brands are operated by the same legal entity and share a single Autonomous System Number (ASN). Standard IP-to-ISP databases resolve every IP in that ASN to one company name, making it impossible to distinguish which brand a given measurement session belongs to. No labelled training set exists. The input is two datasets (IPv4 and IPv6) where each row represents aggregated sessions sharing the same DNS resolver IP, with accompanying fields: IP range bounds, ASN organisation string, geographic region, connection public IP, and a session count. The objective is to assign each row a brand label using only these network-layer signals. Photo by BoliviaInteligente on Unsplash 1. Data Structure and Signal Sources Each row represents one or more device sessions grouped by their configured DNS resolver address. The key fields are: ip_lower_str / ip_upper_str — lower and upper bounds of the IP range; block rows contain wildcard placeholders (e.g. 118.x.x.x) QOS_DNSAddresses — the DNS resolver IP from the device; for home broadband users, this is typically the CPE router’s own IP, not a public upstream resolver. AsnOrganization / isp — ASN-derived organisation strings from a third-party IP database. OneSource_PublicIp — the connection IP observed by the measurement server. Location_Region — geographic region label. numrecords — count of sessions aggregated into this row. The central insight is that the DNS resolver IP is often the home router’s own address rather than a public upstream resolver like 8.8.8.8 or 1.1.1.1. This means the DNS IP encodes information about the CPE hardware and sub-network allocated by the ISP — a signal that is entirely invisible to standard ASN lookups. 2. Feature Engineering 2.1 Wildcard IP Cleaning Block IP ranges use x as a wildcard placeholder in lower-order octets. Before any numeric processing, these are normalised to concrete addresses by substituting 0: clean = lambda s: re.sub(r'\b[xX]\b', '0', str(s).strip()) 2.2 Prefix Extraction Rather than treating IP addresses as opaque strings, the first two octets (IPv4) or first two 16-bit hextets (IPv6) are extracted as compact categorical features. This collapses the address space to a manageable prefix set while preserving the sub-network signal: # IPv4 dns_prefix = '.'.join(ip_str.split('.')[:2]) # e.g. '118.238' # IPv6 - extract first two hextets from packed bytes packed = ipaddress.IPv6Address(ip_str).packed h1 = (packed[0] << 8) | packed[1] For IPv6, the second hextet (h2) emerged during EDA as the primary discriminating feature — different sub-network allocations within the same /20 block correspond to different product lines served by the same parent organisation. 2.3 Log Transformation of Skewed Numerics Two features spanned several orders of magnitude: IP range size (from 256 addresses for a /24 up to 2⁹⁶ for a large IPv6 allocation) and numrecords (session counts from tens to millions). Without transformation, raw differences of this scale would dominate Euclidean distance in clustering. Log transformation compresses the distribution: log_range = math.log2(ip_upper - ip_lower) if ip_upper > ip_lower else 0 log_records = np.log1p(numrecords) Note the use of math.log2() rather than np.log2() for IPv6 range sizes. IPv6 address arithmetic produces Python arbitrary-precision integers, which NumPy cannot process natively — np.log2() raises AttributeError: ‘int’ object has no attribute ‘log2’ on these values. Staying in pure Python before passing to NumPy is required. 2.4 Frequency Encoding of Categoricals Categorical fields (AsnOrganization, isp, region) are encoded by replacing each category with its frequency count in the dataset. This is preferred over one-hot encoding because the cardinality is high (hundreds of distinct ASNs) and frequency itself is a meaningful signal — rare networks are more distinctive than common ones: freq_map = df['AsnOrganization'].value_counts().to_dict() df['asn_freq'] = df['AsnOrganization'].map(freq_map) 2.5 Standardisation All numeric features are passed through StandardScaler (zero mean, unit variance) before clustering. This ensures no single feature dominates the Euclidean distance metric regardless of its original numeric range. 3. Clustering Exploration K-Means and DBSCAN are fundamental clustering algorithms that diverge sharply in their underlying mechanics and applications. K-Means is a centroid-based method that partitions data into a pre-specified number (k) of spherical clusters by minimizing within-cluster distances, making it computationally efficient but highly sensitive to outliers and assuming the clusters are spherical. In contrast, DBSCAN is a density-based algorithm that does not require the user to define the number of clusters; instead, it groups densely packed points based on a distance epsilon (ε) and a minimum point threshold (MinPts), allowing it to discover arbitrarily shaped clusters while robustly labeling sparse points as noise. Consequently, their main differences lie in cluster shape assumptions (spherical vs. arbitrary), the necessity of a predefined cluster count, and their handling of outliers, though DBSCAN’s performance is heavily dependent on proper parameter tuning and can struggle when densities vary significantly across the dataset. 3.1 K-Means (baseline) K-Means with k=3 served as a quick baseline to understand the rough structure of the feature space. It confirmed that the data separates into broadly different network types but was otherwise inadequate: its assumption of spherical, equally-sized clusters produced arbitrary boundaries across the irregular density structure in the data. 3.2 DBSCAN DBSCAN (Density-Based Spatial Clustering of Applications with Noise) was applied next. Its key advantage is producing a noise class (label -1) for points that don’t fit any dense region — useful for flagging outlier rows linked to anomalous ASNs. The limitation is that a single global eps parameter is poorly suited to datasets with clusters of varying density. 3.3 HDBSCAN (final choice) HDBSCAN (Hierarchical DBSCAN) fundamentally improves upon DBSCAN by overcoming its most critical limitation: the reliance on a single, globally applied density threshold (epsilon). Instead of forcing a one-size-fits-all parameter, HDBSCAN constructs a hierarchy of clusters across all possible density scales and uses a stability-based metric to extract the most persistent clusters. This key advantage allows it to automatically discover clusters of widely varying densities within the same dataset — a scenario where traditional DBSCAN inevitably fails — while simultaneously eliminating […]