{"slug": "building-a-vector-search-engine-from-scratch-with-hnsw-in-python", "title": "Building a Vector Search Engine from Scratch with HNSW in Python", "summary": "A developer detailed the construction of a vector search engine from scratch using the HNSW algorithm in Python, explaining the layered graph structure and query-time complexity reduction from O(n) to O(log n). The implementation, while not production-grade, aims to provide a concrete mental model for how approximate nearest neighbor search works under the hood.", "body_md": "Every vector database tutorial opens with `pip install pinecone` and skips the part where you understand why vector search works. That's fine for shipping fast, but it costs you later — when recall drops and you don't know where to look, or when a tuning parameter turns your latency from 10ms to 800ms.\n\nHNSW (Hierarchical Navigable Small World) is the algorithm behind almost every ANN (Approximate Nearest Neighbor) library in production today: FAISS, Qdrant, Weaviate, and hnswlib all use it internally. This post builds a working HNSW index in pure Python — not production-grade, but real enough to give you a concrete mental model.\n\nThe naive approach to nearest neighbor search is brute force: compare your query vector to every stored vector, return the k closest. That's O(n·d) per query — for 1M vectors at 1536 dimensions, expect around 6 seconds per lookup.\n\nHNSW solves this with a layered graph. Vectors live across multiple layers:\n\nAt query time:\n\nExpected complexity drops from O(n) to O(log n). The trade-off: more memory, and approximate results — you may occasionally miss the single true nearest neighbor, but you'll be well within a small margin.\n\nHere's a minimal HNSW implementation that captures the core structure:\n\n``` python\nimport numpy as np\nimport heapq\nimport math\nimport random\nfrom dataclasses import dataclass, field\nfrom typing import Optional\n\n@dataclass\nclass HNSWNode:\n    vector: np.ndarray\n    neighbors: dict = field(default_factory=dict)  # layer -> list of node ids\n\nclass HNSWIndex:\n    def __init__(self, dim: int, M: int = 16, ef_construction: int = 200, max_layers: int = 6):\n        self.dim = dim\n        self.M = M               # max neighbors per layer\n        self.M0 = M * 2          # max neighbors at layer 0\n        self.ef = ef_construction\n        self.max_layers = max_layers\n        self.nodes: list[HNSWNode] = []\n        self.entry_point: Optional[int] = None\n        self.top_layer = 0\n\n    def _distance(self, a: np.ndarray, b: np.ndarray) -> float:\n        diff = a - b\n        return float(np.dot(diff, diff))  # squared L2\n\n    def _random_level(self) -> int:\n        level = 0\n        while random.random() < (1.0 / self.M) and level < self.max_layers - 1:\n            level += 1\n        return level\n\n    def _search_layer(self, query: np.ndarray, entry_id: int, ef: int, layer: int) -> list[int]:\n        visited = {entry_id}\n        entry_dist = self._distance(query, self.nodes[entry_id].vector)\n        candidates = [(entry_dist, entry_id)]\n        dynamic_list = [(entry_dist, entry_id)]\n\n        while candidates:\n            c_dist, c_id = heapq.heappop(candidates)\n            worst_dist = max(d for d, _ in dynamic_list)\n            if c_dist > worst_dist:\n                break\n            for nb_id in self.nodes[c_id].neighbors.get(layer, []):\n                if nb_id not in visited:\n                    visited.add(nb_id)\n                    nb_dist = self._distance(query, self.nodes[nb_id].vector)\n                    if nb_dist < worst_dist or len(dynamic_list) < ef:\n                        heapq.heappush(candidates, (nb_dist, nb_id))\n                        dynamic_list.append((nb_dist, nb_id))\n                        if len(dynamic_list) > ef:\n                            dynamic_list.remove(max(dynamic_list))\n\n        return [nid for _, nid in sorted(dynamic_list)[:ef]]\n\n    def add(self, vector: np.ndarray) -> int:\n        node_id = len(self.nodes)\n        node = HNSWNode(vector=vector)\n        self.nodes.append(node)\n        level = self._random_level()\n\n        if self.entry_point is None:\n            self.entry_point = node_id\n            self.top_layer = level\n            for l in range(level + 1):\n                node.neighbors[l] = []\n            return node_id\n\n        ep = self.entry_point\n        for l in range(self.top_layer, level, -1):\n            candidates = self._search_layer(vector, ep, ef=1, layer=l)\n            ep = candidates[0]\n\n        for l in range(min(level, self.top_layer) + 1):\n            neighbors = self._search_layer(vector, ep, ef=self.ef, layer=l)\n            M = self.M0 if l == 0 else self.M\n            neighbors = sorted(\n                neighbors,\n                key=lambda nid: self._distance(vector, self.nodes[nid].vector)\n            )[:M]\n            node.neighbors[l] = neighbors\n            for nb_id in neighbors:\n                nb_node = self.nodes[nb_id]\n                if l not in nb_node.neighbors:\n                    nb_node.neighbors[l] = []\n                nb_node.neighbors[l].append(node_id)\n                if len(nb_node.neighbors[l]) > M:\n                    nb_node.neighbors[l] = sorted(\n                        nb_node.neighbors[l],\n                        key=lambda nid: self._distance(nb_node.vector, self.nodes[nid].vector)\n                    )[:M]\n            ep = neighbors[0] if neighbors else ep\n\n        if level > self.top_layer:\n            self.top_layer = level\n            self.entry_point = node_id\n        return node_id\n\n    def search(self, query: np.ndarray, k: int = 10, ef: int = 50) -> list[tuple[float, int]]:\n        if self.entry_point is None:\n            return []\n        ep = self.entry_point\n        for l in range(self.top_layer, 0, -1):\n            candidates = self._search_layer(query, ep, ef=1, layer=l)\n            ep = candidates[0]\n        candidates = self._search_layer(query, ep, ef=max(ef, k), layer=0)\n        results = sorted(candidates, key=lambda nid: self._distance(query, self.nodes[nid].vector))\n        return [(self._distance(query, self.nodes[nid].vector), nid) for nid in results[:k]]\npython\nimport numpy as np\nfrom hnsw import HNSWIndex\n\n# Build an index with 128-dimensional vectors\nindex = HNSWIndex(dim=128, M=16, ef_construction=200)\n\n# Insert 10k random vectors (documents, embeddings, etc.)\nnp.random.seed(42)\nn = 10_000\nvectors = np.random.randn(n, 128).astype(np.float32)\n\nprint(\"Building index...\")\nfor i, vec in enumerate(vectors):\n    index.add(vec)\n    if (i + 1) % 1000 == 0:\n        print(f\"  {i + 1}/{n} vectors inserted\")\n\n# Query with a random vector\nquery = np.random.randn(128).astype(np.float32)\nresults = index.search(query, k=5, ef=50)\n\nprint(\"\\nTop 5 nearest neighbors (HNSW):\")\nfor dist, node_id in results:\n    print(f\"  node_id={node_id}, sq_distance={dist:.4f}\")\n\n# Verify against brute force\nall_dists = np.sum((vectors - query) ** 2, axis=1)\nbrute_top5 = sorted(enumerate(all_dists), key=lambda x: x[1])[:5]\nprint(\"\\nBrute-force top 5:\")\nfor nid, dist in brute_top5:\n    print(f\"  node_id={nid}, sq_distance={dist:.4f}\")\n```\n\nOn a 10k-vector dataset, you'll typically see 4 or 5 out of 5 results overlap between HNSW and brute force. That's the \"approximate\" in ANN: for most real workloads, occasionally missing one true neighbor is an acceptable trade-off for a 50x–100x query speedup.\n\n**M — max connections per node per layer.** Higher M means better recall and a denser graph, at the cost of more memory and slower inserts. Start with 16. Go up to 32 if your recall consistently falls below 90%.\n\n**ef_construction — candidate pool size during index build.** Larger values produce a better-connected graph but slow down insertion. 200 is a solid default; lower it to 100 if insertion throughput matters more than recall.\n\n**ef (search-time) — candidate pool size at query time.** This is the most important lever for production tuning because you can adjust it without rebuilding the index. Setting `ef = k` gives the fastest queries; `ef = 500` approaches brute-force recall. Plotting ef vs. recall typically reveals a knee around ef = 50–100 for most datasets — that's your sweet spot.\n\nUnderstanding these parameters matters the moment your AI-powered search starts returning wrong results. Before blaming your embedding model, check `ef` and whether your index was built with sufficient `ef_construction`. It's the same diagnostic discipline that applies across security tooling — knowing the internals saves time when something breaks. The [security hardening checklists](https://ayinedjimi-consultants.fr/checklists) we publish follow the same philosophy: understand the mechanism, then apply the control.\n\nThis implementation deliberately omits several things you'd need in production:\n\n`np.save` for vectors, JSON or msgpack for the neighbor maps).\nFor any production workload, use [hnswlib](https://github.com/nmslib/hnswlib) directly (`pip install hnswlib`) or a vector database that wraps it. The API is almost identical to what's shown above, and you get persistence, filtering, and proper concurrency for free.\n\nHNSW is not magic — it's a graph with controlled connectivity. The layered structure lets queries skip large sections of the search space, and the `ef` parameter gives you a runtime dial between speed and accuracy.\n\nBuilding a stripped-down version reveals what every production vector database is doing under the hood. That understanding pays off when you're debugging recall issues at 3am — and it's faster than re-reading library documentation when you already know what the parameters mean.\n\n*I run [AYI NEDJIMI Consultants](https://ayinedjimi-consultants.fr), a cybersecurity consulting firm. We publish [free security hardening checklists](https://ayinedjimi-consultants.fr/checklists) — PDF and Excel.*", "url": "https://wpnews.pro/news/building-a-vector-search-engine-from-scratch-with-hnsw-in-python", "canonical_source": "https://dev.to/ayinedjimi-consultants/building-a-vector-search-engine-from-scratch-with-hnsw-in-python-18hj", "published_at": "2026-09-09 10:03:05+00:00", "updated_at": "2026-09-09 10:39:02.125877+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/building-a-vector-search-engine-from-scratch-with-hnsw-in-python", "markdown": "https://wpnews.pro/news/building-a-vector-search-engine-from-scratch-with-hnsw-in-python.md", "text": "https://wpnews.pro/news/building-a-vector-search-engine-from-scratch-with-hnsw-in-python.txt", "jsonld": "https://wpnews.pro/news/building-a-vector-search-engine-from-scratch-with-hnsw-in-python.jsonld"}}