{"slug": "building-a-search-engine-for-6000-personal-blogs", "title": "Building a Search Engine for 6,000 Personal Blogs", "summary": "Semsearch, an embedding-based search engine bootstrapped from indieblog.page's blog list, launched at semsearch.blog with roughly 6,000 unique blog sites, 154k pages, and 1.4B tokens, ranking results by semantic relevance alone without link popularity or SEO signals. Built by developer yikerman using FastAPI, Postgres with VectorChord, Trafilatura, and Qwen3-Embedding-4B, the project aims to make indie blogs more discoverable amid declining Google traffic and the rise of AI summarization. The source code is available at github.com/yikerman/semantic-search.", "body_md": "TL;DR: Semsearch is an embedding-based search engine whose corpus is bootstrapped from\n[indieblog.page's blog list](https://indieblog.page/faq#data). It aims to make indie\nblogs more discoverable. Results are ranked by semantic relevance alone, without link popularity\nor SEO-derived signals. At the time of writing, the live instance is up at [semsearch.blog](https://semsearch.blog)\nwith [roughly 6,000 unique blog sites, 154k pages, and 1.4B tokens](https://archive.today/2026.07.29-180627/https://semsearch.blog/status).\nThe source code is at [github.com/yikerman/semantic-search](https://github.com/yikerman/semantic-search).\n\nThe idea for this project emerged when I read about [The Great Blogging Collapse](https://danielstanica.com/posts/Great-Blogging-Collapse)\n([HN discussion](https://news.ycombinator.com/item?id=48758802)). While the post itself\nhas a couple of problems, its broader point resonated with me as a blogger: there are fewer ways\nto promote small and individual blogs. In the past, many visits began when Google led users to\nyour blog. But Google, as shown in *The Great Blogging Collapse*, is doing so less and less\nfor various reasons, such as the rise of AI summarization that completely eliminates visits to\nother sites.\n\nThis aligns with other sources: [people (including me) prefer searching on Reddit](https://dkb.blog/p/google-search-is-dying)\nsince Google's top results are cluttered by ads and SEO-optimized content farms. For example, a\nGoogle search for the old [Postgres vs MySQL](https://www.google.com/search?q=postgres+vs+mysql) debate\nreturns mostly AWS and IBM articles that are of little use. (Interestingly, at the time\nof writing, it seems that Google has boosted Reddit's ranking and put it among the top results.)\n\nWhile the debate has largely settled, this would have been the perfect scenario to display some\nold-n-wise engineer writing about his experience with both in production, even if his webpage is\nstyled uglier than [ The Motherfucking Website](https://motherfuckingwebsite.com/).\n\nRecent embedding models have made it practical to search by meaning rather than exact keywords (which also requires non-trivial tricks to work effectively), and vector databases have also proven to be practical and easy to use. While most existing embedding-based search systems are used to provide agentic LLMs with retrieval capabilities, I want to explore the possibility of using purely embedding-based search to help people discover indie blogs.\n\nI used FastAPI for the web app and [Postgres + VectorChord](https://hub.docker.com/r/tensorchord/vchord-suite)\nfor the database. The set of blogs is bootstrapped from [indieblog.page's blog list](https://indieblog.page/faq#data).\nFor each page discovered through either an RSS feed or a sitemap, I used [Trafilatura](https://github.com/adbar/trafilatura)\nto extract the text content, then chunked it into overlapping 384-token segments. For the\nembedding task, I chose [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B)\nas the embedding model, since it topped the [MTEB leaderboard](http://archive.today/2026.08.01-143048/https://mteb-leaderboard.hf.space/models/Qwen/Qwen3-Embedding-4B)\nat the time of writing, is open source, and is relatively cheap to run. Every chunk is stored in\nthe database as a 2,560-dimensional halfvec.\n\nAt query time, Semsearch embeds the query once and runs designated retrievers. By default, only\nthe dense retriever, based on the cosine similarity between the query embedding and chunk embeddings,\nis used, and the top 64 entries are returned. There is also an optional BM25 [1]\nretriever for both evaluation and combining results through RRF\n\nDeployment is a bit tricky, since the [HNSW index](https://github.com/pgvector/pgvector#hnsw)\ntakes about 30 GB of RAM to load at the current scale, and VPS costs scale a lot with RAM.\nFor reference, an `r6a.xlarge`\n\ninstance on AWS costs $0.252/hr. I ended up getting a\ndedicated i7-6700 box from Hetzner's server auction for €55/mo. While the CPU is crappy by modern\nstandards, it comes with 64 GB RAM and 2 × 512 GB NVMe SSDs, an absolute bargain in the RAMageddon era.\n\nI used [OpenRouter's API](https://openrouter.ai/qwen/qwen3-embedding-4b) to run the\nembedding model, since it is relatively cheap, namely $0.02/M tokens. Indexing the current\ncorpus cost a modest $28.60, which is not bad. The latency and throughput are not\nideal, though: I get roughly 6,000 tokens/sec on OpenRouter, and it bottlenecked the initial\nindexing process.\n\nSo how do the results look? Going back to the [earlier query of Postgres vs MySQL](https://semsearch.blog/?q=Postgres+vs+MySQL&lang=en)\n([archived](http://archive.today/2026.08.03-083542/https://semsearch.blog/?q=Postgres+vs+MySQL&lang=en)),\nthe results are mixed and sit somewhere between useful, fun, and nonsense.\n\nAfter fiddling around, I found that it gives more insightful results when my query is an assertion a blogger might make in a post, and that longer queries resembling a sentence work better. Some extra interesting examples I tried:\n\nIn conclusion, the search definitely works and offers a more personal side of the Internet from time to time. As expected, quality heavily depends on the corpus, while the search can only suggest relevance. The frontend can be confusing sometimes, since it only displays the title and the first few lines of a post, and the user may have a hard time locating the relevant chunk in the original post again.\n\nThis is still an early proof of concept, and some quirks exist. A notable one is that the\nembedding model is not very robust to slightly nonstandard phrasing. For example, the query\n[\"framework laptop review\"](https://semsearch.blog/?q=framework+laptop+review&lang=en)\n([archived](http://archive.today/2026.08.03-121718/https://semsearch.blog/?q=framework+laptop+review&lang=en))\nreturns gibberish results, while the properly capitalized query\n[\"Framework Laptop review\"](https://semsearch.blog/?q=Framework+Laptop+review&lang=en)\n([archived](http://archive.today/2026.08.03-122117/https://semsearch.blog/?q=Framework+Laptop+review&lang=en))\nreturns relevant results. The website would also benefit from displaying relevant chunks in the\nsearch results as discussed above, and some community voting mechanism may be useful to filter\nout low-quality content (though it will introduce other problems).\n\nI plan to build a homelab and get a 3090 for self-hosting and further experiments, namely:", "url": "https://wpnews.pro/news/building-a-search-engine-for-6000-personal-blogs", "canonical_source": "https://ycao.net/posts/creating-semsearch.html", "published_at": "2026-08-03 00:00:00+00:00", "updated_at": "2026-08-03 13:03:05.182119+00:00", "lang": "en", "topics": ["ai-products", "ai-infrastructure", "machine-learning"], "entities": ["Semsearch", "indieblog.page", "yikerman", "FastAPI", "VectorChord", "Trafilatura", "Qwen3-Embedding-4B", "Hetzner"], "alternates": {"html": "https://wpnews.pro/news/building-a-search-engine-for-6000-personal-blogs", "markdown": "https://wpnews.pro/news/building-a-search-engine-for-6000-personal-blogs.md", "text": "https://wpnews.pro/news/building-a-search-engine-for-6000-personal-blogs.txt", "jsonld": "https://wpnews.pro/news/building-a-search-engine-for-6000-personal-blogs.jsonld"}}