If you find value in my newsletter and want to support the work that goes into it consider —> [☕Buy Me a Coffee☕]💡 Want more from me? [SEO Strategy Course] | [Build Your Own SEO Tools (with Python & Agentic AI)] | [Premium SEO Mastermind] Alternatively you can become a paid subscriber:
Intro #
So you’ve heard… I’m building a search engine from scratch. Here’s the tea: I want to build a small search engine for all my content specifically, so you can use it to search for my input about a specific topic.
I’ll be sharing my journey of building this in my newsletter. Every time a new milestone is done, you’ll read about it here first!
I think is is a fun project but also building a search engine is far from simple. One way to simplify the process is by being very specific like in my case, the goal is my content specifically, although there’s still details that go into this.
So how to build a search engine, eh?
Components of a search engine #
We are not re-investing the wheel here by any means, but there are design decisions that allow for creativity.
A search engine is primarily made of 4 components:
- A crawler: of course we need some time a of a system to go find the URLs for our search engine and define the rules this crawler will follow.
- Indexer: we also need a system to process the content on those URLs and produce something that’s simpler and searchable and store it in a database that we may call our “search index”
- Query processor and ranking system: these are two sides of the same coin. What happens to the query when a user searches? how do we interpret it? for example finding the user intent behind the query, so we can match this query against our search index, that will be sorted and filtered based on the “ranking system” we have built.
- Search Interface (SERP) : our search results page, where user enters their search and get back results.So this is generally speaking the 4 components you need to build a search engine. Now let’s talk about my search engine! I don’t have a name for it yet but I’m open for ideas.
My search engine: the crawler #
I started planning with Claude Code. There are few design decisions and the steps I chose to build my crawler:
-
The crawl depth: this is a small search engine, that will rely on two sitemaps for seoriddler.com and sara-taher.com. It will also crawl any specific URLs (not the domains) I have linked to in my content. But that’s as deep as it will go. So the maximum crawl depth is 1.
-
Pre-fetch URL cleanup: before a URL is fetched, it get normalized. This means the URL gets reduced to it’s canonical format by removing tracking parameters for example.
-
Link filtering: I have decided to exclude links to .pdf, .jpg, .zip, .css, .js, etc. from my crawls and only fetch html pages. However, those non-html links are stored and labelled as such.
-
Storage architecture: For each URL we decide to fetch, we store two things:information about the URL and theactual HTML of the page .Information about the URL includes the URL itself, page title, description, crawl depth, crawling errors, status code, and other metadata. This information is stored in an SQLite database. The actual HTML of the page is stored separately as a compressed gzip file . The name and location of that gzip file are stored in the SQLite database entry for the corresponding URL, so we can always connect the URL’s metadata to its raw HTML.
-
Link Graph: I’m also storing which page links to which other page, the anchor text used for the link, and, if we didn’t fetch the target URL, the reason it was skipped. Together, these rows form a graph of how the pages are connected. Right now that’s 2,830 rows from 744 pages. Here’s how this looks in the DB. I’m using https://sqliteviewer.app/ to view the database.
-
Crawl controls: I also decided, with the help of Claude Code, that we would not crawl pages sequentially, because that could take a long time. Instead, we crawl multiple URLs concurrently, with a maximum of 8 requests running at the same time. We also make sure that we do not send multiple requests to the same domain at the same time, and we maintain at least a 1-second gap between requests to the same domain.
-
Respecting robots.txt: because I’m polite 😂 so before crawling a website, we check its robots.txt file to see which pages the site allows crawlers to access. We fetch and cache robots.txt once per domain, then check it before each page request. If a URL is blocked by robots.txt, we don’t request it and record it as blocked_by_robots.If we can’t access a site’s robots.txt, our crawler currently continues rather than blocking the site. This is a deliberate choice for my small, low-volume personal crawler. The crawler also identifies itself with a User-Agent and provides a contact email. If you see sara-taher-seo-crawler as user-agent, that’s me 😉
-
Errors: because every crawl will have errors. My idea is a failed page should not be left out from the dataset, nor it should interrupt the crawl. Every URL we attempt to crawl gets a record in the database, whether the crawl succeeds or fails. If something goes wrong, such as a robots.txt block, HTTP error, network error, or extraction error, we record the reason in the
errorfield instead of silently dropping the URL. -
Stack choices: I chose a relatively simple stack that gives me everything I need for this project. Here’s everything:
Limitations #
Ok so I built my crawler and I ran it. I have the database ready for the next system “indexation” but there is something that I want to include but won’t right now as it will complicate things:
- I want to include my video content in this database such that when you use the search engine to search for a topic, you can get videos back in the results. While video pages are included in the crawl, I didn’t download the transcripts for example.
- I also want to include content I have written on third part websites, this is not included in the current version of this crawler.
- Another thing is, how often should this crawler re-crawl the sitemaps to discover URLs? right now, it’s a one and done thing and to update the database with new URLs I have to manually run a crawl.
So these are few things that come to mind when working on this.
And That’s a Wrap (Almost 😄) #
I really think this is a fun project and I will be recommending it moving forward to anyone who wants to learn SEO. We’ve always recommended things like build your own website, or audit online websites, but building a search engine should definitely come at the top of SEO learning tactics!
I know this is way technical that some of my readers would like, but it’s also a good change of pace, and it may open up ideas and inspire you to build things of your own. If you ever want to vibe code, reach out I’m happy to help!
That’s that for today folks and see you in the next newsletter!
Support the Riddler! #
- Sign up for my newsletter if you’re not already. (Pssst, you can also become a paid subscriber)
- Share the newsletter and invite your friends to signup. Help me reach 2k signups on Substack by end of 2026 please 🙂
- Provide feedback on how I can make this newsletter better!!!
- Buy me coffee .
- If you’re an SEO tool or an SEO service provider, consider sponsoring my newsletter. I’m also open to other partnership ideas as well.
Disclaimer: LLMs were used to assist in wording and phrasing this blog.