{"slug": "i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages", "title": "I counted my 4,323 listings as companies. The biggest one is GitHub Pages.", "summary": "A developer who runs an AI tools directory audited the 4,323 rows in its database and found only 3,317 distinct registrable domains, with 1,200 listings (27.7%) sharing a domain with at least one sibling entry. GitHub emerged as the largest single \"vendor\" at 184 listings once github.com and github.io were merged, ahead of Google and OpenAI, and 337 rows (7.8%) pointed at platform or affiliate-network hosts rather than the vendor's own site, including Adobe Firefly and CapCut.", "body_md": "I run an AI tools directory. The homepage says \"4,300+ tools\" and the number is real: my database has exactly 4,323 rows in the `Tool` table. What I had never done is ask what one row *is*. Yesterday I found that the rank badges on those pages had drifted, so today I went one level down and measured the unit itself.\n\nThe answer: 4,323 listings, 3,317 companies, and the single largest \"vendor\" in my directory is a hosting platform I don't own.\n\nOne row, one name, one slug, one `websiteUrl`, one `pricingType`. The \"Visit Website\" button on a tool page points at `/go/{slug}`, which records a click and then 302s to whatever is in `websiteUrl`:\n\n``` php\nGET /go/descript                   -> 302 https://get.descript.com/sx9ahgvct8j9\nGET /go/underlord-by-descript-ai-video -> 302 https://get.descript.com/sx9ahgvct8j9\n```\n\nThose are two different listings, two different pages, two different slugs. The second one is a feature of the first one. That is the only byte-for-byte `websiteUrl` collision in the whole table - 4,323 rows, 4,322 distinct URLs - so I initially read that as good news. It isn't. Uniqueness at the string level just means the duplicates are one level up: they point at the same *company*, just at different pages of it.\n\nFor every row I took the hostname of `websiteUrl`, dropped a leading `www.`, and then collapsed to the registrable domain with the crudest rule that works: the last two labels, with a short exception list for `co.uk` / `com.cn` style suffixes. No real public-suffix library, no brand knowledge, no verification that two domains belong to the same legal entity. So everything below is a **lower bound on how clustered my catalog is**, not a clean company count - `deepmind.google` and `blog.google` count as two domains here even though they are one company.\n\nNothing is unparseable: 4,323 URLs, 0 failures.\n\n| measure | value | \n|---|---|\n| rows | 4,323 | \n| distinct hostnames | 3,576 | \n| distinct registrable domains | 3,317 | \n| rows that share a domain with at least one sibling row | 1,200 (27.7%) | \n| domains involved in that sharing | 194 | \n\nSo better than one row in four sits behind a domain that already appears elsewhere on my site. The largest clusters are not the things you'd expect from a \"curated list of AI companies\":\n\n```\ngithub.io      103     huggingface.co   78     sjv.io         28\ngithub.com      81     openai.com       50     pxf.io         24\n```\n\nMerge `github.com` and `github.io` and GitHub is my single biggest vendor at 184 listings - more than Google (64 rows on `google.com`, plus 24 on `deepmind.google` and 19 on `blog.google`) and more than OpenAI (50). Not because GitHub has that many products in my catalog, but because a project page on someone's GitHub Pages and a repository on GitHub's own domain both get treated by my crawler as \"the tool's website\".\n\nIf I filter rows whose host is a platform or an affiliate network (`github.io`, `github.com`, `huggingface.co`, `sjv.io`, `pxf.io`, `prf.hn`, `partnerlinks.io`), I get 337 rows - 7.8% of the catalog - where the address I present as \"official website\" is not the vendor's address. Live checks, done today:\n\n``` php\n/go/adobe-firefly      -> 302 https://prf.hn/l/EJXxwOp\n/go/capcut             -> 302 https://capcutaffiliateprogram.pxf.io/9LPy7W\n/go/yue2-ai-model-music-> 302 https://map-yue2.github.io/\n/go/trellis-2-microsoft-> 302 https://microsoft.github.io/TRELLIS.2\n```\n\nAdobe Podcast, Adobe Firefly, Looka, n8n, Klaviyo: their entry in my \"official website\" column is a link belonging to an affiliate network, inherited from the directory I crawl. My own affiliate store is empty - I have not signed up for a single program - so when a reader clicks those, some third party gets attributed for the visit and I get a click in my analytics and nothing else. A looser rule (network host *or* a `fp_ref=` / `ref=aixploria=` query param) catches 69 rows; 62 of them on the network's own domain.\n\nAnd 303 of those 337 rows are labelled `free`, which is the filter most of my readers use. That is the part I'd fix first.\n\nI checked the text too. Zero rows have byte-identical `description` values - but if I replace the tool's own name with a placeholder and the \"ranks #N in the X category\" clause with another, 4,323 descriptions collapse into 4,205 shapes. The largest shape is shared by 18 rows, all sitting in a category literally named `This*DoesNotExist` (21 tools). Their pages open with the same sentence, differing only in the product name.\n\nThe one sentence I'd delete before any of that: 2,780 rows (64.3%) end their description with \"Common tags for this tool include Latest AI.\" \"Latest AI\" is not a tag - it is the name of the source site's chronological feed, and it is attached in my database to 2,860 tools, making it the most common tag I have. 1,446 rows print no tag sentence at all, and 824 have zero tag rows. So two thirds of my catalog tells the reader, in a field labelled \"tags\", that the item appeared recently in somebody else's list.\n\nMy content pipeline today produced 5 more written pages out of the 130 still templated in my top-150-by-traffic queue, which is the number I actually report. But the honest framing for the catalog itself is now: roughly 3,300 destinations, 4,300 listings, about 340 of them pointing at infrastructure rather than a company, and one page that is genuinely a duplicate of another page. The free filter shows 2,198 tools; behind those 2,198 rows there are 1,538 registrable domains, and 101 of them are GitHub Pages sites.\n\nI have not deduplicated anything, and I am not going to pretend a merge is free: \"Underlord by Descript\" might be a listing a reader searches for on purpose. The fix I'd defend is a canonical vendor key stored next to the URL, so \"same company\" is a fact I can query instead of something a reader has to notice. Until then I won't write \"4,323 tools\" and mean 4,323 products - I mean 4,323 rows, and the [free tools list](https://aiverse.mindutil.com/free-ai-tools?utm_source=devto&utm_medium=social&utm_campaign=topic) is the page where that difference shows up most.", "url": "https://wpnews.pro/news/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages", "canonical_source": "https://dev.to/li_yi_dcec96bf2647dec1b7e/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages-136m", "published_at": "2026-11-22 00:40:38+00:00", "updated_at": "2026-11-22 00:41:36.358047+00:00", "lang": "en", "topics": ["ai-tools", "generative-engine-optimization"], "entities": ["GitHub", "GitHub Pages", "Hugging Face", "OpenAI", "Google", "DeepMind", "Adobe", "Microsoft"], "also_reported_by": [], "takeaway": "Collapsing websiteUrl to its registrable domain showed 1,200 of 4,323 directory listings (27.7%) share a domain with a sibling row, and 337 (7.8%) resolve to platform or affiliate hosts instead of the vendor's own site.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages", "markdown": "https://wpnews.pro/news/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages.md", "text": "https://wpnews.pro/news/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages.txt", "jsonld": "https://wpnews.pro/news/i-counted-my-4323-listings-as-companies-the-biggest-one-is-github-pages.jsonld"}}