{"slug": "making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare", "title": "Making 1,200 static pages readable by AI agents: MCP tools + NLWeb on Cloudflare Pages Functions", "summary": "A developer behind OneFindMe, a free AliExpress product search available in 21 languages, built an MCP-based guide retrieval layer that makes roughly 1,200 static pages readable by AI agents without a database or vector store. The implementation generates a 1,213-entry, ~900 KB JavaScript index from sitemap.xml and existing page metadata, adds keyword-scored search_guides and get_guide tools that convert live pages to Markdown on the fly, and exposes an NLWeb-compatible /ask endpoint on Cloudflare Pages Functions.", "body_md": "I run [OneFindMe](https://onefindme.com/en/), a free AliExpress product search that works in 21 languages. Over the past months the site grew a library of about 1,200 static pages: size guides, import-tax pages per country, buying guides, home-problem fixes, all written per language.\n\nThe site already had an [MCP](https://modelcontextprotocol.io/) server at `https://onefindme.com/mcp` with a `search_products` tool. It is listed in the official MCP registry and in about 30 directories. But I noticed a gap: an assistant connected to it could search products, yet it could not read a single one of our guides. When someone asked \"13x4 or 13x6 lace wig?\", the agent had no way to reach the page that answers exactly that.\n\nHere is how I closed the gap without a database, a vector store or a second service.\n\nPages Functions have no file system at runtime, so the index is generated into a JS module that the function imports. A small Python script walks `sitemap.xml`, skips `noindex` pages and the search home pages, and keeps only what each page already publishes about itself:\n\n```\nout.append({\"u\": path, \"l\": lang, \"t\": h1, \"d\": meta_description,\n            \"h\": \" | \".join(h2_headings), \"q\": \" | \".join(faq_questions), \"m\": date_modified})\n```\n\nThe FAQ questions come from the page's own `FAQPage` JSON-LD. Result: 1,213 entries, about 900 KB, written to `functions/_guides.js` as `export default [...]`. The leading underscore keeps it from becoming a route.\n\nI deliberately did **not** store the full text. More on that below.\n\n`search_guides` and `get_guide`\nSearch is plain keyword scoring with field weights. No embeddings: the titles and FAQ questions are written in the reader's language, so they already contain the words people type.\n\n``` js\nconst m = 4 * hit(e.t, w) + 2 * hit(e.u, w) + 2 * hit(e.h, w) + 2 * hit(e.q, w) + 1.5 * hit(e.d, w);\n// ...\ns *= matched / qs.length;                 // pages matching every word rank first\nif (want && g.l === want) s *= 1.6;       // requested language\nelse if (!want && g.l === \"en\") s *= 1.25; // Latin script, no language given: English first\n```\n\n`hit()` accepts exact tokens and, for words of four letters or more, prefix matches, which covers most plurals across European languages. Tokenising with `\\p{L}\\p{N}` and stripping combining marks after `NFD` lets \"perucke\" match \"Perücke\" and keeps Hebrew, Arabic, Japanese and Korean intact. When the query is in a non-Latin script, the language is detected from the script itself.\n\nOne rule is product-specific: an Arabic query never gets Hebrew pages, and the reverse. That is a two-line filter, and it matters more to the readers than any ranking tweak.\n\n`get_guide` takes a URL from the search results and returns the page as Markdown. This is why the index holds no text: the function reads the live page through the static-asset binding and converts it on the fly.\n\n``` js\nlet res = await env.ASSETS.fetch(new Request(page));\nif (res.status >= 300 && res.status < 400 && res.headers.get(\"Location\"))\n  res = await env.ASSETS.fetch(new Request(new URL(res.headers.get(\"Location\"), page)));\nlet md = htmlToMarkdown(await res.text(), url);\n```\n\nThe same `htmlToMarkdown` helper already serves `Accept: text/markdown` requests in the site middleware, so agents that ask for Markdown get it on every page. The answer can never be staler than the page.\n\n[NLWeb](https://github.com/nlweb-ai/NLWeb) is Microsoft's open protocol for asking a website a question in natural language. The minimum is an `/ask` endpoint returning schema.org-shaped results. With the search already written, it is about 30 lines:\n\n``` js\nreturn json({ query_id: crypto.randomUUID(), results: r.results.map((x) => ({\n  url: x.url, name: x.title, site: \"onefindme.com\", score: x.score, description: x.description,\n  schema_object: { \"@type\": \"Article\", headline: x.title, url: x.url, inLanguage: x.language } })) });\n```\n\nIt is advertised with `Link: </ask>; rel=\"nlweb\"` on the home pages, next to the existing `api-catalog`, `service-desc` (OpenAPI) and MCP server-card relations. Try it: `https://onefindme.com/ask?query=dyson+v15`.\n\n[GitMCP](https://gitmcp.io/) turns any public GitHub repo into an MCP server, and it reads `llms.txt` first. Our archive repo had stopped updating, so a coding agent asking GitMCP about OneFindMe got a stale README. One commit fixed it: an `llms.txt` plus one Markdown index per language (title, URL, summary, sections, questions). I kept it to a single commit through the Git Data API (`POST /git/trees` with inline contents, then a commit and a ref update): five API calls for 23 files.\n\nThe same index is also an open dataset (CC BY 4.0) on [Hugging Face](https://huggingface.co/datasets/fufu1976/onefindme-shopping-guides-21-languages) and [Kaggle](https://www.kaggle.com/datasets/onefindme/onefindme-shopping-guides-21-languages), with the FAQ answers included and no product links.\n\nConnect it to any client that supports remote MCP servers:\n\n```\n{ \"mcpServers\": { \"onefindme\": { \"type\": \"http\", \"url\": \"https://onefindme.com/mcp\" } } }\n```\n\n*Disclosure: OneFindMe is free to use; product links on the site and in `search_products` results are affiliate links, at no extra cost to the buyer. The guide tools return no product links of their own.*", "url": "https://wpnews.pro/news/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare", "canonical_source": "https://dev.to/ohadfarkash/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare-pages-functions-oep", "published_at": "2026-11-20 06:36:24+00:00", "updated_at": "2026-11-20 06:42:56.236710+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-search"], "entities": ["OneFindMe", "Cloudflare Pages Functions", "Model Context Protocol", "NLWeb", "Microsoft", "AliExpress"], "also_reported_by": [], "takeaway": "Serving guide content to agents via an MCP search tool plus on-the-fly HTML-to-Markdown conversion avoids a vector store and keeps answers no staler than the live page, with a 1,213-entry index generated from sitemap.xml and existing JSON-LD metadata.", "recommendation": {"advice": "Expose site content to agents with a keyword-scored MCP search tool and a get_guide tool that converts live pages to Markdown, rather than embedding full text in a static index.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare", "markdown": "https://wpnews.pro/news/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare.md", "text": "https://wpnews.pro/news/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare.txt", "jsonld": "https://wpnews.pro/news/making-1200-static-pages-readable-by-ai-agents-mcp-tools-nlweb-on-cloudflare.jsonld"}}