{"slug": "my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa", "title": "My URL-to-Markdown API served a nav menu as an 'article' — detecting JS-rendered SPA shell pages", "summary": "A developer hardened a URL-to-Markdown extraction API after auditing request logs and finding that roughly one in six URLs returned under 60 words of valid-looking Markdown — navigation menus and footers from JavaScript-rendered SPA shell pages — silently poisoning downstream RAG pipelines. The fix adds a low_content flag with word count when extraction falls below a threshold or shows a high boilerplate ratio, a fallback ladder to og:description and meta description labeled as summaries, and a CSS selector escape hatch, while deliberately skipping headless Chromium due to its memory and latency cost. Useful-content yield on a 500-URL sample rose from about 83% to 96% after the flag shipped.", "body_md": "My URL-to-Markdown API began as a scrappy readability pass: fetch a URL, strip nav/ads/sidebars, hand back clean Markdown. It worked beautifully on every blog, docs site, and news article I tested. Then I audited my own request logs and found the failure mode nobody complains about — because it doesn't error.\n\nAbout 1 in 6 URLs came back under 60 words. Not errors — 200 OK, valid Markdown, technically correct output. The \"content\" was `Skip to content / Home / About / Sign in` plus a footer. I had been silently embedding navigation menus into people's RAG pipelines. Valid output, zero information, and worse than an exception because nothing downstream noticed.\n\nThe cause: SPA shells. The server returns real HTML with a mostly-empty root div; the article is assembled client-side. My extraction pass had no actual content to find, so it carved boilerplate into something shaped like an article.\n\nWhat I changed, in order of impact:\n\n**Extraction floor.** If word count falls below a threshold or the boilerplate ratio is too high, return an explicit `low_content` flag with the word count instead of confident garbage. Biggest single win — it made failure visible.\n\n**Fallback ladder.** On a shell page, try `og:description` and meta description, clearly labeled as a summary. An honest 40-word summary beats 40 words of menu pretending to be prose.\n\n**Escape hatch.** A CSS selector option, so callers who know a site can target its article container directly. Print views and RSS endpoints often carry the full text where the main page doesn't.\n\n**What I deliberately didn't do:** add headless Chromium. I benchmarked it — hundreds of MB of RAM per concurrent render and p95 latency from ~800ms to 6s+. For my traffic, loud cheap failure beat expensive half-rendering. I'll revisit if shell pages ever become the majority.\n\nAfter the flag shipped, my retry logic routes shell pages elsewhere instead of embedding empty strings, and useful-content yield on a 500-URL sample went from roughly 83% to 96%. Honest limit: pure client-rendered pages with no SSR and no meta tags will never extract well. The point isn't extracting more — it's never lying about what came back.\n\nI ended up packaging the hardened version as my URL-to-Markdown API: [https://x402.freeq.one/tools/markdown.html](https://x402.freeq.one/tools/markdown.html)", "url": "https://wpnews.pro/news/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa", "canonical_source": "https://dev.to/imapphelp/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa-shell-pages-5ao3", "published_at": "2026-10-01 11:30:19+00:00", "updated_at": "2026-10-01 11:44:25.202791+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure", "natural-language-processing"], "entities": ["x402.freeq.one"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa", "markdown": "https://wpnews.pro/news/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa.md", "text": "https://wpnews.pro/news/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa.txt", "jsonld": "https://wpnews.pro/news/my-url-to-markdown-api-served-a-nav-menu-as-an-article-detecting-js-rendered-spa.jsonld"}}