How to Scrape JavaScript-Heavy Pages for AI Agents

A developer published a TypeScript guide for scraping JavaScript-heavy pages on behalf of AI agents, using a two-tier approach that fetches static HTML first and only falls back to headless Chromium rendering when the result looks thin. The method relies on AgentSearch's Web Extract and Web Render tools on the Pocket Agentic Portal, each priced at $0.005 in USDC via x402 on Base or MPP on Tempo with no signup or API key. The guide detects client-rendered pages heuristically by flagging markdown under 400 characters or text containing no-JavaScript notices.

Point a plain HTTP fetcher at a React or Vue single-page app and you often get back a shell: a title, an empty <div id="root" , and a line asking you to enable JavaScript. Hand that to an LLM and it will either say the page is empty or, worse, answer from memory. The fix is not to run a headless browser on every URL. It is to fetch cheaply first, notice when the result is thin, and render only those pages. This guide builds that fallback in TypeScript on the AgentSearch tools on the Pocket Agentic Portal: Web Extract https://agentsearchhq.com/extract?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents for static HTML and Web Render https://agentsearchhq.com/render?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents for pages that need a browser. Each call is $0.005 in USDC, paid over x402 on Base or MPP on Tempo, with no signup and no API key. A server-rendered page ships its text in the HTML. A client-rendered page ships JavaScript that builds the text after load. Extract reads the HTML it receives and converts it to markdown, so on a client-rendered page it faithfully converts the empty template. Render runs the page in headless Chromium, then returns what the browser ended up with: markdown , text , html , title , links , and an optional screenshot. It is a real browser, so it is the tool for JS-heavy pages. It is also the same $0.005 per call as Extract, which is why it works as a fallback: one thin Extract plus one Render is $0.01 for that page, and pages that extract cleanly never pay for the second call. This is the same wrapper as in Error Handling for AI Agent Web Tools https://agentsearchhq.com/blog/error-handling-ai-agent-web-tools?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents , trimmed. payFetch is the x402 client from Pay-per-call APIs for AI agents https://agentsearchhq.com/blog/x402-pay-per-call-api-for-ai-agents?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents , with a spend cap installed. js // js-pages.ts const PORTAL = "https://agent.pocket.network/v1"; declare const payFetch: typeof fetch; // from the x402 tutorial type PortalFailure = { kind: "portal"; status: number; code: string; retryable: boolean }; type Delivered = { kind: "data"; data: Record<string, unknown }; async function callService serviceId: string, path: string, body: unknown : Promise<PortalFailure | Delivered { const res = await payFetch ${PORTAL}/${serviceId}${path} , { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify body , } ; const json = await res.json ; if res.ok { const err = json.error ?? {}; return { kind: "portal", status: res.status, code: String err.code ?? HTTP ${res.status} , retryable: err.retryable === true, }; } return { kind: "data", data: json.data }; } Extract does not tell you "this page needed JavaScript." You infer it from what came back. Two cheap signals cover most cases: the markdown is very short, or it contains the page's own no-JavaScript notice. type PageData = { url?: string; title?: string | null; markdown?: string; error?: { code: string; retryable?: boolean } | null; }; const THIN = 400; // local heuristic, tune for your sources const JS HINTS = "enable javascript", "requires javascript", "javascript is disabled" ; function looksClientRendered page: PageData : boolean { if page.error return false; // an error is a different problem; see Step 4 const md = page.markdown ?? "" .trim ; const lower = md.toLowerCase ; return md.length < THIN || JS HINTS.some h = lower.includes h ; } THIN is your number, not a service field. A short page with no error is not proof of a JavaScript shell some pages are just short , but it is a reasonable trigger for one Render call. Do not retry Extract on a thin result: it will fetch the same HTML again. Request only the formats you will use. Markdown is what an LLM wants; add links if the agent needs to crawl onward. type ModelPage = | { status: "ok"; url: string; title: string | null; markdown: string; note?: string } | { status: "skip"; url: string; code: string } | { status: "retry later"; url: string; code: string }; export async function readJsPage url: string : Promise<ModelPage { const extract = await callService "agentsearch-web-extract-v1", "/v1/extract", { url, formats: "markdown" , max chars: 8000, } ; if extract.kind === "portal" return portalResult url, extract ; const first = extract.data as PageData; if looksClientRendered first return toModel first, url ; const render = await callService "agentsearch-web-render-v1", "/v1/render", { url, formats: "markdown" , max chars: 8000, } ; if render.kind === "portal" return portalResult url, render ; const page = toModel render.data as PageData, url ; if page.status === "ok" && page.markdown.length < THIN && page.note { return { status: "skip", url: page.url, code: "EMPTY PAGE" }; } return page; } If Render also comes back nearly empty, the page probably has nothing public to read or needs a login, which Render does not do . Skip it rather than paying again. Render accepts a few options for slow apps: wait until domcontentloaded by default, or load or networkidle , wait for selector a CSS selector, waited on for at most 1.5 s and wait ms 0 to 1500 , plus block to skip loading image , media , font or stylesheet requests default "media", "font" . Waiting longer eats into the deadline below, so start with the defaults. Three outcomes need their own handling. robots.txt blocks the page. Render honors robots.txt RFC 9309, user-agent token AgentSearchRender ; its browser identifies itself with a Chrome user-agent string ending in AgentSearchRender/0.1 +https://agentsearchhq.com/agents.md . A disallowed URL is refused before any fetch. Called directly, Render answers HTTP 422: {"error": {"code": "ROBOTS DISALLOWED", "message": "robots.txt disallows this URL for AgentSearchRender.", "retryable": false, "request id": "…"}} Through the Pocket portal this surfaces as HTTP 400 with code UPSTREAM REJECTED , and the call is not charged on x402. For a plain disallow the answer will not change, so treat it as "use a different source" and respect the site's choice. If the site's robots.txt itself answered with a server error, Render also refuses, but marks that one retryable: true . The render hits its deadline. Render's hard deadline is 4.2 s, counted from when the request arrives. When it runs out, the response is still HTTP 200 with error.code set to TARGET DEADLINE , which is always retryable: true , and markdown may already hold the text that rendered in time. Keep that partial page and tell the model it may be incomplete. Other website errors. These arrive inside data.error on an HTTP 200, and the call counts as delivered. Branch on code and retryable , never on message . function portalResult url: string, f: PortalFailure : ModelPage { // robots.txt refusal: 400 UPSTREAM REJECTED via the portal not charged on x402 , 422 ROBOTS DISALLOWED direct if f.code === "UPSTREAM REJECTED" || f.code === "ROBOTS DISALLOWED" return { status: "skip", url, code: f.code }; return f.retryable ? { status: "retry later", url, code: f.code } : { status: "skip", url, code: f.code }; } function toModel page: PageData, requested: string : ModelPage { const url = page.url || requested; // final URL after redirects: cite this const md = page.markdown ?? "" .trim ; const err = page.error ?? null; if err { if err.code === "TARGET DEADLINE" && md { return { status: "ok", url, title: page.title ?? null, markdown: md, note: "partial: render hit its 4.2s deadline" }; } return err.retryable ? { status: "retry later", url, code: err.code } : { status: "skip", url, code: err.code }; } return { status: "ok", url, title: page.title ?? null, markdown: md }; } retry later means "try this URL on another turn," not "loop now." Each delivered retry is a new $0.005 call. The full code list, including Extract's TARGET TIMEOUT and Render's TARGET BOT WALL a bot check or challenge page, which Render reports rather than bypasses; skip that source , is in the error-handling guide https://agentsearchhq.com/blog/error-handling-ai-agent-web-tools?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents . The fallback already does most of the work: static pages cost one call, JS-heavy pages cost two. A few more habits keep a loop predictable: url are one page. markdown alone is enough for most agent reads; skip html and the screenshot unless you need them. Pass the resulting markdown to the model as a tool result, never as part of the system prompt. Rendered or not, it is still someone else's text. If you would rather not write a client, Pocket's MCP server pays over x402 on Base with no backend of your own: { "mcpServers": { "pocket-network": { "command": "npx", "args": "-y", "@pocket-network/agentic-portal-mcp" , "env": { "POCKET PRIVATE KEY": "0x…", "POCKET MAX TOTAL ATOMIC": "1000000", "POCKET MAX PER CALL ATOMIC": "5000" } } } } Amounts are USDC atomic units 6 decimals : POCKET MAX TOTAL ATOMIC of 1000000 caps total signing at $1.00 while the server runs it resets on restart and is required to pay , and POCKET MAX PER CALL ATOMIC of 5000 caps any one call at $0.005. The limits are checked before anything is signed. Use a wallet that holds only what you are willing to spend. Tell the model the same rule in its instructions: call Extract first, and call Render only when the markdown is thin or asks for JavaScript. POST https://agent.pocket.network/v1/agentsearch-web-extract-v1/v1/extract POST https://agent.pocket.network/v1/agentsearch-web-render-v1/v1/render POST https://agent.pocket.network/v1/agentsearch-web-search-v1/v1/search For choosing between the three tools in general, see Search, Extract, Render: three web tools for AI agents https://agentsearchhq.com/blog/search-extract-render-web-tools-for-ai-agents?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents . Fetch it with a static extractor first. If the markdown is very short or asks you to enable JavaScript, send the same URL to a headless browser such as Web Render, which runs the page in Chromium and returns markdown. Look at the extracted text: a near-empty body, an empty app container, or a "please enable JavaScript" notice are the usual signs. Use a length threshold you tune for your sources. Web Render returns markdown, text, HTML, the title, links and an optional screenshot, so an LLM can read the page directly. Not on x402. Render refuses the URL before fetching it HTTP 422 ROBOTS DISALLOWED directly ; through the Pocket portal that surfaces as HTTP 400 UPSTREAM REJECTED and the call is not charged. Render stops at 4.2 s and returns HTTP 200 with error.code TARGET DEADLINE , which is retryable and may include the markdown that rendered in time. Search, Extract and Render are each $0.005 per call in USDC, over x402 on Base or MPP on Tempo. No signup and no API key. Add looksClientRendered and the Render fallback to any loop that already calls Extract. Start with Web Extract https://agentsearchhq.com/extract?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents , fall back to Web Render https://agentsearchhq.com/render?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents , and use Web Search https://agentsearchhq.com/search?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents when the agent does not have a URL yet. More on the AgentSearch blog https://agentsearchhq.com/blog/?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents . Originally published on the AgentSearch blog https://agentsearchhq.com/blog/scrape-javascript-heavy-pages-for-ai-agents?utm source=devto&utm medium=social&utm campaign=scrape-javascript-heavy-pages-for-ai-agents .