Turn Any URL into Clean Markdown for RAG (API Guide)

Raw HTML makes poor context for an LLM. A typical page is mostly navigation, scripts, cookie banners and footer links. Putting all of that into an embedding model or a prompt wastes tokens and lowers retrieval quality. What you want is the content: headings, paragraphs, lists and links, as markdown, plus enough metadata to cite the source and to tell when it goes stale.

This guide uses AgentSearch Web Extract (agentsearch-web-extract-v1) to turn a URL into RAG-ready markdown. It covers the request, the response, chunking and error handling. Each call costs $0.005, paid in USDC over x402 or MPP on the Pocket Agentic Portal, with no account and no API key.

#, ##) tell you where sections begin, which gives you natural chunk boundaries and section titles to store as metadata.[text](url) keeps the citation trail, so an agent can follow a link or quote its source. The endpoint is POST /v1/extract, which takes a JSON body. Only url is required.

curl -X POST https://agent.pocket.network/v1/agentsearch-web-extract-v1/v1/extract \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}'

These are the options from the extract OpenAPI spec:

This is a real response captured on the portal service page (2026-09-25 UTC):

{
  "portal": {
    "provenance": "third-party-supplier",
    "serviceId": "agentsearch-web-extract-v1",
    "schemaCheck": "unchecked"
  },
  "data": {
    "request_id": "05f5a4ebe2684c79867aa9bf3515c33d",
    "url": "https://example.com/",
    "title": "Example Domain",
    "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n",
    "text": "",
    "links": [{ "href": "https://iana.org/domains/example", "text": "Learn more" }],
    "meta": {
      "status_code": 200,
      "content_type": "text/html",
      "fetched_at": "2026-09-25T15:53:07.843243+00:00",
      "chars": 167,
      "truncated": false
    },
    "error": null
  }
}

The fields that matter for RAG:

url is the title is a ready-made citation label.meta.fetched_at lets you re-extract stale documents on a schedule.meta.truncated tells you whether you hit max_chars. If you did, raise the cap and fetch again. error is null on success. Standard x402 v2 clients work with the portal unchanged. Here's @x402/fetch with @x402/evm, paying on Base:

import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

const EXTRACT =
  "https://agent.pocket.network/v1/agentsearch-web-extract-v1/v1/extract";

// Use a dedicated wallet that holds only what you're willing to spend.
const account = privateKeyToAccount(process.env.AGENT_WALLET_KEY as `0x${string}`);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});

export async function extract(url: string, maxChars = 50000) {
  const res = await pay(EXTRACT, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ url, formats: ["markdown"], max_chars: maxChars }),
  });

  if (!res.ok) {
    // Portal errors aren't wrapped in the envelope: { error: { code, message, retryable } }
    const body = await res.json().catch(() => ({}));
    throw new Error(`portal ${res.status}: ${body?.error?.code ?? "unknown"}`);
  }

  const envelope = await res.json();
  if (envelope.portal?.provenance !== "third-party-supplier") {
    throw new Error("Unexpected response shape; refusing to use it.");
  }
  const doc = envelope.data;
  if (doc.error) {
    // The target site failed, but the HTTP status was still 200.
    throw Object.assign(new Error(doc.error.code), { retryable: doc.error.retryable });
  }
  return doc as {
    url: string; title: string | null; markdown: string;
    links: { href: string; text: string }[];
    meta: { fetched_at: string; chars: number; truncated: boolean };
  };
}

If delivery fails, the portal settles nothing, so you aren't charged for it. Payment details are in Pocket's integration guide.

In Claude Desktop, Cursor or Claude Code, add npx -y @pocket-network/agentic-portal-mcp and ask the agent to call_service on agentsearch-web-extract-v1. The full setup, including spending limits, is in How to give your AI agent web search with MCP.

Split on markdown headings first, then on size. Every chunk keeps the path of headings above it, which improves both retrieval and citations.

import re

def chunk_markdown(doc: dict, max_chars: int = 1500, overlap: int = 150):
    """doc is the `data` object returned by agentsearch-web-extract-v1."""
    chunks, path, buf = [], [], []

    def flush():
        text = "\n".join(buf).strip()
        if not text:
            return
        for i in range(0, len(text), max_chars - overlap):
            chunks.append({
                "text": text[i:i + max_chars],
                "source_url": doc["url"],
                "title": doc.get("title"),
                "section": " > ".join(path),
                "fetched_at": doc["meta"]["fetched_at"],
            })

    for line in doc["markdown"].splitlines():
        m = re.match(r"^(#{1,6})\s+(.*)", line)
        if m:
            flush(); buf.clear()
            level = len(m.group(1))
            path[:] = path[:level - 1] + [m.group(2).strip()]
        buf.append(line)
    flush()
    return chunks

Embed chunk["text"] and store the rest as metadata. When your agent answers, cite title and source_url.

Extract separates target-site failures from request problems:

UNSUPPORTED_CONTENT means the URL points to something other than HTML or text, such as a PDF or an image. Send those to a document parser instead.

Any page can contain text written to manipulate your agent. Before content reaches a model:

portal.schemaCheck. Only passed means a schema check ran. Extract fetches the HTML the server sends, which is fast and works for docs, blogs, news and most marketing pages. Single-page apps and dashboards often render their content only after JavaScript runs. For those, AgentSearch has Web Render (agentsearch-web-render-v1), live on Pocket MainNet. It loads the page in headless Chromium and returns the rendered markdown, text, HTML, links and an optional screenshot. It honors robots.txt and doesn't bypass bot walls. Render isn't on the Agentic Portal yet. Its request and response contract is in agents.md and the render spec.

A common pattern is to search with AgentSearch Web Search (up to 5 results per call), pick the most relevant results, then extract them in full. See the search spec, and compare per-call costs across vendors in Cheapest web search API for AI agents in 2026.

How long can an extracted page be?

Up to 100,000 characters per call through max_chars. The default is 50,000.

Does it follow redirects?

Yes. url in the response is the final URL, and requested_url appears on errors.

Can it extract PDFs?

No. Non-HTML content returns UNSUPPORTED_CONTENT.

Who serves the requests?

Decentralized Pocket Network suppliers. Machine-readable details are at agentsearchhq.com/llms.txt.

Service pages: Web Search API · Web Extract API · Web Render API

Originally published at agentsearchhq.com.