{"slug": "how-to-list-a-websites-pages-with-decodos-map-api", "title": "How To List a Website’s Pages With Decodo’s /map API", "summary": "A developer published a guide showing how to use Decodo's /map endpoint of its Web Data API to list a site's pages, filter them by path, and detect changes, aimed at keeping RAG pipelines from indexing stale content. The walkthrough includes a Python script that calls /map with select_paths and exclude_paths filters, retries with exponential backoff on 429 and 5xx responses, and saves sorted URL lists to JSON. The author cites the Model Context Protocol site's sitemap, which lists pages from every spec revision, as an example of why filtering matters.", "body_md": "A RAG pipeline indexes whatever pages you give it. The Model Context Protocol (MCP) site's sitemap listed pages from every specification revision, so a pipeline would index them all. An agent could then answer from an older revision.\n\nA reliable fix is to list the pages and keep only the revision you need. You can parse the sitemap and homepage yourself, but sites with anti-bot protection can return a 403 or a challenge page instead.\n\nThis guide uses Decodo's */map* API to list a website's pages, filter them, and check the list for changes.\n\n*[/map](https://decodo.com/web-data-api/map)* is the URL discovery endpoint of Decodo's Web Data API, which also includes */scrape*, */search*, and */crawl*. It returns URLs from the site's sitemap, the links on the single page you submit, or both, already filtered by any path patterns you send.\n\n*/map* has built-in anti-bot handling and runs on Decodo's proxy network, so it can map sites that block simple crawlers. The [Web Data API page](https://decodo.com/web-data-api) lists the current proxy pool size and success rate. Use */map* to collect publicly available data, and follow each site's terms of service.\n\nThe price of a */map* call is the same for any number of URLs it returns, and you pay only for successful requests. The [billing page](https://help.decodo.com/docs/web-data-api-billing) lists current rates, and the [pricing page](https://decodo.com/web-data-api/pricing) shows what the free plan includes.\n\nCopy your API key from the [Web Data API playground](https://dashboard.decodo.com/web-data/playground?product=map) in the Decodo dashboard. The **Get code** button on the same page shows the */map* request for Decodo's SDKs, [CLI](https://decodo.com/cli), and [MCP server](https://decodo.com/ai/mcp-server).\n\nThen create a virtual environment, install Requests, and set the key as an environment variable, which the Step 2 code reads:\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\npip install requests\nexport DECODO_AUTH_TOKEN=\"your-api-key\"\n```\n\nSave this code as *list_pages.py*. Its list_pages() function calls *[/map](https://help.decodo.com/docs/web-data-api-map-introduction)* once and returns the URLs and 2 metadata fields. The *call_api()* function sends each POST request to the API. After a timeout, a connection error, or a 429 or 5xx status code, it retries with exponential backoff.\n\n``` python\nimport json\nimport os\nimport time\nfrom urllib.parse import urlparse\n\nimport requests\n\nAPI_URL = \"https://data.decodo.com/v1/\"\n\ndef call_api(endpoint, payload, retries=3):\n    headers = {\"Authorization\": f\"Bearer {os.environ['DECODO_AUTH_TOKEN']}\"}\n    for attempt in range(retries):\n        try:\n            resp = requests.post(API_URL + endpoint, json=payload,\n                                 headers=headers, timeout=180)\n            if resp.status_code == 200:\n                return resp.json()[\"results\"][0]\n            if resp.status_code != 429 and resp.status_code < 500:\n                raise RuntimeError(f\"{resp.status_code}: {resp.text[:200]}\")\n        except requests.RequestException:\n            pass\n        if attempt < retries - 1:\n            time.sleep(2 ** attempt)\n    raise RuntimeError(f\"No successful response after {retries} attempts\")\n\ndef list_pages(url, sitemap=\"include\", select_paths=(), exclude_paths=(),\n               domain_filter=\"domain\"):\n    result = call_api(\"map\", {\"url\": url, \"sitemap\": sitemap,\n                              \"select_paths\": list(select_paths),\n                              \"exclude_paths\": list(exclude_paths),\n                              \"domain_filter\": domain_filter})\n    urls = [link[\"url\"] for link in result[\"content\"][\"links\"]]\n    return urls, result[\"sitemap_status\"], result[\"map_id\"]\n\ndef save(urls, url):\n    path = urlparse(url).netloc + \".json\"\n    with open(path, \"w\") as f:\n        json.dump(sorted(urls), f, indent=2)\n    return path\n\nif __name__ == \"__main__\":\n    site = \"https://modelcontextprotocol.io\"\n    urls, sitemap_status, map_id = list_pages(site)\n    print(f\"{len(urls)} URLs, sitemap_status={sitemap_status}, map_id={map_id}\")\n    print(\"saved to\", save(urls, site))\n```\n\nRunning *python list_pages.py* printed:\n\n```\n361 URLs, sitemap_status=include, map_id=a30421ba-49de-459d-be4e-e68346c695f4\nsaved to modelcontextprotocol.io.json\n```\n\nOn a larger site, the same function returned 5,108 URLs for *docs.stripe.com* in one call.\n\n*sitemap_status* shows which sources */map* actually used. *include* means both, *only* means the sitemap, and skip means the links on the page. On a site with no sitemap, */map* returns *skip* even when you send *include*. Use *map_id* to find the call in the dashboard's usage statistics or to ask Decodo's support about it.\n\n*domain_filter* limits the list to the submitted host by default (*domain*). Pass *subdomain* to add its subdomains (e.g., *blog.example.com*), or *all* to add external sites.\n\nOn *modelcontextprotocol.io*, each *sitemap* value returned a different list:\n\n| sitemap | URLs | What the list contained | \n|---|---|---|\n| `include` (default) | 361 | Sitemap and homepage links, merged and deduplicated | \n| `only` | 359 | Every sitemap URL, including pages from 4 older specification revisions and a draft | \n| `skip` | 19 | Only the homepage's links, with no older revisions | \n\nUse *only* for the pages the site owner listed in the sitemap. Use *skip* for the links on the submitted page only. Use *include* for the largest list, then filter it.\n\nThe MCP site's own version menu showed why the sitemap was so large.\n\nIf you build your list from the site's *llms.txt* file, a page index for AI tools, apply the same filter. That file listed the same 359 pages as the sitemap, including older revisions.\n\nIf a site loads its navigation with JavaScript, add *\"JS_rendering\": True* to the payload in *list_pages()*. */map* then renders the page before it collects links. The [billing page](https://help.decodo.com/docs/web-data-api-billing) lists the rate for JavaScript rendering.\n\nThe MCP site puts the revision in the path, so one pattern selects your agent's revision. Save this code as *load_revision.py* in the same folder as *list_pages.py*. It filters the list, fetches each page with one */scrape* request, and saves the loaded URLs as the baseline for Step 5:\n\n``` python\nfrom list_pages import call_api, list_pages, save\n\nSITE = \"https://modelcontextprotocol.io\"\nPATTERN = \"^/(specification|docs)/2026-07-28(/|$)\"\n\ndef fetch_markdown(url):\n    result = call_api(\"scrape\", {\"url\": url, \"markdown\": True})\n    return result[\"content\"] if result[\"status_code\"] == 200 else None\n\nif __name__ == \"__main__\":\n    urls, _, _ = list_pages(SITE, select_paths=[PATTERN])\n    print(len(urls), \"URLs in the 2026-07-28 revision\")\n    loaded = []\n    for url in urls:\n        markdown = fetch_markdown(url)\n        if markdown is not None:\n            # Chunk and embed markdown, then upsert it into your vector store by url\n            loaded.append(url)\n    save(loaded, SITE)\n    print(len(loaded), \"of\", len(urls), \"pages loaded\")\n```\n\nRunning *python load_revision.py* printed:\n\n```\n57 URLs in the 2026-07-28 revision\n57 of 57 pages loaded\n```\n\nThat one pattern reduced the list from 361 URLs to 57. It selects the versioned pages only. To also index unversioned sections, add them to the end of *PATTERN*, such as *|^/extensions(/|$)*. To exclude a section instead, pass its pattern to *exclude_paths*, such as *^/(specification|docs)/draft(/|$)* for the draft. For your own site, change *SITE* and *PATTERN* at the top of this file.\n\nThese 4 rules help you write patterns that match:\n\nTo test a pattern in the dashboard, open **Advanced settings** on Decodo's Playground's Map tab. Set **Sitemap** to *Include* and paste the pattern into **Select paths**.\n\nClick **Map** to run it and see the matching URLs.\n\nThe *fetch_markdown()* function sends one URL to *[/scrape](https://help.decodo.com/docs/web-data-api-scrape-quick-start)* with *\"markdown\": True* and returns that page as Markdown, which uses fewer tokens than the page's HTML. It returns *None* when the page responds with an error, and the script leaves that URL out of the baseline, so Step 5 tries it again.\n\nIf your pipeline uses LangChain or LlamaIndex, Decodo's [DecodoLoader](https://help.decodo.com/docs/langchain-python) and [DecodoWebReader](https://help.decodo.com/docs/llamaindex) load the URLs as Markdown *Document* objects with the same API key instead. To fetch a large section in one job, send the same *select_paths* to *[/crawl](https://help.decodo.com/docs/web-data-api-crawl-quick-start)*.\n\nThe script uses the new list only if it passes 2 checks:\n\n``` python\nimport json\nimport re\nfrom urllib.parse import urlparse\n\nfrom list_pages import list_pages, save\nfrom load_revision import PATTERN, SITE, fetch_markdown\n\nwith open(urlparse(SITE).netloc + \".json\") as f:\n    previous = set(json.load(f))\n\nurls, _, _ = list_pages(SITE, select_paths=[PATTERN])\ncurrent = set(urls)\nif not all(re.search(PATTERN, urlparse(u).path) for u in current):\n    raise SystemExit(\"the new list has a URL outside PATTERN, keeping the baseline\")\nif len(current) < len(previous) / 2:\n    raise SystemExit(f\"{len(current)} URLs vs {len(previous)} saved, keeping the baseline\")\nprint(\"added:  \", sorted(current - previous))\nprint(\"removed:\", sorted(previous - current))\nskipped = set()\nfor url in sorted(current - previous):\n    markdown = fetch_markdown(url)\n    if markdown is None:\n        print(f\"skipped {url}: the page did not return 200\")\n        skipped.add(url)\n        continue\n    print(f\"fetched {url}: {len(markdown)} characters\")\n    # Chunk and embed markdown, then upsert it into your vector store by url\n# Delete the removed URLs from your vector store here\nsave(current - skipped, SITE)\n\n# MCP only: list every revision date to detect a new one\nall_urls, _, _ = list_pages(SITE)\nrevisions = {u.split(\"/\")[4] for u in all_urls if \"/specification/\" in u}\nprint(\"revisions:\", sorted(revisions))\nlatest = max(r for r in revisions if r[:1].isdigit())\nif not re.search(PATTERN, f\"/specification/{latest}\"):\n    print(\"new revision:\", latest)\n```\n\nSave this code as *diff_runs.py*. It loads the saved list (the baseline), maps the site again with the same filter, prints what changed, and fetches each added page as Markdown.\n\nTo test it, delete 2 URLs from *modelcontextprotocol.io.json* and run *python diff_runs.py*. It printed:\n\n```\nadded:   ['https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture', 'https://modelcontextprotocol.io/specification/2026-07-28']\nremoved: []\nfetched https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture: 34494 characters\nfetched https://modelcontextprotocol.io/specification/2026-07-28: 8189 characters\nrevisions: ['2024-11-05', '2025-03-26', '2025-06-18', '2025-11-25', '2026-07-28', 'draft']\n```\n\nThe script skips a page that answers with an error and tries it again on the next run. If your ingestion code raises an exception, the script stops before *save()*, so the next run shows the same changes again.\n\n*/map* normalizes each URL before it removes duplicates, so */page/*, */page#section*, and */page* count as one URL. Use the URLs from the */map* list as the keys in your vector store, so your deletes match.\n\nWhen MCP publishes a new revision, the script prints *new revision:* with its date, so add an alert on that line. To move your agent to that revision, update *PATTERN* only. The next run lists the new revision's pages as added and the old ones as removed. The revision check adds one */map* call to each run.\n\nTo find edits to existing pages, fetch the section again with */crawl*, hash each page's Markdown, and re-ingest the pages whose hash changed.\n\nSchedule *diff_runs.py* with cron or n8n. Run it from the folder that holds the baseline file, on a machine that keeps that file between runs. Set the API key as an environment variable for the job.\n\nUse the scripts on your own site. Run *load_revision.py* once with your *SITE* and *PATTERN* to load your pages and save the first baseline. In *diff_runs.py*, delete the MCP-only revision check, or change *\"/specification/\"* and the index in *split(\"/\")[4]* to match where your site keeps its versions.\n\nThe 2 main decisions are the URL source and the filters, and *list_pages()* takes both as arguments. Coding agents can make the same */map*, */scrape*, and */[crawl](https://help.decodo.com/docs/web-data-api-crawl-introduction)* calls through Decodo's [agent skills](https://help.decodo.com/docs/agent-skills). The dashboard's **Copy agent setup** button copies a setup prompt for your coding agent.\n\nActivate the [free plan](https://dashboard.decodo.com/register?planId=922406), which includes monthly credit and requires no credit card, and map the docs site your agent uses.", "url": "https://wpnews.pro/news/how-to-list-a-websites-pages-with-decodos-map-api", "canonical_source": "https://dev.to/decodo_team/how-to-list-a-websites-pages-with-decodos-map-api-500b", "published_at": "2026-11-16 13:40:39+00:00", "updated_at": "2026-11-16 13:50:52.450965+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools"], "entities": ["Decodo", "Model Context Protocol", "Python", "Requests"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-list-a-websites-pages-with-decodos-map-api", "markdown": "https://wpnews.pro/news/how-to-list-a-websites-pages-with-decodos-map-api.md", "text": "https://wpnews.pro/news/how-to-list-a-websites-pages-with-decodos-map-api.txt", "jsonld": "https://wpnews.pro/news/how-to-list-a-websites-pages-with-decodos-map-api.jsonld"}}