How To List a Website’s Pages With Decodo’s /map API A developer published a guide showing how to use Decodo's /map endpoint of its Web Data API to list a site's pages, filter them by path, and detect changes, aimed at keeping RAG pipelines from indexing stale content. The walkthrough includes a Python script that calls /map with select_paths and exclude_paths filters, retries with exponential backoff on 429 and 5xx responses, and saves sorted URL lists to JSON. The author cites the Model Context Protocol site's sitemap, which lists pages from every spec revision, as an example of why filtering matters. A RAG pipeline indexes whatever pages you give it. The Model Context Protocol MCP site's sitemap listed pages from every specification revision, so a pipeline would index them all. An agent could then answer from an older revision. A reliable fix is to list the pages and keep only the revision you need. You can parse the sitemap and homepage yourself, but sites with anti-bot protection can return a 403 or a challenge page instead. This guide uses Decodo's /map API to list a website's pages, filter them, and check the list for changes. /map https://decodo.com/web-data-api/map is the URL discovery endpoint of Decodo's Web Data API, which also includes /scrape , /search , and /crawl . It returns URLs from the site's sitemap, the links on the single page you submit, or both, already filtered by any path patterns you send. /map has built-in anti-bot handling and runs on Decodo's proxy network, so it can map sites that block simple crawlers. The Web Data API page https://decodo.com/web-data-api lists the current proxy pool size and success rate. Use /map to collect publicly available data, and follow each site's terms of service. The price of a /map call is the same for any number of URLs it returns, and you pay only for successful requests. The billing page https://help.decodo.com/docs/web-data-api-billing lists current rates, and the pricing page https://decodo.com/web-data-api/pricing shows what the free plan includes. Copy your API key from the Web Data API playground https://dashboard.decodo.com/web-data/playground?product=map in the Decodo dashboard. The Get code button on the same page shows the /map request for Decodo's SDKs, CLI https://decodo.com/cli , and MCP server https://decodo.com/ai/mcp-server . Then create a virtual environment, install Requests, and set the key as an environment variable, which the Step 2 code reads: python3 -m venv .venv source .venv/bin/activate pip install requests export DECODO AUTH TOKEN="your-api-key" Save this code as list pages.py . Its list pages function calls /map https://help.decodo.com/docs/web-data-api-map-introduction once and returns the URLs and 2 metadata fields. The call api function sends each POST request to the API. After a timeout, a connection error, or a 429 or 5xx status code, it retries with exponential backoff. python import json import os import time from urllib.parse import urlparse import requests API URL = "https://data.decodo.com/v1/" def call api endpoint, payload, retries=3 : headers = {"Authorization": f"Bearer {os.environ 'DECODO AUTH TOKEN' }"} for attempt in range retries : try: resp = requests.post API URL + endpoint, json=payload, headers=headers, timeout=180 if resp.status code == 200: return resp.json "results" 0 if resp.status code = 429 and resp.status code < 500: raise RuntimeError f"{resp.status code}: {resp.text :200 }" except requests.RequestException: pass if attempt < retries - 1: time.sleep 2 attempt raise RuntimeError f"No successful response after {retries} attempts" def list pages url, sitemap="include", select paths= , exclude paths= , domain filter="domain" : result = call api "map", {"url": url, "sitemap": sitemap, "select paths": list select paths , "exclude paths": list exclude paths , "domain filter": domain filter} urls = link "url" for link in result "content" "links" return urls, result "sitemap status" , result "map id" def save urls, url : path = urlparse url .netloc + ".json" with open path, "w" as f: json.dump sorted urls , f, indent=2 return path if name == " main ": site = "https://modelcontextprotocol.io" urls, sitemap status, map id = list pages site print f"{len urls } URLs, sitemap status={sitemap status}, map id={map id}" print "saved to", save urls, site Running python list pages.py printed: 361 URLs, sitemap status=include, map id=a30421ba-49de-459d-be4e-e68346c695f4 saved to modelcontextprotocol.io.json On a larger site, the same function returned 5,108 URLs for docs.stripe.com in one call. sitemap status shows which sources /map actually used. include means both, only means the sitemap, and skip means the links on the page. On a site with no sitemap, /map returns skip even when you send include . Use map id to find the call in the dashboard's usage statistics or to ask Decodo's support about it. domain filter limits the list to the submitted host by default domain . Pass subdomain to add its subdomains e.g., blog.example.com , or all to add external sites. On modelcontextprotocol.io , each sitemap value returned a different list: | sitemap | URLs | What the list contained | |---|---|---| | include default | 361 | Sitemap and homepage links, merged and deduplicated | | only | 359 | Every sitemap URL, including pages from 4 older specification revisions and a draft | | skip | 19 | Only the homepage's links, with no older revisions | Use only for the pages the site owner listed in the sitemap. Use skip for the links on the submitted page only. Use include for the largest list, then filter it. The MCP site's own version menu showed why the sitemap was so large. If you build your list from the site's llms.txt file, a page index for AI tools, apply the same filter. That file listed the same 359 pages as the sitemap, including older revisions. If a site loads its navigation with JavaScript, add "JS rendering": True to the payload in list pages . /map then renders the page before it collects links. The billing page https://help.decodo.com/docs/web-data-api-billing lists the rate for JavaScript rendering. The MCP site puts the revision in the path, so one pattern selects your agent's revision. Save this code as load revision.py in the same folder as list pages.py . It filters the list, fetches each page with one /scrape request, and saves the loaded URLs as the baseline for Step 5: python from list pages import call api, list pages, save SITE = "https://modelcontextprotocol.io" PATTERN = "^/ specification|docs /2026-07-28 /|$ " def fetch markdown url : result = call api "scrape", {"url": url, "markdown": True} return result "content" if result "status code" == 200 else None if name == " main ": urls, , = list pages SITE, select paths= PATTERN print len urls , "URLs in the 2026-07-28 revision" loaded = for url in urls: markdown = fetch markdown url if markdown is not None: Chunk and embed markdown, then upsert it into your vector store by url loaded.append url save loaded, SITE print len loaded , "of", len urls , "pages loaded" Running python load revision.py printed: 57 URLs in the 2026-07-28 revision 57 of 57 pages loaded That one pattern reduced the list from 361 URLs to 57. It selects the versioned pages only. To also index unversioned sections, add them to the end of PATTERN , such as |^/extensions /|$ . To exclude a section instead, pass its pattern to exclude paths , such as ^/ specification|docs /draft /|$ for the draft. For your own site, change SITE and PATTERN at the top of this file. These 4 rules help you write patterns that match: To test a pattern in the dashboard, open Advanced settings on Decodo's Playground's Map tab. Set Sitemap to Include and paste the pattern into Select paths . Click Map to run it and see the matching URLs. The fetch markdown function sends one URL to /scrape https://help.decodo.com/docs/web-data-api-scrape-quick-start with "markdown": True and returns that page as Markdown, which uses fewer tokens than the page's HTML. It returns None when the page responds with an error, and the script leaves that URL out of the baseline, so Step 5 tries it again. If your pipeline uses LangChain or LlamaIndex, Decodo's DecodoLoader https://help.decodo.com/docs/langchain-python and DecodoWebReader https://help.decodo.com/docs/llamaindex load the URLs as Markdown Document objects with the same API key instead. To fetch a large section in one job, send the same select paths to /crawl https://help.decodo.com/docs/web-data-api-crawl-quick-start . The script uses the new list only if it passes 2 checks: python import json import re from urllib.parse import urlparse from list pages import list pages, save from load revision import PATTERN, SITE, fetch markdown with open urlparse SITE .netloc + ".json" as f: previous = set json.load f urls, , = list pages SITE, select paths= PATTERN current = set urls if not all re.search PATTERN, urlparse u .path for u in current : raise SystemExit "the new list has a URL outside PATTERN, keeping the baseline" if len current < len previous / 2: raise SystemExit f"{len current } URLs vs {len previous } saved, keeping the baseline" print "added: ", sorted current - previous print "removed:", sorted previous - current skipped = set for url in sorted current - previous : markdown = fetch markdown url if markdown is None: print f"skipped {url}: the page did not return 200" skipped.add url continue print f"fetched {url}: {len markdown } characters" Chunk and embed markdown, then upsert it into your vector store by url Delete the removed URLs from your vector store here save current - skipped, SITE MCP only: list every revision date to detect a new one all urls, , = list pages SITE revisions = {u.split "/" 4 for u in all urls if "/specification/" in u} print "revisions:", sorted revisions latest = max r for r in revisions if r :1 .isdigit if not re.search PATTERN, f"/specification/{latest}" : print "new revision:", latest Save this code as diff runs.py . It loads the saved list the baseline , maps the site again with the same filter, prints what changed, and fetches each added page as Markdown. To test it, delete 2 URLs from modelcontextprotocol.io.json and run python diff runs.py . It printed: added: 'https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture', 'https://modelcontextprotocol.io/specification/2026-07-28' removed: fetched https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture: 34494 characters fetched https://modelcontextprotocol.io/specification/2026-07-28: 8189 characters revisions: '2024-11-05', '2025-03-26', '2025-06-18', '2025-11-25', '2026-07-28', 'draft' The script skips a page that answers with an error and tries it again on the next run. If your ingestion code raises an exception, the script stops before save , so the next run shows the same changes again. /map normalizes each URL before it removes duplicates, so /page/ , /page section , and /page count as one URL. Use the URLs from the /map list as the keys in your vector store, so your deletes match. When MCP publishes a new revision, the script prints new revision: with its date, so add an alert on that line. To move your agent to that revision, update PATTERN only. The next run lists the new revision's pages as added and the old ones as removed. The revision check adds one /map call to each run. To find edits to existing pages, fetch the section again with /crawl , hash each page's Markdown, and re-ingest the pages whose hash changed. Schedule diff runs.py with cron or n8n. Run it from the folder that holds the baseline file, on a machine that keeps that file between runs. Set the API key as an environment variable for the job. Use the scripts on your own site. Run load revision.py once with your SITE and PATTERN to load your pages and save the first baseline. In diff runs.py , delete the MCP-only revision check, or change "/specification/" and the index in split "/" 4 to match where your site keeps its versions. The 2 main decisions are the URL source and the filters, and list pages takes both as arguments. Coding agents can make the same /map , /scrape , and / crawl https://help.decodo.com/docs/web-data-api-crawl-introduction calls through Decodo's agent skills https://help.decodo.com/docs/agent-skills . The dashboard's Copy agent setup button copies a setup prompt for your coding agent. Activate the free plan https://dashboard.decodo.com/register?planId=922406 , which includes monthly credit and requires no credit card, and map the docs site your agent uses.