# How To List a Website’s Pages With Decodo’s /map API

> Source: <https://dev.to/decodo_team/how-to-list-a-websites-pages-with-decodos-map-api-500b>
> Published: 2026-11-16 13:40:39+00:00

A RAG pipeline indexes whatever pages you give it. The Model Context Protocol (MCP) site's sitemap listed pages from every specification revision, so a pipeline would index them all. An agent could then answer from an older revision.

A reliable fix is to list the pages and keep only the revision you need. You can parse the sitemap and homepage yourself, but sites with anti-bot protection can return a 403 or a challenge page instead.

This guide uses Decodo's */map* API to list a website's pages, filter them, and check the list for changes.

*[/map](https://decodo.com/web-data-api/map)* is the URL discovery endpoint of Decodo's Web Data API, which also includes */scrape*, */search*, and */crawl*. It returns URLs from the site's sitemap, the links on the single page you submit, or both, already filtered by any path patterns you send.

*/map* has built-in anti-bot handling and runs on Decodo's proxy network, so it can map sites that block simple crawlers. The [Web Data API page](https://decodo.com/web-data-api) lists the current proxy pool size and success rate. Use */map* to collect publicly available data, and follow each site's terms of service.

The price of a */map* call is the same for any number of URLs it returns, and you pay only for successful requests. The [billing page](https://help.decodo.com/docs/web-data-api-billing) lists current rates, and the [pricing page](https://decodo.com/web-data-api/pricing) shows what the free plan includes.

Copy your API key from the [Web Data API playground](https://dashboard.decodo.com/web-data/playground?product=map) in the Decodo dashboard. The **Get code** button on the same page shows the */map* request for Decodo's SDKs, [CLI](https://decodo.com/cli), and [MCP server](https://decodo.com/ai/mcp-server).

Then create a virtual environment, install Requests, and set the key as an environment variable, which the Step 2 code reads:

```
python3 -m venv .venv
source .venv/bin/activate
pip install requests
export DECODO_AUTH_TOKEN="your-api-key"
```

Save this code as *list_pages.py*. Its list_pages() function calls *[/map](https://help.decodo.com/docs/web-data-api-map-introduction)* once and returns the URLs and 2 metadata fields. The *call_api()* function sends each POST request to the API. After a timeout, a connection error, or a 429 or 5xx status code, it retries with exponential backoff.

``` python
import json
import os
import time
from urllib.parse import urlparse

import requests

API_URL = "https://data.decodo.com/v1/"

def call_api(endpoint, payload, retries=3):
    headers = {"Authorization": f"Bearer {os.environ['DECODO_AUTH_TOKEN']}"}
    for attempt in range(retries):
        try:
            resp = requests.post(API_URL + endpoint, json=payload,
                                 headers=headers, timeout=180)
            if resp.status_code == 200:
                return resp.json()["results"][0]
            if resp.status_code != 429 and resp.status_code < 500:
                raise RuntimeError(f"{resp.status_code}: {resp.text[:200]}")
        except requests.RequestException:
            pass
        if attempt < retries - 1:
            time.sleep(2 ** attempt)
    raise RuntimeError(f"No successful response after {retries} attempts")

def list_pages(url, sitemap="include", select_paths=(), exclude_paths=(),
               domain_filter="domain"):
    result = call_api("map", {"url": url, "sitemap": sitemap,
                              "select_paths": list(select_paths),
                              "exclude_paths": list(exclude_paths),
                              "domain_filter": domain_filter})
    urls = [link["url"] for link in result["content"]["links"]]
    return urls, result["sitemap_status"], result["map_id"]

def save(urls, url):
    path = urlparse(url).netloc + ".json"
    with open(path, "w") as f:
        json.dump(sorted(urls), f, indent=2)
    return path

if __name__ == "__main__":
    site = "https://modelcontextprotocol.io"
    urls, sitemap_status, map_id = list_pages(site)
    print(f"{len(urls)} URLs, sitemap_status={sitemap_status}, map_id={map_id}")
    print("saved to", save(urls, site))
```

Running *python list_pages.py* printed:

```
361 URLs, sitemap_status=include, map_id=a30421ba-49de-459d-be4e-e68346c695f4
saved to modelcontextprotocol.io.json
```

On a larger site, the same function returned 5,108 URLs for *docs.stripe.com* in one call.

*sitemap_status* shows which sources */map* actually used. *include* means both, *only* means the sitemap, and skip means the links on the page. On a site with no sitemap, */map* returns *skip* even when you send *include*. Use *map_id* to find the call in the dashboard's usage statistics or to ask Decodo's support about it.

*domain_filter* limits the list to the submitted host by default (*domain*). Pass *subdomain* to add its subdomains (e.g., *blog.example.com*), or *all* to add external sites.

On *modelcontextprotocol.io*, each *sitemap* value returned a different list:

| sitemap | URLs | What the list contained | 
|---|---|---|
| `include` (default) | 361 | Sitemap and homepage links, merged and deduplicated | 
| `only` | 359 | Every sitemap URL, including pages from 4 older specification revisions and a draft | 
| `skip` | 19 | Only the homepage's links, with no older revisions | 

Use *only* for the pages the site owner listed in the sitemap. Use *skip* for the links on the submitted page only. Use *include* for the largest list, then filter it.

The MCP site's own version menu showed why the sitemap was so large.

If you build your list from the site's *llms.txt* file, a page index for AI tools, apply the same filter. That file listed the same 359 pages as the sitemap, including older revisions.

If a site loads its navigation with JavaScript, add *"JS_rendering": True* to the payload in *list_pages()*. */map* then renders the page before it collects links. The [billing page](https://help.decodo.com/docs/web-data-api-billing) lists the rate for JavaScript rendering.

The MCP site puts the revision in the path, so one pattern selects your agent's revision. Save this code as *load_revision.py* in the same folder as *list_pages.py*. It filters the list, fetches each page with one */scrape* request, and saves the loaded URLs as the baseline for Step 5:

``` python
from list_pages import call_api, list_pages, save

SITE = "https://modelcontextprotocol.io"
PATTERN = "^/(specification|docs)/2026-07-28(/|$)"

def fetch_markdown(url):
    result = call_api("scrape", {"url": url, "markdown": True})
    return result["content"] if result["status_code"] == 200 else None

if __name__ == "__main__":
    urls, _, _ = list_pages(SITE, select_paths=[PATTERN])
    print(len(urls), "URLs in the 2026-07-28 revision")
    loaded = []
    for url in urls:
        markdown = fetch_markdown(url)
        if markdown is not None:
            # Chunk and embed markdown, then upsert it into your vector store by url
            loaded.append(url)
    save(loaded, SITE)
    print(len(loaded), "of", len(urls), "pages loaded")
```

Running *python load_revision.py* printed:

```
57 URLs in the 2026-07-28 revision
57 of 57 pages loaded
```

That one pattern reduced the list from 361 URLs to 57. It selects the versioned pages only. To also index unversioned sections, add them to the end of *PATTERN*, such as *|^/extensions(/|$)*. To exclude a section instead, pass its pattern to *exclude_paths*, such as *^/(specification|docs)/draft(/|$)* for the draft. For your own site, change *SITE* and *PATTERN* at the top of this file.

These 4 rules help you write patterns that match:

To test a pattern in the dashboard, open **Advanced settings** on Decodo's Playground's Map tab. Set **Sitemap** to *Include* and paste the pattern into **Select paths**.

Click **Map** to run it and see the matching URLs.

The *fetch_markdown()* function sends one URL to *[/scrape](https://help.decodo.com/docs/web-data-api-scrape-quick-start)* with *"markdown": True* and returns that page as Markdown, which uses fewer tokens than the page's HTML. It returns *None* when the page responds with an error, and the script leaves that URL out of the baseline, so Step 5 tries it again.

If your pipeline uses LangChain or LlamaIndex, Decodo's [DecodoLoader](https://help.decodo.com/docs/langchain-python) and [DecodoWebReader](https://help.decodo.com/docs/llamaindex) load the URLs as Markdown *Document* objects with the same API key instead. To fetch a large section in one job, send the same *select_paths* to *[/crawl](https://help.decodo.com/docs/web-data-api-crawl-quick-start)*.

The script uses the new list only if it passes 2 checks:

``` python
import json
import re
from urllib.parse import urlparse

from list_pages import list_pages, save
from load_revision import PATTERN, SITE, fetch_markdown

with open(urlparse(SITE).netloc + ".json") as f:
    previous = set(json.load(f))

urls, _, _ = list_pages(SITE, select_paths=[PATTERN])
current = set(urls)
if not all(re.search(PATTERN, urlparse(u).path) for u in current):
    raise SystemExit("the new list has a URL outside PATTERN, keeping the baseline")
if len(current) < len(previous) / 2:
    raise SystemExit(f"{len(current)} URLs vs {len(previous)} saved, keeping the baseline")
print("added:  ", sorted(current - previous))
print("removed:", sorted(previous - current))
skipped = set()
for url in sorted(current - previous):
    markdown = fetch_markdown(url)
    if markdown is None:
        print(f"skipped {url}: the page did not return 200")
        skipped.add(url)
        continue
    print(f"fetched {url}: {len(markdown)} characters")
    # Chunk and embed markdown, then upsert it into your vector store by url
# Delete the removed URLs from your vector store here
save(current - skipped, SITE)

# MCP only: list every revision date to detect a new one
all_urls, _, _ = list_pages(SITE)
revisions = {u.split("/")[4] for u in all_urls if "/specification/" in u}
print("revisions:", sorted(revisions))
latest = max(r for r in revisions if r[:1].isdigit())
if not re.search(PATTERN, f"/specification/{latest}"):
    print("new revision:", latest)
```

Save this code as *diff_runs.py*. It loads the saved list (the baseline), maps the site again with the same filter, prints what changed, and fetches each added page as Markdown.

To test it, delete 2 URLs from *modelcontextprotocol.io.json* and run *python diff_runs.py*. It printed:

```
added:   ['https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture', 'https://modelcontextprotocol.io/specification/2026-07-28']
removed: []
fetched https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture: 34494 characters
fetched https://modelcontextprotocol.io/specification/2026-07-28: 8189 characters
revisions: ['2024-11-05', '2025-03-26', '2025-06-18', '2025-11-25', '2026-07-28', 'draft']
```

The script skips a page that answers with an error and tries it again on the next run. If your ingestion code raises an exception, the script stops before *save()*, so the next run shows the same changes again.

*/map* normalizes each URL before it removes duplicates, so */page/*, */page#section*, and */page* count as one URL. Use the URLs from the */map* list as the keys in your vector store, so your deletes match.

When MCP publishes a new revision, the script prints *new revision:* with its date, so add an alert on that line. To move your agent to that revision, update *PATTERN* only. The next run lists the new revision's pages as added and the old ones as removed. The revision check adds one */map* call to each run.

To find edits to existing pages, fetch the section again with */crawl*, hash each page's Markdown, and re-ingest the pages whose hash changed.

Schedule *diff_runs.py* with cron or n8n. Run it from the folder that holds the baseline file, on a machine that keeps that file between runs. Set the API key as an environment variable for the job.

Use the scripts on your own site. Run *load_revision.py* once with your *SITE* and *PATTERN* to load your pages and save the first baseline. In *diff_runs.py*, delete the MCP-only revision check, or change *"/specification/"* and the index in *split("/")[4]* to match where your site keeps its versions.

The 2 main decisions are the URL source and the filters, and *list_pages()* takes both as arguments. Coding agents can make the same */map*, */scrape*, and */[crawl](https://help.decodo.com/docs/web-data-api-crawl-introduction)* calls through Decodo's [agent skills](https://help.decodo.com/docs/agent-skills). The dashboard's **Copy agent setup** button copies a setup prompt for your coding agent.

Activate the [free plan](https://dashboard.decodo.com/register?planId=922406), which includes monthly credit and requires no credit card, and map the docs site your agent uses.
