{"slug": "how-to-make-your-first-web-data-api-request", "title": "How To Make Your First Web Data API Request", "summary": "Decodo published a walkthrough for its Web Data API, which exposes /scrape, /search, /map and /crawl endpoints behind a single API key to return web pages as Markdown or HTML for AI agents and RAG pipelines. The company says proxy rotation, anti-bot handling and JavaScript rendering run server-side across its 125M+ IP network, and reports that /scrape returned a real Walmart results page in all 10 attempts versus 5 of 10 for plain requests with copied Chrome headers.", "body_md": "AI agents and RAG pipelines need web pages they can read. A plain HTTP request to a protected site can get a \"Robot or human?\" page instead of the content. Other pages arrive without their text, because it loads later with JavaScript. The Decodo Web Data API fetches those pages and returns them as Markdown or HTML.\n\nThis guide goes from sign-up to a working Python script in about 5 minutes. You'll send your first request, learn to read the response, and finish with a script that checks each page for the content your agent expects.\n\nThe [Web Data API](https://decodo.com/blog/introducing-web-data-api) gives you */scrape*, */search*, */map*, and */crawl* with a single API key. Pick one based on what your agent needs:\n\n| Endpoint | What it does | Use it when | \n|---|---|---|\n| [/scrape](https://decodo.com/web-data-api/scrape) | Fetches 1 page as HTML, Markdown, or a screenshot | Your agent needs 1 page | \n| [/search](https://decodo.com/web-data-api/search) | Returns Google search results as JSON, including organic results and top stories | Your agent needs current search results | \n| [/map](https://decodo.com/web-data-api/map) | Lists URLs from a page and the site's sitemap, in 1 call | Your agent needs a list of a site's pages first | \n| [/crawl](https://decodo.com/web-data-api/crawl) | Scrapes many pages of a site in 1 job, as Markdown or HTML | Your agent needs a whole docs site or catalog | \n\nProxy rotation, anti-bot handling, and JavaScript rendering run on our servers, so your code stays a few lines long. Every endpoint runs on Decodo's proxy network of [125M+ IPs, with a 99.99% success rate](https://decodo.com/web-data-api).\n\nThe difference is clearest on protected sites. On a Walmart search URL, */scrape* returned the real results page in all 10 attempts. Plain *requests* calls to the same URL from a home connection got HTTP 200 and a \"Robot or human?\" page every time. With headers copied from Chrome, *requests* got the real page in 5 of the 10 attempts.\n\n[Sign up](https://dashboard.decodo.com/register?page=web-data/pricing), then open **Web Data API** in the [Decodo dashboard](https://dashboard.decodo.com/web-data/playground). On the **Playground** tab, click **Activate free plan**:\n\nThe [pricing page](https://decodo.com/web-data-api/pricing) lists what the Free plan includes. Once the plan card shows **ACTIVE**, copy the **API key** from the top right of the same tab:\n\nRun the command for your shell, then paste the key. That way, the key isn't saved in your code or your shell history. On macOS and Linux, nothing appears while you paste. On Windows, the key is visible as you type it. Press Enter, then run the scripts from the same terminal:\n\n```\n# macOS and Linux\nread -rs DECODO_API_KEY && export DECODO_API_KEY\n\n# Windows PowerShell\n$env:DECODO_API_KEY = Read-Host \"Paste your API key\"\n```\n\nThe target, *[https://ip.decodo.com/json](https://ip.decodo.com/json)*, returns the IP and country that a request came from. Save this script as *first_request.py*:\n\n``` python\nimport json\nimport os\nimport sys\n\nimport requests\n\napi_key = os.environ.get(\"DECODO_API_KEY\")\nif not api_key:\n    sys.exit(\"DECODO_API_KEY is not set. Export your Web Data API key first.\")\n\nresponse = requests.post(\n    \"https://data.decodo.com/v1/scrape\",\n    headers={\"Authorization\": f\"Bearer {api_key}\"},\n    json={\"url\": \"https://ip.decodo.com/json\"},\n    timeout=180,  # in an agent tool, keep this below the tool's time limit\n)\nresponse.raise_for_status()\n\nresult = response.json()[\"results\"][0]\nip_info = json.loads(result[\"content\"])\nprint(\"target status code:\", result[\"status_code\"])\nprint(\"exit IP:\", ip_info[\"proxy\"][\"ip\"], ip_info[\"country\"][\"code\"])\n```\n\nWhen you run *python first_request.py*, it prints the IP address and country of the proxy that sent the request:\n\n```\ntarget status code: 200\nexit IP: 38.127.100.236 US\n```\n\nFrom a macOS or Linux terminal with cURL 7.82 or later, run the same request:\n\n```\ncurl https://data.decodo.com/v1/scrape \\\n  -H \"Authorization: Bearer $DECODO_API_KEY\" \\\n  --json '{\"url\": \"https://ip.decodo.com/json\"}'\n```\n\nThat's the whole request.\n\nHere's what step 2's request to *[https://ip.decodo.com/json](https://ip.decodo.com/json)* returned, shortened:\n\n```\n{\n  \"results\": [\n    {\n      \"content\": \"{\\n    \\\"browser\\\": {\\n        \\\"name\\\": \\\"Firefox\\\", ...\",\n      \"type\": \"raw\",\n      \"headers\": {\"content-type\": \"application/json\", \"server\": \"nginx\"},\n      \"cookies\": [],\n      \"status_code\": 200,\n      \"task_id\": \"7518637022634883073\"\n    }\n  ]\n}\n```\n\n**Read the target's status code inside the API's HTTP 200 response**. The API returns HTTP 200 whenever it has processed your request, so an HTTP 200 response can mean 3 different things:\n\nSo your code should check the HTTP status code, the *status* field, and *results[0].status_code*.\n\nYou pay only for requests that return a result, as the [billing page](https://help.decodo.com/docs/web-data-api-billing) explains. A target's error page with content, like the GitHub 404, is a result.\n\nThe practice site *[https://quotes.toscrape.com/js/](https://quotes.toscrape.com/js/)* loads its quotes with JavaScript. Save this as *render.py*. The script sends 2 requests for Markdown, and only the second enables rendering with *\"headless\": \"html\"*:\n\n``` python\nimport os\n\nimport requests\n\nAPI_URL = \"https://data.decodo.com/v1/scrape\"\nHEADERS = {\"Authorization\": f\"Bearer {os.environ['DECODO_API_KEY']}\"}\n\nfor label, extra in ((\"no rendering\", {}), (\"rendered\", {\"headless\": \"html\"})):\n    payload = {\"url\": \"https://quotes.toscrape.com/js/\", \"markdown\": True, **extra}\n    response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=180)\n    response.raise_for_status()\n    result = response.json()[\"results\"][0]\n    quotes = result[\"content\"].count(\"“\")  # each quote starts with a curly “\n    status, kind = result[\"status_code\"], result[\"type\"]\n    print(f\"{label}: target status code {status}, type {kind}, {quotes} quotes\")\n```\n\nWhen you run *python render.py*, it prints:\n\n```\nno rendering: target status code 200, type markdown, 0 quotes\nrendered: target status code 200, type markdown, 10 quotes\n```\n\nWithout rendering, the HTTP status code and the target status code both show success. Only the quote count shows the content is missing. The request with *\"headless\": \"html\"* returns all 10. With *\"markdown\": true*, the rendered page is still returned as Markdown, as the *type markdown* output confirms.\n\nEnable rendering for pages that load their content with JavaScript. Rendering runs a headless browser, so it costs more than a request without it.\n\nMarkdown keeps the structure a model needs in far fewer tokens than HTML. With OpenAI's *o200k_base* tokenizer, the Wikipedia article on web scraping was about 72,000 tokens as HTML and about 16,500 as Markdown. Plain text from Beautiful Soup's *get_text()* was smaller, but it lost the headings and links that Markdown keeps.\n\nTry the standard proxy pool on sites without bot protection. Send *\"proxy_pool\": \"standard\"*, which costs less. By default, requests use the premium pool, which is built for protected sites. If the content you expect is missing on the standard pool, switch to *\"proxy_pool\": \"premium\"*.\n\nOther options, such as *geo* for the proxy location, are in the [parameter reference](https://help.decodo.com/docs/web-data-api-scrape-parameters).\n\nSome errors require a change to your request, while others only require a retry. The table lists the common ones from the [status codes page](https://help.decodo.com/docs/web-data-api-status-codes):\n\n| Code | Meaning | First thing to try | \n|---|---|---|\n| `401` | Missing or invalid credentials | Check the key, the `Authorization: Bearer` header, and the`data.decodo.com` URL | \n| `400` ,`422` | Invalid request or invalid JSON | Read the error details in the response body | \n| `429` | Too many requests | Wait and retry. Your plan's rate limit is on the pricing page | \n| `500` ,`524` | Temporary server error | Retry shortly. These aren't charged | \n| `612` ,`613` | The request ended without the page. Returned inside an HTTP 200 response | Enable or disable rendering, or try another geo, then retry once | \n\nA *401* can also mean the request went to another API's host. Examples written for Decodo's Web Scraping API use *scraper-api.decodo.com* and that API's token. For the Web Data API, use your Web Data API key with *data.decodo.com*. The [migration guide](https://help.decodo.com/docs/web-data-api-migration-guide) explains how to move an existing Web Scraping API or Fast Search API setup to the Web Data API when you're ready.\n\nFor production, save the script below as *scrape.py*. It checks each response in this order:\n\nThe [Retry class](https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry) from urllib3 retries *429* and common *5xx* errors when *allowed_methods* includes POST. A *612* or *613* means our servers have already retried, so *scrape()* changes the request instead of resending it. A *613* isn't charged, so you pay only when an attempt returns a result:\n\n``` python\nimport os\nimport sys\n\nimport requests\nfrom requests.adapters import HTTPAdapter\nfrom urllib3.util import Retry\n\nAPI_URL = \"https://data.decodo.com/v1/scrape\"\n\nsession = requests.Session()\nsession.headers[\"Authorization\"] = f\"Bearer {os.environ['DECODO_API_KEY']}\"\nretry = Retry(\n    total=2,\n    read=False,  # don't resend a request that timed out or lost its connection\n    backoff_factor=2,\n    status_forcelist=[429, 500, 502, 503, 504, 524],\n    allowed_methods={\"POST\"},\n    raise_on_status=False,\n)\nsession.mount(\"https://\", HTTPAdapter(max_retries=retry))\n\ndef scrape(payload, expect=\"\"):\n    for attempt in (1, 2):\n        response = session.post(API_URL, json=payload, timeout=180)\n        try:\n            body = response.json()\n        except ValueError:\n            body = {}\n        if response.ok and body.get(\"status\") != \"failed\" and body.get(\"results\"):\n            result = body[\"results\"][0]\n            if expect in result[\"content\"] or result[\"status_code\"] >= 400:\n                return result\n            status = result[\"status_code\"]\n            note = f\"target status code {status}, {expect!r} not found\"\n            print(f\"attempt {attempt}: {note}\")\n        else:\n            code = body.get(\"status_code\", response.status_code)\n            message = body.get(\"message\", response.text[:200])\n            task_id = body.get(\"task_id\", \"-\")\n            print(f\"attempt {attempt}: {code} {message} (task_id {task_id})\")\n            if code not in (612, 613):\n                return None\n        payload = {**payload, \"headless\": \"html\"}\n    return None\n\nif len(sys.argv) < 2:\n    sys.exit(\"Usage: python scrape.py URL [EXPECTED_TEXT]\")\nurl = sys.argv[1]\nexpect = sys.argv[2] if len(sys.argv) > 2 else \"\"\nresult = scrape({\"url\": url, \"markdown\": True}, expect)\nif result is None:\n    sys.exit(1)\nif result[\"status_code\"] >= 400:\n    print(\"the target returned\", result[\"status_code\"])\nelif expect:\n    print(result[\"content\"].count(expect), \"matches for\", repr(expect))\nprint(len(result[\"content\"]), \"characters of Markdown\")\n```\n\nRun it on the quotes page from Step 4, and pass the curly quote mark (“) as the text the page should contain. Every quote on that page starts with this mark. After the first attempt, the script enables rendering:\n\n``` bash\n$ python scrape.py https://quotes.toscrape.com/js/ \"“\"\nattempt 1: target status code 200, '“' not found\n10 matches for '“'\n1567 characters of Markdown\n```\n\nAgents often search first, then scrape only the result they pick. The next script follows that pattern, then runs */map* and */crawl* on a docs site. Every request uses the same header. Save it as *endpoints.py*:\n\n``` python\nimport os\nimport time\n\nimport requests\n\nBASE_URL = \"https://data.decodo.com/v1\"\nHEADERS = {\"Authorization\": f\"Bearer {os.environ['DECODO_API_KEY']}\"}\n\ndef post(endpoint, payload):\n    url = f\"{BASE_URL}/{endpoint}\"\n    response = requests.post(url, headers=HEADERS, json=payload, timeout=180)\n    response.raise_for_status()\n    return response.json()\n\ndef get(endpoint):\n    url = f\"{BASE_URL}/{endpoint}\"\n    response = requests.get(url, headers=HEADERS, timeout=30)\n    response.raise_for_status()\n    return response.json()\n\nsearch = post(\"search\", {\"query\": \"mcp server oauth\"})\ntop_link = search[\"organic\"][0][\"link\"]\npage = post(\"scrape\", {\"url\": top_link, \"markdown\": True})[\"results\"][0]\nsite = top_link.split(\"/\")[2]\nprint(\"search: top result on\", site, \"| target status code\", page[\"status_code\"])\n\nsite_map = post(\"map\", {\"url\": \"https://help.decodo.com\"})\nprint(\"map:\", len(site_map[\"results\"][0][\"content\"][\"links\"]), \"URLs\")\n\ncrawl_id = post(\"crawl\", {\"url\": \"https://help.decodo.com\", \"limit\": 5})[\"crawl_id\"]\njob = get(f\"crawl/{crawl_id}/status\")\nwhile job[\"status\"] == \"pending\":  # every other status means the crawl has ended\n    time.sleep(10)\n    job = get(f\"crawl/{crawl_id}/status\")\npages = get(f\"crawl/{crawl_id}?type=content\")[\"url\"]  # \"url\" holds the page list\nprint(\"crawl:\", job[\"status\"], \"|\", len(pages), \"pages, first:\", pages[0][\"url\"])\n```\n\nThe crawl runs as a background job. The script checks its status every 10 seconds and fetches the crawled pages once the crawl ends. When you run *python endpoints.py*, it prints output like this. Your results and counts will differ as the sites change:\n\n```\nsearch: top result on modelcontextprotocol.io | target status code 200\nmap: 801 URLs\ncrawl: done | 5 pages, first: https://help.decodo.com/\n```\n\nFor larger crawls, follow the *next* link in the response to get the remaining pages, as the [crawl results docs](https://help.decodo.com/docs/web-data-api-crawl-status-results-cancellation) explain. For the full list of fields that each endpoint returns, see the [/search docs](https://help.decodo.com/docs/web-data-api-search-introduction), [/map docs](https://help.decodo.com/docs/web-data-api-map-introduction), and [/crawl docs](https://help.decodo.com/docs/web-data-api-crawl-introduction).\n\nEvery later request works like the first one in step 2. It uses the same key, header, and JSON body across all 4 endpoints.\n\nTo give your agent our web data tools over the Model Context Protocol (MCP), connect it to our hosted [MCP server](https://decodo.com/ai/mcp-server). It runs at *[https://mcp.decodo.com/mcp](https://mcp.decodo.com/mcp)* and accepts your Web Data API key as a Bearer token. It's also listed in [GitHub's MCP Registry](https://github.com/mcp/decodo/mcp-server).\n\nInstall *langchain-decodo* for the [LangChain integration](https://decodo.com/integrations/langchain), or *llama-index-readers-decodo* and *llama-index-tools-decodo* for LlamaIndex. These packages read the key from the *DECODO_API_TOKEN* environment variable, so set that variable to your key too. Then pass *auth_mode=\"token\"* when you create a tool, for example, *DecodoWebScrapeTool(auth_mode=\"token\")*. The [n8n node](https://decodo.com/integrations/n8n) accepts the key with no extra settings.\n\nNext, run *scrape.py* with a page your agent needs and a piece of text that page should contain. If you haven't signed up yet, start with the [Web Data API free plan](https://dashboard.decodo.com/register?page=web-data/pricing). The [Web Data API documentation](https://help.decodo.com/docs/web-data-api-introduction) describes the endpoints and their parameters in detail.", "url": "https://wpnews.pro/news/how-to-make-your-first-web-data-api-request", "canonical_source": "https://dev.to/decodo_official/how-to-make-your-first-web-data-api-request-ool", "published_at": "2026-11-04 10:47:57+00:00", "updated_at": "2026-11-04 10:52:38.425748+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-crawlers", "developer-tools"], "entities": ["Decodo", "Walmart", "Google"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-make-your-first-web-data-api-request", "markdown": "https://wpnews.pro/news/how-to-make-your-first-web-data-api-request.md", "text": "https://wpnews.pro/news/how-to-make-your-first-web-data-api-request.txt", "jsonld": "https://wpnews.pro/news/how-to-make-your-first-web-data-api-request.jsonld"}}