How To Make Your First Web Data API Request

Decodo published a walkthrough for its Web Data API, which exposes /scrape, /search, /map and /crawl endpoints behind a single API key to return web pages as Markdown or HTML for AI agents and RAG pipelines. The company says proxy rotation, anti-bot handling and JavaScript rendering run server-side across its 125M+ IP network, and reports that /scrape returned a real Walmart results page in all 10 attempts versus 5 of 10 for plain requests with copied Chrome headers.

AI agents and RAG pipelines need web pages they can read. A plain HTTP request to a protected site can get a "Robot or human?" page instead of the content. Other pages arrive without their text, because it loads later with JavaScript. The Decodo Web Data API fetches those pages and returns them as Markdown or HTML. This guide goes from sign-up to a working Python script in about 5 minutes. You'll send your first request, learn to read the response, and finish with a script that checks each page for the content your agent expects. The Web Data API https://decodo.com/blog/introducing-web-data-api gives you /scrape , /search , /map , and /crawl with a single API key. Pick one based on what your agent needs: | Endpoint | What it does | Use it when | |---|---|---| | /scrape https://decodo.com/web-data-api/scrape | Fetches 1 page as HTML, Markdown, or a screenshot | Your agent needs 1 page | | /search https://decodo.com/web-data-api/search | Returns Google search results as JSON, including organic results and top stories | Your agent needs current search results | | /map https://decodo.com/web-data-api/map | Lists URLs from a page and the site's sitemap, in 1 call | Your agent needs a list of a site's pages first | | /crawl https://decodo.com/web-data-api/crawl | Scrapes many pages of a site in 1 job, as Markdown or HTML | Your agent needs a whole docs site or catalog | Proxy rotation, anti-bot handling, and JavaScript rendering run on our servers, so your code stays a few lines long. Every endpoint runs on Decodo's proxy network of 125M+ IPs, with a 99.99% success rate https://decodo.com/web-data-api . The difference is clearest on protected sites. On a Walmart search URL, /scrape returned the real results page in all 10 attempts. Plain requests calls to the same URL from a home connection got HTTP 200 and a "Robot or human?" page every time. With headers copied from Chrome, requests got the real page in 5 of the 10 attempts. Sign up https://dashboard.decodo.com/register?page=web-data/pricing , then open Web Data API in the Decodo dashboard https://dashboard.decodo.com/web-data/playground . On the Playground tab, click Activate free plan : The pricing page https://decodo.com/web-data-api/pricing lists what the Free plan includes. Once the plan card shows ACTIVE , copy the API key from the top right of the same tab: Run the command for your shell, then paste the key. That way, the key isn't saved in your code or your shell history. On macOS and Linux, nothing appears while you paste. On Windows, the key is visible as you type it. Press Enter, then run the scripts from the same terminal: macOS and Linux read -rs DECODO API KEY && export DECODO API KEY Windows PowerShell $env:DECODO API KEY = Read-Host "Paste your API key" The target, https://ip.decodo.com/json https://ip.decodo.com/json , returns the IP and country that a request came from. Save this script as first request.py : python import json import os import sys import requests api key = os.environ.get "DECODO API KEY" if not api key: sys.exit "DECODO API KEY is not set. Export your Web Data API key first." response = requests.post "https://data.decodo.com/v1/scrape", headers={"Authorization": f"Bearer {api key}"}, json={"url": "https://ip.decodo.com/json"}, timeout=180, in an agent tool, keep this below the tool's time limit response.raise for status result = response.json "results" 0 ip info = json.loads result "content" print "target status code:", result "status code" print "exit IP:", ip info "proxy" "ip" , ip info "country" "code" When you run python first request.py , it prints the IP address and country of the proxy that sent the request: target status code: 200 exit IP: 38.127.100.236 US From a macOS or Linux terminal with cURL 7.82 or later, run the same request: curl https://data.decodo.com/v1/scrape \ -H "Authorization: Bearer $DECODO API KEY" \ --json '{"url": "https://ip.decodo.com/json"}' That's the whole request. Here's what step 2's request to https://ip.decodo.com/json https://ip.decodo.com/json returned, shortened: { "results": { "content": "{\n \"browser\": {\n \"name\": \"Firefox\", ...", "type": "raw", "headers": {"content-type": "application/json", "server": "nginx"}, "cookies": , "status code": 200, "task id": "7518637022634883073" } } Read the target's status code inside the API's HTTP 200 response . The API returns HTTP 200 whenever it has processed your request, so an HTTP 200 response can mean 3 different things: So your code should check the HTTP status code, the status field, and results 0 .status code . You pay only for requests that return a result, as the billing page https://help.decodo.com/docs/web-data-api-billing explains. A target's error page with content, like the GitHub 404, is a result. The practice site https://quotes.toscrape.com/js/ https://quotes.toscrape.com/js/ loads its quotes with JavaScript. Save this as render.py . The script sends 2 requests for Markdown, and only the second enables rendering with "headless": "html" : python import os import requests API URL = "https://data.decodo.com/v1/scrape" HEADERS = {"Authorization": f"Bearer {os.environ 'DECODO API KEY' }"} for label, extra in "no rendering", {} , "rendered", {"headless": "html"} : payload = {"url": "https://quotes.toscrape.com/js/", "markdown": True, extra} response = requests.post API URL, headers=HEADERS, json=payload, timeout=180 response.raise for status result = response.json "results" 0 quotes = result "content" .count "“" each quote starts with a curly “ status, kind = result "status code" , result "type" print f"{label}: target status code {status}, type {kind}, {quotes} quotes" When you run python render.py , it prints: no rendering: target status code 200, type markdown, 0 quotes rendered: target status code 200, type markdown, 10 quotes Without rendering, the HTTP status code and the target status code both show success. Only the quote count shows the content is missing. The request with "headless": "html" returns all 10. With "markdown": true , the rendered page is still returned as Markdown, as the type markdown output confirms. Enable rendering for pages that load their content with JavaScript. Rendering runs a headless browser, so it costs more than a request without it. Markdown keeps the structure a model needs in far fewer tokens than HTML. With OpenAI's o200k base tokenizer, the Wikipedia article on web scraping was about 72,000 tokens as HTML and about 16,500 as Markdown. Plain text from Beautiful Soup's get text was smaller, but it lost the headings and links that Markdown keeps. Try the standard proxy pool on sites without bot protection. Send "proxy pool": "standard" , which costs less. By default, requests use the premium pool, which is built for protected sites. If the content you expect is missing on the standard pool, switch to "proxy pool": "premium" . Other options, such as geo for the proxy location, are in the parameter reference https://help.decodo.com/docs/web-data-api-scrape-parameters . Some errors require a change to your request, while others only require a retry. The table lists the common ones from the status codes page https://help.decodo.com/docs/web-data-api-status-codes : | Code | Meaning | First thing to try | |---|---|---| | 401 | Missing or invalid credentials | Check the key, the Authorization: Bearer header, and the data.decodo.com URL | | 400 , 422 | Invalid request or invalid JSON | Read the error details in the response body | | 429 | Too many requests | Wait and retry. Your plan's rate limit is on the pricing page | | 500 , 524 | Temporary server error | Retry shortly. These aren't charged | | 612 , 613 | The request ended without the page. Returned inside an HTTP 200 response | Enable or disable rendering, or try another geo, then retry once | A 401 can also mean the request went to another API's host. Examples written for Decodo's Web Scraping API use scraper-api.decodo.com and that API's token. For the Web Data API, use your Web Data API key with data.decodo.com . The migration guide https://help.decodo.com/docs/web-data-api-migration-guide explains how to move an existing Web Scraping API or Fast Search API setup to the Web Data API when you're ready. For production, save the script below as scrape.py . It checks each response in this order: The Retry class https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html urllib3.util.Retry from urllib3 retries 429 and common 5xx errors when allowed methods includes POST. A 612 or 613 means our servers have already retried, so scrape changes the request instead of resending it. A 613 isn't charged, so you pay only when an attempt returns a result: python import os import sys import requests from requests.adapters import HTTPAdapter from urllib3.util import Retry API URL = "https://data.decodo.com/v1/scrape" session = requests.Session session.headers "Authorization" = f"Bearer {os.environ 'DECODO API KEY' }" retry = Retry total=2, read=False, don't resend a request that timed out or lost its connection backoff factor=2, status forcelist= 429, 500, 502, 503, 504, 524 , allowed methods={"POST"}, raise on status=False, session.mount "https://", HTTPAdapter max retries=retry def scrape payload, expect="" : for attempt in 1, 2 : response = session.post API URL, json=payload, timeout=180 try: body = response.json except ValueError: body = {} if response.ok and body.get "status" = "failed" and body.get "results" : result = body "results" 0 if expect in result "content" or result "status code" = 400: return result status = result "status code" note = f"target status code {status}, {expect r} not found" print f"attempt {attempt}: {note}" else: code = body.get "status code", response.status code message = body.get "message", response.text :200 task id = body.get "task id", "-" print f"attempt {attempt}: {code} {message} task id {task id} " if code not in 612, 613 : return None payload = { payload, "headless": "html"} return None if len sys.argv < 2: sys.exit "Usage: python scrape.py URL EXPECTED TEXT " url = sys.argv 1 expect = sys.argv 2 if len sys.argv 2 else "" result = scrape {"url": url, "markdown": True}, expect if result is None: sys.exit 1 if result "status code" = 400: print "the target returned", result "status code" elif expect: print result "content" .count expect , "matches for", repr expect print len result "content" , "characters of Markdown" Run it on the quotes page from Step 4, and pass the curly quote mark “ as the text the page should contain. Every quote on that page starts with this mark. After the first attempt, the script enables rendering: bash $ python scrape.py https://quotes.toscrape.com/js/ "“" attempt 1: target status code 200, '“' not found 10 matches for '“' 1567 characters of Markdown Agents often search first, then scrape only the result they pick. The next script follows that pattern, then runs /map and /crawl on a docs site. Every request uses the same header. Save it as endpoints.py : python import os import time import requests BASE URL = "https://data.decodo.com/v1" HEADERS = {"Authorization": f"Bearer {os.environ 'DECODO API KEY' }"} def post endpoint, payload : url = f"{BASE URL}/{endpoint}" response = requests.post url, headers=HEADERS, json=payload, timeout=180 response.raise for status return response.json def get endpoint : url = f"{BASE URL}/{endpoint}" response = requests.get url, headers=HEADERS, timeout=30 response.raise for status return response.json search = post "search", {"query": "mcp server oauth"} top link = search "organic" 0 "link" page = post "scrape", {"url": top link, "markdown": True} "results" 0 site = top link.split "/" 2 print "search: top result on", site, "| target status code", page "status code" site map = post "map", {"url": "https://help.decodo.com"} print "map:", len site map "results" 0 "content" "links" , "URLs" crawl id = post "crawl", {"url": "https://help.decodo.com", "limit": 5} "crawl id" job = get f"crawl/{crawl id}/status" while job "status" == "pending": every other status means the crawl has ended time.sleep 10 job = get f"crawl/{crawl id}/status" pages = get f"crawl/{crawl id}?type=content" "url" "url" holds the page list print "crawl:", job "status" , "|", len pages , "pages, first:", pages 0 "url" The crawl runs as a background job. The script checks its status every 10 seconds and fetches the crawled pages once the crawl ends. When you run python endpoints.py , it prints output like this. Your results and counts will differ as the sites change: search: top result on modelcontextprotocol.io | target status code 200 map: 801 URLs crawl: done | 5 pages, first: https://help.decodo.com/ For larger crawls, follow the next link in the response to get the remaining pages, as the crawl results docs https://help.decodo.com/docs/web-data-api-crawl-status-results-cancellation explain. For the full list of fields that each endpoint returns, see the /search docs https://help.decodo.com/docs/web-data-api-search-introduction , /map docs https://help.decodo.com/docs/web-data-api-map-introduction , and /crawl docs https://help.decodo.com/docs/web-data-api-crawl-introduction . Every later request works like the first one in step 2. It uses the same key, header, and JSON body across all 4 endpoints. To give your agent our web data tools over the Model Context Protocol MCP , connect it to our hosted MCP server https://decodo.com/ai/mcp-server . It runs at https://mcp.decodo.com/mcp https://mcp.decodo.com/mcp and accepts your Web Data API key as a Bearer token. It's also listed in GitHub's MCP Registry https://github.com/mcp/decodo/mcp-server . Install langchain-decodo for the LangChain integration https://decodo.com/integrations/langchain , or llama-index-readers-decodo and llama-index-tools-decodo for LlamaIndex. These packages read the key from the DECODO API TOKEN environment variable, so set that variable to your key too. Then pass auth mode="token" when you create a tool, for example, DecodoWebScrapeTool auth mode="token" . The n8n node https://decodo.com/integrations/n8n accepts the key with no extra settings. Next, run scrape.py with a page your agent needs and a piece of text that page should contain. If you haven't signed up yet, start with the Web Data API free plan https://dashboard.decodo.com/register?page=web-data/pricing . The Web Data API documentation https://help.decodo.com/docs/web-data-api-introduction describes the endpoints and their parameters in detail.