A scraper gives you a page as one HTML string: markup, inline styles, scripts, and somewhere inside it the text you wanted. Many pipelines send that string directly to an LLM. That works, so the cost rarely gets measured. But on the 10 pages I measured, the HTML used 3 to 24 times more tokens than the same pages as Markdown. A product page arrived as 267,361 tokens, of which 3,393 were visible text. This guide shows where those tokens go, what they cost, and how to stop sending them.
*Every number here is from my own runs, September 2026.
I fetched each page once with requests.get and a Chrome User-Agent, then removed one category at a time and re-counted with tiktoken's o200k_base. Each category is measured after the ones above it, so an SVG inside nav counts as SVG, and rows sum up to the fetched total:
6 of 10 pages. Markdown keeps structure, so it’s more than that share.
The Hydration JSON is the largest cost on the Next.js docs page, most of it in 65 *self.*next_f.push calls repeating the content as escaped strings. The Hacker News front page has almost no scripts or styles and is still 90% markup, because it's built from layout tables. Docs, news, and storefronts would each need a different rule.
Tokens. At $2.00 per 1M input tokens, a rate several current models charge, those 10 pages cost $3.04 as HTML and $0.25 as Markdown. Per 1K of the heaviest pages, that's $1,157 instead of $49, plus scrapes at current rates. A different tokenizer changes the totals: across OpenAI, Anthropic, and Google the same HTML was up to 50% larger, while the pooled ratio stayed between 11× and 14×. So, trust the ratio and re-count the dollars for your model.
Time to first token. I asked one question per page on both versions, with a unique first line per call to prevent caching. On gpt-5.6-terra, the raw version took 2.0 to 2.8 times longer to start on the 4 pages above 130K tokens, and 1.0 to 1.4 times longer on the 4 below 75K. The other 2 models were slower, up to 5.7 times.
Rate limits. A 57,266-token page was refused on a gpt-4.1 account plan: Request too large … Limit 30000, Requested 57266. It had 622 tokens of text.
All 3 models across 2 vendors answered every question correctly from the HTML and Markdown, 60 answers each way. With no page attached they answered 1 or 2 of them, so they were reading rather than recalling. I put 4 raw pages into 396K-token prompts and got the right answer in all 3 runs.
Markdown conversion removes the markup around the content, including style blocks, SVG paths, class and data- attributes, and the scripts. It keeps headings, lists, tables, and link URLs. On the Hacker News front page, it returned Markdown tables with the story titles, URLs, scores, and comment counts in 3,644 tokens instead of 11,817.
Markdown keeps prose, and JSON-LD is in a script block. When I asked for a product's price, rating, and review count, the model got all 3 from the raw HTML and only the price from the Markdown, because the rest are in JSON-LD. On 3 of the 5 pages with it, ratings, stock status, and publication dates were absent. Markdown still kept 99.4% of that page’s words, because the rating and review count are 2 of 16,844.
Read the application/ld+json blocks with an HTML parser, since quoting varies, and send the model the prose.
Decodo's Web Scraping API takes a markdown flag that runs the conversion server-side, so the parsing happens outside your pipeline. The free plan starts immediately after signup.
DECODO_TOKEN is the Basic authentication token in the Playground tab, which builds the same request:
Create a virtual environment, install the packages, and export the token:
python3 -m venv .venv && source .venv/bin/activate
pip install requests tiktoken
export DECODO_TOKEN="paste-your-basic-auth-token"
Once you’ve set up your environment and .env file, create a script file with the following code:
import os
import requests
import tiktoken
API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = os.environ["DECODO_TOKEN"]
enc = tiktoken.get_encoding("o200k_base")
def fetch(url, markdown):
payload = {
"url": url,
"proxy_pool": "standard",
"markdown": markdown,
}
try:
response = requests.post(
API_URL,
json=payload,
headers={"Authorization": f"Basic {TOKEN}"},
timeout=120,
)
except requests.RequestException as e:
raise SystemExit(f"Request failed before a reply: {e}")
if response.status_code >= 400:
raise SystemExit(f"HTTP {response.status_code}: {response.text[:200]}")
try:
data = response.json()
except ValueError:
raise SystemExit(f"Reply was not JSON: {response.text[:200]}")
if not data.get("results"):
raise SystemExit(f"Scrape failed: {data.get('message')}")
result = data["results"][0]
code = result.get("status_code", 200)
if code != 200:
raise SystemExit(f"Target returned {code}, not the page")
if result.get("url", url) != url:
print(f"note: redirected to {result['url']}")
content = result.get("content")
if not content or not content.strip():
raise SystemExit("Empty body: a shell or a challenge, not the page")
return content
url = "https://en.wikipedia.org/wiki/Web_scraping"
html = fetch(url, markdown=False)
md = fetch(url, markdown=True)
print(f"HTML: {len(enc.encode(html, disallowed_special=())):>7,} tokens")
print(f"Markdown: {len(enc.encode(md, disallowed_special=())):>7,} tokens")
Running it prints both counts:
HTML: 72,201 tokens
Markdown: 16,553 tokens
Reduction across my 10 pages was 3× to 24×, median 7×. A ratio alone means little, since deleting content improves it. A local extractor reached 856× on a storefront listing by keeping 3.6% of its words, whereas this Markdown kept 99.4% at 24×.
Fetching is the harder half. From one home connection, a plain fetch reached the page on 4 of the 20 commercial sites I tested. It missed a job board at 59× and a property listing at 61×. Free html2text kept more of the words than the API's median 98.4% on pages a plain fetch reaches. The parameters reference lists the fields.
Decodo's MCP server offers this conversion as an agent tool.
A 200 that isn't the page. A plain GET to a home-improvement retailer returned HTTP 200 with an "Access Denied" body of 67 visible-text tokens, whereas the API on the premium pool returned the homepage, 3 runs of 3. A department store served a challenge script both ways, 152,000 tokens of it and no visible text. No single limit separated them – a real docs page had 622 visible-text tokens, fewer than a 909-token "browser not supported" page. Check the target's own status code in results[0], then compare each URL with its last good fetch.
Rendering worth confirming. "headless": "html" on a client-rendered page returned the rendered content in 12 of 14 runs. The other 2 returned the unrendered page, 62 tokens, at HTTP 200. The rendered page had 6 times more text.
Extractor coverage. A readability-style extractor kept a median 13% of these pages' words, and 0% on Hacker News. Another extractor kept 99.7% there, and the whole-page conversion kept 69.6%, its lowest. Test yours on a listings page.
Raw HTML in a prompt is mostly markup. On the pages I measured, visible text was a median 2.3% of what got sent. That waste costs you a larger bill, a slower start, and a rate limit you didn't plan for, not a wrong answer on the 3 models I tested, so check accuracy on yours. Converting server-side removes the cleanup code from your pipeline, and the HTML response has JSON-LD for pages where you need exact fields. When a page needs rendering or a different pool, check the target's own status and the returned content, because both failures return HTTP 200. Run that snippet on a URL your pipeline already fetches, and compare the 2 numbers before you change anything.