{"slug": "web-scraping-7000-pages-in-3-minutes-with-python-and-olostep", "title": "Web Scraping 7,000 Pages in 3 Minutes with Python and Olostep", "summary": "A test using Python 3.13.15 and Olostep SDK 1.2.2 processed 7,225 of 10,000 MDN documentation pages in under 3 minutes through the Olostep Batch API, but the full job — scraping, downloading and verifying — took 22 minutes and 26 seconds, with 9,938 pages saved as Markdown files and 62 missing. Olostep, which is positioning itself as web data infrastructure for enterprises, says the platform can process 100,000 pages in under 25 minutes for enterprise workloads, according to its co-founder. The test underscores that a completed batch does not guarantee every page was saved, making verification as important as scraping speed for reliable data pipelines.", "body_md": "# Web Scraping 7,000 Pages in 3 Minutes with Python and Olostep\n\nScrape thousands of pages in a few minutes with the Olostep Batch API, download Markdown, and measure what actually finished.\n\nScraping thousands of web pages can take hours, especially when you are collecting data for enterprise applications, market research, price monitoring, or large-scale AI systems. The challenge is not just scraping pages quickly, but making sure the process is reliable, scalable, and easy to manage.\n\nI wanted to see how fast I could scrape 10,000 web pages using Python and the **[Olostep Batch API](https://www.olostep.com/api-endpoints/batches)**. In my test, **7,225 pages were processed in under 3 minutes**, which was impressive. However, completing the entire job, including scraping, downloading, and verifying the results, took 22 minutes and 26 seconds. Out of 10,000 URLs, 9,938 were successfully saved as Markdown files.\n\nIn this tutorial, I will show you how to use Python and Olostep to scrape thousands of pages in a single batch, track progress, download the content, and verify the results. I will also share the actual performance numbers, including what worked well and where the process slowed down.\n\n## Understanding How the Olostep Batch API Works\n\nThe [Olostep Batch API](https://docs.olostep.com/features/batches) lets you scrape up to 10,000 URLs in a single batch instead of sending thousands of individual requests. It handles the processing while you track progress, retrieve completed results, and download the extracted content.\n\nOlostep is also expanding beyond developer-focused scraping tools to provide **web data infrastructure for enterprises**. According to its co-founder, the platform can process 100,000 pages in under 25 minutes for enterprise workloads. This kind of scale is useful for companies that regularly collect large amounts of web data for market research, competitor monitoring, product intelligence, and AI applications.\n\nFor this tutorial, I used the Olostep Python SDK to scrape **10,000 [MDN](https://developer.mozilla.org/) documentation pages** and save their content as Markdown files. The workflow involves five steps:\n\n1. **Prepare the URLs:** Download the MDN sitemap and select 10,000 unique pages.\n2. **Submit the batch:** Send all URLs to Olostep in a single job.\n3. **Track progress:** Monitor completed pages until scraping finishes.\n4. **Download the results:** Retrieve the processed content and save it as Markdown.\n5. **Verify the files:** Check the saved files, identify missing results, and measure the total execution time.\n\nThe workflow diagram below shows the complete process, from collecting URLs to verifying the final output.\n\nOne important thing to understand is that **a completed batch does not necessarily mean every page was successfully saved**. In my test, 9,938 out of 10,000 pages were saved, with 62 missing. This is why verifying the results matters just as much as scraping speed, especially when building reliable data pipelines.\n\n## Scraping 10,000 Pages with the Olostep Batch API\n\nWe will use the Olostep Batch API to scrape 10,000 MDN documentation pages, download their content as Markdown files, and verify how many pages were successfully collected.\n\nFor this experiment, I used **Python 3.13.15** and **Olostep SDK 1.2.2**. The process is divided into six steps, from setting up the environment to checking the final results. You will need an [Olostep API key](https://docs.olostep.com/get-started/authentication) to follow along. You can either set it as an environment variable (`OLOSTEP_API_KEY`) or enter it securely when prompted.\n\n**Note:** Submitting a new batch consumes API credits, so make sure your account has sufficient credits before running the code.\n\n#### 1. Install the SDK\n\nStart by opening a new **[Jupyter Notebook](https://jupyter.org/)** or launching the notebook provided with this tutorial. In the first cell, install the Olostep Python SDK, along with **[requests](https://requests.readthedocs.io/)** for fetching the MDN sitemap and **[tqdm](https://github.com/tqdm/tqdm)** for tracking scraping progress.\n\n```\n%pip -q install olostep==1.2.2 requests tqdm\n```\n\n#### 2. Configure the Client\n\nNext, we will set up the Olostep client using our API key, create a folder to store the scraped data, and define a few helper functions to save and validate the results. We will also add a small workaround for a response-handling issue in SDK version 1.2.2 to make sure we can check the batch status without errors.\n\n``` python\nimport asyncio, csv, gzip, hashlib, json, os, random, time\nfrom datetime import datetime, timezone\nfrom getpass import getpass\nfrom pathlib import Path\nfrom xml.etree import ElementTree as ET\n\nimport requests\nfrom olostep import AsyncOlostep\nfrom olostep.errors import OlostepClientError_ResponseValidationFailed\nfrom olostep.models.response import BatchInfoResponse\nfrom tqdm import tqdm\n\nN = 10000\nMAX_WAIT_SECONDS = 2400\nDATA = Path(\"data\"); DATA.mkdir(exist_ok=True)\nclient = AsyncOlostep(api_key=os.getenv(\"OLOSTEP_API_KEY\") or getpass(\"Olostep API key: \"))\n\ndef write_json(path, value):\n    path.write_text(json.dumps(value, indent=2), encoding=\"utf-8\")\n\ndef valid_file(path):\n    try:\n        return path.is_file() and bool(path.read_text(encoding=\"utf-8\").strip())\n    except (OSError, UnicodeError):\n        return False\n\nasync def batch_info(batch_id):\n    try:\n        return await client.batches.info(batch_id)\n    except OlostepClientError_ResponseValidationFailed as error:\n        return BatchInfoResponse.model_validate({\"parser\": \"none\", **json.loads(error.response.body)})\n```\n\n#### 3. Build the URL Dataset\n\nNow, we will download the MDN sitemap and randomly select 10,000 unique documentation pages for our experiment.\n\nThe code saves these URLs in a CSV file so we can reuse the same list without downloading it again. It also checks for duplicates and makes sure we are working with the correct dataset.\n\n```\nsource = \"https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz\"\n\nif not (DATA / \"urls.csv\").exists():\n    response = requests.get(source, timeout=60); response.raise_for_status()\n    payload = response.content\n    xml = gzip.decompress(payload) if payload.startswith(b\"\\x1f\\x8b\") else payload\n    (DATA / \"mdn_sitemap.xml\").write_bytes(xml)\n    ns = {\"s\": \"http://www.sitemaps.org/schemas/sitemap/0.9\"}\n    nodes = ET.fromstring(xml).findall(\"s:url\", ns)\n    urls = sorted({node.findtext(\"s:loc\", namespaces=ns) for node in nodes})\n    urls = [url for url in urls if url and url.startswith(\"https://developer.mozilla.org/en-US/docs/\")]\n    random.Random(42).shuffle(urls)\n    assert len(urls) >= N, f\"Only {len(urls)} unique documentation URLs available.\"\n    with (DATA / \"urls.csv\").open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n        writer = csv.writer(f); writer.writerow([\"url\"]); writer.writerows([url] for url in urls[:N])\n    write_json(DATA / \"dataset.json\", {\"source\": source, \"downloaded_utc\": datetime.now(timezone.utc).isoformat(),\n                                     \"available_urls\": len(urls), \"selected_urls\": N, \"random_seed\": 42})\n\nwith (DATA / \"urls.csv\").open(encoding=\"utf-8\", newline=\"\") as f:\n    urls = [row[\"url\"] for row in csv.DictReader(f)]\n\nassert len(urls) == len(set(urls)) == N\nfingerprint = hashlib.sha256(\"\\n\".join(urls).encode()).hexdigest()\nexpected = {f\"{i:05d}\": url for i, url in enumerate(urls)}\nprint(f\"Ready: {len(urls):,} unique URLs; source: {source}\")\n```\n\nOutput:\n\n```\nReady: 10,000 unique URLs; source: https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz\n```\n\nThe output confirms that our dataset contains 10,000 unique URLs, ready for batch scraping.\n\n#### 4. Submit and Monitor the Batch\n\nNext, we will submit all 10,000 URLs to Olostep in a single batch and monitor the scraping progress using a progress bar.\n\nThe code also saves the batch ID, allowing us to resume the same job instead of starting over if the process gets interrupted.\n\n```\ncheckpoint = Path(\"batch.json\")\nresumed = checkpoint.exists()\n\nif resumed:\n    run = json.loads(checkpoint.read_text(encoding=\"utf-8\"))\n    assert run[\"urls_sha256\"] == fingerprint, \"URL list changed; preserve the checkpoint before starting another batch.\"\nelse:\n    started = time.time()\n    batch = await client.batches.create(urls=[{\"url\": url, \"custom_id\": key} for key, url in expected.items()])\n    run = {\"batch_id\": batch.id, \"requested\": N, \"urls_sha256\": fingerprint, \"started_wall\": started}\n    write_json(checkpoint, run)\n\nOUT = Path(\"output\") / run[\"batch_id\"]; OUT.mkdir(parents=True, exist_ok=True)\nprint(\"Resuming\" if resumed else \"Submitted\", run[\"batch_id\"])\nwait_started = time.monotonic()\n\nwith tqdm(total=N, desc=\"Scraping\", unit=\"page\") as bar:\n    while True:\n        info = await batch_info(run[\"batch_id\"])\n        bar.update(max(0, info.completed_urls - bar.n))\n        write_json(OUT / \"progress.json\", {\"phase\": \"scraping\", \"status\": info.status,\n                   \"processed\": info.completed_urls, \"total\": N, \"elapsed_seconds\": round(time.time() - run[\"started_wall\"], 2)})\n        if info.status in (\"completed\", \"failed\"):\n            break\n        if time.monotonic() - wait_started > MAX_WAIT_SECONDS:\n            raise TimeoutError(\"Batch still running. Rerun this cell to continue the same batch.\")\n        await asyncio.sleep(10)\n\nrun.setdefault(\"scrape_seconds\", time.time() - run[\"started_wall\"])\nwrite_json(checkpoint, run)\nprint(f\"Scraping stage finished: {info.status}; {info.completed_urls:,}/{N:,} completed.\")\n```\n\n**What the output showed:** After approximately 2 minutes and 52 seconds, Olostep had completed 7,225 pages. However, reaching 9,938 completed pages took around 18 minutes and 30 seconds. This shows that the initial scraping speed was impressive, but the remaining pages took considerably longer to process.\n\n#### 5. Download the Markdown\n\nOnce scraping finishes, we will retrieve the completed pages and save their content as individual Markdown files.\n\nTo speed up downloading, the code handles up to 96 retrieval requests concurrently. It also skips files that have already been saved, making it easier to resume an interrupted download.\n\n```\ndownload_started = time.perf_counter()\nlimit = asyncio.Semaphore(96)\nerrors, scheduled, tasks = {}, set(), []\nsaved = set()\nbar = tqdm(total=N, desc=\"Markdown saved\", unit=\"file\")\n\nasync def save_markdown(item):\n    key = item.custom_id\n    if key not in expected:\n        return\n    async with limit:\n        path = OUT / f\"{key}.md\"\n        if valid_file(path):\n            saved.add(key); bar.update(1)\n            return\n        for attempt in range(2):\n            try:\n                async with asyncio.timeout(40):\n                    content = (await item.retrieve([\"markdown\"])).markdown_content\n                if not content or not content.strip():\n                    errors[key] = \"No Markdown content returned\"\n                    return\n                path.write_text(content, encoding=\"utf-8\")\n                saved.add(key); bar.update(1)\n                if len(saved) % 50 == 0:\n                    write_json(OUT / \"progress.json\", {\"phase\": \"downloading\", \"saved\": len(saved), \"total\": N,\n                               \"elapsed_seconds\": round(time.time() - run[\"started_wall\"], 2)})\n                return\n            except Exception as error:\n                errors[key] = str(error)\n                if attempt == 0:\n                    await asyncio.sleep(1)\n\nasync for item in client.batches.items(run[\"batch_id\"], status=\"completed\", batch_size=50, wait_for_completion=False):\n    if item.retrieve_id and item.custom_id not in scheduled:\n        scheduled.add(item.custom_id)\n        tasks.append(asyncio.create_task(save_markdown(item)))\n\nawait asyncio.gather(*tasks)\nbar.close()\ndownload_seconds = time.perf_counter() - download_started\n```\n\n**Output:** We successfully downloaded 9,938 Markdown files in approximately 2 minutes and 25 seconds after the scraping stage finished.\n\n#### 6. Verify the Results\n\nFinally, we will check how many of the original 10,000 URLs produced valid Markdown files.\n\nThe code creates a summary of the scraping results, a CSV file linking each URL to its saved file, and a separate report for missing pages. It also calculates the total time taken, including scraping, downloading, and verification.\n\n```\nmissing = {key for key in expected if not valid_file(OUT / f\"{key}.md\")}\n\nwith (OUT / \"manifest.csv\").open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n    writer = csv.writer(f); writer.writerow([\"file\", \"url\", \"saved\"])\n    writer.writerows((f\"{key}.md\", url, key not in missing) for key, url in expected.items())\n\nwith (OUT / \"failures.csv\").open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n    writer = csv.writer(f); writer.writerow([\"url\", \"error\"])\n    writer.writerows((expected[key], errors.get(key, \"Missing result or failed scrape\")) for key in sorted(missing))\n\ntotal_seconds = time.time() - run[\"started_wall\"]\nsummary = {\"batch_id\": run[\"batch_id\"], \"requested\": N, \"batch_status\": info.status,\n           \"api_completed_urls\": info.completed_urls, \"saved_markdown\": N - len(missing),\n           \"missing_markdown\": len(missing), \"all_saved\": not missing, \"resumed\": resumed,\n           \"scrape_seconds\": round(run[\"scrape_seconds\"], 2), \"download_seconds\": round(download_seconds, 2),\n           \"total_seconds\": round(total_seconds, 2),\n           \"all_saved_under_5_minutes\": not resumed and not missing and total_seconds < 300}\n\nwrite_json(OUT / \"summary.json\", summary)\nwrite_json(OUT / \"progress.json\", {\"phase\": \"finished\", **summary})\nprint(json.dumps(summary, indent=2))\nprint(\"Files:\", OUT.resolve())\nprint(\"All 10,000 Markdown files saved.\" if not missing else f\"{len(missing):,} missing; see failures.csv.\")\nawait client.close()\n```\n\nThe results show that 9,938 out of 10,000 pages (99.38%) were successfully saved, while 62 were missing. The complete workflow took 22 minutes and 26 seconds, which is significantly longer than the initial three-minute scraping milestone.\n\nThis final check is important because a completed batch does not necessarily mean every requested page was successfully collected.\n\n## Final Thoughts\n\nI tested the Olostep Batch API with 1,000, 5,000, and 10,000 URLs, and what surprised me most was how much the scraping time varied between runs. My 5,000-URL batch took around 14 minutes to scrape and 16 minutes overall, while the 10,000-URL batch took approximately 18 minutes and 30 seconds to process and 22 minutes and 26 seconds from submission to final verification.\n\nThe most interesting result was the initial speed. Olostep processed over 7,000 pages in under 3 minutes, but the remaining pages took considerably longer. My guess is that the platform might dynamically allocate workers to handle large batches, although I cannot confirm how its infrastructure works internally. Slower pages, retries, or other processing delays could also explain the difference.\n\nOverall, setting up Olostep with Python was surprisingly simple. Most of my time went into benchmarking different batch sizes, tracking progress, and validating the downloaded files rather than writing the scraping code.\n\nThe biggest takeaway is that **batch scraping can deliver impressive initial throughput, but you should measure the entire workflow rather than judging performance by the first few minutes**. In my test, 9,938 out of 10,000 URLs produced nonempty Markdown files, giving me 99.38% file coverage. For enterprise data pipelines, that visibility into both performance and missing results is just as important as scraping speed.\n\n \n\n \n\n[**\\[Abid Ali Awan\\](https://abid.work)**](https://abid.work) ([@1abidaliawan](https://www.linkedin.com/in/1abidaliawan)) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.", "url": "https://wpnews.pro/news/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep", "canonical_source": "https://www.kdnuggets.com/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep", "published_at": "2026-11-05 15:00:01+00:00", "updated_at": "2026-11-05 15:25:31.992377+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Olostep", "Olostep Batch API", "Olostep Python SDK", "Python", "MDN", "Jupyter Notebook"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep", "markdown": "https://wpnews.pro/news/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep.md", "text": "https://wpnews.pro/news/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep.txt", "jsonld": "https://wpnews.pro/news/web-scraping-7000-pages-in-3-minutes-with-python-and-olostep.jsonld"}}