{"slug": "python-web-scraping-what-actually-works", "title": "Python Web Scraping: What Actually Works", "summary": "A developer's practical guide to Python web scraping in 2026 warns that basic techniques like `time.sleep()` lead to blocks, recommending `httpx` with `selectolax` for static pages, Playwright for client-side rendering, and checking the Network tab for JSON APIs as a cheat code. The guide advises using residential proxies and adaptive pacing with a custom `AdaptiveLimiter` class to beat advanced bot detection systems like DataDome and PerimeterX.", "body_md": "# Python Web Scraping: What Actually Works\n\n`time.sleep()`\n\non your code is a great way to get blocked in 2026. Modern sites aren't just looking at your headers; they're fingerprinting your TLS handshake and tracking your mouse movements like some digital private eye. If you're still scraping like it's 2018, you're basically just donating your IP addresses to a blacklist.My current real-world AI workflow for data extraction focuses on picking the right tool for the target, rather than just defaulting to whatever tutorial I found on Medium.\n\n## Tooling Selection\n\nStop using Playwright or Selenium for everything. Launching a full browser is a resource hog and usually overkill.\n\n**Static/Server-Rendered Pages:** I use`httpx`\n\nwith`selectolax`\n\n. Selectolax is C-backed and destroys BeautifulSoup in terms of speed on large pages, while`httpx`\n\nhandles HTTP/2 and async without the drama.\n\n``` python\nimport httpx\nfrom selectolax.parser import HTMLParser\n\ndef scrape_static(url):\n    resp = httpx.get(url, headers={\"User-Agent\": \"Mozilla/5.0\"})\n    tree = HTMLParser(resp.text)\n    for node in tree.css(\".listing\"):\n        print(node.text())\n```\n\n**Client-Side Rendering:** Playwright is the only sane choice here. It's faster than Selenium and the auto-waiting logic means I spend less time debugging flaky selectors.\n\n``` python\nfrom playwright.sync_api import sync_playwright\n\ndef scrape_with_playwright(url):\n    with sync_playwright() as p:\n        browser = p.chromium.launch(headless=True)\n        page = browser.new_page()\n        page.goto(url, wait_until=\"networkidle\")\n        results = [\n            item.query_selector(\"h2\").text_content()\n            for item in page.query_selector_all(\".job-item\")\n        ]\n        browser.close()\n        return results\n```\n\n**The \"Cheat Code\":** Always check the Network tab in DevTools first. Half the time, the site is just hitting a JSON API. If you can find that endpoint, you can skip the HTML parsing nightmare entirely.\n\n```\nurl = \"https://www.freelancer.com/api/projects/0.1/projects/active/?query=python\"\ndata = httpx.get(url).json()\n```\n\n## Beating Bot Detection\n\nWhen you're up against DataDome or PerimeterX, a random User-Agent is basically a placebo. They know you're a bot because your TLS fingerprint screams \"Python library\" and your IP comes from a datacenter.\n\nResidential proxies are non-negotiable for high-tier targets. Pair those with adaptive pacing so you don't look like a clockwork robot:\n\n``` python\nimport time\n\nclass AdaptiveLimiter:\n    def __init__(self, min_delay=1.0, max_delay=5.0):\n        self.min_delay = min_delay\n        self.max_delay = max_delay\n        self.current_delay = min_delay\n\n    def wait(self):\n        time.sleep(self.current_delay)\n\n    def on_success(self):\n        self.current_delay = max(self.min_delay, self.current_delay * 0.9)\n\n    def on_block(self):\n        self.current_delay = min(self.max_delay, self.current_delay * 1.5)\n```\n\nThe logic is simple: if you get blocked, back off. If it's working, gradually speed up. It's a practical tutorial in not getting banned.\n\n[Next Claude Code: Managing Multi-Repo Workspaces →](/en/threads/3217/)", "url": "https://wpnews.pro/news/python-web-scraping-what-actually-works", "canonical_source": "https://promptcube3.com/en/threads/3234/", "published_at": "2026-07-25 14:47:19+00:00", "updated_at": "2026-07-25 15:05:22.604063+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Playwright", "Selenium", "httpx", "selectolax", "BeautifulSoup", "DataDome", "PerimeterX", "Freelancer"], "alternates": {"html": "https://wpnews.pro/news/python-web-scraping-what-actually-works", "markdown": "https://wpnews.pro/news/python-web-scraping-what-actually-works.md", "text": "https://wpnews.pro/news/python-web-scraping-what-actually-works.txt", "jsonld": "https://wpnews.pro/news/python-web-scraping-what-actually-works.jsonld"}}