{"slug": "crawl4ai-vs-website-content-crawler", "title": "Crawl4AI vs. Website Content Crawler", "summary": "A benchmark of Apify's Website Content Crawler and Crawl4AI's open-source Python library on Shopify Help Center pages found both tools returned every requested page with zero failed requests yet lost content: at initial cleaning settings on 20 URLs, Apify passed 33 of 40 content checks and Crawl4AI passed 34 of 40, with both omitting the instruction that stores with more than 50 locations must save a product first. Switching from Crawl4AI's pruning filter to direct selection of Shopify's article.article section, and applying the same change to Apify, was tested in a second round covering 1,000 URLs, where Apify crawled in 6 min 48 sec and 6 min 18 sec and Crawl4AI in 7 min 51 sec and 7 min 38 sec, each returning 1,000 records per run. The author's point is that a successful crawl confirms retrieval, not that every detail survived.", "body_md": "I wanted to build a support chatbot on top of [Shopify's help center](https://help.shopify.com/en/). It needed to give visitors clear instructions without losing important conditions and exceptions that matter.\n\nFor example, if a merchant asks how to add inventory quantities, the chatbot needs to know that [stores with more than 50 locations must save the product first](https://help.shopify.com/en/manual/products/add-update-products).\n\nApify's Website Content Crawler and Crawl4AI fetched every page I gave them. Neither reported a failed request. Yet with the initial cleaning settings, both left out that instruction.\n\nA successful crawl tells you a page was retrieved. It doesn't tell you whether every detail survived. A run can look perfect and still leave gaps in the information your application relies on.\n\n## How the two crawlers differ in practice\n\n[Website Content Crawler](https://apify.com/apify/website-content-crawler) is an Actor you configure in Apify Console. You supply URLs, choose whether to follow links, and set rules for what content to keep. Apify runs it, stores the results in a dataset, and keeps a record of each run.\n\n[Crawl4AI](https://docs.crawl4ai.com/) is an open-source Python library. You write the code that controls page loading and extraction, then run it on your own machine or infrastructure. [Crawl4AI Cloud](https://api.crawl4ai.com/) offers a hosted API, but **this benchmark tested the Python library**.\n\n## What I tested\n\nI tested [Crawl4AI](https://docs.crawl4ai.com/)'s open-source Python library against [Apify's Website Content Crawler](https://apify.com/apify/website-content-crawler) using the same Shopify Help Center URLs. I ran each tool twice at separate scales: **20 articles covering 15 topics**, before expanding to **1,000 URLs**.\n\nI checked content quality on a sample of the 1,000 pages. I chose two details from each of the first 20 articles that the collected text needed to preserve. After adjusting the crawlers, I checked one detail in each of 20 additional articles. That gave me **60 content checks across 40 pages**.\n\nTo keep the comparison consistent, both tools fetched HTML without running JavaScript or following links. Each was allowed two concurrent requests, one retry, and a 60-second timeout per page.\n\n## The results\n\n### Round 1: initial cleaning settings, 20 URLs\n\n| Measure | Apify | Crawl4AI | \n|---|---|---|\n| Crawl time, runs 1 / 2 | 15 s / 12 s | 15.1 s / 24.2 s | \n| Pages returned per run | 20 | 20 | \n| **Content checks passed** | **33 of 40** | **34 of 40** | \n| Final failed requests | 0 | 0 | \n\nBoth tools returned all 20 pages without reporting a failed request. But Apify's output missed 7 of the 40 checked details, and Crawl4AI's missed 6. Both omitted the exception for stores with more than 50 locations. Apify also left out later steps for connecting a domain and some section headings.\n\nCrawl4AI's run summary explains why its number came out where it did, and it records the hardware caveat that applies to every timing in this article.\n\n### Where the missing details went\n\nI used Crawl4AI's pruning filter to remove navigation and other page clutter. It saved the unfiltered Markdown in `.raw.md` and the pruned version in `.fit.md`.\n\nAll 40 checked details appeared in the raw files, but only 34 remained after pruning. The crawler had fetched those six details; the filter removed them.\n\nTurning off the filter alone could bring back menus and other clutter. Shopify keeps the help article inside an `article.article` section, so I could select that section directly instead of relying on a filter to decide what mattered.\n\nApify's cleaned output had also missed details, so I made the same change to both setups and checked the results in the 1,000-URL crawl.\n\n### Round 2: targeted extraction, 1,000 URLs\n\n| Measure | Apify | Crawl4AI | \n|---|---|---|\n| Crawl time, runs 1 / 2 | 6 min 48 sec / 6 min 18 sec | 7 min 51 sec / 7 min 38 sec | \n| Returned records per run | 1,000 | 1,000 | \n| Records passing article inspection, first run | 999 | 999 | \n| **Content checks passed, first run** | **60 of 60** | **60 of 60** | \n| Retried requests, runs 1 / 2 | 45 / 22 | 0 / 0 | \n| Final failed requests, runs 1 / 2 | 0 / 0 | 0 / 0 | \n\nThe first round passed 33 of 40 checks for Apify and 34 of 40 for Crawl4AI. After the adjustment, both passed **all 60 checks across 40 reviewed pages** in the larger crawl.\n\nThree other results stood out:\n\n- **Speed:** Apify finished both 1,000-URL runs sooner, taking about 15% less time on average.\n- **Retries:** Apify recorded 45 retries in its first large run and 22 in its second. Crawl4AI recorded none.\n- **A page that needed a closer look:** The inspection flagged the same`pos-hardware-support` URL in both outputs. The request returned HTTP 200, but neither record contained an article body.\n\n## What the runs cost\n\nApify's two 1,000-URL runs cost **$0.249 combined**, well within its [Free plan's $5 monthly credit](https://apify.com/pricing); Crawl4AI had no software licence fee when I ran it on my computer, though I managed the script myself and would pay extra for hosting or proxies.\n\n| Setup | Cost for one 1,000-URL run | Cost for both runs | \n|---|---|---|\n| **Apify** | $0.128 for the first; $0.121 for the second | **$0.249** | \n| **Crawl4AI Python library** | No software license fee, but you pay externally for compute, storage, and maintenance | varies | \n| **Crawl4AI Cloud** | $0.50 at the basic rate | **$1.00** | \n\n[Crawl4AI Cloud's pricing page](https://api.crawl4ai.com/#pricing) lists basic page fetches at **0.5 credits**, with each credit worth **$0.001**. Browser-based fetches and harder sites cost more.\n\n## Which crawler should you choose?\n\nWith the adjusted settings, both tools returned 1,000 records and passed all 60 content checks; the choice then comes down to how you want to run and manage future crawls:\n\n| **Need** | **Winner** | **Why** | \n|---|---|---|\n| Refresh a full collection daily or weekly | **Apify** | [Built-in schedules](https://docs.apify.com/actors/running/schedules) can rerun a saved crawl without a separate scheduling tool. | \n| Build crawling into a Python application or run it on your own infrastructure | **Crawl4AI** | The open-source library gives you control over the code and where it runs. | \n| Finished faster within its test environment | **Apify** | Apify finished sooner in both small and larger runs of this benchmark. | \n| Set up crawls, review results, and adjust settings through an interface | **Apify** | The Console brings settings, run history, and datasets together. | \n| Preserve important instructions and exceptions | **Tie** | Both passed all 60 checks after the extraction settings were adjusted. | \n| Let teammates manage collections without depending on the original developer | **Apify** | [Access rights](https://docs.apify.com/account/collaboration/access-rights) let teammates work with tasks and results without maintaining the Python script. | \n| Collect data from sites that require custom scrapers | **Apify (Store Actors)** | [Apify Store](https://apify.com/store) offers site-specific Actors with ready-made extraction logic. | \n\nCrawl4AI makes sense if you want crawling built into a script you already maintain, on infrastructure you already run. Apify is best if your collection needs to keep running without a developer attached to it, with scheduled reruns, run history, shared access, and no server to maintain.\n\n### Methodology: reproduce the benchmark yourself\n\nEverything above is reproducible. The benchmark ran on the files below — the exact URL lists, the content checklists I scored against, and the scripts. Your numbers may differ, because Shopify's pages and network conditions change.\n\n| File | What it contains or does | How you'll use it | \n|---|---|---|\n| `urls-20.json` | 20 Shopify help articles covering 15 topics | Feed this list to both crawlers for the first test. | \n| `urls-1000.json` | The original 20 URLs plus 980 more from Shopify's help center. | Use this for the larger test run. | \n| `quality-checks-initial-40.json` | Two details to check in each of the first 20 articles | Compare each detail with the scraped result. | \n| `quality-checks-additional-20.json` | One content check for each of 20 other pages in the larger list. | Use it alongside the original 40 checks when reviewing the 1,000-URL runs. | \n| `requirements.txt` | The versions of Crawl4AI and two supporting Python packages used for this test. | Install them with pip using the command below. | \n| `run_crawl4ai.py` | The script that runs Crawl4AI and saves its output and run summary | Run it from the terminal. | \n| `export_articles.py` | A script that exports either crawler's articles with matching fields | Optional, if you want matching fields from both crawlers. | \n| `inspect_shopify_articles.py` | A script that flags missing, duplicate, or questionable article records | Use it to find records that need a closer look. | \n\nThe checklists guide your review; they don't score the crawlers automatically. A check passes only if all its required details are present and clear enough to follow. You'll also create `prepare_apify.py` along the way to format the URL lists for Apify.\n\n### Step 1. Set up your workspace\n\nInstall [Python 3.12](https://www.python.org/downloads/release/python-31210/) and [PyCharm](https://www.jetbrains.com/pycharm/download/). VS Code works too.\n\nCreate a folder for the benchmark and put the files from the table above in it. In PyCharm, choose **File → Open** and select that folder. In the **Project** panel, double-click `urls-20.json`. These are the 20 addresses both crawlers get.\n\nClick the **Terminal icon** near the bottom-left. On Windows, if the prompt starts with `PS`, type `cmd` and press **Enter**. Then:\n\n```\npy -3.12 -m venv .venv\n.venv\\Scripts\\activate.bat\npython -m pip install -r requirements.txt\n```\n\nOn macOS or Linux:\n\n```\npython3.12 -m venv .venv\nsource .venv/bin/activate\npython -m pip install -r requirements.txt\n```\n\nAfter installing you should see `(.venv)` at the start of the prompt. Leave this terminal open.\n\n### Step 2. Collect the first 20 pages with Apify\n\nApify expects addresses and crawl settings in a specific format. A short script prepares that input, so you don't format 20 addresses by hand now and 1,000 later.\n\nIn PyCharm's **Project** panel, right-click the folder containing `urls-20.json` and choose **New → Python File**. Name it `prepare_apify.py`, then paste:\n\n``` python\nimport argparse\nimport json\nfrom pathlib import Path\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"urls\", choices=[\"urls-20.json\", \"urls-1000.json\"])\nparser.add_argument(\"--article-only\", action=\"store_true\")\nargs = parser.parse_args()\n\nurls = json.loads(Path(args.urls).read_text(encoding=\"utf-8-sig\"))\ncount = len(urls)\nif count not in (20, 1000) or len(set(urls)) != count:\n    raise ValueError(\"Use the unchanged list of 20 or 1,000 unique URLs.\")\n\nsettings = {\n    \"includeUrlGlobs\": [{\"glob\": \"https://help.shopify.com/en/manual/**\"}],\n    \"crawlerType\": \"cheerio\",\n    \"maxCrawlDepth\": 0,\n    \"initialConcurrency\": 2,\n    \"maxConcurrency\": 2,\n    \"requestTimeoutSecs\": 60,\n    \"maxRequestRetries\": 1,\n    \"maxSessionRotations\": 1,\n    \"respectRobotsTxtFile\": True,\n    \"proxyConfiguration\": {\"useApifyProxy\": True},\n    \"htmlTransformer\": \"readableText\",\n    \"saveMarkdown\": True,\n    \"saveHtml\": True,\n    \"useSitemaps\": False,\n    \"summarize\": False,\n    \"startUrls\": [{\"url\": url} for url in urls],\n    \"maxCrawlPages\": count,\n    \"maxResults\": count,\n}\n\nif args.article_only:\n    settings[\"keepElementsCssSelector\"] = \"article.article\"\n    settings[\"htmlTransformer\"] = \"none\"\n\nsuffix = \"-article-only\" if args.article_only else \"\"\noutput = Path(f\"apify-input-{count}{suffix}.json\")\noutput.write_text(json.dumps(settings, indent=2) + \"\\n\", encoding=\"utf-8\")\nprint(f\"Saved {output} with {count} URLs\")\n```\n\nSave, then run:\n\n```\npython prepare_apify.py urls-20.json\n```\n\nWhen you see **Saved apify-input-20.json with 20 URLs**, open the new file and copy its entire contents.\n\nOpen [Website Content Crawler](https://apify.com/apify/website-content-crawler), click **Try for free** or **Open in Console**, and sign in. In the input editor, switch to **JSON** and replace the existing input with what you copied.\n\nSwitch back to the input form and confirm **HTML transformer** is set to **Readable text**, the default cleaning option for this first run. Then open **Run options**:\n\n| Run option | Value | \n|---|---|\n| Build | 0.3.98 | \n| Memory | 4 GB | \n| Timeout | 300 seconds | \n\nType `0.3.98` into the **Build** field to match my test. If you use `version-0`, record the build number from your first run and keep it for all remaining tests.\n\nClick **Start**. When it finishes, open **Storage → Dataset**, pick an article, and read its `markdown` field. Then choose **Export → JSON → Download**, leaving **Select fields** and **Omit fields** empty so the saved HTML is included. Save it as `apify-20-1.json`.\n\nRun it a second time with the same input and save as `apify-20-2.json`. Record both durations, since two runs give you a range rather than a single figure.\n\n### Step 3. Run Crawl4AI and score both outputs\n\nIn the terminal, run these one at a time, waiting for each to finish:\n\n```\npython run_crawl4ai.py --urls urls-20.json --output crawl4ai-20-1\npython run_crawl4ai.py --urls urls-20.json --output crawl4ai-20-2\n```\n\nExpand `crawl4ai-20-1` and open `summary.json`. The `elapsed_seconds` field gives the duration; `results` lists each URL and its saved files. Do the same for `crawl4ai-20-2`.\n\nNow score. Use `quality-checks-initial-40.json` against the original Shopify articles. For Apify, read the `markdown` field in `apify-20-1.json`. For Crawl4AI, use the filtered `.fit.md` files. Give each crawler a score out of 40 and record the IDs of failed checks.\n\nWhen a detail is missing from a `.fit.md` file, open the matching `.raw.md` to see the text before filtering. That comparison is what showed the pruning filter was responsible. Keep your score based on the filtered output, since that is what you would ship.\n\n### Step 4. Target the article and turn the cleaning off\n\nShopify wraps each article in an `<article>` element with the class `article`. Open any help article, right-click a paragraph, choose **Inspect**, and find that element in the developer tools. Hover it to confirm the region it covers.\n\n`urls-1000.json` contains the original 20 pages plus 980 more, so the adjusted run re-checks the same details at scale. Generate the new Apify input:\n\n```\npython prepare_apify.py urls-1000.json --article-only\n```\n\nOpen `apify-input-1000-article-only.json`, copy its contents, and paste them over the existing input in Apify's **JSON** editor. Back in the input form, open the **HTML Extraction** section and set **Keep HTML elements (CSS selector)** to `article.article` and **HTML transformer** to **None**.\n\nUnder **Run options**, keep the same build and memory but raise the timeout to **1,800 seconds**. Leave the other limits alone.\n\n### Step 5. Run the larger crawls and verify\n\nRun Apify twice with the new input, one at a time, saving the full JSON exports as `apify-1000-1.json` and `apify-1000-2.json`. Then run Crawl4AI with the same list. The `--article-only` option applies the same selector and disables pruning:\n\n```\npython run_crawl4ai.py --urls urls-1000.json --article-only --output crawl4ai-1000-1\npython run_crawl4ai.py --urls urls-1000.json --article-only --output crawl4ai-1000-2\n```\n\nThe script already caps these runs at 1,800 seconds. Record durations from Apify's run pages and from `elapsed_seconds` in each Crawl4AI `summary.json`.\n\n**Check retries and failures.** For Apify, open each run's **Log** tab, click **View full log**, press **Ctrl+F**, search for `Final request statistics`, and use the last match near the bottom. Look for:\n\n- `retryHistogram` : the first number counts requests that needed no retries; add the rest to get the retried total.`[968, 32]` means 968 requests needed no retry and 32 needed one.\n- `requestsFailed` : requests that still failed after the allowed retries.`0` means none.\n\nFor Crawl4AI, `summary.json` reports `retries` and `failed_results` directly.\n\n**Check that records actually contain articles.** This step is what caught `pos-hardware-support`:\n\n```\npython inspect_shopify_articles.py --apify apify-1000-1.json --urls urls-1000.json --output apify-1000-1-inspection.json\npython inspect_shopify_articles.py --crawl4ai crawl4ai-1000-1 --urls urls-1000.json --output crawl4ai-1000-1-inspection.json\n```\n\nCompare `record_count` with `structurally_valid_article_count` in each inspection file. If they differ, open the entries under `flagged_urls` and read the saved records.\n\n**Score the content again.** Use `quality-checks-initial-40.json` for the original 20 pages and `quality-checks-additional-20.json` for 20 more pages. That gives you 60 checks across 40 pages. This time, review Apify's exported `markdown` and Crawl4AI's `.raw.md` files, since pruning is off.\n\nCompare your findings with the results above. If you want to try the same approach on your own collection, find a scraper in [Apify Store](https://apify.com/store), start with a few pages, and check the output before scaling up. Apify's [Free plan](https://apify.com/pricing) includes $5 in monthly usage credit and requires no credit card.", "url": "https://wpnews.pro/news/crawl4ai-vs-website-content-crawler", "canonical_source": "https://blog.apify.com/crawl4ai-vs-website-content-crawler/", "published_at": "2026-10-15 00:00:00+00:00", "updated_at": "2026-10-15 09:18:46.221787+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-tools", "developer-tools"], "entities": ["Apify", "Website Content Crawler", "Crawl4AI", "Shopify", "Crawl4AI Cloud"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crawl4ai-vs-website-content-crawler", "markdown": "https://wpnews.pro/news/crawl4ai-vs-website-content-crawler.md", "text": "https://wpnews.pro/news/crawl4ai-vs-website-content-crawler.txt", "jsonld": "https://wpnews.pro/news/crawl4ai-vs-website-content-crawler.jsonld"}}