Crawl4AI vs. Website Content Crawler A benchmark of Apify's Website Content Crawler and Crawl4AI's open-source Python library on Shopify Help Center pages found both tools returned every requested page with zero failed requests yet lost content: at initial cleaning settings on 20 URLs, Apify passed 33 of 40 content checks and Crawl4AI passed 34 of 40, with both omitting the instruction that stores with more than 50 locations must save a product first. Switching from Crawl4AI's pruning filter to direct selection of Shopify's article.article section, and applying the same change to Apify, was tested in a second round covering 1,000 URLs, where Apify crawled in 6 min 48 sec and 6 min 18 sec and Crawl4AI in 7 min 51 sec and 7 min 38 sec, each returning 1,000 records per run. The author's point is that a successful crawl confirms retrieval, not that every detail survived. I wanted to build a support chatbot on top of Shopify's help center https://help.shopify.com/en/ . It needed to give visitors clear instructions without losing important conditions and exceptions that matter. For example, if a merchant asks how to add inventory quantities, the chatbot needs to know that stores with more than 50 locations must save the product first https://help.shopify.com/en/manual/products/add-update-products . Apify's Website Content Crawler and Crawl4AI fetched every page I gave them. Neither reported a failed request. Yet with the initial cleaning settings, both left out that instruction. A successful crawl tells you a page was retrieved. It doesn't tell you whether every detail survived. A run can look perfect and still leave gaps in the information your application relies on. How the two crawlers differ in practice Website Content Crawler https://apify.com/apify/website-content-crawler is an Actor you configure in Apify Console. You supply URLs, choose whether to follow links, and set rules for what content to keep. Apify runs it, stores the results in a dataset, and keeps a record of each run. Crawl4AI https://docs.crawl4ai.com/ is an open-source Python library. You write the code that controls page loading and extraction, then run it on your own machine or infrastructure. Crawl4AI Cloud https://api.crawl4ai.com/ offers a hosted API, but this benchmark tested the Python library . What I tested I tested Crawl4AI https://docs.crawl4ai.com/ 's open-source Python library against Apify's Website Content Crawler https://apify.com/apify/website-content-crawler using the same Shopify Help Center URLs. I ran each tool twice at separate scales: 20 articles covering 15 topics , before expanding to 1,000 URLs . I checked content quality on a sample of the 1,000 pages. I chose two details from each of the first 20 articles that the collected text needed to preserve. After adjusting the crawlers, I checked one detail in each of 20 additional articles. That gave me 60 content checks across 40 pages . To keep the comparison consistent, both tools fetched HTML without running JavaScript or following links. Each was allowed two concurrent requests, one retry, and a 60-second timeout per page. The results Round 1: initial cleaning settings, 20 URLs | Measure | Apify | Crawl4AI | |---|---|---| | Crawl time, runs 1 / 2 | 15 s / 12 s | 15.1 s / 24.2 s | | Pages returned per run | 20 | 20 | | Content checks passed | 33 of 40 | 34 of 40 | | Final failed requests | 0 | 0 | Both tools returned all 20 pages without reporting a failed request. But Apify's output missed 7 of the 40 checked details, and Crawl4AI's missed 6. Both omitted the exception for stores with more than 50 locations. Apify also left out later steps for connecting a domain and some section headings. Crawl4AI's run summary explains why its number came out where it did, and it records the hardware caveat that applies to every timing in this article. Where the missing details went I used Crawl4AI's pruning filter to remove navigation and other page clutter. It saved the unfiltered Markdown in .raw.md and the pruned version in .fit.md . All 40 checked details appeared in the raw files, but only 34 remained after pruning. The crawler had fetched those six details; the filter removed them. Turning off the filter alone could bring back menus and other clutter. Shopify keeps the help article inside an article.article section, so I could select that section directly instead of relying on a filter to decide what mattered. Apify's cleaned output had also missed details, so I made the same change to both setups and checked the results in the 1,000-URL crawl. Round 2: targeted extraction, 1,000 URLs | Measure | Apify | Crawl4AI | |---|---|---| | Crawl time, runs 1 / 2 | 6 min 48 sec / 6 min 18 sec | 7 min 51 sec / 7 min 38 sec | | Returned records per run | 1,000 | 1,000 | | Records passing article inspection, first run | 999 | 999 | | Content checks passed, first run | 60 of 60 | 60 of 60 | | Retried requests, runs 1 / 2 | 45 / 22 | 0 / 0 | | Final failed requests, runs 1 / 2 | 0 / 0 | 0 / 0 | The first round passed 33 of 40 checks for Apify and 34 of 40 for Crawl4AI. After the adjustment, both passed all 60 checks across 40 reviewed pages in the larger crawl. Three other results stood out: - Speed: Apify finished both 1,000-URL runs sooner, taking about 15% less time on average. - Retries: Apify recorded 45 retries in its first large run and 22 in its second. Crawl4AI recorded none. - A page that needed a closer look: The inspection flagged the same pos-hardware-support URL in both outputs. The request returned HTTP 200, but neither record contained an article body. What the runs cost Apify's two 1,000-URL runs cost $0.249 combined , well within its Free plan's $5 monthly credit https://apify.com/pricing ; Crawl4AI had no software licence fee when I ran it on my computer, though I managed the script myself and would pay extra for hosting or proxies. | Setup | Cost for one 1,000-URL run | Cost for both runs | |---|---|---| | Apify | $0.128 for the first; $0.121 for the second | $0.249 | | Crawl4AI Python library | No software license fee, but you pay externally for compute, storage, and maintenance | varies | | Crawl4AI Cloud | $0.50 at the basic rate | $1.00 | Crawl4AI Cloud's pricing page https://api.crawl4ai.com/ pricing lists basic page fetches at 0.5 credits , with each credit worth $0.001 . Browser-based fetches and harder sites cost more. Which crawler should you choose? With the adjusted settings, both tools returned 1,000 records and passed all 60 content checks; the choice then comes down to how you want to run and manage future crawls: | Need | Winner | Why | |---|---|---| | Refresh a full collection daily or weekly | Apify | Built-in schedules https://docs.apify.com/actors/running/schedules can rerun a saved crawl without a separate scheduling tool. | | Build crawling into a Python application or run it on your own infrastructure | Crawl4AI | The open-source library gives you control over the code and where it runs. | | Finished faster within its test environment | Apify | Apify finished sooner in both small and larger runs of this benchmark. | | Set up crawls, review results, and adjust settings through an interface | Apify | The Console brings settings, run history, and datasets together. | | Preserve important instructions and exceptions | Tie | Both passed all 60 checks after the extraction settings were adjusted. | | Let teammates manage collections without depending on the original developer | Apify | Access rights https://docs.apify.com/account/collaboration/access-rights let teammates work with tasks and results without maintaining the Python script. | | Collect data from sites that require custom scrapers | Apify Store Actors | Apify Store https://apify.com/store offers site-specific Actors with ready-made extraction logic. | Crawl4AI makes sense if you want crawling built into a script you already maintain, on infrastructure you already run. Apify is best if your collection needs to keep running without a developer attached to it, with scheduled reruns, run history, shared access, and no server to maintain. Methodology: reproduce the benchmark yourself Everything above is reproducible. The benchmark ran on the files below — the exact URL lists, the content checklists I scored against, and the scripts. Your numbers may differ, because Shopify's pages and network conditions change. | File | What it contains or does | How you'll use it | |---|---|---| | urls-20.json | 20 Shopify help articles covering 15 topics | Feed this list to both crawlers for the first test. | | urls-1000.json | The original 20 URLs plus 980 more from Shopify's help center. | Use this for the larger test run. | | quality-checks-initial-40.json | Two details to check in each of the first 20 articles | Compare each detail with the scraped result. | | quality-checks-additional-20.json | One content check for each of 20 other pages in the larger list. | Use it alongside the original 40 checks when reviewing the 1,000-URL runs. | | requirements.txt | The versions of Crawl4AI and two supporting Python packages used for this test. | Install them with pip using the command below. | | run crawl4ai.py | The script that runs Crawl4AI and saves its output and run summary | Run it from the terminal. | | export articles.py | A script that exports either crawler's articles with matching fields | Optional, if you want matching fields from both crawlers. | | inspect shopify articles.py | A script that flags missing, duplicate, or questionable article records | Use it to find records that need a closer look. | The checklists guide your review; they don't score the crawlers automatically. A check passes only if all its required details are present and clear enough to follow. You'll also create prepare apify.py along the way to format the URL lists for Apify. Step 1. Set up your workspace Install Python 3.12 https://www.python.org/downloads/release/python-31210/ and PyCharm https://www.jetbrains.com/pycharm/download/ . VS Code works too. Create a folder for the benchmark and put the files from the table above in it. In PyCharm, choose File → Open and select that folder. In the Project panel, double-click urls-20.json . These are the 20 addresses both crawlers get. Click the Terminal icon near the bottom-left. On Windows, if the prompt starts with PS , type cmd and press Enter . Then: py -3.12 -m venv .venv .venv\Scripts\activate.bat python -m pip install -r requirements.txt On macOS or Linux: python3.12 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt After installing you should see .venv at the start of the prompt. Leave this terminal open. Step 2. Collect the first 20 pages with Apify Apify expects addresses and crawl settings in a specific format. A short script prepares that input, so you don't format 20 addresses by hand now and 1,000 later. In PyCharm's Project panel, right-click the folder containing urls-20.json and choose New → Python File . Name it prepare apify.py , then paste: python import argparse import json from pathlib import Path parser = argparse.ArgumentParser parser.add argument "urls", choices= "urls-20.json", "urls-1000.json" parser.add argument "--article-only", action="store true" args = parser.parse args urls = json.loads Path args.urls .read text encoding="utf-8-sig" count = len urls if count not in 20, 1000 or len set urls = count: raise ValueError "Use the unchanged list of 20 or 1,000 unique URLs." settings = { "includeUrlGlobs": {"glob": "https://help.shopify.com/en/manual/ "} , "crawlerType": "cheerio", "maxCrawlDepth": 0, "initialConcurrency": 2, "maxConcurrency": 2, "requestTimeoutSecs": 60, "maxRequestRetries": 1, "maxSessionRotations": 1, "respectRobotsTxtFile": True, "proxyConfiguration": {"useApifyProxy": True}, "htmlTransformer": "readableText", "saveMarkdown": True, "saveHtml": True, "useSitemaps": False, "summarize": False, "startUrls": {"url": url} for url in urls , "maxCrawlPages": count, "maxResults": count, } if args.article only: settings "keepElementsCssSelector" = "article.article" settings "htmlTransformer" = "none" suffix = "-article-only" if args.article only else "" output = Path f"apify-input-{count}{suffix}.json" output.write text json.dumps settings, indent=2 + "\n", encoding="utf-8" print f"Saved {output} with {count} URLs" Save, then run: python prepare apify.py urls-20.json When you see Saved apify-input-20.json with 20 URLs , open the new file and copy its entire contents. Open Website Content Crawler https://apify.com/apify/website-content-crawler , click Try for free or Open in Console , and sign in. In the input editor, switch to JSON and replace the existing input with what you copied. Switch back to the input form and confirm HTML transformer is set to Readable text , the default cleaning option for this first run. Then open Run options : | Run option | Value | |---|---| | Build | 0.3.98 | | Memory | 4 GB | | Timeout | 300 seconds | Type 0.3.98 into the Build field to match my test. If you use version-0 , record the build number from your first run and keep it for all remaining tests. Click Start . When it finishes, open Storage → Dataset , pick an article, and read its markdown field. Then choose Export → JSON → Download , leaving Select fields and Omit fields empty so the saved HTML is included. Save it as apify-20-1.json . Run it a second time with the same input and save as apify-20-2.json . Record both durations, since two runs give you a range rather than a single figure. Step 3. Run Crawl4AI and score both outputs In the terminal, run these one at a time, waiting for each to finish: python run crawl4ai.py --urls urls-20.json --output crawl4ai-20-1 python run crawl4ai.py --urls urls-20.json --output crawl4ai-20-2 Expand crawl4ai-20-1 and open summary.json . The elapsed seconds field gives the duration; results lists each URL and its saved files. Do the same for crawl4ai-20-2 . Now score. Use quality-checks-initial-40.json against the original Shopify articles. For Apify, read the markdown field in apify-20-1.json . For Crawl4AI, use the filtered .fit.md files. Give each crawler a score out of 40 and record the IDs of failed checks. When a detail is missing from a .fit.md file, open the matching .raw.md to see the text before filtering. That comparison is what showed the pruning filter was responsible. Keep your score based on the filtered output, since that is what you would ship. Step 4. Target the article and turn the cleaning off Shopify wraps each article in an