{"slug": "i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s", "title": "I Built a CLI Tool That Checks Any Domain for Machine-Readable Infrastructure. Here's the Code.", "summary": "A developer has released an open-source CLI tool that scans websites for machine-readable infrastructure, checking for JSON-LD blocks, llms.txt files, and other signals. The tool found that 42% of domains in a recent batch returned no valid JSON-LD and 82% lacked an llms.txt file, highlighting gaps in structured data availability for automated systems.", "body_md": "I scan websites for a living. 42% of domains in my most recent batch returned zero valid JSON-LD. 82% served no llms.txt file. Finance, healthcare, government, e-commerce. The regulated industries.\n\nEvery retrieval pipeline assumes the source delivers structured data. I stopped assuming and started measuring. Five signals, one binary result, a CSV for batch runs. Below is the tool and every line of reasoning behind it.\n\nFive signals determine whether a domain provides the minimum infrastructure an automated system can rely on:\n\n`application/ld+json`\n\nblocks? Which Schema.org types?`llms.txt`\n\nfile at the root?`Content-Type`\n\nand security headers?A domain passes if at least 3 of 5 checks succeed. The threshold is configurable in the code.\n\nPython 3.8+ and the `requests`\n\nlibrary:\n\n```\npip install requests\n```\n\nThat is all. Everything runs locally, with zero external dependencies beyond `requests`\n\n.\n\nThe homepage gets one GET request. Every `<script type=\"application/ld+json\">`\n\nblock goes through `json.loads`\n\n. If the JSON parses and carries an `@type`\n\n, the check passes.\n\n``` php\ndef check_json_ld(domain: str, session: requests.Session) -> dict:\n    \"\"\"Fetch the homepage and look for valid JSON-LD blocks.\"\"\"\n    url = f\"https://{domain}\"\n    result = {\n        \"signal\": \"json_ld\",\n        \"found\": False,\n        \"valid\": False,\n        \"types\": [],\n        \"error\": None,\n    }\n\n    try:\n        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)\n        resp.raise_for_status()\n    except requests.RequestException as e:\n        result[\"error\"] = str(e)[:120]\n        return result\n\n    pattern = re.compile(\n        r'<script[^>]+type=[\"\\']application/ld\\+json[\"\\'][^>]*>'\n        r'(.*?)</script>',\n        re.DOTALL | re.IGNORECASE,\n    )\n    matches = pattern.findall(resp.text)\n\n    if not matches:\n        return result\n\n    result[\"found\"] = True\n    types_found = []\n\n    for raw in matches:\n        try:\n            data = json.loads(raw.strip())\n            items = data if isinstance(data, list) else [data]\n            for item in items:\n                t = item.get(\"@type\", \"\")\n                if t:\n                    types_found.append(\n                        t if isinstance(t, str) else str(t)\n                    )\n        except (json.JSONDecodeError, AttributeError):\n            continue\n\n    if types_found:\n        result[\"valid\"] = True\n        result[\"types\"] = types_found\n\n    return result\n```\n\n168 domains had a JSON-LD block in the HTML. The block existed. It contained empty types, broken nesting, truncated strings. A malformed block is the more dangerous case: your pipeline sees data, tries to parse it, and extracts garbage.\n\nThe `llms.txt`\n\nstandard (proposed by Jeremy Howard) gives language models a machine-readable summary of a website. Adoption sits at 18% across 559 scanned domains in my most recent batch run.\n\n``` php\ndef check_llms_txt(domain: str, session: requests.Session) -> dict:\n    \"\"\"Check whether the domain serves an llms.txt file at the root.\"\"\"\n    url = f\"https://{domain}/llms.txt\"\n    result = {\n        \"signal\": \"llms_txt\",\n        \"found\": False,\n        \"size_bytes\": 0,\n        \"error\": None,\n    }\n\n    try:\n        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)\n        if (\n            resp.status_code == 200\n            and \"text\" in resp.headers.get(\"content-type\", \"\")\n        ):\n            body = resp.text.strip()\n            if len(body) > 20 and not body.startswith(\"<!\"):\n                result[\"found\"] = True\n                result[\"size_bytes\"] = len(body.encode(\"utf-8\"))\n    except requests.RequestException as e:\n        result[\"error\"] = str(e)[:120]\n\n    return result\n```\n\nTwo guard clauses matter here. First, the content-type check filters out HTML error pages served with a 200 status. Second, `body.startswith(\"<!\")`\n\ncatches soft-404 pages that return an HTML document instead of a plain-text file.\n\nTwo questions matter here. Can a generic bot crawl the site at all? And does the site explicitly shut the door on known AI crawlers like GPTBot, CCBot, ClaudeBot, or PerplexityBot?\n\n```\nKNOWN_AI_BOTS = [\n    \"gptbot\", \"chatgpt-user\", \"claudebot\", \"anthropic\",\n    \"google-extended\", \"ccbot\", \"bytespider\", \"cohere-ai\",\n    \"perplexitybot\", \"amazonbot\",\n]\n\ndef check_robots_txt(domain: str, session: requests.Session) -> dict:\n    \"\"\"Parse robots.txt for bot-related directives.\"\"\"\n    url = f\"https://{domain}/robots.txt\"\n    result = {\n        \"signal\": \"robots_txt\",\n        \"found\": False,\n        \"allows_generic_bots\": True,\n        \"blocks_ai_bots\": False,\n        \"ai_bot_rules\": [],\n        \"error\": None,\n    }\n\n    try:\n        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)\n        if resp.status_code != 200:\n            return result\n        if \"text\" not in resp.headers.get(\"content-type\", \"\"):\n            return result\n    except requests.RequestException as e:\n        result[\"error\"] = str(e)[:120]\n        return result\n\n    result[\"found\"] = True\n    lines = resp.text.lower().splitlines()\n    current_agent = None\n\n    for line in lines:\n        line = line.split(\"#\")[0].strip()\n        if line.startswith(\"user-agent:\"):\n            current_agent = line.split(\":\", 1)[1].strip()\n        elif line.startswith(\"disallow:\") and current_agent:\n            path = line.split(\":\", 1)[1].strip()\n            if path == \"/\" and current_agent == \"*\":\n                result[\"allows_generic_bots\"] = False\n            if path == \"/\" and current_agent in KNOWN_AI_BOTS:\n                result[\"blocks_ai_bots\"] = True\n                result[\"ai_bot_rules\"].append(current_agent)\n\n    return result\n```\n\n`KNOWN_AI_BOTS`\n\nwill grow. I update the list every time a new crawler shows up in my server logs.\n\nA domain that ships `Strict-Transport-Security`\n\nand `X-Content-Type-Options`\n\nruns a maintained stack. I track 10 sector lists. The pattern repeats in every single one: domains that fail on structured data also fail on security headers.\n\n```\nSECURITY_HEADERS = [\n    \"strict-transport-security\",\n    \"x-content-type-options\",\n    \"x-frame-options\",\n    \"content-security-policy\",\n]\n\ndef check_headers(domain: str, session: requests.Session) -> dict:\n    url = f\"https://{domain}\"\n    result = {\n        \"signal\": \"headers\",\n        \"status_code\": None,\n        \"has_security_headers\": False,\n        \"content_type_valid\": False,\n        \"server\": None,\n        \"error\": None,\n    }\n\n    try:\n        resp = session.head(url, timeout=TIMEOUT, allow_redirects=True)\n        result[\"status_code\"] = resp.status_code\n        ct = resp.headers.get(\"content-type\", \"\")\n        result[\"content_type_valid\"] = \"text/html\" in ct\n        present = sum(\n            1 for h in SECURITY_HEADERS if h in resp.headers\n        )\n        result[\"has_security_headers\"] = present >= 2\n        result[\"server\"] = resp.headers.get(\"server\", \"\")[:60]\n    except requests.RequestException as e:\n        result[\"error\"] = str(e)[:120]\n\n    return result\n```\n\nThe shortest function in the file. If the HTTPS handshake breaks, everything else is academic.\n\n``` php\ndef check_ssl(domain: str, session: requests.Session) -> dict:\n    url = f\"https://{domain}\"\n    result = {\"signal\": \"ssl\", \"valid\": False, \"error\": None}\n\n    try:\n        resp = session.head(url, timeout=TIMEOUT, allow_redirects=True)\n        result[\"valid\"] = True\n    except requests.exceptions.SSLError as e:\n        result[\"error\"] = f\"SSL error: {str(e)[:100]}\"\n    except requests.RequestException as e:\n        result[\"error\"] = str(e)[:100]\n\n    return result\n```\n\nA domain passes when at least 3 of 5 checks succeed. The threshold lives in a constant at the top of the script. Adjust it based on your pipeline's tolerance.\n\n``` php\nPASS_THRESHOLD = 3\n\ndef evaluate_domain(domain: str) -> dict:\n    session = requests.Session()\n    session.headers.update({\"User-Agent\": USER_AGENT})\n\n    checks = {\n        \"json_ld\": check_json_ld(domain, session),\n        \"llms_txt\": check_llms_txt(domain, session),\n        \"robots_txt\": check_robots_txt(domain, session),\n        \"headers\": check_headers(domain, session),\n        \"ssl\": check_ssl(domain, session),\n    }\n\n    passing = 0\n    if checks[\"json_ld\"][\"found\"] and checks[\"json_ld\"][\"valid\"]:\n        passing += 1\n    if checks[\"llms_txt\"][\"found\"]:\n        passing += 1\n    if (\n        checks[\"robots_txt\"][\"found\"]\n        and checks[\"robots_txt\"][\"allows_generic_bots\"]\n    ):\n        passing += 1\n    if (\n        checks[\"headers\"][\"content_type_valid\"]\n        and checks[\"headers\"][\"has_security_headers\"]\n    ):\n        passing += 1\n    if checks[\"ssl\"][\"valid\"]:\n        passing += 1\n\n    verdict = \"PASS\" if passing >= PASS_THRESHOLD else \"FAIL\"\n\n    return {\n        \"domain\": domain,\n        \"verdict\": verdict,\n        \"passing\": passing,\n        \"total\": 5,\n        \"checks\": checks,\n    }\npython infra-check.py example.com\npython infra-check.py example.com another.com third.com\n```\n\nCreate a `domains.txt`\n\nwith one domain per line:\n\n```\nexample.com\nanother.com\nthird.com\n```\n\nThen:\n\n```\npython infra-check.py --file domains.txt --csv results.csv\npython infra-check.py --file domains.txt --json-out results.json\npython infra-check.py --file domains.txt -q --csv results.csv\n```\n\nPrints only the verdict line per domain. Useful for CI pipelines.\n\n```\nChecking 3 domain(s)...\n\n❌  example-bank.de  [FAIL]  (2/5 checks passed)\n   JSON-LD:     missing\n   llms.txt:    missing\n   robots.txt:  open\n   Headers:     HTTP 200, security headers: yes\n   SSL/TLS:     valid\n\n✅  example-saas.com  [PASS]  (4/5 checks passed)\n   JSON-LD:     valid (Organization, WebSite)\n   llms.txt:    found (2340 bytes)\n   robots.txt:  open, blocks 2 AI bots\n   Headers:     HTTP 200, security headers: yes\n   SSL/TLS:     valid\n\n❌  example-gov.de  [FAIL]  (1/5 checks passed)\n   JSON-LD:     missing\n   llms.txt:    missing\n   robots.txt:  blocks all bots\n   Headers:     HTTP 200, security headers: incomplete\n   SSL/TLS:     valid\n\n==================================================\nTotal: 3 | Passed: 1 | Failed: 2\n```\n\nFive binary signals. One question: can an automated system extract structured data from this source?\n\nThis is a pre-flight check. My production scanner (SOVP) runs 180+ signals across 21 audit clusters and produces cryptographically signed attestations. This tool extracts 5 of those signals into a standalone script you can run without an account, an API key, or my infrastructure.\n\nContent quality, factual accuracy, and freshness sit on a higher layer. This layer sits beneath all of them. If a domain fails here, your pipeline will burn tokens on extraction and get noise in return.\n\nThe user-agent string identifies itself honestly. The script waits 1 second between domains, follows redirects, and respects a 15-second timeout.\n\nThe complete `infra-check.py`\n\nwith CLI argument parsing, CSV/JSON export, and deduplication:\n\n**infra-check** is a single-file command-line tool that inspects any domain for\nmachine-readable infrastructure signals — the kind of things that determine how\nwell a site can be understood by crawlers, AI agents, and automated clients\nIt runs five checks per domain and produces a binary **PASS / FAIL** verdict\nwith optional CSV or JSON export for batch runs.\n\n| # | Check | What it looks for |\n|---|---|---|\n| 1 | JSON-LD |\nPresence of valid `application/ld+json` structured data on the homepage, and the `@type` s it declares. |\n| 2 | llms.txt |\nWhether the domain serves a non-trivial `/llms.txt` file at the root. |\n| 3 | robots.txt |\nWhether `/robots.txt` exists, whether generic bots are allowed, and which known AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, …) are blocked. |\n| 4 | Headers |\nHTTP status, a valid `text/html` content type, and the presence of common security headers (HSTS, X-Content-Type-Options, X-Frame-Options, CSP). |\n| 5 | SSL/TLS |\nThat the domain responds over HTTPS without certificate |\n\nYou feed domains into your pipeline. How many of them pass all five checks?\n\nRun the script. Look at the CSV. That number is the actual size of your usable source list.\n\nIf you run it, drop your numbers in the comments. I am genuinely curious how other people's source lists hold up.\n\n*Thorsten Litzki is the founder of Litzki Systems LLC and builds a cryptographically signed infrastructure verification engine. He scans regulated websites for a living and writes about what the data reveals.*", "url": "https://wpnews.pro/news/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s", "canonical_source": "https://dev.to/litzki-systems/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-heres-the-code-3cep", "published_at": "2026-08-26 23:36:08+00:00", "updated_at": "2026-08-27 00:19:08.959552+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Jeremy Howard"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s", "markdown": "https://wpnews.pro/news/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s.md", "text": "https://wpnews.pro/news/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s.txt", "jsonld": "https://wpnews.pro/news/i-built-a-cli-tool-that-checks-any-domain-for-machine-readable-infrastructure-s.jsonld"}}