I Built a CLI Tool That Checks Any Domain for Machine-Readable Infrastructure. Here's the Code.

A developer has released an open-source CLI tool that scans websites for machine-readable infrastructure, checking for JSON-LD blocks, llms.txt files, and other signals. The tool found that 42% of domains in a recent batch returned no valid JSON-LD and 82% lacked an llms.txt file, highlighting gaps in structured data availability for automated systems.

I scan websites for a living. 42% of domains in my most recent batch returned zero valid JSON-LD. 82% served no llms.txt file. Finance, healthcare, government, e-commerce. The regulated industries. Every retrieval pipeline assumes the source delivers structured data. I stopped assuming and started measuring. Five signals, one binary result, a CSV for batch runs. Below is the tool and every line of reasoning behind it. Five signals determine whether a domain provides the minimum infrastructure an automated system can rely on: application/ld+json blocks? Which Schema.org types? llms.txt file at the root? Content-Type and security headers?A domain passes if at least 3 of 5 checks succeed. The threshold is configurable in the code. Python 3.8+ and the requests library: pip install requests That is all. Everything runs locally, with zero external dependencies beyond requests . The homepage gets one GET request. Every <script type="application/ld+json" block goes through json.loads . If the JSON parses and carries an @type , the check passes. php def check json ld domain: str, session: requests.Session - dict: """Fetch the homepage and look for valid JSON-LD blocks.""" url = f"https://{domain}" result = { "signal": "json ld", "found": False, "valid": False, "types": , "error": None, } try: resp = session.get url, timeout=TIMEOUT, allow redirects=True resp.raise for status except requests.RequestException as e: result "error" = str e :120 return result pattern = re.compile r'<script ^ +type= "\' application/ld\+json "\' ^ ' r' . ? </script ', re.DOTALL | re.IGNORECASE, matches = pattern.findall resp.text if not matches: return result result "found" = True types found = for raw in matches: try: data = json.loads raw.strip items = data if isinstance data, list else data for item in items: t = item.get "@type", "" if t: types found.append t if isinstance t, str else str t except json.JSONDecodeError, AttributeError : continue if types found: result "valid" = True result "types" = types found return result 168 domains had a JSON-LD block in the HTML. The block existed. It contained empty types, broken nesting, truncated strings. A malformed block is the more dangerous case: your pipeline sees data, tries to parse it, and extracts garbage. The llms.txt standard proposed by Jeremy Howard gives language models a machine-readable summary of a website. Adoption sits at 18% across 559 scanned domains in my most recent batch run. php def check llms txt domain: str, session: requests.Session - dict: """Check whether the domain serves an llms.txt file at the root.""" url = f"https://{domain}/llms.txt" result = { "signal": "llms txt", "found": False, "size bytes": 0, "error": None, } try: resp = session.get url, timeout=TIMEOUT, allow redirects=True if resp.status code == 200 and "text" in resp.headers.get "content-type", "" : body = resp.text.strip if len body 20 and not body.startswith "< " : result "found" = True result "size bytes" = len body.encode "utf-8" except requests.RequestException as e: result "error" = str e :120 return result Two guard clauses matter here. First, the content-type check filters out HTML error pages served with a 200 status. Second, body.startswith "< " catches soft-404 pages that return an HTML document instead of a plain-text file. Two questions matter here. Can a generic bot crawl the site at all? And does the site explicitly shut the door on known AI crawlers like GPTBot, CCBot, ClaudeBot, or PerplexityBot? KNOWN AI BOTS = "gptbot", "chatgpt-user", "claudebot", "anthropic", "google-extended", "ccbot", "bytespider", "cohere-ai", "perplexitybot", "amazonbot", def check robots txt domain: str, session: requests.Session - dict: """Parse robots.txt for bot-related directives.""" url = f"https://{domain}/robots.txt" result = { "signal": "robots txt", "found": False, "allows generic bots": True, "blocks ai bots": False, "ai bot rules": , "error": None, } try: resp = session.get url, timeout=TIMEOUT, allow redirects=True if resp.status code = 200: return result if "text" not in resp.headers.get "content-type", "" : return result except requests.RequestException as e: result "error" = str e :120 return result result "found" = True lines = resp.text.lower .splitlines current agent = None for line in lines: line = line.split " " 0 .strip if line.startswith "user-agent:" : current agent = line.split ":", 1 1 .strip elif line.startswith "disallow:" and current agent: path = line.split ":", 1 1 .strip if path == "/" and current agent == " ": result "allows generic bots" = False if path == "/" and current agent in KNOWN AI BOTS: result "blocks ai bots" = True result "ai bot rules" .append current agent return result KNOWN AI BOTS will grow. I update the list every time a new crawler shows up in my server logs. A domain that ships Strict-Transport-Security and X-Content-Type-Options runs a maintained stack. I track 10 sector lists. The pattern repeats in every single one: domains that fail on structured data also fail on security headers. SECURITY HEADERS = "strict-transport-security", "x-content-type-options", "x-frame-options", "content-security-policy", def check headers domain: str, session: requests.Session - dict: url = f"https://{domain}" result = { "signal": "headers", "status code": None, "has security headers": False, "content type valid": False, "server": None, "error": None, } try: resp = session.head url, timeout=TIMEOUT, allow redirects=True result "status code" = resp.status code ct = resp.headers.get "content-type", "" result "content type valid" = "text/html" in ct present = sum 1 for h in SECURITY HEADERS if h in resp.headers result "has security headers" = present = 2 result "server" = resp.headers.get "server", "" :60 except requests.RequestException as e: result "error" = str e :120 return result The shortest function in the file. If the HTTPS handshake breaks, everything else is academic. php def check ssl domain: str, session: requests.Session - dict: url = f"https://{domain}" result = {"signal": "ssl", "valid": False, "error": None} try: resp = session.head url, timeout=TIMEOUT, allow redirects=True result "valid" = True except requests.exceptions.SSLError as e: result "error" = f"SSL error: {str e :100 }" except requests.RequestException as e: result "error" = str e :100 return result A domain passes when at least 3 of 5 checks succeed. The threshold lives in a constant at the top of the script. Adjust it based on your pipeline's tolerance. php PASS THRESHOLD = 3 def evaluate domain domain: str - dict: session = requests.Session session.headers.update {"User-Agent": USER AGENT} checks = { "json ld": check json ld domain, session , "llms txt": check llms txt domain, session , "robots txt": check robots txt domain, session , "headers": check headers domain, session , "ssl": check ssl domain, session , } passing = 0 if checks "json ld" "found" and checks "json ld" "valid" : passing += 1 if checks "llms txt" "found" : passing += 1 if checks "robots txt" "found" and checks "robots txt" "allows generic bots" : passing += 1 if checks "headers" "content type valid" and checks "headers" "has security headers" : passing += 1 if checks "ssl" "valid" : passing += 1 verdict = "PASS" if passing = PASS THRESHOLD else "FAIL" return { "domain": domain, "verdict": verdict, "passing": passing, "total": 5, "checks": checks, } python infra-check.py example.com python infra-check.py example.com another.com third.com Create a domains.txt with one domain per line: example.com another.com third.com Then: python infra-check.py --file domains.txt --csv results.csv python infra-check.py --file domains.txt --json-out results.json python infra-check.py --file domains.txt -q --csv results.csv Prints only the verdict line per domain. Useful for CI pipelines. Checking 3 domain s ... ❌ example-bank.de FAIL 2/5 checks passed JSON-LD: missing llms.txt: missing robots.txt: open Headers: HTTP 200, security headers: yes SSL/TLS: valid ✅ example-saas.com PASS 4/5 checks passed JSON-LD: valid Organization, WebSite llms.txt: found 2340 bytes robots.txt: open, blocks 2 AI bots Headers: HTTP 200, security headers: yes SSL/TLS: valid ❌ example-gov.de FAIL 1/5 checks passed JSON-LD: missing llms.txt: missing robots.txt: blocks all bots Headers: HTTP 200, security headers: incomplete SSL/TLS: valid ================================================== Total: 3 | Passed: 1 | Failed: 2 Five binary signals. One question: can an automated system extract structured data from this source? This is a pre-flight check. My production scanner SOVP runs 180+ signals across 21 audit clusters and produces cryptographically signed attestations. This tool extracts 5 of those signals into a standalone script you can run without an account, an API key, or my infrastructure. Content quality, factual accuracy, and freshness sit on a higher layer. This layer sits beneath all of them. If a domain fails here, your pipeline will burn tokens on extraction and get noise in return. The user-agent string identifies itself honestly. The script waits 1 second between domains, follows redirects, and respects a 15-second timeout. The complete infra-check.py with CLI argument parsing, CSV/JSON export, and deduplication: infra-check is a single-file command-line tool that inspects any domain for machine-readable infrastructure signals — the kind of things that determine how well a site can be understood by crawlers, AI agents, and automated clients It runs five checks per domain and produces a binary PASS / FAIL verdict with optional CSV or JSON export for batch runs. | | Check | What it looks for | |---|---|---| | 1 | JSON-LD | Presence of valid application/ld+json structured data on the homepage, and the @type s it declares. | | 2 | llms.txt | Whether the domain serves a non-trivial /llms.txt file at the root. | | 3 | robots.txt | Whether /robots.txt exists, whether generic bots are allowed, and which known AI crawlers GPTBot, ClaudeBot, Google-Extended, CCBot, … are blocked. | | 4 | Headers | HTTP status, a valid text/html content type, and the presence of common security headers HSTS, X-Content-Type-Options, X-Frame-Options, CSP . | | 5 | SSL/TLS | That the domain responds over HTTPS without certificate | You feed domains into your pipeline. How many of them pass all five checks? Run the script. Look at the CSV. That number is the actual size of your usable source list. If you run it, drop your numbers in the comments. I am genuinely curious how other people's source lists hold up. Thorsten Litzki is the founder of Litzki Systems LLC and builds a cryptographically signed infrastructure verification engine. He scans regulated websites for a living and writes about what the data reveals.