Building AI Website Audits with Python: 5 Lessons About Reliability

AI automation engineer Ozan Orhan outlined five engineering principles for building trustworthy automated website audits in Python, FastAPI, and Playwright, centered on distinguishing real site failures from checks the audit could not verify. He recommends reserving LLMs for interpreting evidence rather than inventing observations, testing against previously unseen sites to avoid overfitting, and reporting "not enough evidence" instead of confident but unsupported warnings. "Reliable automation should make uncertainty visible rather than hide it behind a score," Orhan wrote.

Building an automated website audit sounds straightforward: fetch a page, run checks, and generate a report. But creating an audit that developers can actually trust is a different challenge. While working with Python, FastAPI, Playwright, and automated website analysis, I’ve been exploring an important question: How do we distinguish a real website problem from something our audit simply couldn’t verify? Here are five engineering principles that make automated audits more reliable. A common mistake is treating every unsuccessful check as a website failure. For example, a request timeout does not prove that a website is broken. An audit should distinguish between: This distinction prevents unreliable conclusions. Not everything needs an LLM. HTTP response codes, HTML elements, redirects, and response headers can be checked using conventional Python code. import requests def check robots txt domain: str : url = f"https://{domain}/robots.txt" try: response = requests.get url, timeout=10 except requests.RequestException: return {"status": "not tested"} if response.status code == 200: return {"status": "pass"} if response.status code == 404: return {"status": "fail"} return {"status": "not tested"} This example only checks whether robots.txt is accessible. It does not validate the file’s content. For subjective assessments, LLMs can help interpret evidence, but they should not invent missing observations. A useful finding should answer three questions: Instead of reporting: Your website is not AI-ready. An audit should identify a specific, verifiable condition and explain its relevance. A system can appear highly accurate when developers repeatedly test it against the same websites. This creates the risk of optimizing for known examples rather than general reliability. A better approach is to test against previously unseen websites, classify false positives, and document the results before changing the implementation. A trustworthy audit does not need to produce an answer for every check. Sometimes the correct answer is: Not enough evidence to determine the result. That is more useful than a confident but unsupported warning. Final thoughts The challenge in AI-assisted auditing is not generating more findings. It is making sure those findings are reproducible, explainable, and supported by evidence. Reliable automation should make uncertainty visible rather than hide it behind a score. ⸻ About the author I’m Ozan Orhan, an AI Automation Engineer based in Hannover, Germany. I work with Python, AI agents, APIs, RAG, MCP, and workflow automation. My technical projects and case studies: ozan-orhan.com https://ozan-orhan.com