{"slug": "building-ai-website-audits-with-python-5-lessons-about-reliability", "title": "Building AI Website Audits with Python: 5 Lessons About Reliability", "summary": "AI automation engineer Ozan Orhan outlined five engineering principles for building trustworthy automated website audits in Python, FastAPI, and Playwright, centered on distinguishing real site failures from checks the audit could not verify. He recommends reserving LLMs for interpreting evidence rather than inventing observations, testing against previously unseen sites to avoid overfitting, and reporting \"not enough evidence\" instead of confident but unsupported warnings. \"Reliable automation should make uncertainty visible rather than hide it behind a score,\" Orhan wrote.", "body_md": "Building an automated website audit sounds straightforward: fetch a page, run checks, and generate a report.\n\nBut creating an audit that developers can actually trust is a different challenge.\n\nWhile working with Python, FastAPI, Playwright, and automated website analysis, I’ve been exploring an important question:\n\nHow do we distinguish a real website problem from something our audit simply couldn’t verify?\n\nHere are five engineering principles that make automated audits more reliable.\n\nA common mistake is treating every unsuccessful check as a website failure.\n\nFor example, a request timeout does not prove that a website is broken.\n\nAn audit should distinguish between:\n\nThis distinction prevents unreliable conclusions.\n\nNot everything needs an LLM.\n\nHTTP response codes, HTML elements, redirects, and response headers can be checked using conventional Python code.\n\nimport requests\n\ndef check_robots_txt(domain: str):\n\n    url = f\"https://{domain}/robots.txt\"\n\n    try:\n\n        response = requests.get(url, timeout=10)\n\n    except requests.RequestException:\n\n        return {\"status\": \"not_tested\"}\n\n    if response.status_code == 200:\n\n        return {\"status\": \"pass\"}\n\n    if response.status_code == 404:\n\n        return {\"status\": \"fail\"}\n\n    return {\"status\": \"not_tested\"}\n\nThis example only checks whether robots.txt is accessible. It does not validate the file’s content.\n\nFor subjective assessments, LLMs can help interpret evidence, but they should not invent missing observations.\n\nA useful finding should answer three questions:\n\nInstead of reporting:\n\nYour website is not AI-ready.\n\nAn audit should identify a specific, verifiable condition and explain its relevance.\n\nA system can appear highly accurate when developers repeatedly test it against the same websites.\n\nThis creates the risk of optimizing for known examples rather than general reliability.\n\nA better approach is to test against previously unseen websites, classify false positives, and document the results before changing the implementation.\n\nA trustworthy audit does not need to produce an answer for every check.\n\nSometimes the correct answer is:\n\nNot enough evidence to determine the result.\n\nThat is more useful than a confident but unsupported warning.\n\nFinal thoughts\n\nThe challenge in AI-assisted auditing is not generating more findings.\n\nIt is making sure those findings are reproducible, explainable, and supported by evidence.\n\nReliable automation should make uncertainty visible rather than hide it behind a score.\n\n⸻\n\nAbout the author\n\nI’m Ozan Orhan, an AI Automation Engineer based in Hannover, Germany. I work with Python, AI agents, APIs, RAG, MCP, and workflow automation.\n\nMy technical projects and case studies: [ozan-orhan.com](https://ozan-orhan.com)", "url": "https://wpnews.pro/news/building-ai-website-audits-with-python-5-lessons-about-reliability", "canonical_source": "https://dev.to/ozanorhn/building-ai-website-audits-with-python-5-lessons-about-reliability-35nj", "published_at": "2026-11-04 08:50:09+00:00", "updated_at": "2026-11-04 08:58:49.891244+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Ozan Orhan", "Python", "FastAPI", "Playwright"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-ai-website-audits-with-python-5-lessons-about-reliability", "markdown": "https://wpnews.pro/news/building-ai-website-audits-with-python-5-lessons-about-reliability.md", "text": "https://wpnews.pro/news/building-ai-website-audits-with-python-5-lessons-about-reliability.txt", "jsonld": "https://wpnews.pro/news/building-ai-website-audits-with-python-5-lessons-about-reliability.jsonld"}}