{"slug": "classify-web-pages-before-adding-them-to-an-ai-agent-s-context", "title": "Classify web pages before adding them to an AI agent's context", "summary": "Sifty launched a URL classification API that labels a web page from its address alone, returning a chosen label plus a probability for each without returning the page content, so an AI agent's context stays clean. The service accepts a POST to https://sifty.dev/ with a url, a question and user-defined labels, and a separate /v1/systemone endpoint answers multiple yes/no questions about one page as probabilities. Sifty pitches the tool for pre-screening links before an agent reads them, filtering search results in parallel, detecting silently broken links, and sorting user-submitted links.", "body_md": "# Ask about a web page before you read it\n\nAn agent that follows a link finds out what is behind it only after reading it. By then it has spent seconds, and thousands of tokens of context, on a login wall, a cookie notice, a \"page not found\" that answered `200 OK`, or the wrong product. Pipelines that sort links have the same problem: status codes say little, and keyword rules break on the first page written differently.\n\nsifty can now classify a web page from its address. Send `url` with labels you choose, and get back the label that fits, with a probability for each. Only the answer comes back, never the page, so your agent's context stays clean.\n\n```\ncurl https://sifty.dev/ -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"url\":\"https://example.com/pricing\",\"labels\":[\"pricing page\",\"login wall\",\"other\"]}'\n```\n\nHere is what we built it for. The shorter examples below are request bodies for `POST https://sifty.dev/`, sent like the one above.\n\n## Check a link before your agent opens it\n\nBefore an agent spends its context on a page, ask what the page is. Labels like these catch the pages that waste an agent's time, including the ones that answer `200 OK`:\n\n```\n{\"url\": \"https://example.com/docs/install\",\n \"question\": \"What is this page?\",\n \"labels\": [\"installation guide\", \"login or paywall\",\n            \"error or empty page\", \"something else\"]}\n```\n\nOpen the page only when the answer is the one you want. Otherwise move on to the next link, without reading the page.\n\n## Pick the right search results\n\nA search returns ten links, and an agent usually needs one or two of them. Classify each result, in parallel, and read only the ones that match:\n\n``` python\nimport requests\nfrom concurrent.futures import ThreadPoolExecutor\n\nLABELS = [\"official documentation\", \"blog or forum post\", \"login or paywall\", \"other\"]\n\ndef kind(url):\n    response = requests.post(\n        \"https://sifty.dev/\",\n        headers={\"Authorization\": \"Bearer YOUR_SIFTY_API_KEY\"},\n        json={\"url\": url, \"question\": \"What kind of page is this?\", \"labels\": LABELS},\n    )\n    response.raise_for_status()\n    return response.json()[\"results\"][0][\"label\"]\n\nresults = [\"https://example.com/docs/install\", \"https://example.org/forum/thread/42\"]\nwith ThreadPoolExecutor(8) as pool:\n    for url, label in zip(results, pool.map(kind, results)):\n        print(label, url)\n```\n\n## Ask several questions about one page\n\nResearching a company, a vendor or a lead usually means opening their site and looking for the same few things. System One asks all of them in one request, and each answer comes back as a probability of yes:\n\n```\ncurl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"url\":\"https://example.com/pricing\",\n       \"questions\":{\n         \"prices\":{\"type\":\"noul\",\"instructions\":\"Does this page list prices?\"},\n         \"free_plan\":{\"type\":\"noul\",\"instructions\":\"Is there a free plan?\"},\n         \"enterprise\":{\"type\":\"noul\",\"instructions\":\"Is there an enterprise plan?\"}}}'\n```\n\nsifty reads only the page you name and doesn't follow its links, so point it at the page most likely to answer: `/pricing` for prices, `/security` for certifications.\n\n## Find links that quietly broke\n\nA link checker sees status codes. It can't tell that a documentation link now leads to a redesigned landing page, or that a \"not found\" page answers `200 OK`. Ask whether each page still says what you linked it for:\n\n```\n{\"url\": \"https://example.com/docs/export\",\n \"question\": \"Does this page document the export command?\",\n \"labels\": [\"yes\", \"no\", \"page not found or moved\"]}\n```\n\n## Sort links people submit\n\nWhen users post links in a community, a marketplace or a support form, a first pass can sort them before anyone opens them:\n\n```\n{\"url\": \"https://example.com/listing/123\",\n \"labels\": [\"product listing\", \"article\", \"spam or scam\", \"other\"]}\n```\n\nUse it to order the review queue, not to replace it: a page can say whatever it likes about itself.\n\n## Good to know\n\n- **Cost.** Each page adds 2 units to the request; the free tier is 1,000 units per IP a day. A page counts once per request, however many questions ask about it. If a page can't be read, the request fails with a`502` , its 2 units are kept, and the rest is refunded.\n- **Addresses in your text are read too.** Write them in`input` or a System One`state` and sifty reads those pages as well, up to 5 a request. If you classify emails, that includes unsubscribe and one-time sign-in links: remove links you don't want opened.\n- **Public pages only.** sifty can't read pages behind a login, and on a long page it uses the parts that best match your question and labels, so a fact far from them can be missed.\n- **A page can argue for its own label.** Nothing stops a page from saying \"this is not a login wall\". Don't let sifty's answer alone decide whether a page is safe.\n- **We haven't measured accuracy on pages yet.** Check your labels on a few pages you know before relying on the answers.\n\n## Try it\n\nGet an API key from a verified account at [sifty.dev/account](https://sifty.dev/account); at 3 units for a page and its labels, the free tier covers about 330 page checks a day. Reading pages is part of [syn](https://github.com/b0bleet/syn), the open-source service behind sifty, if you would rather run it yourself. Questions and corrections are welcome through the [contact page](https://sifty.dev/contact) or GitHub issues.", "url": "https://wpnews.pro/news/classify-web-pages-before-adding-them-to-an-ai-agent-s-context", "canonical_source": "https://sifty.dev/blog/web-pages", "published_at": "2026-10-15 20:04:13+00:00", "updated_at": "2026-10-15 20:28:17.863808+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["sifty", "System One"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/classify-web-pages-before-adding-them-to-an-ai-agent-s-context", "markdown": "https://wpnews.pro/news/classify-web-pages-before-adding-them-to-an-ai-agent-s-context.md", "text": "https://wpnews.pro/news/classify-web-pages-before-adding-them-to-an-ai-agent-s-context.txt", "jsonld": "https://wpnews.pro/news/classify-web-pages-before-adding-them-to-an-ai-agent-s-context.jsonld"}}