Classify web pages before adding them to an AI agent's context

Sifty launched a URL classification API that labels a web page from its address alone, returning a chosen label plus a probability for each without returning the page content, so an AI agent's context stays clean. The service accepts a POST to https://sifty.dev/ with a url, a question and user-defined labels, and a separate /v1/systemone endpoint answers multiple yes/no questions about one page as probabilities. Sifty pitches the tool for pre-screening links before an agent reads them, filtering search results in parallel, detecting silently broken links, and sorting user-submitted links.

Ask about a web page before you read it An agent that follows a link finds out what is behind it only after reading it. By then it has spent seconds, and thousands of tokens of context, on a login wall, a cookie notice, a "page not found" that answered 200 OK , or the wrong product. Pipelines that sort links have the same problem: status codes say little, and keyword rules break on the first page written differently. sifty can now classify a web page from its address. Send url with labels you choose, and get back the label that fits, with a probability for each. Only the answer comes back, never the page, so your agent's context stays clean. curl https://sifty.dev/ -H 'Authorization: Bearer YOUR SIFTY API KEY' \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/pricing","labels": "pricing page","login wall","other" }' Here is what we built it for. The shorter examples below are request bodies for POST https://sifty.dev/ , sent like the one above. Check a link before your agent opens it Before an agent spends its context on a page, ask what the page is. Labels like these catch the pages that waste an agent's time, including the ones that answer 200 OK : {"url": "https://example.com/docs/install", "question": "What is this page?", "labels": "installation guide", "login or paywall", "error or empty page", "something else" } Open the page only when the answer is the one you want. Otherwise move on to the next link, without reading the page. Pick the right search results A search returns ten links, and an agent usually needs one or two of them. Classify each result, in parallel, and read only the ones that match: python import requests from concurrent.futures import ThreadPoolExecutor LABELS = "official documentation", "blog or forum post", "login or paywall", "other" def kind url : response = requests.post "https://sifty.dev/", headers={"Authorization": "Bearer YOUR SIFTY API KEY"}, json={"url": url, "question": "What kind of page is this?", "labels": LABELS}, response.raise for status return response.json "results" 0 "label" results = "https://example.com/docs/install", "https://example.org/forum/thread/42" with ThreadPoolExecutor 8 as pool: for url, label in zip results, pool.map kind, results : print label, url Ask several questions about one page Researching a company, a vendor or a lead usually means opening their site and looking for the same few things. System One asks all of them in one request, and each answer comes back as a probability of yes: curl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR SIFTY API KEY' \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/pricing", "questions":{ "prices":{"type":"noul","instructions":"Does this page list prices?"}, "free plan":{"type":"noul","instructions":"Is there a free plan?"}, "enterprise":{"type":"noul","instructions":"Is there an enterprise plan?"}}}' sifty reads only the page you name and doesn't follow its links, so point it at the page most likely to answer: /pricing for prices, /security for certifications. Find links that quietly broke A link checker sees status codes. It can't tell that a documentation link now leads to a redesigned landing page, or that a "not found" page answers 200 OK . Ask whether each page still says what you linked it for: {"url": "https://example.com/docs/export", "question": "Does this page document the export command?", "labels": "yes", "no", "page not found or moved" } Sort links people submit When users post links in a community, a marketplace or a support form, a first pass can sort them before anyone opens them: {"url": "https://example.com/listing/123", "labels": "product listing", "article", "spam or scam", "other" } Use it to order the review queue, not to replace it: a page can say whatever it likes about itself. Good to know - Cost. Each page adds 2 units to the request; the free tier is 1,000 units per IP a day. A page counts once per request, however many questions ask about it. If a page can't be read, the request fails with a 502 , its 2 units are kept, and the rest is refunded. - Addresses in your text are read too. Write them in input or a System One state and sifty reads those pages as well, up to 5 a request. If you classify emails, that includes unsubscribe and one-time sign-in links: remove links you don't want opened. - Public pages only. sifty can't read pages behind a login, and on a long page it uses the parts that best match your question and labels, so a fact far from them can be missed. - A page can argue for its own label. Nothing stops a page from saying "this is not a login wall". Don't let sifty's answer alone decide whether a page is safe. - We haven't measured accuracy on pages yet. Check your labels on a few pages you know before relying on the answers. Try it Get an API key from a verified account at sifty.dev/account https://sifty.dev/account ; at 3 units for a page and its labels, the free tier covers about 330 page checks a day. Reading pages is part of syn https://github.com/b0bleet/syn , the open-source service behind sifty, if you would rather run it yourself. Questions and corrections are welcome through the contact page https://sifty.dev/contact or GitHub issues.