# Classify web pages before adding them to an AI agent's context > Source: > Published: 2026-10-15 20:04:13+00:00 # Ask about a web page before you read it An agent that follows a link finds out what is behind it only after reading it. By then it has spent seconds, and thousands of tokens of context, on a login wall, a cookie notice, a "page not found" that answered `200 OK`, or the wrong product. Pipelines that sort links have the same problem: status codes say little, and keyword rules break on the first page written differently. sifty can now classify a web page from its address. Send `url` with labels you choose, and get back the label that fits, with a probability for each. Only the answer comes back, never the page, so your agent's context stays clean. ``` curl https://sifty.dev/ -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/pricing","labels":["pricing page","login wall","other"]}' ``` Here is what we built it for. The shorter examples below are request bodies for `POST https://sifty.dev/`, sent like the one above. ## Check a link before your agent opens it Before an agent spends its context on a page, ask what the page is. Labels like these catch the pages that waste an agent's time, including the ones that answer `200 OK`: ``` {"url": "https://example.com/docs/install", "question": "What is this page?", "labels": ["installation guide", "login or paywall", "error or empty page", "something else"]} ``` Open the page only when the answer is the one you want. Otherwise move on to the next link, without reading the page. ## Pick the right search results A search returns ten links, and an agent usually needs one or two of them. Classify each result, in parallel, and read only the ones that match: ``` python import requests from concurrent.futures import ThreadPoolExecutor LABELS = ["official documentation", "blog or forum post", "login or paywall", "other"] def kind(url): response = requests.post( "https://sifty.dev/", headers={"Authorization": "Bearer YOUR_SIFTY_API_KEY"}, json={"url": url, "question": "What kind of page is this?", "labels": LABELS}, ) response.raise_for_status() return response.json()["results"][0]["label"] results = ["https://example.com/docs/install", "https://example.org/forum/thread/42"] with ThreadPoolExecutor(8) as pool: for url, label in zip(results, pool.map(kind, results)): print(label, url) ``` ## Ask several questions about one page Researching a company, a vendor or a lead usually means opening their site and looking for the same few things. System One asks all of them in one request, and each answer comes back as a probability of yes: ``` curl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \ -H 'Content-Type: application/json' \ -d '{"url":"https://example.com/pricing", "questions":{ "prices":{"type":"noul","instructions":"Does this page list prices?"}, "free_plan":{"type":"noul","instructions":"Is there a free plan?"}, "enterprise":{"type":"noul","instructions":"Is there an enterprise plan?"}}}' ``` sifty reads only the page you name and doesn't follow its links, so point it at the page most likely to answer: `/pricing` for prices, `/security` for certifications. ## Find links that quietly broke A link checker sees status codes. It can't tell that a documentation link now leads to a redesigned landing page, or that a "not found" page answers `200 OK`. Ask whether each page still says what you linked it for: ``` {"url": "https://example.com/docs/export", "question": "Does this page document the export command?", "labels": ["yes", "no", "page not found or moved"]} ``` ## Sort links people submit When users post links in a community, a marketplace or a support form, a first pass can sort them before anyone opens them: ``` {"url": "https://example.com/listing/123", "labels": ["product listing", "article", "spam or scam", "other"]} ``` Use it to order the review queue, not to replace it: a page can say whatever it likes about itself. ## Good to know - **Cost.** Each page adds 2 units to the request; the free tier is 1,000 units per IP a day. A page counts once per request, however many questions ask about it. If a page can't be read, the request fails with a`502` , its 2 units are kept, and the rest is refunded. - **Addresses in your text are read too.** Write them in`input` or a System One`state` and sifty reads those pages as well, up to 5 a request. If you classify emails, that includes unsubscribe and one-time sign-in links: remove links you don't want opened. - **Public pages only.** sifty can't read pages behind a login, and on a long page it uses the parts that best match your question and labels, so a fact far from them can be missed. - **A page can argue for its own label.** Nothing stops a page from saying "this is not a login wall". Don't let sifty's answer alone decide whether a page is safe. - **We haven't measured accuracy on pages yet.** Check your labels on a few pages you know before relying on the answers. ## Try it Get an API key from a verified account at [sifty.dev/account](https://sifty.dev/account); at 3 units for a page and its labels, the free tier covers about 330 page checks a day. Reading pages is part of [syn](https://github.com/b0bleet/syn), the open-source service behind sifty, if you would rather run it yourself. Questions and corrections are welcome through the [contact page](https://sifty.dev/contact) or GitHub issues.