Classify web pages before adding them to an AI agent's context

An agent that follows a link finds out what is behind it only after reading it. By then it has spent seconds, and thousands of tokens of context, on a login wall, a cookie notice, a "page not found" that answered 200 OK, or the wrong product. Pipelines that sort links have the same problem: status codes say little, and keyword rules break on the first page written differently.

sifty can now classify a web page from its address. Send url with labels you choose, and get back the label that fits, with a probability for each. Only the answer comes back, never the page, so your agent's context stays clean.

curl https://sifty.dev/ -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/pricing","labels":["pricing page","login wall","other"]}'

Here is what we built it for. The shorter examples below are request bodies for POST https://sifty.dev/, sent like the one above.

Check a link before your agent opens it #

Before an agent spends its context on a page, ask what the page is. Labels like these catch the pages that waste an agent's time, including the ones that answer 200 OK:

{"url": "https://example.com/docs/install",
 "question": "What is this page?",
 "labels": ["installation guide", "login or paywall",
            "error or empty page", "something else"]}

Open the page only when the answer is the one you want. Otherwise move on to the next link, without reading the page.

Pick the right search results #

A search returns ten links, and an agent usually needs one or two of them. Classify each result, in parallel, and read only the ones that match:

import requests
from concurrent.futures import ThreadPoolExecutor

LABELS = ["official documentation", "blog or forum post", "login or paywall", "other"]

def kind(url):
    response = requests.post(
        "https://sifty.dev/",
        headers={"Authorization": "Bearer YOUR_SIFTY_API_KEY"},
        json={"url": url, "question": "What kind of page is this?", "labels": LABELS},
    )
    response.raise_for_status()
    return response.json()["results"][0]["label"]

results = ["https://example.com/docs/install", "https://example.org/forum/thread/42"]
with ThreadPoolExecutor(8) as pool:
    for url, label in zip(results, pool.map(kind, results)):
        print(label, url)

Ask several questions about one page #

Researching a company, a vendor or a lead usually means opening their site and looking for the same few things. System One asks all of them in one request, and each answer comes back as a probability of yes:

curl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/pricing",
       "questions":{
         "prices":{"type":"noul","instructions":"Does this page list prices?"},
         "free_plan":{"type":"noul","instructions":"Is there a free plan?"},
         "enterprise":{"type":"noul","instructions":"Is there an enterprise plan?"}}}'

sifty reads only the page you name and doesn't follow its links, so point it at the page most likely to answer: /pricing for prices, /security for certifications.

Find links that quietly broke #

A link checker sees status codes. It can't tell that a documentation link now leads to a redesigned landing page, or that a "not found" page answers 200 OK. Ask whether each page still says what you linked it for:

{"url": "https://example.com/docs/export",
 "question": "Does this page document the export command?",
 "labels": ["yes", "no", "page not found or moved"]}

Sort links people submit #

When users post links in a community, a marketplace or a support form, a first pass can sort them before anyone opens them:

{"url": "https://example.com/listing/123",
 "labels": ["product listing", "article", "spam or scam", "other"]}

Use it to order the review queue, not to replace it: a page can say whatever it likes about itself.

Good to know #

• Cost. Each page adds 2 units to the request; the free tier is 1,000 units per IP a day. A page counts once per request, however many questions ask about it. If a page can't be read, the request fails with a502 , its 2 units are kept, and the rest is refunded.
• Addresses in your text are read too. Write them ininput or a System Onestate and sifty reads those pages as well, up to 5 a request. If you classify emails, that includes unsubscribe and one-time sign-in links: remove links you don't want opened.
• Public pages only. sifty can't read pages behind a login, and on a long page it uses the parts that best match your question and labels, so a fact far from them can be missed.
• A page can argue for its own label. Nothing stops a page from saying "this is not a login wall". Don't let sifty's answer alone decide whether a page is safe.
• We haven't measured accuracy on pages yet. Check your labels on a few pages you know before relying on the answers.

Try it #

Get an API key from a verified account at sifty.dev/account; at 3 units for a page and its labels, the free tier covers about 330 page checks a day. Reading pages is part of syn, the open-source service behind sifty, if you would rather run it yourself. Questions and corrections are welcome through the contact page or GitHub issues.