AIArticle
A new YC-backed API attempts to turn the entire web into a structured, queryable database for AI agents.
Web scraping has always been a chore. Most developers have written a script that worked perfectly on a Tuesday, only to break on Thursday because a marketing team changed a class name in a CMS template. For years, the recipe for extracting web data remained unchanged: spin up a headless browser, rotate residential proxies to bypass Cloudflare, write fragile CSS selectors, and hope for the best.
AI agents and LLM-based applications have turned this minor annoyance into a production bottleneck. LLMs do not want raw, messy HTML. They want clean, token-efficient Markdown, or better yet, strictly typed JSON. Feeding raw DOM trees into a context window is an expensive way to get poor results.
Context.dev, a member of the Y Combinator S26 batch, is trying to abstract this entire pipeline away. Instead of managing headless browsers and writing custom parsers, it exposes a single API to scrape, crawl, and extract structured data from any website. It represents a broader architectural shift in how software interacts with the web: moving from imperative DOM parsing to declarative, schema-driven extraction.
From DOM Parsing to Schema-Driven Extraction #
Traditional scraping tools require you to know the structure of the target website beforehand. If you want to extract pricing data from a competitor, you have to inspect their page, find the container element, and extract the text. If they switch from a table to a flexbox layout, your scraper breaks.
Context.dev bypasses this by using LLMs to perform schema-driven extraction. Instead of telling the scraper how to find the data, you tell it what the data should look like. You define a JSON Schema, and the API handles the rendering, parsing, and extraction to match that schema.
This approach is not entirely new. Frameworks like Instructor and Biconomy have popularized structured outputs using LLMs. However, Context.dev bundles the entire infrastructure layer: proxy rotation, Javascript rendering, sitemap crawling, and LLM extraction, into a single API endpoint.
The Developer Angle: Integration and the Agentic Workflow #
For developers building AI agents, the integration path is straightforward. The platform supports standard JSON Schema, which means you can use validation libraries like Zod to define your target data structure directly in your codebase.
Here is how you would extract structured pricing tiers from a target URL using the Node.js SDK:
import ContextDev from 'context.dev';
import { z } from 'zod';
const contextDevClient = new ContextDev({
apiKey: process.env['CONTEXT_DEV_API_KEY']
});
// Define the exact shape you want to extract
const schema = z.object({
company_name: z.string(),
pricing_tiers: z.array(
z.object({
tier_name: z.string(),
tier_description: z.string(),
tier_price: z.number(),
tier_currency: z.string(),
tier_billing_frequency: z.string(),
tier_billing_model: z.enum(['monthly', 'yearly', 'one_time', 'usage_based']),
})
),
});
// Convert the Zod schema to JSON Schema and call the API
const result = await contextDevClient.web.extract({
url: 'https://www.context.dev',
schema: schema.toJSONSchema(),
});
console.log(result.data.pricing_tiers);
Beyond standard extraction, Context.dev includes a meta-feature called "agentic setup." Instead of forcing developers to manually sign up, copy API keys, and write boilerplate integration code, they provide a single-line command designed for coding assistants like Cursor or GitHub Copilot. You paste the instruction into your AI assistant, and the assistant autonomously handles the signup, retrieves the API key, and integrates the SDK into your codebase. It is a clever nod to how modern software is actually being written.
Bundling Web Context: Scraping Meets Enrichment #
What makes Context.dev different from pure-play scraping APIs like Firecrawl or Jina Reader is its focus on identity and brand enrichment. It does not just return Markdown; it attempts to resolve the identity of the entity behind the URL.
Its API endpoints include features for:
Brand Intelligence: Extracting logos, brand colors, fonts, and styleguides from a domain.Firmographics: Resolving a domain to a typed profile containing company name, location, industry, employee count, and social links.Transaction Enrichment: Matching messy transaction strings (like "SQ *BLUE BOTTLE 8xx") to clean brand profiles, complete with logos and merchant category codes.
This bundling suggests that Context.dev is positioning itself as a utility layer for both AI agents and B2B SaaS applications. If you are building a CRM, you can use it to auto-fill onboarding forms from a user's email domain. If you are building a fintech app, you can use it to clean up credit card statements.
The Real-World Trade-offs #
While delegating web scraping to an LLM-backed API is highly convenient, developers must consider the trade-offs before replacing their existing data pipelines.
The first major hurdle is latency. Traditional HTTP requests and DOM parsing take milliseconds. When you introduce Javascript rendering, proxy routing, and LLM-based extraction, latency jumps. Context.dev's own examples show simple markdown scrapes taking around 247ms, but complex schema extractions on heavy pages can easily stretch into several seconds. This makes it unsuitable for real-time, user-facing critical paths where sub-second response times are required.
The second issue is cost. Running LLM extraction on every page visit is expensive. If you are crawling thousands of pages a day, the API costs of structured extraction will quickly dwarf the cost of running a basic proxy pool and a custom parser. For high-volume pipelines, the smart play is to use a hybrid approach: use schema-driven extraction to bootstrap your data collection or handle highly variable sites, and fall back to traditional, cheaper parsing methods for high-volume, static targets.
Finally, there is the risk of model hallucination. While JSON Schema enforcement guarantees the shape of the output, it does not guarantee the accuracy of the data. If a website has ambiguous pricing terms, the LLM might map them to the wrong fields in your schema. Developers must still implement validation logic to ensure the extracted data makes sense before writing it to a production database.
The Verdict #
Context.dev is a compelling entry into the emerging "web context" space. By combining scraping, crawling, brand enrichment, and transaction cleaning into a single API, it eliminates the need to stitch together multiple third-party data providers.
For teams building AI agents that need to browse the live web, or startups looking to quickly enrich user profiles during onboarding, the platform is production-ready and highly practical. However, for high-throughput enterprise data ingestion, the latency and cost overhead of LLM-based parsing mean that traditional, self-hosted scraping pipelines are not going away anytime soon.
Sources & further reading #
Launch HN: Context.dev (YC S26) – API to get structured data from any website— context.dev - Context.dev - Context.dev— docs.context.dev - Context.dev - Launches by UIComet— launches.uicomet.com
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.