{"slug": "show-hn-runo-open-source-web-scraping-that-returns-typed-json", "title": "Show HN: Runo – open-source web scraping that returns typed JSON", "summary": "Developer rhymeswithlimo open-sourced Runo, a web scraping tool that uses LLMs to extract typed JSON from any URL based on a user-defined schema, after initially developing it as a closed-source SaaS. The tool requires a Google Gemini API key and offers CLI, local server, and Python library interfaces.", "body_md": "English ·\n[简体中文](/rhymeswithlimo/runo/blob/main/README.zh-CN.md) ·\n[Español](/rhymeswithlimo/runo/blob/main/README.es.md) ·\n[Français](/rhymeswithlimo/runo/blob/main/README.fr.md) ·\n[Deutsch](/rhymeswithlimo/runo/blob/main/README.de.md) ·\n[日本語](/rhymeswithlimo/runo/blob/main/README.ja.md)\n\nExtract structured, typed JSON from any URL using a schema you define.\n\nNote\n\nI'm a sole maintainer on this project.\nIt started as a closed-source SaaS ([scrapewithruno.com](https://scrapewithruno.com)), but I decided to open-source it :).\n\nYou describe what you want (a field name, a type, an example value) and Runo fetches the page, renders JavaScript if the site needs it, extracts the data with an LLM, and coerces every value to the type you asked for. You get clean, flat JSON back.\n\nNo selectors, no XPath, nothing to maintain. Since the LLM reads for meaning instead of DOM position, your schema doesn't break the next time someone redesigns the site. A field it can't find comes back `null`\n\ninstead of just vanishing.\n\nThis is the open-source build you run yourself. You'll need a Google Gemini API key, that's the only main requirement.\n\n**Typed output**: strings, ints, floats, booleans, ISO 8601 dates, typed arrays, all coerced strictly.** Plain schema**: name, type, example. No DSL.** Semantic extraction**: reads meaning, not DOM position, so redesigns don't break it.** Smart rendering**: plain HTTP first, headless browser only if the page needs it.** Fast paths**: checks JSON-LD, OpenGraph, Twitter Cards, oEmbed before ever calling an LLM.** Three modes**: extract (single URL), batch (one schema across many URLs), or crawl (follows links from a seed URL).** Async built in**:`extract_async`\n\n,`batch_async`\n\n,`crawl_async`\n\nfor anyone running this inside their own event loop.**Three interfaces**: CLI, local server, or Python library.\n\nRequires Python 3.11+ (python 3.14 recommended).\n\n```\npip install -e \".[tls,patchright]\"   # the extras improve anti-bot fetching\nplaywright install chromium           # one-time browser download for JS pages\n```\n\nIMPORTANT!: Runo requires a Gemini API key to function. Get one at\n\n[https://aistudio.google.com/apikey].\n\n```\ncp .env.example .env\n# edit .env and set GEMINI_API_KEY=...\n```\n\n(`.env`\n\nis loaded automatically. You can also just export `GEMINI_API_KEY`\n\nin\nyour shell.)\n\nThe `pip install -e .`\n\nabove registers a `runo`\n\ncommand on your PATH, so you can\nrun it from **any** directory, no need to `cd`\n\ninto the clone. Just keep the\ncloned folder in place (the editable install links back to it) and the same\nPython environment active.\n\n** .env is read from the current directory you're in**,\nso to run\n\n`runo`\n\nfrom anywhere, export the key globally instead:\n\n```\nexport GEMINI_API_KEY=your_key      # Unix/macOS\nsetx GEMINI_API_KEY your_key        # Windows (applies to new terminals)\n```\n\nThe **command line** is the quickest way to try Runo. Alternatively, reach for the\n**Python library** when you're building it into your own code, or the **local\nserver** when you want a language-agnostic HTTP endpoint.\n\n```\nruno serve                    # http://127.0.0.1:8000\ncurl -X POST http://127.0.0.1:8000/v1/extract \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\":\"https://example.com\",\"schema\":[{\"field\":\"title\",\"type\":\"string\",\"example\":\"x\"}]}'\n{\n  \"url\": \"https://example.com\",\n  \"status\": \"success\",\n  \"render_mode\": \"plain\",\n  \"data\": {\"title\": \"Example Domain\"}\n}\n```\n\nNo API key or auth header, it's your local server. Endpoints: `/v1/extract`\n\n,\n`/v1/batch`\n\n, `/v1/crawl`\n\n. Pass per-request settings in an `options`\n\nobject (see\n[Options](#options)).\n\nRuno works as a python library.\n\n``` python\nfrom runo import extract\n\ndata = extract(\"https://example.com\", [\n    {\"field\": \"title\",     \"type\": \"string\",        \"example\": \"Example Domain\"},\n    {\"field\": \"paragraph\", \"type\": \"string\",        \"example\": \"This domain is...\"},\n])\nprint(data)   # {\"title\": \"Example Domain\", \"paragraph\": \"...\"}\n```\n\n`batch`\n\nruns one schema across many URLs; `crawl`\n\nfollows links from a seed URL.\nEach has an `_async`\n\nvariant (`extract_async`\n\n, `batch_async`\n\n, `crawl_async`\n\n) for\nuse inside your own event loop.\n\n``` python\nfrom runo import batch, crawl\n\nrows = batch([\"https://a.com\", \"https://b.com\"], schema, concurrency=5)\nsite = crawl(\"https://blog.com\", \"https://blog.com/posts/*\", schema, max_pages=50)\n```\n\nYou can also run Runo from the command line.\n\n```\n# single URL (schema from a file)\nruno extract https://example.com --schema schema.json\n\n# inline schema, force the headless browser, write result to a file\nruno extract https://example.com --schema '[{\"field\":\"title\",\"type\":\"string\",\"example\":\"x\"}]' \\\n  --render-js always -o out.json\n\n# many URLs with one schema (urls.txt is one URL per line)\nruno batch --urls urls.txt --schema schema.json --concurrency 5\n\n# follow links from a seed\nruno crawl https://blog.com --pattern \"https://blog.com/posts/*\" --schema schema.json --max-pages 50\n\n# run the local HTTP server\nruno serve --host 127.0.0.1 --port 8000\n```\n\n`--schema`\n\ntakes a path to a JSON file or an inline JSON string. Common flags:\n`--render-js auto|always|never`\n\n, `--timeout-ms`\n\n, `--no-cache`\n\n, and `-o out.json`\n\nto write output to a file instead of stdout (`--concurrency`\n\nfor batch;\n`--max-pages`\n\n, `--max-depth`\n\n, `--use-sitemap`\n\n, `--ignore-robots`\n\nfor crawl).\n\nA `schema.json`\n\nfile is just a JSON array of field objects:\n\n```\n[\n  {\"field\": \"title\", \"type\": \"string\", \"example\": \"Example Domain\"},\n  {\"field\": \"price\", \"type\": \"float\",  \"example\": 29.99, \"hint\": \"Use the sale price if present.\"}\n]\n```\n\nEach field has a `field`\n\nname, a `type`\n\n, an `example`\n\nvalue (a one-shot anchor\nfor the LLM), and an optional `hint`\n\n. A good example disambiguates format, for\ninstance `35`\n\nvs `\"35 years old\"`\n\n, or `2024-01-31`\n\nvs `January 31`\n\n.\n\n| Type | Coercion |\n|---|---|\n`string` |\nAlways a string |\n`integer` |\nParsed from text (`\"35 years old\"` -> `35` ) |\n`float` |\nParsed from text (`\"$1.2M\"` -> `1200000.0` ) |\n`boolean` |\nNormalised (`\"✓ Verified\"` -> `true` ) |\n`date` |\nISO 8601 (`YYYY-MM-DD` ); relative dates resolved |\n`array<string>` / `array<integer>` / `array<float>` |\nJSON array (empty `[]` if nothing matched) |\n\nUnresolvable fields come back as `null`\n\n, never dropped, so `data`\n\nalways has the\nsame keys as your schema.\n\nDefault behaviour is usually fine. Reach for `hint`\n\nwhen a page shows two values\nfor the same concept and you want a specific one (`\"Use sale price if present.\"`\n\n),\nwhen the field name is ambiguous (`author`\n\non a republished article), or when the\nsite uses non-obvious wording (`likes`\n\nvs `reactions`\n\n). Keep hints short and use\nthem only when needed.\n\nProduct page:\n\n```\n[\n  { \"field\": \"title\",    \"type\": \"string\",        \"example\": \"MacBook Pro 14\\\"\" },\n  { \"field\": \"price\",    \"type\": \"float\",         \"example\": 1999.00, \"hint\": \"Use sale price if present.\" },\n  { \"field\": \"inStock\",  \"type\": \"boolean\",       \"example\": true },\n  { \"field\": \"rating\",   \"type\": \"float\",         \"example\": 4.6 },\n  { \"field\": \"tags\",     \"type\": \"array<string>\", \"example\": [\"laptop\", \"apple\"] }\n]\n```\n\nArticle / blog post:\n\n```\n[\n  { \"field\": \"headline\",    \"type\": \"string\", \"example\": \"OpenAI ships o3\" },\n  { \"field\": \"author\",      \"type\": \"string\", \"example\": \"Cade Metz\" },\n  { \"field\": \"publishedAt\", \"type\": \"date\",   \"example\": \"2024-12-20\" },\n  { \"field\": \"summary\",     \"type\": \"string\", \"example\": \"A short summary.\", \"hint\": \"1-3 sentences.\" }\n]\n```\n\n**Keep schemas tight.** 4-10 fields extract more accurately than 30. Split large ones into two calls.**Prefer** Declare`array<T>`\n\nover delimiter strings.`array<string>`\n\nand let Runo build the list instead of splitting a joined string yourself.**Names matter.**`firstName`\n\nvs`givenName`\n\nproduce subtly different extractions. Use the term your target sites use; camelCase works well.\n\n```\nruno <command> [flags]\n\nCOMMANDS:\n   extract <url>            extract from a single URL\n   batch                    extract from many URLs with one schema\n   crawl <seed_url>         crawl from a seed URL, following a link pattern\n   serve                    run the local HTTP server\n\nCOMMON (extract, batch, crawl):\n   --schema string          JSON schema: a .json file path or inline JSON (required)\n   --render-js string       JS render mode: auto, always, never (default \"auto\")\n   --timeout-ms int         per-page timeout in milliseconds (default 15000)\n   --locale string          BCP-47 locale for the browser context (default \"en-US\")\n   --no-cache               bypass the in-memory result cache\n   -o, --output string      write JSON output to a file instead of stdout\n\nEXTRACT:\n   --process-images         vision pass to fill null fields from page images (extra tokens)\n\nBATCH:\n   --urls string            URL list: a file (one per line) or comma-separated (required)\n   --concurrency int        URLs fetched in parallel (default 5)\n\nCRAWL:\n   --pattern string         glob for links to follow, e.g. https://site.com/* (required)\n   --max-pages int          hard ceiling on pages visited (default 50)\n   --max-depth int          link hops from the seed URL (default 2)\n   --use-sitemap            also seed URLs from the site's sitemap.xml\n   --ignore-robots          ignore robots.txt disallow rules\n\nSERVE:\n   --host string            bind address (default \"127.0.0.1\")\n   --port int               port to listen on (default 8000)\n   --reload                 auto-reload on code changes (development)\n```\n\nThe flags above aren't CLI-only. The same options are available from the Python library and over HTTP, under the same names.\n\n**Python library.** Pass options as keyword arguments in snake_case (the CLI's\n`--render-js`\n\nbecomes `render_js`\n\n, `--max-pages`\n\nbecomes `max_pages`\n\n, and so on).\nThe URL or URL list, the schema, and the crawl follow-pattern are positional\narguments:\n\n``` python\nfrom runo import extract, crawl\n\n# force headless, longer timeout, fill nulls from images, skip the cache\nextract(\"https://example.com\", schema,\n        render_js=\"always\", timeout_ms=30000, process_images=True, no_cache=True)\n\n# crawl-specific settings are keyword arguments too\ncrawl(\"https://blog.com\", \"https://blog.com/posts/*\", schema,\n      max_pages=100, max_depth=3, use_sitemap=True)\n```\n\nEach function has an `_async`\n\ntwin (`extract_async`\n\n, `batch_async`\n\n,\n`crawl_async`\n\n) with an identical signature for use inside your own event loop.\n\n**Over HTTP.** Options go in the request body. `extract`\n\nand `batch`\n\ntake an\n`options`\n\nobject; `crawl`\n\nkeeps its crawl settings in a `crawl`\n\nobject alongside\na shared `options`\n\nobject:\n\n```\n{\n  \"seed_url\": \"https://blog.com\",\n  \"schema\": [{ \"field\": \"title\", \"type\": \"string\", \"example\": \"Example post\" }],\n  \"crawl\": { \"follow_pattern\": \"https://blog.com/posts/*\", \"max_pages\": 100, \"use_sitemap\": true },\n  \"options\": { \"render_js\": \"always\", \"timeout_ms\": 30000 }\n}\n```\n\n**Batch API.** `batch`\n\nand `crawl`\n\nalso accept `async_mode`\n\n, which routes\nextractions through Gemini's Batch API: cheaper, up to 24h latency, with a\ntransparent fallback to sync on failure.\n\nEvery extract returns the same shape:\n\n```\n{\n  \"url\": \"https://example.com\",\n  \"status\": \"success\",\n  \"render_mode\": \"plain\",\n  \"data\": { \"title\": \"Example Domain\" },\n  \"images_processed\": null\n}\n```\n\n| Field | Notes |\n|---|---|\n`status` |\n`\"success\"` or `\"error\"` . |\n`render_mode` |\n`\"plain\"` (plain HTTP fetch was enough) or `\"headless\"` (escalated to a browser). |\n`data` |\nKeyed by your schema's `field` names; unresolvable fields are `null` . |\n`warnings` |\nOptional array of coercion notes (e.g. `\"coerced 'price' from '$19.99' to 19.99\"` ); omitted when empty. |\n`images_processed` |\nNumber of images read by the vision pass; `null` when it didn't run. |\n\nThe Python library's `extract()`\n\nreturns just the `data`\n\ndict and raises\n`RunoError`\n\non failure; `batch()`\n\n/`crawl()`\n\nreturn the full result objects and\ndon't raise on a single page's failure (check each entry's `status`\n\n).\n\nRuno tries the cheapest path first and escalates only when a page needs it.\nUnder `render_js: \"auto\"`\n\n(the default), it starts with a plain HTTP fetch and\nswitches to a stealth headless browser when it sees signs of trouble:\n\n- A known anti-bot block signature (Cloudflare, Datadome, PerimeterX, Akamai, Incapsula).\n- A body under ~500 characters, or sparse visible text behind a large HTML payload (a JS shell).\n- JavaScript-framework markers in the HTML.\n- An HTTP\n`402`\n\n,`403`\n\n,`406`\n\n,`429`\n\n, or`503`\n\n. - A schema asking for numbers on a page with almost no digits (weather/dashboard widgets).\n\nEscalation is transparent: the response shape is identical, only `render_mode`\n\nchanges from `plain`\n\nto `headless`\n\n.\n\nTo get past bot protection it works up from the cheapest option, stopping at the first that succeeds, and remembers per host what a site needs so later calls skip the attempts that won't work:\n\n- A\n**plain HTTP fetch** for static HTML. - A\n**TLS-impersonating fetch**(`[tls]`\n\nextra) that mimics a real browser's TLS fingerprint, defeating passive JA3/JA4 checks. - A\n**hardened headless browser**(`[patchright]`\n\n/ camoufox) with a canvas/WebGL/audio fingerprint bundle, defeating CDP detection and fingerprint walls. **Per-host cookie persistence**, so progressive-trust challenges only have to be cleared once.- An\n**archive fallback**(Wayback / reader view) as a last resort when the live site is unreachable.\n\nAggressively protected sites (some Cloudflare/Datadome setups, large retail like\nAmazon/Walmart) can still defeat all of these and come back as `FETCH_BLOCKED`\n\n.\n\n`batch`\n\nruns one schema across a list of URLs you already have. `crawl`\n\nstarts\nfrom a seed URL, follows links matching a pattern, and discovers pages for you.\n\n``` python\nfrom runo import batch, crawl\n\nrows = batch([\"https://a.com\", \"https://b.com\"], schema, concurrency=5)\n\nsite = crawl(\"https://blog.com\", \"https://blog.com/posts/*\", schema,\n             max_pages=50, max_depth=2, use_sitemap=False, ignore_robots=False)\n```\n\nThe crawler respects `robots.txt`\n\n(unless `ignore_robots=True`\n\n), can seed from\n`sitemap.xml`\n\n(`use_sitemap=True`\n\n), and applies per-host jitter plus adaptive\nback-off so you don't hammer a site. `crawl`\n\nreturns per-page results plus a\n`crawl_meta`\n\nblock:\n\n```\n{\n  \"results\": [ { \"url\": \"...\", \"status\": \"success\", \"data\": { } } ],\n  \"crawl_meta\": { \"pages_visited\": 17, \"pages_skipped\": 3, \"pages_failed\": 0, \"cancelled\": false }\n}\n```\n\nIt's best to use `batch`\n\nwhen you have the URL list (including paginated feeds\nyou can build as `?page=1..N`\n\n) and use `crawl`\n\nwhen you have one URL and want to\ndiscover related pages.\n\nSet `process_images=true`\n\n(option / `--process-images`\n\nflag) and, after the text\npass, any fields still `null`\n\ntrigger a vision pass: Runo scores the page's\n`<img>`\n\ntags against the missing field names, fetches up to 3 of the best\ncandidates, and sends them to Gemini in a single multimodal call targeting only\nthose fields. It merges anything it finds and reports the count in\n`images_processed`\n\n. If the image pass fails, the original text-only result is\nreturned unchanged. Best for data baked into images (price overlays, stats on a\nposter, marketplace cards); it costs extra tokens, so it's off by default.\n\nFailures use a consistent envelope. On a single extract the top-level `status`\n\nis `\"error\"`\n\n; inside a `batch`\n\n/`crawl`\n\n, individual entries carry the same\n`error`\n\nobject while the overall call still succeeds.\n\n```\n{ \"status\": \"error\", \"error\": { \"code\": \"FETCH_BLOCKED\", \"message\": \"...\", \"retryable\": true } }\n```\n\n| Code | Retryable | Meaning |\n|---|---|---|\n`SCHEMA_INVALID` |\nno | Schema is malformed (missing `field` , unknown type). |\n`TYPE_COERCION_FAILED` |\nno | A value couldn't be coerced to its declared type. |\n`URL_UNREACHABLE` |\nyes | DNS/network failure, or blocked by the SSRF guard. |\n`TIMEOUT` |\nyes | Page exceeded `timeout_ms` . |\n`FETCH_BLOCKED` |\nyes | Anti-bot defeated every free fetch strategy. |\n`LLM_UNAVAILABLE` / `LLM_RATE_LIMITED` / `LLM_TIMEOUT` / `LLM_EMPTY` / `LLM_ERROR` |\nyes | Gemini was overloaded, rate-limited, slow, or returned an unusable response. |\n`LLM_TRUNCATED` / `LLM_BLOCKED` / `LLM_BAD_REQUEST` |\nno | Output couldn't be parsed, a safety/policy block, or a bad request (prompt too long). |\n\nRetry the `retryable: true`\n\ncodes with exponential back-off (1s, 2s, 4s, 8s, cap\nat ~4 attempts); treat the rest as terminal. When a call succeeds but something\nlooked off (e.g. a currency symbol stripped), the fix is reported in the\noptional `warnings`\n\narray rather than failing the call.\n\nEverything is driven by environment variables (or `.env`\n\n). Only `GEMINI_API_KEY`\n\nis required; see [ .env.example](/rhymeswithlimo/runo/blob/main/.env.example) for the documented tunables,\nincluding\n\n`GEMINI_API_KEYS`\n\nfor round-robin across several keys, `HEADLESS_ENGINE`\n\n,\n`TLS_IMPERSONATE`\n\n, and `SSRF_GUARD_ENABLED`\n\n.**JS-heavy sites need the browser.** Plain-HTML pages work with just`pip install`\n\n, but sites that render content with JavaScript need`playwright install chromium`\n\n. Without it, those pages come back empty.**Hard anti-bot walls may fail.** Aggressively protected sites (some Cloudflare/Datadome setups, large retail like Amazon/Walmart) can defeat every built-in fetch strategy and return`FETCH_BLOCKED`\n\n.**Caching is in-memory.** Results and per-field values are cached within a running process to avoid repeat LLM calls, but the cache resets on restart.**You pay Google for tokens.** Extraction quality and cost track whatever Gemini model is configured (Flash-Lite by default).", "url": "https://wpnews.pro/news/show-hn-runo-open-source-web-scraping-that-returns-typed-json", "canonical_source": "https://github.com/rhymeswithlimo/runo", "published_at": "2026-07-08 17:16:08+00:00", "updated_at": "2026-07-08 17:42:19.045602+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-tools"], "entities": ["Runo", "Google Gemini", "scrapewithruno.com"], "alternates": {"html": "https://wpnews.pro/news/show-hn-runo-open-source-web-scraping-that-returns-typed-json", "markdown": "https://wpnews.pro/news/show-hn-runo-open-source-web-scraping-that-returns-typed-json.md", "text": "https://wpnews.pro/news/show-hn-runo-open-source-web-scraping-that-returns-typed-json.txt", "jsonld": "https://wpnews.pro/news/show-hn-runo-open-source-web-scraping-that-returns-typed-json.jsonld"}}