Show HN: I built a scraping engine where the LLM writes configs, not code

A developer released Fitter, an open-source scraping engine that converts any website or API into structured JSON using declarative YAML/JSON configs instead of code, enabling LLMs like Claude Code to author and run scraping pipelines locally. The tool supports HTTP, headless browser, and file connectors with JSON, XML, HTML, and XPath parsing, and is available as a static binary with MCP server integration.

Fitter turns any website or API into structured JSON — declaratively. One JSON/YAML config describes where the data lives HTTP request, headless browser, file, static value and what to extract JSON paths, CSS selectors, XPath . No code, no brittle scraping scripts. Because configs are plain data, LLMs can author them . The built-in MCP server how-to-use-fitter mcp lets Claude Code, Claude Desktop, or any MCP client write and run scraping pipelines on your machine, on demand: "Get the top 5 HackerNews stories with titles and scores"→ the model authors a fitter config, validates it, runs it locally, and gets clean JSON back. One engine, five ways to use it: 🤖 Fitter MCP | | Fitter Agent Fitter CLI Fitter Lib Fitter Why fitter for AI agents? Declarative & auditable — the agent produces a config you can read, save and re-run, not throwaway code Local-first — all fetching happens on your machine; no third-party scraping API, no keys, no per-request billing Batteries included — HTTP client, headless browser Playwright/Chromium/Docker , JSON/HTML/XML/XPath parsing, pagination, cached references, host rate-limits — in a single static binary Reusable — what the agent authored today becomes tomorrow's cron job or service config Server - parsing response from some API's or http request usage of http.Client Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information Static - parsing static string as data JSON - parsing JSON to get specific information XML - parsing xml tree to get specific information HTML - parsing dom tree to get specific information XPath - parsing dom tree to get specific information but by xpath go get github.com/PxyUp/fitter package main import "fmt" "github.com/PxyUp/fitter/lib" "github.com/PxyUp/fitter/pkg/config" "log" "net/http" func main { res, err := lib.Parse &config.Item{ ConnectorConfig: &config.ConnectorConfig{ ResponseType: config.Json, Url: "https://random-data-api.com/api/appliance/random appliance", ServerConfig: &config.ServerConnectorConfig{ Method: http.MethodGet, }, }, Model: &config.Model{ ObjectConfig: &config.ObjectConfig{ Fields: map string config.Field{ "my id": { BaseField: &config.BaseField{ Type: config.Int, Path: "id", }, }, "generated id": { BaseField: &config.BaseField{ Generated: &config.GeneratedFieldConfig{ UUID: &config.UUIDGeneratedFieldConfig{}, }, }, }, "generated array": { ArrayConfig: &config.ArrayConfig{ RootPath: "@this|@keys", ItemConfig: &config.ObjectConfig{ Field: &config.BaseField{ Type: config.String, }, }, }, }, }, }, }, }, nil, nil, nil, nil if err = nil { log.Fatal err } fmt.Println res.ToJson } Output: { "generated array": "id","uid","brand","equipment" , "my id": 6000, "generated id": "26b08b73-2f2e-444d-bcf2-dac77ac3130e" } Download latest version from the release page https://github.com/PxyUp/fitter/releases or locally: go run cmd/fitter/main.go --path=./examples/config api.json --path - string "" - path for the configuration of the Fitter --url - string "" - url for the configuration of the Fitter --verbose - bool false - enable logging --plugins - string "" - path for plugins for Fitter https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md --log-level - enum "info", "error", "debug", "fatal" - set log level only if verbose set to true Download latest version from the release page https://github.com/PxyUp/fitter/releases or locally: go run cmd/cli/main.go --path=./examples/cli/config cli.json --path - string "" - path for the configuration of the Fitter CLI --url - string "" - url for the configuration of the Fitter CLI --copy - bool false - copy information into clipboard --pretty - bool true - make readable result also affect on copy --verbose - bool false - enable logging --omit-error-pretty - bool false - Provide pure value if pretty is invalid --plugins - string "" - path for plugins for Fitter https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md --log-level - enum "info", "error", "debug", "fatal" - set log level only if verbose set to true --input - string "" - specify input value for formatting placeholder-list . Examples: --input=\""124"\" --input=124 --input='{"test": 5}' ./fitter cli ${VERSION} --path=./examples/cli/config cli.json --copy=true Examples: Server version HackerNews + Quotes + Guardian News https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json - using API + HTML + XPath parsing Chromium version Guardian News + Quotes https://github.com/PxyUp/fitter/blob/master/examples/cli/config browser.json - using HTML parsing + browser emulation Docker version Docker version: Guardian News + Quotes https://github.com/PxyUp/fitter/blob/master/examples/cli/config docker.json - using HTML parsing + browser from Docker image Playwright version Playwright version: Guardian News + Quotes https://github.com/PxyUp/fitter/blob/master/examples/cli/config playwright.json - using HTML parsing + browser from Playwright framework Playwright version Playwright version: England Cities + Weather https://github.com/PxyUp/fitter/blob/master/examples/cli/config weather.json - using HTML + XPath parsing + browser from Playwright framework JSON version Generate pagination https://github.com/PxyUp/fitter/blob/master/examples/cli/config static connector.json - using static connector for generate pagination array Server version Get current time https://github.com/PxyUp/fitter/blob/master/examples/cli/config current time.json - get time from url and format it Fitter Agent is an AI-powered CLI that uses Claude to convert natural language requests into Fitter configurations and execute them automatically. Download latest version from the release page https://github.com/PxyUp/fitter/releases or locally: export ANTHROPIC API KEY=<your-anthropic-api-key go run cmd/agent/main.go --api-key - string "" - Anthropic API key. Prefer the ANTHROPIC API KEY environment variable so the key does not end up in your shell history --model - string "claude-opus-4-8" - Claude model to use --effort - enum "low", "medium", "high", "xhigh", "max" - reasoning effort, default "high". Lower it for faster/cheaper configs, raise it for harder extractions --verbose - bool false - enable logging --log-level - enum "info", "error", "debug", "fatal" - set log level --plugins - string "" - path for plugins for Fitter --chromium-limit - uint 0 - limit concurrent Chromium instances --docker-limit - uint 0 - limit concurrent Docker containers --playwright-limit - uint 0 - limit concurrent Playwright instances ┌─────────────────────────────────────────────────────────────────┐ │ 1. User enters natural language request │ │ "Get top 5 HackerNews stories with titles and scores" │ │ ↓ │ │ 2. Claude returns a config in a schema-constrained response │ │ ↓ │ │ 3. Agent validates it; on failure the error is handed back │ │ to Claude to repair up to 3 attempts │ │ ↓ │ │ 4. Agent displays config and asks for confirmation │ │ ↓ │ │ 5. On confirmation, executes via lib.Parse │ │ ↓ │ │ 6. Returns structured JSON result │ └─────────────────────────────────────────────────────────────────┘ The agent keeps the conversation, so after a config is generated you can just say what to change instead of restating the whole request: Get top 3 HackerNews stories with titles and scores refine Only return 5 items and also include the article URL Use new to forget the current config and start a fresh session. help - show help message new/reset - forget the current config and start fresh clear - clear the screen exit/quit/q - exit the agent bash $ export ANTHROPIC API KEY=sk-ant-... $ ./fitter agent ╔══════════════════════════════════════════════════════════════╗ ║ Fitter Agent - AI-Powered Data Extraction ║ ╚══════════════════════════════════════════════════════════════╝ Describe what you want to extract. Follow-up messages refine the previous config. Type 'help' for commands. Get top 3 HackerNews stories with titles and scores ┌─ Generated Fitter Config ─────────────────────────────────────── { "item": { "connector config": { "response type": "json", "url": "https://hacker-news.firebaseio.com/v0/topstories.json", "server config": { "method": "GET" } }, "model": { "array config": { "root path": "@this", "length limit": 3, "item config": { "fields": { "id": { "base field": { "type": "int" } }, "story": { "base field": { "type": "int", "generated": { "model": { "type": "object", "connector config": { "response type": "json", "url": "https://hacker-news.firebaseio.com/v0/item/{PL}.json", "server config": { "method": "GET" } }, "model": { "object config": { "fields": { "title": { "base field": { "type": "string", "path": "title" } }, "score": { "base field": { "type": "int", "path": "score" } } } } } } } } } } } } } } } └────────────────────────────────────────────────────────────────── Execute this config? y/n : y ┌─ Result ──────────────────────────────────────────────────────── { "id": 46740029, "story": { "title": "Show HN: Open-source project", "score": 161 } }, { "id": 46737630, "story": { "title": "Interesting article", "score": 237 } }, { "id": 46735644, "story": { "title": "New technology release", "score": 192 } } └────────────────────────────────────────────────────────────────── exit Goodbye | Request | What it does | |---|---| Get Bitcoin price from CoinGecko API | Fetches current BTC price | Scrape headlines from news.ycombinator.com with links | HTML scraping with CSS selectors | Get top 5 stories from HackerNews with titles | Nested API calls | Fetch weather data from wttr.in for London | Simple API extraction | Scrape product names and prices from example.com | Web scraping | The agent can generate configs for: JSON APIs - REST APIs with GET/POST methods HTML Scraping - CSS selector-based extraction XPath Scraping - XPath-based extraction Nested API Calls - Fetch details for each item in a list Browser Emulation - Playwright for JS-rendered pages Formatted Fields - URL templates with placeholders Array Limiting - Limit results to N items Fitter MCP is a Model Context Protocol https://modelcontextprotocol.io server stdio transport which lets any MCP client — Claude Code, Claude Desktop, IDE assistants, custom agents — run Fitter configs and get structured JSON back. Download latest version from the release page https://github.com/PxyUp/fitter/releases or build locally: go build -o bin/fitter mcp ./cmd/mcp available in every project claude mcp add fitter -s user -- /path/to/fitter mcp { "mcpServers": { "fitter": { "command": "/path/to/fitter mcp" } } } | Tool | Description | |---|---| fitter run | Run a Fitter config passed inline JSON or YAML string and return the extracted data as JSON. Accepts an optional input value available in the config via {{{FromInput=.}}} / {{{FromInput=json.path}}} | fitter run file | Same as fitter run but reads the config from a local .json / .yaml file | fitter validate config | Validate a config without executing it structure, response type , connector data source, model . Useful while iterating on a config | fitter config reference | Return a condensed reference of the whole config format connectors, parsers, model/field schema, placeholders, references, limits with working examples, so the model can author configs without external docs | The config format is exactly the same as for Fitter CLI how-to-use-fitter cli : a top-level object with item required , limits and references . FITTER PLUGINS - string "" - path for plugins folder https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md , same as the --plugins flag of Fitter/Fitter CLI Ask your MCP client something like: Get the top 5 HackerNews stories with titles and scores using fitter The model calls fitter config reference , authors a config, optionally checks it with fitter validate config and executes it via fitter run — all data fetching happens locally on your machine. It is the way how you fetch the data type ConnectorConfig struct { ResponseType ParserType json:"response type" yaml:"response type" Url string json:"url" yaml:"url" Attempts uint32 json:"attempts" yaml:"attempts" NullOnError bool yaml:"null on error" json:"null on error" StaticConfig StaticConnectorConfig json:"static config" yaml:"static config" IntSequenceConfig IntSequenceConnectorConfig json:"int sequence config" yaml:"int sequence config" ServerConfig ServerConnectorConfig json:"server config" yaml:"server config" BrowserConfig BrowserConnectorConfig yaml:"browser config" json:"browser config" PluginConnectorConfig PluginConnectorConfig json:"plugin connector config" yaml:"plugin connector config" ReferenceConfig ReferenceConnectorConfig yaml:"reference config" json:"reference config" FileConfig FileConnectorConfig json:"file config" yaml:"file config" } - NullOnError false - if set to true then all errors a ignored - ResponseType - enum "HTML", "json","xpath" - in which format data comes from the connector - Attempts - how many attempts to use for fetch data by connector - Url - define which address to request. Important: can be with inject of the parent value as a string placeholder-list https://api.open-meteo.com/v1/forecast?latitude={{{latitude}}}&longitude={{{longitude}}}&hourly=temperature 2m&forecast days=1 Config can be one of: ServerConfig serverconnectorconfig BrowserConfig browserconnectorconfig StaticConfig staticconnectorconfig PluginConnectorConfig pluginconnectorconfig ReferenceConfig referenceconnectorconfig IntSequenceConfig intsequenceconnectorconfig FileConfig fileconnectorconfig Example: { "response type": "xpath", "attempts": 3, "url": "https://openweathermap.org/find?q={PL}", "browser config": { "playwright": { "timeout": 30, "wait": 30, "install": false, "browser": "Chromium" } } } Connector can be defined via plugin system. For use that you need apply next flags to Fitter/Cli location of the plugins : ... --plugins=./examples/plugin --plugins - looking for all files with ".so" extension in provided folder subdirs excluded type PluginConnectorConfig struct { Name string json:"name" yaml:"name" Config json.RawMessage json:"config" yaml:"config" } { "name": "connector", "config": { "name": "Elon" } } - Name - name of the plugin - Config - json config of the plugin Build plugin go build -buildmode=plugin -gcflags="all=-N -l" -o examples/plugin/connector.so examples/plugin/hardcoder/connector.go Make sure you export Plugin variable which implements pl.ConnectorPlugin interface Example for CLI: https://github.com/PxyUp/fitter/blob/master/examples/cli/config plugin.json L5 https://github.com/PxyUp/fitter/blob/master/examples/cli/config plugin.json L5 Plugin example: package main import "encoding/json" "fmt" "github.com/PxyUp/fitter/pkg/config" "github.com/PxyUp/fitter/pkg/logger" "github.com/PxyUp/fitter/pkg/builder" pl "github.com/PxyUp/fitter/pkg/plugins/plugin" var pl.ConnectorPlugin = &plugin{} Plugin plugin type plugin struct { log logger.Logger Name string json:"name" yaml:"name" } func pl plugin Get parsedValue builder.Interfacable, index uint32, input builder.Interfacable byte, error { return byte fmt.Sprintf {"name": "%s"} , pl.Name , nil } func pl plugin SetConfig cfg config.PluginConnectorConfig, logger logger.Logger { pl.log = logger if cfg.Config = nil { err := json.Unmarshal cfg.Config, pl if err = nil { pl.log.Errorw "cant unmarshal plugin configuration", "error", err.Error return } } } Connector which allow get prefetched data from references references type ReferenceConnectorConfig struct { Name string yaml:"name" json:"name" } Example https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L66 https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L66 - Name - reference name from references references map Improved version of static connector which generate int sequence as result type IntSequenceConnectorConfig struct { Start int json:"start" yaml:"start" End int json:"end" yaml:"end" Step int json:"step" yaml:"step" } - Start 0 - start point for generation included - End 0 - end point for generation excluded from final result like range in any lang - Step 1 - interval for sequence Example { "start": 0, "end": 2 // Generate 0, 1 } Connector type which fetch data from provided file type FileConnectorConfig struct { Path string yaml:"path" json:"path" UseFormatting bool yaml:"use formatting" json:"use formatting" } - Path - file path. Support formatting placeholder-list - UseFormatting false - use formatting placeholder-list file content or not Connector type which fetch data from provided string type StaticConnectorConfig struct { Value string json:"value" yaml:"value" Raw json.RawMessage json:"raw" yaml:"raw" } - Value - static string as data, can be html, json - Raw - accept raw json. Example https://github.com/PxyUp/fitter/blob/master/examples/config static.json . Also support formatting placeholder-list Example: https://github.com/PxyUp/fitter/blob/master/examples/cli/config static connector.json L5 https://github.com/PxyUp/fitter/blob/master/examples/cli/config static connector.json L5 { "value": " 1,2,3,4,5 " } Connector type which fetch data using golang http.Client server side request like curl type ServerConnectorConfig struct { Method string json:"method" yaml:"method" Headers map string string yaml:"headers" json:"headers" Timeout uint32 yaml:"timeout" json:"timeout" JsonRawBody json.RawMessage json:"json raw body" yaml:"json raw body" Body string yaml:"body" json:"body" Proxy ProxyConfig yaml:"proxy" json:"proxy" } - Method - supported all http methods: GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD - Headers - predefine headers for using during request can be injected into key/value placeholder-list - Timeout sec - default 60sec timeout or used provided - Body - body of the request, parsed value can be injected placeholder-list - JsonRawBody - body of the request in json format; value can be injected placeholder-list - Proxy - setup proxy for request config proxy-config Example: { "method": "GET", "proxy": { "server": "http://localhost:8080", "username": "pyx" } } Right now default timeout it is 10 sec. type ProxyConfig struct { // Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example // http://myproxy.com:3128 or socks5://myproxy.com:3128 . Short form myproxy.com:3128 // is considered an HTTP proxy. Server string json:"server" yaml:"server" // Optional username to use if HTTP proxy requires authentication. Username string json:"username" yaml:"username" // Optional password to use if HTTP proxy requires authentication. Password string json:"password" yaml:"password" } - Server - address with schema of proxy server. Also support formatting placeholder-list - Username - username for proxy can be empty . Also support formatting placeholder-list - Password - password for proxy can be empty . Also support formatting placeholder-list { "server": "http://localhost:8080", "username": "pyx" } FITTER HTTP WORKER - int 1000 - default concurrent HTTP workers Connector type which emulate fetching of data via browser type BrowserConnectorConfig struct { Chromium ChromiumConfig json:"chromium" yaml:"chromium" Docker DockerConfig json:"docker" yaml:"docker" Playwright PlaywrightConfig json:"playwright" yaml:"playwright" } Config can be one of: Chromium chromium - use local installed Chromium for fetch data Docker docker - use docker as service for spin up container for fetch data Playwright playwright - use playwright framework for fetch data Example: { "docker": { "wait": 10000, "image": "docker.io/zenika/alpine-chrome:with-node", "entry point": "chromium-browser", "purge": true } } Use locally installed Chromium for fetch the data type ChromiumConfig struct { Path string yaml:"path" json:"path" Timeout uint32 yaml:"timeout" json:"timeout" Wait uint32 yaml:"wait" json:"wait" Flags string yaml:"flags" json:"flags" } - Path - path to binary of Chromium - Timeout sec - timeout for execution of the chromium - Wait msec - timeout of page loading - Flags - flags for Chromium default: "--headless", "--proxy-auto-detect", "--temp-profile", "--incognito", "--disable-logging", "--disable-extensions", "--no-sandbox" Example: { "path": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "wait": 10000 } Use Docker for spin up container for fetch data type DockerConfig struct { Image string yaml:"image" json:"image" EntryPoint string json:"entry point" yaml:"entry point" Timeout uint32 yaml:"timeout" json:"timeout" Wait uint32 yaml:"wait" json:"wait" Flags string yaml:"flags" json:"flags" Purge bool json:"purge" yaml:"purge" NoPull bool yaml:"no pull" json:"no pull" PullTimeout uint32 yaml:"pull timeout" json:"pull timeout" } Docker default image : docker.io/zenika/alpine-chrome - Image - image for the docker registry provide with registry host - EntryPoint - cmd which will be run inside container - Timeout sec - timeout for run container without pulling image - Wait msec - timeout of page loading works just for Chromium based containers - Flags - cmd arguments for run containers, default for Chromium based: "--no-sandbox","--headless", "--proxy-auto-detect", "--temp-profile", "--incognito", "--disable-logging", "--disable-gpu" - Purge - should we remove container after work done like docker rm - NoPull - prevent pulling of the image - PullTimeout - define timeout for pull contains DOCKER HOST - string - EnvOverrideHost to set the URL to the docker server. DOCKER API VERSION - string - EnvOverrideAPIVersion to set the version of the API to use, leave empty for latest. DOCKER CERT PATH - string - EnvOverrideCertPath to specify the directory from which to load the TLS certificates ca.pem, cert.pem, key.pem . DOCKER TLS VERIFY - bool - EnvTLSVerify to enable or disable TLS verification off by default Example: { "wait": 10000, "image": "docker.io/zenika/alpine-chrome:with-node", "entry point": "chromium-browser", "purge": true } Run browsers via playwright framework type PlaywrightConfig struct { Browser PlaywrightBrowser json:"browser" yaml:"browser" Install bool yaml:"install" json:"install" Timeout uint32 yaml:"timeout" json:"timeout" Wait uint32 yaml:"wait" json:"wait" TypeOfWait playwright.WaitUntilState json:"type of wait" yaml:"type of wait" PreRunScript string json:"pre run script" yaml:"pre run script" Stealth bool json:"stealth" yaml:"stealth" Proxy ProxyConfig yaml:"proxy" json:"proxy" } - Browser - enum "Chromium", "FireFox", "WebKit" - which browser to use - Install - should we install browser - Timeout sec - timeout to run playwright - Wait sec - timeout of page loading - TypeOfWait - enum "load", "domcontentloaded", "networkidle", "commit" which state of page we waiting, default is "load" - PreRunScript "" - script which will be executed before reading content of the page. Also support placeholder {PL} placeholder-list - Stealth false - add script for trying passing bot defends - Proxy - setup proxy for request config proxy-config Example { "timeout": 30, "wait": 30, "install": false, "browser": "Chromium" } With model we define result of the scrapping type Model struct { ObjectConfig ObjectConfig yaml:"object config" json:"object config" ArrayConfig ArrayConfig json:"array config" yaml:"array config" BaseField BaseField json:"base field" yaml:"base field" IsArray bool json:"is array" yaml:"is array" } Config can be one of: ObjectConfig objectconfig - configuration of object format ArrayConfig arrayconfig - configuration of array format BaseField basefield - configuration of single/generated field- IsArray - bool false - force indicate that field is array usable in case of model field model-field with base field basefield Example: { "object config": {} } Configuration of the object and fields type ObjectConfig struct { Fields map string Field json:"fields" yaml:"fields" Field BaseField json:"field" yaml:"field" ArrayConfig ArrayConfig json:"array config" yaml:"array config" } Config can be one of: Fields field - map of each field definition; key - field name, value - configuration Field basefield - used for element of array; fields which will be deserialized like basic type like "string", "int" and etc used here for case array of basic types ArrayConfig arrayconfig - used for element of array; deserialization array of array Example: { "fields": { "title": { "base field": { "type": "string", "path": "type" } } } } Configuration of the array and fields type ArrayConfig struct { RootPath string json:"root path" yaml:"root path" Reverse bool yaml:"reverse" json:"reverse" ItemConfig ObjectConfig json:"item config" yaml:"item config" LengthLimit uint32 json:"length limit" yaml:"length limit" StaticConfig StaticArrayConfig json:"static array" yaml:"static array" } - RootPath - selector for find root element of the array or repeated element in case of html parsing, size of array will be amount of children element under the root - Reverse - bool false - indicate that need use reverse iteration n to 1 - LengthLimit - for define size of array only for generated not working for static Config can be one of: ItemConfig objectconfig - configuration of each element of the array StaticConfig static-array-config - configuration of the static array Example: { "root path": " content dt.quote a", "item config": { "field": { "type": "string" } } } Common of the field type Field struct { BaseField BaseField json:"base field" yaml:"base field" ObjectConfig ObjectConfig json:"object config" yaml:"object config" ArrayConfig ArrayConfig json:"array config" yaml:"array config" FirstOf Field json:"first of" yaml:"first of" } Config can be one of: BaseField basefield - fields which will be deserialized like basic type like "string", "int" and etc ObjectConfig objectconfig - in case our field in nested object ArrayConfig arrayconfig - in case our field in array FirstOf field - first not empty resolved field will be selected Example: { "base field": { "type": "string", "path": "div.current-temp span.heading" } } In case we want get some static information or generate new one type BaseField struct { Type FieldType yaml:"type" json:"type" Path string yaml:"path" json:"path" HTMLAttribute string json:"html attribute" yaml:"html attribute" Generated GeneratedFieldConfig yaml:"generated" json:"generated" FirstOf BaseField json:"first of" yaml:"first of" } - FieldType - enum "null", "boolean", "string", "int", "int64", "float", "float64", "array", "object", "html", "raw string" - static field for parse. Important : type html will only works from connector which return HTML HTMLAttribute - have no effect in this case . Example https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L25 - Path - selector relative in case it is array child for parsing - HTMLAttribute - extra value which have effect only in HTML parsing via goquery . Here you can specify which attribute need to be parsed. Important : by default "string" type trimmed and all special chars is replaced, if you need plain string use "raw string" Config can be one of or empty: Generated generatedfieldconfig - field can be generated one which custom configuration FirstOf basefield - first not empty resolved field will be selected Examples { "generated": { "uuid": {} } } { "type": "string", "path": "text " } Provide functionality of generating field on the flight type GeneratedFieldConfig struct { UUID UUIDGeneratedFieldConfig yaml:"uuid" json:"uuid" Static StaticGeneratedFieldConfig yaml:"static" json:"static" Formatted FormattedFieldConfig json:"formatted" yaml:"formatted" Plugin PluginFieldConfig yaml:"plugin" json:"plugin" Calculated CalculatedConfig yaml:"calculated" json:"calculated" File FileFieldConfig yaml:"file" json:"file" Model ModelField yaml:"model" json:"model" FileStorageField FileStorageField json:"file storage" yaml:"file storage" } Config can be one of: UUID uuid - generate random UUID V4 Static static - generate static field Formatted formatted-field-config - format field Model model-field - model generated from the other connector and model Plugin plugin-field - plugin field Calculated calculated-field - calculated field File file-field - file field for download file from server FileStorage file-storage-field - file field which can be saved to local file Examples: { "uuid": {} } https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L58 https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L58 { "model": { "type": "array", "model": { "array config": { "root path": " content dt.quote a", "item config": { "field": { "type": "string" } } } }, "connector config": { "response type": "HTML", "attempts": 3, "browser config": { "url": "http://www.quotationspage.com/random.php", "chromium": { "path": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "wait": 10000 } } } } } Generate random UUID V4 on the flight, can be used for generate uniq id type UUIDGeneratedFieldConfig struct { Regexp string yaml:"regexp" json:"regexp" } - Regexp - provide matcher which can be used for get part of generated uuid Generate static field type StaticGeneratedFieldConfig struct { Type FieldType yaml:"type" json:"type" Value string json:"value" yaml:"value" Raw json.RawMessage json:"raw" yaml:"raw" } - Type - enum "null", "boolean", "string", "int","int64","float","float64", "array", "object" - type of the field - Value - string value of the field - Raw - pure json value of the field Example { "type": "int", "value": "65" } { "type": "array", "value": " 65,45 " } { "type": "array", "raw": 65,45 } Generate formatted field which will pass value from parent base field basefield type FormattedFieldConfig struct { Template string yaml:"template" json:"template" } Example: https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L98 https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L98 { "template": "https://news.ycombinator.com/item?id={PL}" } Field can be used for store field result as local file type FileStorageField struct { Content string json:"content" yaml:"content" Raw json.RawMessage yaml:"raw" yaml:"raw" FileName string json:"file name" yaml:"file name" Path string json:"path" yaml:"path" Append bool json:"append" yaml:"append" } - Content - template string for content. Important: can be with inject of the parent value as a string placeholder-list - Raw - raw json content of field. Important: can be with inject of the parent value as a string placeholder-list - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with inject of the parent value as a string placeholder-list . - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with inject of the parent value as a string placeholder-list - Append false - append to file or not { "content": "{{{id}}}, {{{message}}}\n", "append": true, "file name": "{{{id}}}.csv", "path": "/Users/pxyup/fitter/examples/cli/test/csv" } Field can be used for download file from server locally type FileFieldConfig struct { Config ServerConnectorConfig yaml:"config" json:"config" Url string yaml:"url" json:"url" FileName string json:"file name" yaml:"file name" Path string json:"path" yaml:"path" } - Config - ServerConfig serverconnectorconfig use default fitter http.Client for send request - Url - url of the image. Important: URL in the connector can be with inject of the parent value as a string placeholder-list - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with inject of the parent value as a string placeholder-list . - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with inject of the parent value as a string placeholder-list Result of the field will be local file path as string { "url": "https://images.shcdn.de/resized/w680/p/dekostoff-gobelinstoff-panel-oriental-cat-46-x-46 P19-KP 2.jpg", "path": "/Users/pxyup/fitter/bin", "config": { "method": "GET" } } With propagated URL inject of the parent value as a string placeholder-list { "url": "https://picsum.photos{PL}", "path": "/Users/pxyup/fitter/bin", "config": { "method": "GET" } } Config example: https://github.com/PxyUp/fitter/blob/master/examples/cli/config image.json https://github.com/PxyUp/fitter/blob/master/examples/cli/config image.json https://github.com/PxyUp/fitter/blob/master/examples/cli/config image multiple.json https://github.com/PxyUp/fitter/blob/master/examples/cli/config image multiple.json Field can generate different types depends from expression type CalculatedConfig struct { Type FieldType yaml:"type" json:"type" Expression string yaml:"expression" json:"expression" } - Type - resulting type of expression\ - Expression - expression for calculation we use this lib https://github.com/expr-lang/expr for calculated expression FNull - alias for builder.Nullvalue FNil - alias for nil isNull value T - function for check is value is FNull fRes - it is raw with proper type result from the parsing base field basefield fIndex - it is index in parent array only if parent was array field fResJson - it is JSON string representation of the raw result fResRaw - result in bytes format FNewLine - new line separator { "type": "bool", "expression": "fRes 500" } Field can be some external plugin for fitter type PluginFieldConfig struct { Name string json:"name" yaml:"name" Config json.RawMessage json:"config" yaml:"config" } - Name - name of the plugin without extension just name - Config - json config of the plugin Field type which can be generated on the flight by news model model and connector connector type ModelField struct { // Type of parsing ConnectorConfig ConnectorConfig yaml:"connector config" json:"connector config" // Model of the response Model Model yaml:"model" json:"model" Type FieldType yaml:"type" json:"type" Path string yaml:"path" json:"path" Expression string yaml:"expression" json:"expression" } ConnectorConfig connector - which connector to use. Important: URL in the connector can be with inject of the parent value as a string placeholder-list Model model - configuration of the underhood model- Type - enum "null", "boolean", "string", "int", "int64", "float", "float64", "array", "object" - type of generated field - Path - in case we cant extract some information from generated field we can use json selector for extract - Expression - string which can be used for post processing of the Model ignoring path field Examples: https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L60 https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L60 { "type": "array", "model": { "array config": { "root path": " content dt.quote a", "item config": { "field": { "type": "string" } } } } } https://github.com/PxyUp/fitter/blob/master/examples/cli/config weather.json L37 https://github.com/PxyUp/fitter/blob/master/examples/cli/config weather.json L37 { "type": "string", "path": "temp.temp", "model": { "object config": { "fields": { "temp": { "base field": { "type": "string", "path": "//div @id='forecast list ul' //td/b/a/@href", "generated": { "model": { "type": "string", "model": { "object config": { "fields": { "temp": { "base field": { "type": "string", "path": "div.current-temp span.heading" } } } } }, "connector config": { "response type": "HTML", "attempts": 4, "url": "https://openweathermap.org{PL}", "browser config": { "playwright": { "timeout": 30, "wait": 30, "install": false, "browser": "FireFox", "type of wait": "networkidle" } } } } } } } } } }, "connector config": { "response type": "xpath", "attempts": 3, "url": "https://openweathermap.org/find?q={PL}", "browser config": { "playwright": { "timeout": 30, "wait": 30, "install": false, "browser": "Chromium" } } } } Provide static fixed length array generation type StaticArrayConfig struct { Items map uint32 Field yaml:"items" json:"items" Length uint32 yaml:"length" json:"length" } Items field - map uint32 Field field - key is index in array, value is field definition- Length - if set 1+ can be used for define custom length of array Examples: { "0": { "base field": { "type": "string", "path": "div.current-temp span.heading" } } } { "length": 4, "0": { "base field": { "type": "string", "path": "div.current-temp span.heading" } } } { "length": 4, "2": { "base field": { "type": "string", "path": "div.current-temp span.heading" } } } - {PL} - for inject value - {INDEX} - for inject index in parent array - {HUMAN INDEX} - for inject index in parent array in human way - {{{json path}}} - will get information from propagated "object"/"array" field - {{{RefName=SomeName}}} - get reference references value by name. Example https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L67 - {{{RefName=SomeName json.path}}} - get reference references value by name and extract value by json path. Example https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L67 - {{{FromEnv=ENV KEY}}} - get value from environment variable - {{{FromExp=fRes + 5 + fIndex}}} - get value from the expression https://github.com/expr-lang/expr . Predefined values predefined-values - {{{FromInput=.}}} or {{{FromInput=json.path}}} - get value from input of trigger or library - {{{FromFile=./test file.log}}} - get value from file by path. Content of file also can contain placeholders - {{{FromURL= http://localhost:8081}}} http://localhost:8081%7D%7D%7D - get response from url Examples: {{{FromExp="{{{FromEnv=TEST VAL}}}" + "hello"}}} Current time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}} Current time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}} TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}} Object={{{value}}} {PL} Env={{{FromEnv=TEST VAL}}} {INDEX} {HUMAN INDEX} Special map which prefetched before any processing and can be user for connector referenceconnectorconfig or for placeholder placeholder-list Can be used for: - Cache jwt token and use them in headers - Cache values - Etc type Reference struct { ModelField Expire uint32 yaml:"expire" json:"expire" } ModelField model-field - is embedded struct, you can use same fields- Expire sec - duration when reference is expired after fetching. Not set = forever cached . Set to 0 = every time re-fetch . Set to n 0 = cached for n second For Fitter how-to-use-fitter type RefMap map string Reference type Config struct { // Other Config Fields Limits Limits yaml:"limits" json:"limits" References RefMap json:"references" yaml:"references" } For Fitter Cli how-to-use-fittercli type RefMap map string Reference type CliItem struct { // Other Config Fields Limits Limits yaml:"limits" json:"limits" References RefMap json:"references" yaml:"references" } - References - map string Reference - object where is key if ReferenceName can be user for connector referenceconnectorconfig or placeholder placeholder-list and value is Reference reference Limits limits Example https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L2 https://github.com/PxyUp/fitter/blob/master/examples/cli/config ref.json L2 { "references": { "TokenRef": { "expire": 10, "connector config": { "response type": "json", "static config": { "value": "\"plain token\"" } }, "model": { "base field": { "type": "string" } } }, "TokenObjectRef": { "connector config": { "response type": "json", "static config": { "value": "{\"token\":\"token from object\"}" } }, "model": { "object config": { "fields": { "token": { "base field": { "type": "string", "path": "token" } } } } } } } } Provide limitation for prevent DDOS, big usage of memory type Limits struct { HostRequestLimiter HostRequestLimiter yaml:"host request limiter" json:"host request limiter" ChromiumInstance uint32 yaml:"chromium instance" json:"chromium instance" DockerContainers uint32 yaml:"docker containers" json:"docker containers" PlaywrightInstance uint32 yaml:"playwright instance" json:"playwright instance" } - HostRequestLimiter - map string int64 - limitation per host name, key is host, value is amount of parallel request usage for server connector serverconnectorconfig - ChromiumInstance - amount of parallel chromium chromium instance - DockerContainers - amount of parallel docker docker instance - PlaywrightInstance - amount of parallel playwright playwright instance https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L2 https://github.com/PxyUp/fitter/blob/master/examples/cli/config cli.json L2 { "limits": { "host request limiter": { "hacker-news.firebaseio.com": 5 }, "chromium instance": 3, "docker containers": 3, "playwright instance": 3 } } - Add browser scenario for preparing, after parsing - Add scrolling support for scenario - Add pagination support for scenario - Add notification methods for Fitter: Webhook/Queue