{"slug": "show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code", "title": "Show HN: I built a scraping engine where the LLM writes configs, not code", "summary": "A developer released Fitter, an open-source scraping engine that converts any website or API into structured JSON using declarative YAML/JSON configs instead of code, enabling LLMs like Claude Code to author and run scraping pipelines locally. The tool supports HTTP, headless browser, and file connectors with JSON, XML, HTML, and XPath parsing, and is available as a static binary with MCP server integration.", "body_md": "**Fitter turns any website or API into structured JSON — declaratively.** One JSON/YAML config describes where the data lives (HTTP request, headless browser, file, static value) and what to extract (JSON paths, CSS selectors, XPath). No code, no brittle scraping scripts.\n\nBecause configs are plain data, **LLMs can author them**. The built-in [MCP server](#how-to-use-fitter_mcp) lets Claude Code, Claude Desktop, or any MCP client write and run scraping pipelines on your machine, on demand:\n\n\"Get the top 5 HackerNews stories with titles and scores\"→ the model authors a fitter config, validates it, runs it locally, and gets clean JSON back.\n\n**One engine, five ways to use it:**\n\n🤖 Fitter MCP |\n|\n\n**Fitter Agent****Fitter CLI****Fitter Lib****Fitter****Why fitter for AI agents?**\n\n**Declarative & auditable**— the agent produces a config you can read, save and re-run, not throwaway code** Local-first**— all fetching happens on your machine; no third-party scraping API, no keys, no per-request billing** Batteries included**— HTTP client, headless browser (Playwright/Chromium/Docker), JSON/HTML/XML/XPath parsing, pagination, cached references, host rate-limits — in a single static binary**Reusable**— what the agent authored today becomes tomorrow's cron job or service config\n\n**Server**- parsing response from some API's or http request(usage of http.Client)** Browser**- emulate real browser using chromium + docker + playwright/cypress and get DOM information** Static**- parsing static string as data\n\n**JSON**- parsing JSON to get specific information** XML**- parsing xml tree to get specific information** HTML**- parsing dom tree to get specific information** XPath**- parsing dom tree to get specific information but by xpath\n\n```\ngo get github.com/PxyUp/fitter\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/PxyUp/fitter/lib\"\n\t\"github.com/PxyUp/fitter/pkg/config\"\n\t\"log\"\n\t\"net/http\"\n)\n\nfunc main() {\n\tres, err := lib.Parse(&config.Item{\n\t\tConnectorConfig: &config.ConnectorConfig{\n\t\t\tResponseType:  config.Json,\n\t\t\tUrl:           \"https://random-data-api.com/api/appliance/random_appliance\",\n\t\t\tServerConfig: &config.ServerConnectorConfig{\n\t\t\t\tMethod: http.MethodGet,\n\t\t\t},\n\t\t},\n\t\tModel: &config.Model{\n\t\t\tObjectConfig: &config.ObjectConfig{\n\t\t\t\tFields: map[string]*config.Field{\n\t\t\t\t\t\"my_id\": {\n\t\t\t\t\t\tBaseField: &config.BaseField{\n\t\t\t\t\t\t\tType: config.Int,\n\t\t\t\t\t\t\tPath: \"id\",\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t\"generated_id\": {\n\t\t\t\t\t\tBaseField: &config.BaseField{\n\t\t\t\t\t\t\tGenerated: &config.GeneratedFieldConfig{\n\t\t\t\t\t\t\t\tUUID: &config.UUIDGeneratedFieldConfig{},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t\"generated_array\": {\n\t\t\t\t\t\tArrayConfig: &config.ArrayConfig{\n\t\t\t\t\t\t\tRootPath: \"@this|@keys\",\n\t\t\t\t\t\t\tItemConfig: &config.ObjectConfig{\n\t\t\t\t\t\t\t\tField: &config.BaseField{\n\t\t\t\t\t\t\t\t\tType: config.String,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t}, nil, nil, nil, nil)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Println(res.ToJson())\n}\n```\n\nOutput:\n\n```\n{\n  \"generated_array\": [\"id\",\"uid\",\"brand\",\"equipment\"],\n  \"my_id\": 6000,\n  \"generated_id\": \"26b08b73-2f2e-444d-bcf2-dac77ac3130e\"\n}\n```\n\n[Download latest version from the release page](https://github.com/PxyUp/fitter/releases)\n\nor locally:\n\n```\ngo run cmd/fitter/main.go --path=./examples/config_api.json\n```\n\n**--path**- string[\"\"] - path for the configuration of the Fitter**--url**- string[\"\"] - url for the configuration of the Fitter**--verbose**- bool[false] - enable logging**--plugins**- string[\"\"] -[path for plugins for Fitter](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md)**--log-level**- enum[\"info\", \"error\", \"debug\", \"fatal\"] - set log level(only if verbose set to true)\n\n[Download latest version from the release page](https://github.com/PxyUp/fitter/releases)\n\nor locally:\n\n```\ngo run cmd/cli/main.go --path=./examples/cli/config_cli.json\n```\n\n**--path**- string[\"\"] - path for the configuration of the Fitter_CLI**--url**- string[\"\"] - url for the configuration of the Fitter_CLI**--copy**- bool[false] - copy information into clipboard**--pretty**- bool[true] - make readable result(also affect on copy)**--verbose**- bool[false] - enable logging**--omit-error-pretty**- bool[false] - Provide pure value if pretty is invalid**--plugins**- string[\"\"] -[path for plugins for Fitter](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md)**--log-level**- enum[\"info\", \"error\", \"debug\", \"fatal\"] - set log level(only if verbose set to true)**--input**- string[\"\"] - specify input value for[formatting](#placeholder-list). Examples:`--input=\\\"\"124\"\\\"`\n\n`--input=124`\n\n`--input='{\"test\": 5}'`\n\n```\n./fitter_cli_${VERSION} --path=./examples/cli/config_cli.json --copy=true\n```\n\nExamples:\n\n**Server version**[HackerNews + Quotes + Guardian News](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json)- using API + HTML + XPath parsing**Chromium version**[Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_browser.json)- using HTML parsing + browser emulation**Docker version**[Docker version: Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_docker.json)- using HTML parsing + browser from Docker image**Playwright version**[Playwright version: Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_playwright.json)- using HTML parsing + browser from Playwright framework**Playwright version**[Playwright version: England Cities + Weather](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_weather.json)- using HTML + XPath parsing + browser from Playwright framework**JSON version**[Generate pagination](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_static_connector.json)- using static connector for generate pagination array**Server version**[Get current time](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_current_time.json)- get time from url and format it\n\nFitter Agent is an AI-powered CLI that uses Claude to convert natural language requests into Fitter configurations and execute them automatically.\n\n[Download latest version from the release page](https://github.com/PxyUp/fitter/releases)\n\nor locally:\n\n```\nexport ANTHROPIC_API_KEY=<your-anthropic-api-key>\ngo run cmd/agent/main.go\n```\n\n**--api-key**- string[\"\"] - Anthropic API key. Prefer the`ANTHROPIC_API_KEY`\n\nenvironment variable so the key does not end up in your shell history**--model**- string[\"claude-opus-4-8\"] - Claude model to use**--effort**- enum[\"low\", \"medium\", \"high\", \"xhigh\", \"max\"] - reasoning effort, default \"high\". Lower it for faster/cheaper configs, raise it for harder extractions**--verbose**- bool[false] - enable logging**--log-level**- enum[\"info\", \"error\", \"debug\", \"fatal\"] - set log level**--plugins**- string[\"\"] - path for plugins for Fitter**--chromium-limit**- uint[0] - limit concurrent Chromium instances**--docker-limit**- uint[0] - limit concurrent Docker containers**--playwright-limit**- uint[0] - limit concurrent Playwright instances\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│  1. User enters natural language request                       │\n│     \"Get top 5 HackerNews stories with titles and scores\"      │\n│                              ↓                                  │\n│  2. Claude returns a config in a schema-constrained response   │\n│                              ↓                                  │\n│  3. Agent validates it; on failure the error is handed back    │\n│     to Claude to repair (up to 3 attempts)                     │\n│                              ↓                                  │\n│  4. Agent displays config and asks for confirmation            │\n│                              ↓                                  │\n│  5. On confirmation, executes via lib.Parse()                  │\n│                              ↓                                  │\n│  6. Returns structured JSON result                             │\n└─────────────────────────────────────────────────────────────────┘\n```\n\nThe agent keeps the conversation, so after a config is generated you can just say what to change instead of restating the whole request:\n\n```\n> Get top 3 HackerNews stories with titles and scores\nrefine> Only return 5 items and also include the article URL\n```\n\nUse **new** to forget the current config and start a fresh session.\n\n**help**- show help message** new/reset**- forget the current config and start fresh** clear**- clear the screen** exit/quit/q**- exit the agent\n\n``` bash\n$ export ANTHROPIC_API_KEY=sk-ant-...\n$ ./fitter_agent\n\n╔══════════════════════════════════════════════════════════════╗\n║           Fitter Agent - AI-Powered Data Extraction           ║\n╚══════════════════════════════════════════════════════════════╝\n\nDescribe what you want to extract. Follow-up messages refine the\nprevious config. Type 'help' for commands.\n\n> Get top 3 HackerNews stories with titles and scores\n\n┌─ Generated Fitter Config ───────────────────────────────────────\n{\n  \"item\": {\n    \"connector_config\": {\n      \"response_type\": \"json\",\n      \"url\": \"https://hacker-news.firebaseio.com/v0/topstories.json\",\n      \"server_config\": { \"method\": \"GET\" }\n    },\n    \"model\": {\n      \"array_config\": {\n        \"root_path\": \"@this\",\n        \"length_limit\": 3,\n        \"item_config\": {\n          \"fields\": {\n            \"id\": { \"base_field\": { \"type\": \"int\" } },\n            \"story\": {\n              \"base_field\": {\n                \"type\": \"int\",\n                \"generated\": {\n                  \"model\": {\n                    \"type\": \"object\",\n                    \"connector_config\": {\n                      \"response_type\": \"json\",\n                      \"url\": \"https://hacker-news.firebaseio.com/v0/item/{PL}.json\",\n                      \"server_config\": { \"method\": \"GET\" }\n                    },\n                    \"model\": {\n                      \"object_config\": {\n                        \"fields\": {\n                          \"title\": { \"base_field\": { \"type\": \"string\", \"path\": \"title\" } },\n                          \"score\": { \"base_field\": { \"type\": \"int\", \"path\": \"score\" } }\n                        }\n                      }\n                    }\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n}\n└──────────────────────────────────────────────────────────────────\n\nExecute this config? [y/n]: y\n\n┌─ Result ────────────────────────────────────────────────────────\n[\n  {\n    \"id\": 46740029,\n    \"story\": { \"title\": \"Show HN: Open-source project\", \"score\": 161 }\n  },\n  {\n    \"id\": 46737630,\n    \"story\": { \"title\": \"Interesting article\", \"score\": 237 }\n  },\n  {\n    \"id\": 46735644,\n    \"story\": { \"title\": \"New technology release\", \"score\": 192 }\n  }\n]\n└──────────────────────────────────────────────────────────────────\n\n> exit\nGoodbye!\n```\n\n| Request | What it does |\n|---|---|\n`Get Bitcoin price from CoinGecko API` |\nFetches current BTC price |\n`Scrape headlines from news.ycombinator.com with links` |\nHTML scraping with CSS selectors |\n`Get top 5 stories from HackerNews with titles` |\nNested API calls |\n`Fetch weather data from wttr.in for London` |\nSimple API extraction |\n`Scrape product names and prices from example.com` |\nWeb scraping |\n\nThe agent can generate configs for:\n\n**JSON APIs**- REST APIs with GET/POST methods** HTML Scraping**- CSS selector-based extraction** XPath Scraping**- XPath-based extraction** Nested API Calls**- Fetch details for each item in a list** Browser Emulation**- Playwright for JS-rendered pages** Formatted Fields**- URL templates with placeholders** Array Limiting**- Limit results to N items\n\nFitter MCP is a [Model Context Protocol](https://modelcontextprotocol.io) server (stdio transport) which lets any MCP client — Claude Code, Claude Desktop, IDE assistants, custom agents — run Fitter configs and get structured JSON back.\n\n[Download latest version from the release page](https://github.com/PxyUp/fitter/releases)\n\nor build locally:\n\n```\ngo build -o bin/fitter_mcp ./cmd/mcp\n# available in every project\nclaude mcp add fitter -s user -- /path/to/fitter_mcp\n{\n  \"mcpServers\": {\n    \"fitter\": {\n      \"command\": \"/path/to/fitter_mcp\"\n    }\n  }\n}\n```\n\n| Tool | Description |\n|---|---|\n`fitter_run` |\nRun a Fitter config passed inline (JSON or YAML string) and return the extracted data as JSON. Accepts an optional `input` value available in the config via `{{{FromInput=.}}}` / `{{{FromInput=json.path}}}` |\n`fitter_run_file` |\nSame as `fitter_run` but reads the config from a local `.json` /`.yaml` file |\n`fitter_validate_config` |\nValidate a config without executing it (structure, `response_type` , connector data source, model). Useful while iterating on a config |\n`fitter_config_reference` |\nReturn a condensed reference of the whole config format (connectors, parsers, model/field schema, placeholders, references, limits) with working examples, so the model can author configs without external docs |\n\nThe config format is exactly the same as for [Fitter_CLI](#how-to-use-fitter_cli): a top-level object with `item`\n\n(required), `limits`\n\nand `references`\n\n.\n\n**FITTER_PLUGINS**- string[\"\"] -[path for plugins folder](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md), same as the`--plugins`\n\nflag of Fitter/Fitter_CLI\n\nAsk your MCP client something like:\n\nGet the top 5 HackerNews stories with titles and scores using fitter\n\nThe model calls `fitter_config_reference`\n\n, authors a config, optionally checks it with `fitter_validate_config`\n\nand executes it via `fitter_run`\n\n— all data fetching happens locally on your machine.\n\nIt is the way how you fetch the data\n\n```\ntype ConnectorConfig struct {\n    ResponseType ParserType `json:\"response_type\" yaml:\"response_type\"`\n    Url          string     `json:\"url\" yaml:\"url\"`\n    Attempts     uint32     `json:\"attempts\" yaml:\"attempts\"`\n    \n    NullOnError bool `yaml:\"null_on_error\" json:\"null_on_error\"`\n    \n    StaticConfig          *StaticConnectorConfig      `json:\"static_config\" yaml:\"static_config\"`\n    IntSequenceConfig     *IntSequenceConnectorConfig `json:\"int_sequence_config\" yaml:\"int_sequence_config\"`\n    ServerConfig          *ServerConnectorConfig      `json:\"server_config\" yaml:\"server_config\"`\n    BrowserConfig         *BrowserConnectorConfig     `yaml:\"browser_config\" json:\"browser_config\"`\n    PluginConnectorConfig *PluginConnectorConfig      `json:\"plugin_connector_config\" yaml:\"plugin_connector_config\"`\n    ReferenceConfig       *ReferenceConnectorConfig   `yaml:\"reference_config\" json:\"reference_config\"`\n    FileConfig            *FileConnectorConfig        `json:\"file_config\" yaml:\"file_config\"`\n}\n```\n\n- NullOnError[false] - if set to true then all errors a ignored\n- ResponseType - enum[\"HTML\", \"json\",\"xpath\"] - in which format data comes from the connector\n- Attempts - how many attempts to use for fetch data by connector\n- Url - define which address to request. Important: can be with\n[inject of the parent value as a string](#placeholder-list)`https://api.open-meteo.com/v1/forecast?latitude={{{latitude}}}&longitude={{{longitude}}}&hourly=temperature_2m&forecast_days=1`\n\nConfig can be one of:\n\n[ServerConfig](#serverconnectorconfig)[BrowserConfig](#browserconnectorconfig)[StaticConfig](#staticconnectorconfig)[PluginConnectorConfig](#pluginconnectorconfig)[ReferenceConfig](#referenceconnectorconfig)[IntSequenceConfig](#intsequenceconnectorconfig)[FileConfig](#fileconnectorconfig)\n\nExample:\n\n```\n{\n  \"response_type\": \"xpath\",\n  \"attempts\": 3,\n  \"url\": \"https://openweathermap.org/find?q={PL}\",\n  \"browser_config\": {\n    \"playwright\": {\n      \"timeout\": 30,\n      \"wait\": 30,\n      \"install\": false,\n      \"browser\": \"Chromium\"\n    }\n  }\n}\n```\n\nConnector can be defined via plugin system. For use that you need apply next flags to Fitter/Cli(location of the plugins):\n\n```\n... --plugins=./examples/plugin\n```\n\n--plugins - looking for all files with \".so\" extension in provided folder(subdirs excluded)\n\n```\ntype PluginConnectorConfig struct {\n\tName   string          `json:\"name\" yaml:\"name\"`\n\tConfig json.RawMessage `json:\"config\" yaml:\"config\"`\n}\n{\n    \"name\": \"connector\",\n    \"config\": {\n      \"name\": \"Elon\"\n    }\n}\n```\n\n- Name - name of the plugin\n- Config - json config of the plugin\n\nBuild plugin\n\n```\ngo build -buildmode=plugin -gcflags=\"all=-N -l\" -o examples/plugin/connector.so examples/plugin/hardcoder/connector.go\n```\n\nMake sure you export **Plugin** variable which implements **pl.ConnectorPlugin** interface\n\nExample for CLI:\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_plugin.json#L5](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_plugin.json#L5)\n\nPlugin example:\n\n```\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"github.com/PxyUp/fitter/pkg/config\"\n\t\"github.com/PxyUp/fitter/pkg/logger\"\n\t\"github.com/PxyUp/fitter/pkg/builder\"\n\tpl \"github.com/PxyUp/fitter/pkg/plugins/plugin\"\n)\n\nvar (\n\t_ pl.ConnectorPlugin = &plugin{}\n\n\tPlugin plugin\n)\n\ntype plugin struct {\n\tlog  logger.Logger\n\tName string `json:\"name\" yaml:\"name\"`\n}\n\nfunc (pl *plugin) Get(parsedValue builder.Interfacable, index *uint32, input builder.Interfacable) ([]byte, error) {\n\treturn []byte(fmt.Sprintf(`{\"name\": \"%s\"}`, pl.Name)), nil\n}\n\nfunc (pl *plugin) SetConfig(cfg *config.PluginConnectorConfig, logger logger.Logger) {\n\tpl.log = logger\n\n\tif cfg.Config != nil {\n\t\terr := json.Unmarshal(cfg.Config, pl)\n\t\tif err != nil {\n\t\t\tpl.log.Errorw(\"cant unmarshal plugin configuration\", \"error\", err.Error())\n\t\t\treturn\n\t\t}\n\t}\n}\n```\n\nConnector which allow get prefetched data from [references](#references)\n\n```\ntype ReferenceConnectorConfig struct {\n\tName string `yaml:\"name\" json:\"name\"`\n}\n```\n\nExample\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L66](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L66)\n\n- Name - reference name from\n[references](#references)map\n\nImproved version of static connector which generate int sequence as result\n\n```\ntype IntSequenceConnectorConfig struct {\n\tStart int `json:\"start\" yaml:\"start\"`\n\tEnd   int `json:\"end\" yaml:\"end\"`\n\tStep  int `json:\"step\" yaml:\"step\"`\n}\n```\n\n- Start[0] - start point for generation(\n**included**) - End[0] - end point for generation(\n**excluded** from final result like range in any lang) - Step[1] - interval for sequence\n\nExample\n\n```\n{\n    \"start\": 0,\n    \"end\": 2 \n    // Generate [0, 1]\n}\n```\n\nConnector type which fetch data from provided file\n\n```\ntype FileConnectorConfig struct {\n    Path          string `yaml:\"path\" json:\"path\"`\n    UseFormatting bool   `yaml:\"use_formatting\" json:\"use_formatting\"`\n}\n```\n\n- Path - file path. Support\n[formatting](#placeholder-list) - UseFormatting[false] - use\n[formatting](#placeholder-list)file content or not\n\nConnector type which fetch data from provided string\n\n```\ntype StaticConnectorConfig struct {\n    Value string `json:\"value\" yaml:\"value\"`\n    Raw   json.RawMessage `json:\"raw\" yaml:\"raw\"`\n}\n```\n\n- Value - static string as data, can be html, json\n- Raw - accept raw json.\n[Example](https://github.com/PxyUp/fitter/blob/master/examples/config_static.json). Also support[formatting](#placeholder-list)\n\nExample:\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_static_connector.json#L5](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_static_connector.json#L5)\n\n```\n{\n  \"value\": \"[1,2,3,4,5]\"\n}\n```\n\nConnector type which fetch data using golang http.Client(server side request like curl)\n\n```\ntype ServerConnectorConfig struct {\n    Method      string            `json:\"method\" yaml:\"method\"`\n    Headers     map[string]string `yaml:\"headers\" json:\"headers\"`\n    Timeout     uint32            `yaml:\"timeout\" json:\"timeout\"`\n    JsonRawBody json.RawMessage   `json:\"json_raw_body\" yaml:\"json_raw_body\"`\n    Body        string            `yaml:\"body\" json:\"body\"`\n    \n    Proxy *ProxyConfig `yaml:\"proxy\" json:\"proxy\"`\n}\n```\n\n- Method - supported all http methods: GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD\n- Headers - predefine headers for using during request\n[can be injected into key/value](#placeholder-list) - Timeout[sec] - default 60sec timeout or used provided\n- Body - body of the request, parsed value\n[can be injected](#placeholder-list) - JsonRawBody - body of the request in json format; value\n[can be injected](#placeholder-list) - Proxy - setup proxy for request\n[config](#proxy-config)\n\nExample:\n\n```\n{\n  \"method\": \"GET\",\n  \"proxy\": {\n    \"server\": \"http://localhost:8080\",\n    \"username\": \"pyx\"\n  }\n}\n```\n\nRight now default timeout it is 10 sec.\n\n```\ntype ProxyConfig struct {\n    // Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example\n    // `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128`\n    // is considered an HTTP proxy.\n    Server string `json:\"server\" yaml:\"server\"`\n    // Optional username to use if HTTP proxy requires authentication.\n    Username string `json:\"username\" yaml:\"username\"`\n    // Optional password to use if HTTP proxy requires authentication.\n    Password string `json:\"password\" yaml:\"password\"`\n}\n```\n\n- Server - address with schema of proxy server. Also support\n[formatting](#placeholder-list) - Username - username for proxy(can be empty). Also support\n[formatting](#placeholder-list) - Password - password for proxy(can be empty). Also support\n[formatting](#placeholder-list)\n\n```\n{\n  \"server\": \"http://localhost:8080\",\n  \"username\": \"pyx\"\n}\n```\n\n**FITTER_HTTP_WORKER**- int[1000] - default concurrent HTTP workers\n\nConnector type which emulate fetching of data via browser\n\n```\ntype BrowserConnectorConfig struct {\n\tChromium   *ChromiumConfig   `json:\"chromium\" yaml:\"chromium\"`\n\tDocker     *DockerConfig     `json:\"docker\" yaml:\"docker\"`\n\tPlaywright *PlaywrightConfig `json:\"playwright\" yaml:\"playwright\"`\n}\n```\n\nConfig can be one of:\n\n[Chromium](#chromium)- use local installed Chromium for fetch data[Docker](#docker)- use docker as service for spin up container for fetch data[Playwright](#playwright)- use playwright framework for fetch data\n\nExample:\n\n```\n{\n    \"docker\": {\n      \"wait\": 10000,\n      \"image\": \"docker.io/zenika/alpine-chrome:with-node\",\n      \"entry_point\": \"chromium-browser\",\n      \"purge\": true\n    }\n}\n```\n\nUse locally installed Chromium for fetch the data\n\n```\ntype ChromiumConfig struct {\n\tPath    string   `yaml:\"path\" json:\"path\"`\n\tTimeout uint32   `yaml:\"timeout\" json:\"timeout\"`\n\tWait    uint32   `yaml:\"wait\" json:\"wait\"`\n\tFlags   []string `yaml:\"flags\" json:\"flags\"`\n}\n```\n\n- Path - path to binary of Chromium\n- Timeout[sec] - timeout for execution of the chromium\n- Wait[msec] - timeout of page loading\n- Flags - flags for Chromium default: \"--headless\", \"--proxy-auto-detect\", \"--temp-profile\", \"--incognito\", \"--disable-logging\", \"--disable-extensions\", \"--no-sandbox\"\n\nExample:\n\n```\n{\n  \"path\": \"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome\",\n  \"wait\": 10000\n}\n```\n\nUse Docker for spin up container for fetch data\n\n```\ntype DockerConfig struct {\n\tImage       string   `yaml:\"image\" json:\"image\"`\n\tEntryPoint  string   `json:\"entry_point\" yaml:\"entry_point\"`\n\tTimeout     uint32   `yaml:\"timeout\" json:\"timeout\"`\n\tWait        uint32   `yaml:\"wait\" json:\"wait\"`\n\tFlags       []string `yaml:\"flags\" json:\"flags\"`\n\tPurge       bool     `json:\"purge\" yaml:\"purge\"`\n\tNoPull      bool     `yaml:\"no_pull\" json:\"no_pull\"`\n\tPullTimeout uint32   `yaml:\"pull_timeout\" json:\"pull_timeout\"`\n}\n```\n\n**Docker default image**: docker.io/zenika/alpine-chrome\n\n- Image - image for the docker registry(provide with registry host)\n- EntryPoint - cmd which will be run inside container\n- Timeout[sec] - timeout for run container(without pulling image)\n- Wait[msec] - timeout of page loading (works just for Chromium based containers)\n- Flags - cmd arguments for run containers, default for Chromium based: \"--no-sandbox\",\"--headless\", \"--proxy-auto-detect\", \"--temp-profile\", \"--incognito\", \"--disable-logging\", \"--disable-gpu\"\n- Purge - should we remove container after work done(like docker rm)\n- NoPull - prevent pulling of the image\n- PullTimeout - define timeout for pull contains\n\n**DOCKER_HOST**- string - (EnvOverrideHost) to set the URL to the docker server.** DOCKER_API_VERSION**- string - (EnvOverrideAPIVersion) to set the version of the API to use, leave empty for latest.** DOCKER_CERT_PATH**- string - (EnvOverrideCertPath) to specify the directory from which to load the TLS certificates (ca.pem, cert.pem, key.pem).**DOCKER_TLS_VERIFY**- bool - (EnvTLSVerify) to enable or disable TLS verification (off by default)\n\nExample:\n\n```\n{\n  \"wait\": 10000,\n  \"image\": \"docker.io/zenika/alpine-chrome:with-node\",\n  \"entry_point\": \"chromium-browser\",\n  \"purge\": true\n}\n```\n\nRun browsers via playwright framework\n\n```\ntype PlaywrightConfig struct {\n    Browser      PlaywrightBrowser          `json:\"browser\" yaml:\"browser\"`\n    Install      bool                       `yaml:\"install\" json:\"install\"`\n    Timeout      uint32                     `yaml:\"timeout\" json:\"timeout\"`\n    Wait         uint32                     `yaml:\"wait\" json:\"wait\"`\n    TypeOfWait   *playwright.WaitUntilState `json:\"type_of_wait\" yaml:\"type_of_wait\"`\n    PreRunScript string                     `json:\"pre_run_script\" yaml:\"pre_run_script\"`\n    Stealth      bool                       `json:\"stealth\" yaml:\"stealth\"`\n    \n    Proxy *ProxyConfig `yaml:\"proxy\" json:\"proxy\"`\n}\n```\n\n- Browser - enum[\"Chromium\", \"FireFox\", \"WebKit\"] - which browser to use\n- Install - should we install browser\n- Timeout[sec] - timeout to run playwright\n- Wait[sec] - timeout of page loading\n- TypeOfWait - enum[\"load\", \"domcontentloaded\", \"networkidle\", \"commit\"] which state of page we waiting, default is \"load\"\n- PreRunScript[\"\"] - script which will be executed before reading content of the page. Also support placeholder\n[{PL}](#placeholder-list) - Stealth[false] - add script for trying passing bot defends\n- Proxy - setup proxy for request\n[config](#proxy-config)\n\nExample\n\n```\n{\n  \"timeout\": 30,\n  \"wait\": 30,\n  \"install\": false,\n  \"browser\": \"Chromium\"\n}\n```\n\nWith model we define result of the scrapping\n\n```\ntype Model struct {\n    ObjectConfig *ObjectConfig `yaml:\"object_config\" json:\"object_config\"`\n    ArrayConfig  *ArrayConfig  `json:\"array_config\" yaml:\"array_config\"`\n    BaseField    *BaseField    `json:\"base_field\" yaml:\"base_field\"`\n    IsArray      bool          `json:\"is_array\" yaml:\"is_array\"`\n}\n```\n\nConfig can be one of:\n\n[ObjectConfig](#objectconfig)- configuration of object format[ArrayConfig](#arrayconfig)- configuration of array format[BaseField](#basefield)- configuration of single/generated field- IsArray - bool[false] - force indicate that field is array(usable in case of\n[model field](#model-field)with[base field](#basefield))\n\nExample:\n\n```\n{\n  \"object_config\": {}\n}\n```\n\nConfiguration of the object and fields\n\n```\ntype ObjectConfig struct {\n    Fields      map[string]*Field `json:\"fields\" yaml:\"fields\"`\n    Field       *BaseField        `json:\"field\" yaml:\"field\"`\n    ArrayConfig *ArrayConfig      `json:\"array_config\" yaml:\"array_config\"`\n}\n```\n\nConfig can be one of:\n\n[Fields](#field)- map of each field definition; key - field name, value - configuration[Field](#basefield)- used for element of array; fields which will be deserialized like basic type like \"string\", \"int\" and etc (used here for case array of basic types)[ArrayConfig](#arrayconfig)- used for element of array; deserialization array of array\n\nExample:\n\n```\n{\n  \"fields\": {\n    \"title\": {\n      \"base_field\": {\n        \"type\": \"string\",\n        \"path\": \"type\"\n      }\n    }\n  }\n}\n```\n\nConfiguration of the array and fields\n\n```\ntype ArrayConfig struct {\n    RootPath    string        `json:\"root_path\" yaml:\"root_path\"`\n    Reverse     bool          `yaml:\"reverse\" json:\"reverse\"`\n    \n    ItemConfig  *ObjectConfig `json:\"item_config\" yaml:\"item_config\"`\n    LengthLimit uint32        `json:\"length_limit\" yaml:\"length_limit\"`\n    \n    StaticConfig *StaticArrayConfig `json:\"static_array\"  yaml:\"static_array\"`\n}\n```\n\n- RootPath - selector for find root element of the array or repeated element in case of html parsing, size of array will be amount of children element under the root\n- Reverse - bool[false] - indicate that need use reverse iteration(n to 1)\n- LengthLimit - for define size of array only for generated(not working for static)\n\nConfig can be one of:\n\n[ItemConfig](#objectconfig)- configuration of each element of the array[StaticConfig](#static-array-config)- configuration of the static array\n\nExample:\n\n```\n{\n  \"root_path\": \"#content dt.quote > a\",\n  \"item_config\": {\n    \"field\": {\n      \"type\": \"string\"\n    }\n  }\n}\n```\n\nCommon of the field\n\n```\ntype Field struct {\n\tBaseField    *BaseField    `json:\"base_field\" yaml:\"base_field\"`\n\tObjectConfig *ObjectConfig `json:\"object_config\" yaml:\"object_config\"`\n\tArrayConfig  *ArrayConfig  `json:\"array_config\" yaml:\"array_config\"`\n\n\tFirstOf []*Field `json:\"first_of\" yaml:\"first_of\"`\n}\n```\n\nConfig can be one of:\n\n[BaseField](#basefield)- fields which will be deserialized like basic type like \"string\", \"int\" and etc[ObjectConfig](#objectconfig)- in case our field in nested object[ArrayConfig](#arrayconfig)- in case our field in array[FirstOf](#field)- first not empty resolved field will be selected\n\nExample:\n\n```\n{\n  \"base_field\": {\n    \"type\": \"string\",\n    \"path\": \"div.current-temp span.heading\"\n  }\n}\n```\n\nIn case we want get some static information or generate new one\n\n```\ntype BaseField struct {\n\tType FieldType `yaml:\"type\" json:\"type\"`\n\tPath string    `yaml:\"path\" json:\"path\"`\n\n\tHTMLAttribute string `json:\"html_attribute\" yaml:\"html_attribute\"`\n\n\tGenerated *GeneratedFieldConfig `yaml:\"generated\" json:\"generated\"`\n\n\tFirstOf []*BaseField `json:\"first_of\" yaml:\"first_of\"`\n}\n```\n\n- FieldType - enum[\"null\", \"boolean\", \"string\", \"int\", \"int64\", \"float\", \"float64\", \"array\", \"object\", \"html\", \"raw_string\"] - static field for parse.\n**Important**: type html will only works from connector which return HTML (HTMLAttribute - have no effect in this case).[Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L25) - Path - selector(relative in case it is array child) for parsing\n- HTMLAttribute - extra value which have effect only in HTML parsing via\n**goquery**. Here you can specify which attribute need to be parsed.\n\n**Important**: by default \"string\" type trimmed and all special chars is replaced, if you need plain string use \"raw_string\"\n\nConfig can be one of or empty:\n\n[Generated](#generatedfieldconfig)- field can be generated one which custom configuration[FirstOf](#basefield)- first not empty resolved field will be selected\n\nExamples\n\n```\n{\n  \"generated\": {\n    \"uuid\": {}\n  }\n}\n{\n  \"type\": \"string\",\n  \"path\": \"text()\"\n}\n```\n\nProvide functionality of generating field on the flight\n\n```\ntype GeneratedFieldConfig struct {\n    UUID             *UUIDGeneratedFieldConfig   `yaml:\"uuid\" json:\"uuid\"`\n    Static           *StaticGeneratedFieldConfig `yaml:\"static\" json:\"static\"`\n    Formatted        *FormattedFieldConfig       `json:\"formatted\" yaml:\"formatted\"`\n    Plugin           *PluginFieldConfig          `yaml:\"plugin\" json:\"plugin\"`\n    Calculated       *CalculatedConfig           `yaml:\"calculated\" json:\"calculated\"`\n    File             *FileFieldConfig            `yaml:\"file\" json:\"file\"`\n    Model            *ModelField                 `yaml:\"model\" json:\"model\"`\n    FileStorageField *FileStorageField           `json:\"file_storage\" yaml:\"file_storage\"`\n}\n```\n\nConfig can be one of:\n\n[UUID](#uuid)- generate random UUID V4[Static](#static)- generate static field[Formatted](#formatted-field-config)- format field[Model](#model-field)- model generated from the other connector and model[Plugin](#plugin-field)- plugin field[Calculated](#calculated-field)- calculated field[File](#file-field)- file field (for download file from server)[FileStorage](#file-storage-field)- file field which can be saved to local file\n\nExamples:\n\n```\n{\n    \"uuid\": {}\n}\n```\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L58](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L58)\n\n```\n{\n    \"model\": {\n      \"type\": \"array\",\n      \"model\": {\n        \"array_config\": {\n          \"root_path\": \"#content dt.quote > a\",\n          \"item_config\": {\n            \"field\": {\n              \"type\": \"string\"\n            }\n          }\n        }\n      },\n      \"connector_config\": {\n        \"response_type\": \"HTML\",\n        \"attempts\": 3,\n        \"browser_config\": {\n          \"url\": \"http://www.quotationspage.com/random.php\",\n          \"chromium\": {\n            \"path\": \"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome\",\n            \"wait\": 10000\n          }\n        }\n      }\n    }\n}\n```\n\nGenerate random UUID V4 on the flight, can be used for generate uniq id\n\n```\ntype UUIDGeneratedFieldConfig struct {\n\tRegexp string `yaml:\"regexp\" json:\"regexp\"`\n}\n```\n\n- Regexp - provide matcher which can be used for get part of generated uuid\n\nGenerate static field\n\n```\ntype StaticGeneratedFieldConfig struct {\n    Type  FieldType       `yaml:\"type\" json:\"type\"`\n    Value string          `json:\"value\" yaml:\"value\"`\n    Raw   json.RawMessage `json:\"raw\" yaml:\"raw\"`\n}\n```\n\n- Type - enum[\"null\", \"boolean\", \"string\", \"int\",\"int64\",\"float\",\"float64\", \"array\", \"object\"] - type of the field\n- Value - string value of the field\n- Raw - pure json value of the field\n\nExample\n\n```\n{\n  \"type\": \"int\",\n  \"value\": \"65\"\n}\n{\n  \"type\": \"array\",\n  \"value\": \"[65,45]\"\n}\n{\n  \"type\": \"array\",\n  \"raw\": [65,45]\n}\n```\n\nGenerate formatted field which will pass value from parent [base field](#basefield)\n\n```\ntype FormattedFieldConfig struct {\n\tTemplate string `yaml:\"template\" json:\"template\"`\n}\n```\n\nExample:\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L98](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L98)\n\n```\n{\n  \"template\": \"https://news.ycombinator.com/item?id={PL}\"\n}\n```\n\nField can be used for store field result as local file\n\n```\ntype FileStorageField struct {\n    Content string          `json:\"content\" yaml:\"content\"`\n    Raw     json.RawMessage `yaml:\"raw\" yaml:\"raw\"`\n    \n    FileName string `json:\"file_name\" yaml:\"file_name\"`\n    Path     string `json:\"path\" yaml:\"path\"`\n    Append   bool   `json:\"append\" yaml:\"append\"`\n}\n```\n\n- Content - template string for content. Important: can be with\n[inject of the parent value as a string](#placeholder-list) - Raw - raw json content of field. Important: can be with\n[inject of the parent value as a string](#placeholder-list) - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with\n[inject of the parent value as a string](#placeholder-list). - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with\n[inject of the parent value as a string](#placeholder-list) - Append[false] - append to file or not\n\n```\n{\n  \"content\": \"{{{id}}}, {{{message}}}\\n\",\n  \"append\": true,\n  \"file_name\": \"{{{id}}}.csv\",\n  \"path\": \"/Users/pxyup/fitter/examples/cli/test/csv\"\n}\n```\n\nField can be used for download file from server locally\n\n```\ntype FileFieldConfig struct {\n\tConfig *ServerConnectorConfig `yaml:\"config\" json:\"config\"`\n\n\tUrl      string `yaml:\"url\" json:\"url\"`\n\tFileName string `json:\"file_name\" yaml:\"file_name\"`\n\tPath     string `json:\"path\" yaml:\"path\"`\n}\n```\n\n- Config -\n[ServerConfig](#serverconnectorconfig)use default fitter http.Client for send request - Url - url of the image. Important: URL in the connector can be with\n[inject of the parent value as a string](#placeholder-list) - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with\n[inject of the parent value as a string](#placeholder-list). - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with\n[inject of the parent value as a string](#placeholder-list)\n\nResult of the field will be local file path as string\n\n```\n{\n  \"url\": \"https://images.shcdn.de/resized/w680/p/dekostoff-gobelinstoff-panel-oriental-cat-46-x-46_P19-KP_2.jpg\",\n  \"path\": \"/Users/pxyup/fitter/bin\",\n  \"config\": {\n    \"method\": \"GET\"\n  }\n}\n```\n\nWith propagated URL ([inject of the parent value as a string](#placeholder-list))\n\n```\n{\n  \"url\": \"https://picsum.photos{PL}\",\n  \"path\": \"/Users/pxyup/fitter/bin\",\n  \"config\": {\n    \"method\": \"GET\"\n  }\n}\n```\n\nConfig example:\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image.json](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image.json)\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image_multiple.json](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image_multiple.json)\n\nField can generate different types depends from expression\n\n```\ntype CalculatedConfig struct {\n\tType       FieldType `yaml:\"type\" json:\"type\"`\n\tExpression string    `yaml:\"expression\" json:\"expression\"`\n}\n```\n\n- Type - resulting type of expression\\\n- Expression - expression for calculation (we use\n[this lib](https://github.com/expr-lang/expr)for calculated expression)\n\n**FNull** - alias for builder.Nullvalue\n\n**FNil** - alias for nil\n\n**isNull(value T)** - function for check is value is FNull\n\n**fRes** - it is raw(with proper type) result from the parsing [base field](#basefield)\n\n**fIndex** - it is index in parent array(only if parent was array field)\n\n**fResJson** - it is JSON string representation of the raw result\n\n**fResRaw** - result in bytes format\n\n**FNewLine** - new line separator\n\n```\n{\n  \"type\": \"bool\",\n  \"expression\": \"fRes > 500\"\n}\n```\n\nField can be some external plugin for fitter\n\n```\ntype PluginFieldConfig struct {\n\tName string `json:\"name\" yaml:\"name\"`\n\tConfig json.RawMessage `json:\"config\" yaml:\"config\"`\n}\n```\n\n- Name - name of the plugin(without extension just name)\n- Config - json config of the plugin\n\nField type which can be generated on the flight by news [model](#model) and [connector](#connector)\n\n```\ntype ModelField struct {\n\t// Type of parsing\n\tConnectorConfig *ConnectorConfig `yaml:\"connector_config\" json:\"connector_config\"`\n\t// Model of the response\n\tModel *Model `yaml:\"model\" json:\"model\"`\n\n\tType FieldType `yaml:\"type\" json:\"type\"`\n\tPath string             `yaml:\"path\" json:\"path\"`\n\n\tExpression string    `yaml:\"expression\" json:\"expression\"`\n}\n```\n\n[ConnectorConfig](#connector)- which connector to use. Important: URL in the connector can be with[inject of the parent value as a string](#placeholder-list)[Model](#model)- configuration of the underhood model- Type - enum[\"null\", \"boolean\", \"string\", \"int\", \"int64\", \"float\", \"float64\", \"array\", \"object\"] - type of generated field\n- Path - in case we cant extract some information from generated field we can use json selector for extract\n- Expression - string which can be used for post processing of the Model (\n**ignoring path field**)\n\nExamples:\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L60](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L60)\n\n```\n{\n  \"type\": \"array\",\n  \"model\": {\n    \"array_config\": {\n      \"root_path\": \"#content dt.quote > a\",\n      \"item_config\": {\n        \"field\": {\n          \"type\": \"string\"\n        }\n      }\n    }\n  }\n}\n```\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_weather.json#L37](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_weather.json#L37)\n\n```\n{\n    \"type\": \"string\",\n    \"path\": \"temp.temp\",\n    \"model\": {\n       \"object_config\": {\n        \"fields\": {\n          \"temp\": {\n            \"base_field\": {\n              \"type\": \"string\",\n              \"path\": \"//div[@id='forecast_list_ul']//td/b/a/@href\",\n              \"generated\": {\n                \"model\": {\n                  \"type\": \"string\",\n                  \"model\": {\n                    \"object_config\": {\n                      \"fields\": {\n                        \"temp\": {\n                          \"base_field\": {\n                            \"type\": \"string\",\n                            \"path\": \"div.current-temp span.heading\"\n                          }\n                        }\n                      }\n                    }\n                  },\n                  \"connector_config\": {\n                    \"response_type\": \"HTML\",\n                    \"attempts\": 4,\n                    \"url\": \"https://openweathermap.org{PL}\",\n                    \"browser_config\": {\n                      \"playwright\": {\n                        \"timeout\": 30,\n                        \"wait\": 30,\n                        \"install\": false,\n                        \"browser\": \"FireFox\",\n                        \"type_of_wait\": \"networkidle\"\n                      }\n                    }\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n    },\n    \"connector_config\": {\n      \"response_type\": \"xpath\",\n      \"attempts\": 3,\n      \"url\": \"https://openweathermap.org/find?q={PL}\",\n      \"browser_config\": {\n        \"playwright\": {\n          \"timeout\": 30,\n          \"wait\": 30,\n          \"install\": false,\n          \"browser\": \"Chromium\"\n        }\n      }\n    }\n}\n```\n\nProvide static(fixed length) array generation\n\n```\ntype StaticArrayConfig struct {\n    Items map[uint32]*Field `yaml:\"items\" json:\"items\"`\n    Length uint32            `yaml:\"length\" json:\"length\"`\n}\n```\n\n[Items](#field)- map[uint32]*[Field](#field)- key is index in array, value is field definition- Length - if set(1+) can be used for define custom length of array\n\nExamples:\n\n```\n{\n  \"0\": {\n    \"base_field\": {\n      \"type\": \"string\",\n      \"path\": \"div.current-temp span.heading\"\n    }\n  }\n}\n{\n  \"length\": 4,\n  \"0\": {\n    \"base_field\": {\n      \"type\": \"string\",\n      \"path\": \"div.current-temp span.heading\"\n    }\n  }\n}\n{\n  \"length\": 4,\n  \"2\": {\n    \"base_field\": {\n      \"type\": \"string\",\n      \"path\": \"div.current-temp span.heading\"\n    }\n  }\n}\n```\n\n- {PL} - for inject value\n- {INDEX} - for inject index in parent array\n- {HUMAN_INDEX} - for inject index in parent array in human way\n- {{{json_path}}} - will get information from propagated \"object\"/\"array\" field\n- {{{RefName=SomeName}}} - get\n[reference](#references)value by name.[Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L67) - {{{RefName=SomeName json.path}}} - get\n[reference](#references)value by name and extract value by json path.[Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L67) - {{{FromEnv=ENV_KEY}}} - get value from environment variable\n- {{{FromExp=fRes + 5 + fIndex}}} - get value from the\n[expression](https://github.com/expr-lang/expr).[Predefined values](#predefined-values) - {{{FromInput=.}}} or {{{FromInput=json.path}}} - get value from input of trigger or library\n- {{{FromFile=./test_file.log}}} - get value from file by path. Content of file also can contain placeholders\n- {{{FromURL=\n[http://localhost:8081}}}](http://localhost:8081%7D%7D%7D)- get response from url\n\nExamples:\n\n```\n{{{FromExp=\"{{{FromEnv=TEST_VAL}}}\" + \"hello\"}}}\nCurrent time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}}\nCurrent time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}}\nTokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}} Object={{{value}}} {PL} Env={{{FromEnv=TEST_VAL}}} {INDEX} {HUMAN_INDEX}\n```\n\nSpecial map which **prefetched**(before any processing) and can be user for [connector](#referenceconnectorconfig) or for [placeholder](#placeholder-list)\n\nCan be used for:\n\n- Cache jwt token and use them in headers\n- Cache values\n- Etc\n\n```\ntype Reference struct {\n    *ModelField\n    \n    Expire uint32 `yaml:\"expire\" json:\"expire\"`\n}\n```\n\n[ModelField](#model-field)- is embedded struct, you can use same fields- Expire[sec] - duration when reference is expired after fetching. Not set =>\n**forever cached**. Set to 0 =>** every time re-fetch**. Set to n > 0 =>** cached for n second**\n\nFor [Fitter](#how-to-use-fitter)\n\n```\ntype RefMap map[string]*Reference\n\ntype Config struct {\n    // Other Config Fields\n\n    Limits     *Limits `yaml:\"limits\" json:\"limits\"`\n    References RefMap  `json:\"references\" yaml:\"references\"`\n}\n```\n\nFor [Fitter Cli](#how-to-use-fittercli)\n\n```\ntype RefMap map[string]*Reference\n\ntype CliItem struct {\n    // Other Config Fields\n\n    Limits     *Limits `yaml:\"limits\" json:\"limits\"`\n    References RefMap  `json:\"references\" yaml:\"references\"`\n}\n```\n\n- References - map[string]*Reference - object where is key if ReferenceName (can be user for\n[connector](#referenceconnectorconfig)or[placeholder](#placeholder-list)) and value is[Reference](#reference) [Limits](#limits)\n\nExample\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L2](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L2)\n\n```\n{\n  \"references\": {\n    \"TokenRef\": {\n      \"expire\": 10,\n      \"connector_config\": {\n        \"response_type\": \"json\",\n        \"static_config\": {\n          \"value\": \"\\\"plain token\\\"\"\n        }\n      },\n      \"model\": {\n        \"base_field\": {\n          \"type\": \"string\"\n        }\n      }\n    },\n    \"TokenObjectRef\": {\n      \"connector_config\": {\n        \"response_type\": \"json\",\n        \"static_config\": {\n          \"value\": \"{\\\"token\\\":\\\"token from object\\\"}\"\n        }\n      },\n      \"model\": {\n        \"object_config\": {\n          \"fields\": {\n            \"token\": {\n              \"base_field\": {\n                \"type\": \"string\",\n                \"path\": \"token\"\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n}\n```\n\nProvide limitation for prevent DDOS, big usage of memory\n\n```\ntype Limits struct {\n\tHostRequestLimiter HostRequestLimiter `yaml:\"host_request_limiter\" json:\"host_request_limiter\"`\n\tChromiumInstance   uint32             `yaml:\"chromium_instance\" json:\"chromium_instance\"`\n\tDockerContainers   uint32             `yaml:\"docker_containers\" json:\"docker_containers\"`\n\tPlaywrightInstance uint32             `yaml:\"playwright_instance\" json:\"playwright_instance\"`\n}\n```\n\n- HostRequestLimiter - map[string]int64 - limitation per host name, key is host, value is amount of parallel request(usage for\n[server connector](#serverconnectorconfig)) - ChromiumInstance - amount of parallel\n[chromium](#chromium)instance - DockerContainers - amount of parallel\n[docker](#docker)instance - PlaywrightInstance - amount of parallel\n[playwright](#playwright)instance\n\n[https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L2](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L2)\n\n```\n{\n  \"limits\": {\n    \"host_request_limiter\": {\n      \"hacker-news.firebaseio.com\": 5\n    },\n    \"chromium_instance\": 3,\n    \"docker_containers\": 3,\n    \"playwright_instance\": 3\n  }\n}\n```\n\n- Add browser scenario for preparing, after parsing\n- Add scrolling support for scenario\n- Add pagination support for scenario\n- Add notification methods for Fitter: Webhook/Queue", "url": "https://wpnews.pro/news/show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code", "canonical_source": "https://github.com/PxyUp/fitter", "published_at": "2026-07-23 01:03:01+00:00", "updated_at": "2026-07-23 01:22:35.483689+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-agents"], "entities": ["Fitter", "Claude Code", "MCP", "Playwright", "Chromium", "Docker", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code", "markdown": "https://wpnews.pro/news/show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code.md", "text": "https://wpnews.pro/news/show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-built-a-scraping-engine-where-the-llm-writes-configs-not-code.jsonld"}}