{"slug": "show-hn-sjq-jq-but-for-ai-streaming", "title": "Show HN: Sjq – Jq, but for AI Streaming", "summary": "A developer released sjq, a command-line tool that parses streaming LLM responses the way jq parses whole JSON documents, extracting only the tokens each Server-Sent Events chunk adds. The tool's author measured that envelope data accounts for 99.1% of a 3,590-byte short chat-completion stream (31 bytes of generated text), 98.6% of a 10,446-byte reasoning-plus-answer stream (142 bytes), and 98.8% of a 24,726-byte Responses API stream (300 bytes), and sjq also reports timing such as connect, ttfb, ttft, first_content, total and tokens-per-second. sjq is positioned as a drop-in replacement for jq, which fails on streaming responses with errors like 'Invalid numeric literal at line 1, column 5' for /v1/chat/completions chunks.", "body_md": "**jq, but for AI streaming.**\n\nPipe an LLM stream in. Get the tokens out, one per line, and a stopwatch.\n\n[Why](#why) · [jq parse error?](#jq-parse-error-invalid-numeric-literal) · [Install](#install) · [Usage](#usage) · [Reading the output](#reading-the-output) · [Responses API](#responses-api) · [Timing](#timing)\n\nA streaming LLM response is mostly envelope. Every `data:` line repeats the id, the model,\nthe object type and a row of `null` s, just to deliver a few characters of new text:\n\n| stream | on the wire | generated text | envelope | \n|---|---|---|---|\n| chat completion, short answer | 3,590 B | 31 B | **99.1%** | \n| chat completion, reasoning + answer | 10,446 B | 142 B | **98.6%** | \n| Responses API, reasoning + answer | 24,726 B | 300 B | **98.8%** | \n\n`jq` wants whole documents, and `grep` can't tell a reasoning token from an answer token. sjq\nreads the stream as it arrives and keeps only what each event *adds*:\n\n```\n- data: {\"id\":\"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d\",\"created\":1792034296,\"model\":\"viv-fast-2609\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":null,\"obfuscation\":null,\"choices\":[{\"delta\":{\"role\":\"assistant\",\"reasoning_content\":\"The\",\"refusal\":null},\"index\":0,\"logprobs\":null}]}\n+ meta: id=021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d created=1792034296 model=viv-fast-2609\n+ data: role=assistant, reasoning_content='The'\n- data: {\"id\":\"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d\",\"created\":1792034296,\"model\":\"viv-fast-2609\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":null,\"obfuscation\":null,\"choices\":[{\"delta\":{\"reasoning_content\":\" user\",\"refusal\":null},\"index\":0,\"logprobs\":null}]}\n+ data: reasoning_content=' user'\n- data: {\"id\":\"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d\",\"created\":1792034296,\"model\":\"viv-fast-2609\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":null,\"obfuscation\":null,\"choices\":[{\"delta\":{\"content\":\"Hola\",\"refusal\":null},\"index\":0,\"logprobs\":null}]}\n+ data: 'Hola'\n- data: {\"id\":\"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d\",\"created\":1792034296,\"model\":\"viv-fast-2609\",\"object\":\"chat.completion.chunk\",\"system_fingerprint\":null,\"obfuscation\":null,\"choices\":[{\"delta\":{\"refusal\":null},\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null}],\"usage\":{\"prompt_tokens\":232,\"completion_tokens\":44,\"total_tokens\":276,\"prompt_tokens_details\":{\"audio_tokens\":0,\"cached_tokens\":0},\"completion_tokens_details\":{\"accepted_prediction_tokens\":0,\"audio_tokens\":0,\"reasoning_tokens\":24,\"rejected_prediction_tokens\":0}}}\n+ data: finish_reason=stop, usage: {\"prompt_tokens\":232,\"completion_tokens\":44,\"total_tokens\":276,\"completion_tokens_details\":{\"reasoning_tokens\":24}}\n  data: [DONE]\n+ time: connect=0.3ms ttfb=308.7ms ttft=531.8ms first_content=1.119s total=1.416s speed=49.8tok/s\n```\n\nSearched for this error? You piped a streaming response into `jq`:\n\n``` bash\n$ curl -i \"$BASE_URL/v1/chat/completions\" -d '{\"stream\":true, …}' | jq\njq: parse error: Invalid numeric literal at line 1, column 9\n```\n\nWith `\"stream\": true` the response isn't one JSON document. It's\n[Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events):\n`data: {…}` lines (plus `event: …` lines on `/v1/responses`), separated by blank lines and\nended by `data: [DONE]`. jq gives up at the first word that isn't JSON, and the column\ntells you which word it was:\n\n| the input starts with | jq says | \n|---|---|\n| `HTTP/1.1 200 OK` , the status line from`curl -i` | `jq: parse error: Invalid numeric literal at line 1, column 9` | \n| `data: {\"id\":…` , a`/v1/chat/completions` chunk | `jq: parse error: Invalid numeric literal at line 1, column 5` | \n| `event: response.created` , a`/v1/responses` event | `jq: parse error: Invalid numeric literal at line 1, column 6` | \n\nThe fix is one word: put `sjq` where `jq` was. Headers from `-i` pass through untouched.\n\n```\ncurl -iN \"$BASE_URL/v1/chat/completions\" … | sjq\n```\n\nRather stay in jq? Read the lines raw, strip `data:`  and skip whatever isn't JSON:\n\n```\ncurl -sN \"$BASE_URL/v1/chat/completions\" … | jq -cR 'ltrimstr(\"data: \") | fromjson? // empty'\n```\n\nThat gets you valid JSON again, but still a few hundred bytes of envelope per token.\n\n```\nbrew tap vivgrid/tap && brew trust vivgrid/tap  # Homebrew 6+ asks you to trust third-party taps; once is enough\nbrew install sjq\n```\n\nOr build from source:\n\n```\ncargo install --git https://github.com/vivgrid/sjq\n```\n\nOne binary, one dependency (`serde_json`). Needs a recent stable Rust (edition 2024).\n\n| run | what you get | \n|---|---|\n| `sjq curl …` | sjq runs curl for you, so the `time:` line also gets connect, TLS and TTFB | \n| `curl -sN … \\| sjq` | pipe in any stream ( `-N` stops curl from buffering it) | \n| `sjq capture.txt` | replay a saved stream, or several | \n\n```\nsjq curl -s \"$BASE_URL/v1/chat/completions\" \\\n  -H \"Authorization: Bearer $API_KEY\" -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"'\"$MODEL\"'\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\n```\n\n`-t` prefixes each line with the seconds since start, handy for spotting stalls.\n`--color auto|always|never` sets coloring, and `NO_COLOR` is honored.\n\n- **Text is quoted and escaped** , so leading spaces, empty deltas (`''` ) and newlines are\nvisible, and every chunk stays on one line.\n- **Reasoning keeps the provider's field name** :`reasoning_content` ,`reasoning` ,`reasoning_details` , whatever the server sends.\n- **`meta:` appears once per stream** , and again only when`id` ,`model` or anything else\nin the envelope changes.\n- **`usage` drops zero counters** inside`*_details` .\n- **Nothing unusual is hidden.** Unrecognized non-null fields print as`key=value` , and\nnon-`data:` lines (`event:` ,`: ping` , a plain-text error body) pass through. The only\nlines dropped are blank lines and`event:` lines that repeat the next payload's`type` .\n\n`/v1/responses` streams typed events instead of chunks. sjq drops the `response.` prefix\nfrom event names and prints answer text the same way as chat content:\n\n```\nmeta: created_at=1792038227 id=resp_0217… model=… reasoning={\"effort\":\"medium\"} service_tier=auto\ndata: created status=in_progress\ndata: output_item.added reasoning id=rs_0217… status=in_progress\ndata: reasoning_summary_text='Simple'\ndata: reasoning_summary_text.done\ndata: output_item.added message role=assistant status=in_progress id=msg_0217…\ndata: content_part.added output_text\ndata: 'Here'\ndata: output_text.done\ndata: output_item.done message status=completed\ndata: completed, usage: {\"input_tokens\":18,\"output_tokens\":87,\"output_tokens_details\":{\"reasoning_tokens\":3},\"total_tokens\":105}\n```\n\nThe Responses API repeats itself a lot. Every `.done` event echoes the full text again, at\nevery level (text, part, item, response), and every event carries a `sequence_number`.\nsjq checks instead of printing: an echo that matches the deltas is hidden, and so is a\n`sequence_number` that is one more than the last. You only see them when something is off:\n\n`item_id`, `output_index`, `content_index` and `summary_index` work the same way: they only\nshow when they change, for example when two function calls stream at once.\n\nWhen the stream ends, sjq prints one more line:\n\n| field | meaning | \n|---|---|\n| `connect`` tls``ttfb` | from curl, only with `sjq curl …` | \n| `ttft` | first generated token: reasoning, content or tool call | \n| `first_content` | first answer token, shown when reasoning or a tool call came first | \n| `total` | last line received | \n| `speed` | `completion_tokens` (or`output_tokens` )`/ (total − ttft)` | \n\nA pipe only carries the response body. With `curl … | sjq`, sjq counts from its own start,\nwhich is when the shell started curl, but it can't see the TCP connect. `sjq curl …` adds\n`-N -sS -w …` to your curl arguments (replacing any `-w` of yours) to get connection\ntimings, and exits with curl's status. Replayed files get no `time:` line.\n\n`data: [DONE]`", "url": "https://wpnews.pro/news/show-hn-sjq-jq-but-for-ai-streaming", "canonical_source": "https://github.com/vivgrid/sjq", "published_at": "2026-10-16 00:34:51+00:00", "updated_at": "2026-10-16 00:58:20.582158+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["sjq", "jq", "Responses API", "chat completions API", "Server-Sent Events"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-sjq-jq-but-for-ai-streaming", "markdown": "https://wpnews.pro/news/show-hn-sjq-jq-but-for-ai-streaming.md", "text": "https://wpnews.pro/news/show-hn-sjq-jq-but-for-ai-streaming.txt", "jsonld": "https://wpnews.pro/news/show-hn-sjq-jq-but-for-ai-streaming.jsonld"}}