Show HN: Sjq – Jq, but for AI Streaming

A developer released sjq, a command-line tool that parses streaming LLM responses the way jq parses whole JSON documents, extracting only the tokens each Server-Sent Events chunk adds. The tool's author measured that envelope data accounts for 99.1% of a 3,590-byte short chat-completion stream (31 bytes of generated text), 98.6% of a 10,446-byte reasoning-plus-answer stream (142 bytes), and 98.8% of a 24,726-byte Responses API stream (300 bytes), and sjq also reports timing such as connect, ttfb, ttft, first_content, total and tokens-per-second. sjq is positioned as a drop-in replacement for jq, which fails on streaming responses with errors like 'Invalid numeric literal at line 1, column 5' for /v1/chat/completions chunks.

jq, but for AI streaming. Pipe an LLM stream in. Get the tokens out, one per line, and a stopwatch. Why why · jq parse error? jq-parse-error-invalid-numeric-literal · Install install · Usage usage · Reading the output reading-the-output · Responses API responses-api · Timing timing A streaming LLM response is mostly envelope. Every data: line repeats the id, the model, the object type and a row of null s, just to deliver a few characters of new text: | stream | on the wire | generated text | envelope | |---|---|---|---| | chat completion, short answer | 3,590 B | 31 B | 99.1% | | chat completion, reasoning + answer | 10,446 B | 142 B | 98.6% | | Responses API, reasoning + answer | 24,726 B | 300 B | 98.8% | jq wants whole documents, and grep can't tell a reasoning token from an answer token. sjq reads the stream as it arrives and keeps only what each event adds : - data: {"id":"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d","created":1792034296,"model":"viv-fast-2609","object":"chat.completion.chunk","system fingerprint":null,"obfuscation":null,"choices": {"delta":{"role":"assistant","reasoning content":"The","refusal":null},"index":0,"logprobs":null} } + meta: id=021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d created=1792034296 model=viv-fast-2609 + data: role=assistant, reasoning content='The' - data: {"id":"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d","created":1792034296,"model":"viv-fast-2609","object":"chat.completion.chunk","system fingerprint":null,"obfuscation":null,"choices": {"delta":{"reasoning content":" user","refusal":null},"index":0,"logprobs":null} } + data: reasoning content=' user' - data: {"id":"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d","created":1792034296,"model":"viv-fast-2609","object":"chat.completion.chunk","system fingerprint":null,"obfuscation":null,"choices": {"delta":{"content":"Hola","refusal":null},"index":0,"logprobs":null} } + data: 'Hola' - data: {"id":"021792034294732770d3492782dd10cc0a489fec0a72ead7e3e0d","created":1792034296,"model":"viv-fast-2609","object":"chat.completion.chunk","system fingerprint":null,"obfuscation":null,"choices": {"delta":{"refusal":null},"finish reason":"stop","index":0,"logprobs":null} ,"usage":{"prompt tokens":232,"completion tokens":44,"total tokens":276,"prompt tokens details":{"audio tokens":0,"cached tokens":0},"completion tokens details":{"accepted prediction tokens":0,"audio tokens":0,"reasoning tokens":24,"rejected prediction tokens":0}}} + data: finish reason=stop, usage: {"prompt tokens":232,"completion tokens":44,"total tokens":276,"completion tokens details":{"reasoning tokens":24}} data: DONE + time: connect=0.3ms ttfb=308.7ms ttft=531.8ms first content=1.119s total=1.416s speed=49.8tok/s Searched for this error? You piped a streaming response into jq : bash $ curl -i "$BASE URL/v1/chat/completions" -d '{"stream":true, …}' | jq jq: parse error: Invalid numeric literal at line 1, column 9 With "stream": true the response isn't one JSON document. It's Server-Sent Events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent events/Using server-sent events : data: {…} lines plus event: … lines on /v1/responses , separated by blank lines and ended by data: DONE . jq gives up at the first word that isn't JSON, and the column tells you which word it was: | the input starts with | jq says | |---|---| | HTTP/1.1 200 OK , the status line from curl -i | jq: parse error: Invalid numeric literal at line 1, column 9 | | data: {"id":… , a /v1/chat/completions chunk | jq: parse error: Invalid numeric literal at line 1, column 5 | | event: response.created , a /v1/responses event | jq: parse error: Invalid numeric literal at line 1, column 6 | The fix is one word: put sjq where jq was. Headers from -i pass through untouched. curl -iN "$BASE URL/v1/chat/completions" … | sjq Rather stay in jq? Read the lines raw, strip data: and skip whatever isn't JSON: curl -sN "$BASE URL/v1/chat/completions" … | jq -cR 'ltrimstr "data: " | fromjson? // empty' That gets you valid JSON again, but still a few hundred bytes of envelope per token. brew tap vivgrid/tap && brew trust vivgrid/tap Homebrew 6+ asks you to trust third-party taps; once is enough brew install sjq Or build from source: cargo install --git https://github.com/vivgrid/sjq One binary, one dependency serde json . Needs a recent stable Rust edition 2024 . | run | what you get | |---|---| | sjq curl … | sjq runs curl for you, so the time: line also gets connect, TLS and TTFB | | curl -sN … \| sjq | pipe in any stream -N stops curl from buffering it | | sjq capture.txt | replay a saved stream, or several | sjq curl -s "$BASE URL/v1/chat/completions" \ -H "Authorization: Bearer $API KEY" -H 'Content-Type: application/json' \ -d '{"model":"'"$MODEL"'","stream":true,"messages": {"role":"user","content":"hi"} }' -t prefixes each line with the seconds since start, handy for spotting stalls. --color auto|always|never sets coloring, and NO COLOR is honored. - Text is quoted and escaped , so leading spaces, empty deltas '' and newlines are visible, and every chunk stays on one line. - Reasoning keeps the provider's field name : reasoning content , reasoning , reasoning details , whatever the server sends. - meta: appears once per stream , and again only when id , model or anything else in the envelope changes. - usage drops zero counters inside details . - Nothing unusual is hidden. Unrecognized non-null fields print as key=value , and non- data: lines event: , : ping , a plain-text error body pass through. The only lines dropped are blank lines and event: lines that repeat the next payload's type . /v1/responses streams typed events instead of chunks. sjq drops the response. prefix from event names and prints answer text the same way as chat content: meta: created at=1792038227 id=resp 0217… model=… reasoning={"effort":"medium"} service tier=auto data: created status=in progress data: output item.added reasoning id=rs 0217… status=in progress data: reasoning summary text='Simple' data: reasoning summary text.done data: output item.added message role=assistant status=in progress id=msg 0217… data: content part.added output text data: 'Here' data: output text.done data: output item.done message status=completed data: completed, usage: {"input tokens":18,"output tokens":87,"output tokens details":{"reasoning tokens":3},"total tokens":105} The Responses API repeats itself a lot. Every .done event echoes the full text again, at every level text, part, item, response , and every event carries a sequence number . sjq checks instead of printing: an echo that matches the deltas is hidden, and so is a sequence number that is one more than the last. You only see them when something is off: item id , output index , content index and summary index work the same way: they only show when they change, for example when two function calls stream at once. When the stream ends, sjq prints one more line: | field | meaning | |---|---| | connect tls ttfb | from curl, only with sjq curl … | | ttft | first generated token: reasoning, content or tool call | | first content | first answer token, shown when reasoning or a tool call came first | | total | last line received | | speed | completion tokens or output tokens / total − ttft | A pipe only carries the response body. With curl … | sjq , sjq counts from its own start, which is when the shell started curl, but it can't see the TCP connect. sjq curl … adds -N -sS -w … to your curl arguments replacing any -w of yours to get connection timings, and exits with curl's status. Replayed files get no time: line. data: DONE