{"slug": "i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq", "title": "I kept hitting jq: parse error on AI streams, so I built sjq", "summary": "A developer built sjq, an MIT-licensed Rust CLI that parses OpenAI-compatible server-sent event streams from /v1/chat/completions and /v1/responses, printing one short line per event showing only the delta content instead of the full JSON envelope. The tool renders tool calls, reasoning_content, refusals, logprobs, usage blocks and multiple choices, and adds timing output including TTFT, total time, tokens/sec, plus connect, TLS handshake and TTFB when run through sjq curl. It ships 14 test fixtures covering multi-line data payloads, mid-stream errors, parallel tool calls, logprobs and CJK/emoji text.", "body_md": "If you work with LLMs, you've done this: `curl -N <streaming endpoint> | jq`. And you've seen this:\n\n```\njq: parse error: Invalid numeric literal at line 1, column 9\n```\n\nColumn 9. That's where `data:` ends and the JSON begins. An SSE stream isn't JSON — it's a framing protocol: `data: {...}` lines separated by blank lines, `[DONE]` at the end. jq never stood a chance.\n\nSo I built **sjq** — jq, but for AI streaming. A small Rust CLI that parses OpenAI-compatible server-sent event streams (`/v1/chat/completions` and `/v1/responses`) and prints one short line per event, showing only what each event *added*.\n\n```\nsjq curl -s $BASE/v1/chat/completions \\\n  -H \"Authorization: Bearer $KEY\" -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"gemini-4-argon\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}'\nmodel: gemini-4-argon\nrole: assistant\ndata: 'Hi'\ndata: ' there'\ndata: ' 👋'\n…\ntime: ttft=0.42s total=3.1s tok/s=258.2\n```\n\nInstead of 99% envelope JSON, you watch the actual content arrive — plus a timing line: TTFT, total, tokens/sec. Run through `sjq curl` and it also breaks out connect, TLS handshake, and TTFB, which is genuinely useful when you're debugging a slow provider.\n\nIt understands tool calls (name + arguments as they stream in), `reasoning_content`, refusals, logprobs, usage blocks, and multiple choices — each rendered as its own labeled line.\n\n```\nbrew tap vivgrid/tap\nbrew trust vivgrid/tap  # Homebrew 6+ requires trusting third-party taps\nbrew install sjq\n```\n\nOr from source: `cargo install --git https://github.com/vivgrid/sjq` (one dependency: `serde_json`).\n\nYou can — `sed 's/^data: //' | jq` — and plenty of people do. But then you're staring at full envelope objects per chunk: ids, timestamps, `object: \"chat.completion.chunk\"`, `finish_reason: null` fifty times over. The signal (the tokens) drowns in the envelope. sjq inverts that: the delta is the output; the envelope gets summarized once.\n\nMIT licensed, single binary. The repo ships 14 test fixtures covering the gnarly edges: multi-line data payloads, mid-stream errors, parallel tool calls, logprobs, CJK/emoji text.\n\nRepo: [https://github.com/vivgrid/sjq](https://github.com/vivgrid/sjq) — feedback and weird streams welcome.", "url": "https://wpnews.pro/news/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq", "canonical_source": "https://dev.to/fanweixiao/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq-166m", "published_at": "2026-10-16 00:44:53+00:00", "updated_at": "2026-10-16 00:57:29.464364+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents"], "entities": ["sjq", "jq", "Rust", "OpenAI", "GitHub", "Homebrew", "vivgrid"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq", "markdown": "https://wpnews.pro/news/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq.md", "text": "https://wpnews.pro/news/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq.txt", "jsonld": "https://wpnews.pro/news/i-kept-hitting-jq-parse-error-on-ai-streams-so-i-built-sjq.jsonld"}}