Streaming vs JSON: Trade-offs in AI-Powered Apps

This is a breakdown of a decision I made building LogicVisor, not a general "streaming vs JSON" guide. AI API responses aren't limited to text: some return images, audio, structured data. I only needed text, so streaming vs JSON became a real fork in the road. If I'd needed image responses too, this calculus changes entirely.

Two constraints were in tension from the start:

JSON has to be parsed before it's usable, and you can't parse it in pieces as it arrives. It needs to be complete, start to finish, before JSON.parse()

does anything with it. That forces a binary choice: stream, or don't. Streaming, in my case, meant raw text, because that's all the underlying APIs stream.

I started with streaming. Feedback was near-instant on the user end, and it felt great. Then I needed a more structured response, for both the user view and the database, so I switched the request over to ask for JSON instead of free text:

const groqParams: ChatCompletionCreateParamsNonStreaming = {
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: false,
  response_format: {
    type: "json_object",
  },
};

const groqResponse = await groq.chat.completions.create(groqParams);
aiReviewText = groqResponse.choices[0].message.content ?? "";

stream: false

plus response_format: json_object

is the whole trick. The model still wraps its answer in a markdown code fence half the time regardless of what you ask for, so I still need to strip that off before parsing:

export function extractJSONFromMarkdown(markdownString: string) {
  let jsonString = markdownString.trim();

  if (jsonString.startsWith("```

json")) {
    jsonString = jsonString.substring(7);
  }
  if (jsonString.endsWith("

```")) {
    jsonString = jsonString.substring(0, jsonString.length - 3);
  }

  const parsedData: PromptResult = JSON.parse(jsonString.trim());
  return parsedData;
}

I got the data shape I wanted. What I lost was the user experience. Now the user was staring at a state for the full response time instead of watching text stream in.

The streaming version was never faster in absolute terms. It just felt faster, because you're more patient when you can see progress happening. Streaming exploited that. JSON removed it.

Since I couldn't stream JSON (parsing requires a complete payload) and I couldn't get structured data out of raw streamed text without hacking something together after the fact, I stopped trying to force one approach to do both jobs. I fixed the UX problem directly instead: spinners, pulsers, rotating status text. Not real progress, but it reads as progress, and that's what the user actually responds to.

When I added anonymous, free-trial reviews, the storage requirements dropped. Anonymous users didn't need the same depth of data persisted as authenticated ones. So that path defaults to streaming, for the fastest perceived response, which is exactly what you want when you're trying to hook a new user during a free trial:

const groqStream = await groq.chat.completions.create({
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: true,
});

for await (const chunk of groqStream) {
  const text = chunk.choices[0]?.delta?.content ?? "";
  fullText += text;

  controller.enqueue(
    encoder.encode(`data: ${JSON.stringify({ type: "content", content: text })}\n\n`),
  );
}

Same groq.chat.completions.create

call as before, just stream: true

instead of false

, and no response_format

. Each chunk gets pushed to the client over server-sent events as it arrives, and I accumulate fullText

on the server side so I've still got the complete response to hand off for logging once the stream closes.

(The real implementation also retries on 503s with backoff and sends a "waking up" notice on cold starts. Left out here since it's not specific to the streaming vs JSON decision, just general resilience around a serverless AI provider.)

Authenticated users, who need full structured data persisted, get the non-streaming JSON path from the first section, feeding straight into a database write:

const result = extractJSONFromMarkdown(aiReviewText ?? "");

const review = await createAIReview({
  user_id: user.id,
  time_complexity: result?.time_complexity ?? "",
  overall_grade: result?.overall_grade ?? "",
  markdown_review: result?.markdown_review ?? "",
  submission_id: submission.id,
});

Two code paths, one architecture, each one earning its keep for a different constraint: speed for anonymous trial users, structure for authenticated ones.

Building AI-powered products (or honestly, any product) comes down to picking the trade-off that serves your actual goal, not the one that looks cleanest on paper. Streaming and JSON aren't competing solutions here, they're two tools solving two different problems in the same app.

Cover photo by Myles Bloomfield on Unsplash