Customer Support Evidence: Summarize Large Documents with an LLM API, Batch and Streaming

TL;DR: For large customer-support documents, I split and score evidence in a queued batch first, then stream only the compact result the reviewer needs. Streaming the entire document through one request feels responsive, but it couples extraction, retry behavior, and UI state. The useful decision is not batch versus streaming in the abstract. It is which stage has a human waiting.

My concrete job is scoring support-agent candidates against a rubric. A resume, work sample, and interview transcript can be much larger than the final scorecard. The score must point back to evidence, survive a retry, and remain portable across model providers. That last constraint changes the design: the application owns the job format and event format, while a provider adapter owns request details.

I use a queue for document preparation, chunk extraction, and rubric scoring. Those steps are repeatable work. They can be retried with an idempotency key, and a reviewer does not need to watch every token. I stream the finished scorecard, or a small set of progress events, when a person is looking at the result now.

That split avoids a common failure mode. A long streaming request can disconnect after the model has produced most of an answer. If the server treats partial text as the record of truth, a reconnect may create a second score or silently lose the first one. A queued stage writes structured intermediate records instead: document hash, chunk number, rubric version, provider request id, and status. The UI can reconnect and read the same state.

There is a second trap: output speed is not the same as useful latency. Showing a partial sentence about a candidate before all evidence has been considered can anchor a reviewer on an incomplete score. For an evaluation workflow, early narrative is often worse than a clear “processing” state. I reserve live output for bounded work: a short explanation assembled after the score has passed schema validation. I first assumed a streaming LLM API would make every step feel faster; the useful approach turned out to be streaming only the final, reviewable explanation.

This is a revenue-per-hour choice. A one-person SaaS cannot spend an afternoon making a token-by-token transcript durable when the customer needs a defensible scorecard. I ship the narrow workflow weekly, and outsource the undifferentiated transport details to a provider adapter with a small contract.

No token stream is sacred.

The application has three boundaries: a queue job, a model adapter, and a stream of application events. None of them knows a vendor-specific route. The adapter returns text plus usage metadata; the parser decides if that text is acceptable.

First, define the data that must survive a retry. A stable candidateId, documentHash, and rubricVersion make duplicate work visible. The chunk index is part of the key, not a field added for logging later.

type EvidenceChunk = {
  candidateId: string;
  documentHash: string;
  rubricVersion: string;
  index: number;
  total: number;
  text: string;
};

type ChunkScore = {
  chunkKey: string;
  criterion: string;
  score: number;
  evidence: string;
};

type ModelResult = {
  text: string;
  inputTokens?: number;
  outputTokens?: number;
};

type ModelAdapter = {
  complete(input: { system: string; user: string }): Promise<ModelResult>;
};

The queue worker can then be boring. Boring is good here. It claims one chunk, asks for JSON, validates the result, and stores a record under a deterministic key. A retry is allowed to call the adapter again, but it cannot create a second logical score.

async function scoreChunk(
  chunk: EvidenceChunk,
  adapter: ModelAdapter,
  save: (key: string, value: ChunkScore) => Promise<void>,
): Promise<ChunkScore> {
  const chunkKey = [
    chunk.candidateId,
    chunk.documentHash,
    chunk.rubricVersion,
    chunk.index,
  ].join(":");

  const result = await adapter.complete({
    system: "Return one JSON object. Score only the supplied evidence.",
    user: JSON.stringify({
      rubricVersion: chunk.rubricVersion,
      chunkIndex: chunk.index,
      evidence: chunk.text,
    }),
  });

  const parsed: unknown = JSON.parse(result.text);
  if (!isChunkScore(parsed) || parsed.score < 0 || parsed.score > 5) {
    throw new Error("model output failed score validation");
  }

  const score: ChunkScore = { ...parsed, chunkKey };
  await save(chunkKey, score);
  return score;
}

function isChunkScore(value: unknown): value is Omit<ChunkScore, "chunkKey"> {
  if (!value || typeof value !== "object") return false;
  const item = value as Record<string, unknown>;
  return (
    typeof item.criterion === "string" &&
    typeof item.score === "number" &&
    typeof item.evidence === "string"
  );
}

The real implementation needs a transactional claim and an outbox or equivalent durable event record. The example leaves those pieces behind the save function on purpose: they are storage concerns, and pretending an in-memory promise is a queue would teach the wrong lesson.

After all chunks are stored, a reducer combines scores by criterion. It should retain the source chunk keys and evidence excerpts. I do not ask the final call to rediscover the whole document. That reduces prompt size, makes the final response auditable, and gives me a useful boundary for evaluating a new provider.

The stream is a separate concern:

type ScoreEvent =
  | { type: "status"; value: "queued" | "scoring" | "ready" }
  | { type: "text"; value: string }
  | { type: "error"; message: string };

async function* scorecardEvents(
  scorecard: Promise<string>,
): AsyncGenerator<ScoreEvent> {
  yield { type: "status", value: "scoring" };

  try {
    const text = await scorecard;
    yield { type: "text", value: text };
    yield { type: "status", value: "ready" };
  } catch {
    yield { type: "error", message: "Scorecard generation failed" };
  }
}

In production, the HTTP layer can map these events to a standard event-stream response, but it should not expose provider chunks as application truth. The client can reconnect, display status, and render the final validated scorecard. A stream is a delivery mechanism. It is not a persistence layer.

Chunking is the first place I would test aggressively. Splitting on a fixed character count can separate a rating from the paragraph that justifies it. I keep headings, paragraph boundaries, and document offsets where possible. I also include the rubric version in every prompt. Otherwise a rubric edit can mix old chunk scores with new final aggregation. This approach is a poor fit for a tiny document whose complete answer is already short; the queue adds state and delay, so I would use one bounded call there.

One fixture is deliberately annoying: the candidate claims ownership of escalation work in the interview transcript, but the work sample describes a handoff to someone else. The extractor should preserve both excerpts and their offsets. The scorer should flag the conflict for review instead of averaging two persuasive sentences into a confident number. Another fixture has no evidence for a criterion at all. That result is “insufficient evidence,” not zero by default, because a missing paragraph is a data condition rather than proof of poor performance. These cases take longer to design than a happy-path prompt, but they protect the scorecard when the documents stop looking tidy.

The second break is partial failure. One failed chunk should produce a visible incomplete state, not a confident overall score. The reducer should require every expected criterion or mark the result as incomplete. Retry only transient adapter failures; malformed JSON needs a validation path, because repeating the same request without changing the input rarely fixes it.

The third break is provider drift. Different APIs expose different batching limits, streaming event shapes, context windows, and usage fields. I normalize only what the workflow needs: completion, structured text, usage when available, cancellation, and an error category. I keep raw responses in a restricted diagnostic store with retention limits. The core tables never depend on a provider-specific field.

For evaluation, I keep a small fixture set with deliberately awkward cases: a candidate who meets a criterion in two distant sections, a document with no evidence, and a transcript containing contradictory answers. Each adapter runs against the same fixture and the same rubric version. The test asks whether the score and citations are acceptable, not whether two providers produce identical prose.

At higher volume, I would add a scheduler that groups independent chunk jobs, a dead-letter queue for repeated validation failures, and an explicit cancellation state. I would also separate extraction from scoring so a rubric change can reuse normalized evidence without reprocessing the source document. Those are worthwhile once queue depth and reviewer latency are real constraints, not because a diagram looks more complete with them.

I would keep the user-facing contract stable: submit a job, observe durable status, fetch a scorecard with evidence links, and optionally open a bounded live explanation. That contract lets the underlying adapter change without rewriting the dashboard or the audit trail. It also makes the batch-versus-streaming decision reversible, which matters more to a solo founder than squeezing a little more novelty out of the first integration.

The decision rule is simple. Queue work that must be complete, retryable, and auditable. Stream work that is already bounded and has a waiting reader. For candidate scoring, the long document belongs in the first category; the final explanation can belong in the second. That is the trade-off: batch protects completeness, while streaming improves perceived responsiveness but exposes more connection and partial-state failure modes.