NarrateAI: production-ready LLM quality assurance on Amazon Bedrock AWS detailed five quality-assurance techniques built on Amazon Bedrock that give its NarrateAI conversational assistant approximately 99 percent numerical accuracy while streaming responses in real time for more than 4,000 AWS executive leaders. The techniques are adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, a composite evaluation framework, and data accuracy verification, each targeting a distinct failure mode such as hallucinated metrics, API throttling, validation latency, and subjective language. NarrateAI runs on Amazon Bedrock AgentCore in a two-layer architecture with an Automated Narrative Generation Layer for batch processing and a Conversational AI Interface Layer for real-time interaction. Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ NarrateAI: production-ready LLM quality assurance on Amazon Bedrock Executives need to make data-driven decisions during live business reviews, where accuracy and speed matter. A conversational agentic AI assistant can meet this need by answering data questions instantly. But the stakes are high: a wrong number or a slow response in front of leadership carries immediate professional consequences, and a capable large language model LLM alone can’t guarantee either. In production, that gap shows up as hallucinated metrics, API throttling, validation latency, and subjective language. Closing it requires production-ready quality assurance built into every step, from data retrieval to response delivery. This post details five techniques, implemented on Amazon Bedrock https://aws.amazon.com/bedrock/ , that work together to deliver it: adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, composite evaluation framework, and data accuracy verification. Each addresses a distinct failure mode while operating as a coordinated system. This post is the second in our NarrateAI series. NarrateAI transforms business intelligence for over 4,000 AWS executive leaders through a two-layer architecture built on Amazon Bedrock AgentCore https://aws.amazon.com/bedrock/agentcore/ , a platform to build, connect, and optimize agents at scale, with any framework or model. The architecture comprises an Automated Narrative Generation Layer for batch processing and a Conversational AI Interface Layer for real-time interaction. Our previous post https://aws.amazon.com/blogs/machine-learning/how-aws-smgs-uses-an-ai-powered-conversational-assistant-to-transform-business-management-with-amazon-bedrock-agentcore/ covered the business challenges, overall architecture, user experience, and enterprise deployment. This one goes deeper into the engineering, showing how the five techniques achieve approximately 99 percent numerical accuracy while streaming responses in real time. It’s written for engineers and architects building LLM applications who are familiar with LLM APIs and streaming responses. From business vision to technical implementation This post focuses exclusively on the advanced quality assurance mechanisms within the real-time layer. Consider an executive asking “Which regions aren’t meeting their targets and why?” The five techniques work together so that the response is numerically accurate and arrives in real time. It also maintains throughput under concurrent global usage and is professionally formatted for direct use in business reviews. Figure 1 shows how these five techniques form a layered dependency chain, where each layer feeds the one below it to produce a validated real-time response from raw queries. Adaptive Pipeline Orchestration routes queries by data volume so that most complete in a single fast pass while complex queries receive full parallel treatment. Cross-Account Multi-Model Failover expands available inference capacity across independent model-account quota spaces, reducing user-visible throttling. Real-Time Streaming Evaluation validates each paragraph the moment it’s produced, overlapping quality checks with generation. The Composite Evaluation Framework runs multiple independent evaluators in parallel against each paragraph. Data Accuracy Verification catches numerical hallucinations through a two-stage cascade. It starts with cheap exact matching and escalates to semantic verification only when needed. The following sections walk through each technique in the order it appears in the pipeline, from data retrieval through generation to quality assurance. Adaptive pipeline orchestration Enterprise knowledge documents contain extensive information and queries vary enormously in how much of it they need. A focused question like “What’s my team’s quarterly attainment?” might draw on a handful of sections discrete chunks of enterprise knowledge documents retrieved . A comprehensive request like “Give me a full regional performance analysis” requires synthesis across hundreds. Users expect the same response time either way. This heterogeneity requires a routing strategy that handles the common case quickly without sacrificing thoroughness for complex scenarios. No single processing strategy works for all queries, because the right approach depends on how much data a query actually requires. Single-pass concatenation is fast and cheap, but it breaks down when aggregated sections exceed the model’s context window typically 200K tokens , forcing truncation and quality loss. Fixed multi-pass batch processing avoids truncation, but it pays for multiple LLM invocations on every query, including the roughly 90 percent that don’t need them. Our approach routes each query based on its total aggregated section volume ∣D∣ where D is the set of retrieved document sections : single pass when the data fits, batch processing only when necessary. The result is a three-phase pipeline that classifies each query once and then processes it along the cheapest path that preserves quality. Three-phase adaptive processing The pipeline processes every query in up to three phases: Mode-Aware Consolidation packs retrieved sections and picks a route, Bifurcated Analysis executes that route as either a single fast-path call or parallel normal-path batches, and Conditional Consolidation merges the parallel results when needed. The following table summarizes how the two paths move through these phases before we walk through each phase in detail. | | Fast path ~90% of queries | Normal path ~10% of queries | | Trigger | Retrieved volume ∣D∣ ≤ θ | Retrieved volume ∣D∣ θ | | Phase 1: Mode-Aware Consolidation | All sections concatenated into a single chunk | Sections packed into N batches respecting document boundaries | | Phase 2: Bifurcated Analysis | Single LLM call, streams directly to evaluation | N parallel LLM calls, one per batch | | Phase 3: Conditional Consolidation | Skipped entirely | Merges N analyses, resolves conflicts, streams the final answer | The first phase, Mode-Aware Consolidation, operates on retrieved document sections. It applies a greedy first-fit packing strategy to fill context windows up to the token limit while preserving section priority order and section boundaries. For smaller volumes ∣D∣≤θ, where θ is an empirically calibrated character-count threshold , all sections are concatenated into a single chunk, and this is called “fast path”. For large volumes ∣D∣ θ , sections are packed into optimal batches that respect document boundaries and this is called “normal path”. Threshold θ is set relative to the model’s context window limit and calibrated to capture 70–90 percent of queries on the fast path in practice, approximately 90 percent take this path, as shown in the Results section . Because the execution mode is fixed in this first phase, each downstream phase can optimize for its specific path rather than handling both. The second phase, Bifurcated Analysis, executes the chosen path. The fast path makes a single LLM call with the complete context and streams results directly to the evaluation pipeline. The normal path distributes batches across N parallel LLM invocations where N is the number of batches produced by Phase 1 that analyze chunks independently to achieve near-linear speedup. The third phase, Conditional Consolidation, operates exclusively on the normal path by synthesizing N independent analyses into a unified response. The consolidation LLM receives partial analyses with the original question and applies conflict resolution heuristics while streaming the final answer. The fast path bypasses this phase entirely because the final response only uses one analysis. Figure 2 shows the complete flow, with both paths converging on the streaming response. In production, this threshold-based routing delivers outsized efficiency. Approximately 90 percent of queries take the fast path single LLM call, time-to-first-token TTFT within a few seconds, total latency typically under 25 seconds . The remaining 10 percent complex multi-document queries receive the full parallel batch treatment approximately 50–75 seconds compared to under 25 seconds on the fast path commensurate with their complexity. With N=4 typical batches for normal-path queries, the blended cost per query is approximately 1.4 LLM invocations. That is a 72 percent reduction in LLM invocations per query versus always using the multi-pass strategy, while maintaining full quality coverage for complex queries. Teams should calibrate their own threshold to match their workload’s query volume distribution and keep it below the model’s context window limit. Cross-account multi-model failover The adaptive pipeline consolidates information efficiently, but a fast pipeline is only useful if the model behind it stays available. During peak review periods, when thousands of users run their analyses simultaneously, a single throttled request erodes confidence in the tool and drives users back to manual spreadsheet analysis. The standard mitigation, exponential backoff with jitter, proved insufficient. Amazon Bedrock assigns independent quotas per model and per account. Treating each model-account pair as its own capacity space multiplies available throughput without provisioning new infrastructure. Rather than waiting for capacity to free up, we set out to discover capacity that was never contended in the first place. This section shows how treating every model-account pair as its own quota space multiplies capacity without provisioning new infrastructure. That capacity hides in plain sight. Amazon Bedrock quotas are independent across two axes: each model has its own limits, and each AWS account receives its own quota. A 3-model × 3-account configuration therefore provides nine independent quota spaces. At runtime, the system explores this grid by attempting the highest-ranking model first, with accounts randomized to help prevent hot-spotting. The implementation is a custom Strands https://strandsagents.com/ model provider, a drop-in replacement for the standard BedrockModel https://github.com/strands-agents that adds transparent capacity expansion while maintaining full compatibility with existing agent code. Figure 3 shows the grid and the cascade path a request follows through it. Three-mechanism coordination Three mechanisms coordinate to make this work. The first is model ranking, which establishes a quality-speed hierarchy. Queries attempt the highest-ranking model first and cascade only when necessary, so users receive the best available model automatically. The second is account-level distribution, which helps prevent hot-spotting through stochastic load balancing. Before each failover attempt, Python’s random.shuffle randomizes the AWS role ARN list order by creating a copy and shuffling it in place. Load then distributes uniformly across accounts over time without requiring complex traffic-shaping algorithms or centralized coordination. Each request gets fresh randomization, naturally achieving approximately 1/N traffic distribution where N is the number of configured accounts. This maximizes aggregate Amazon Bedrock API quota utilization across the account pool. The third is detection and fast recovery, which avoids backoff entirely. The ThrottlingDetector catches ThrottlingException , ServiceQuotaExceededException , and TooManyRequestsException , immediately attempting the next model-account combination. AWS Security Token Service STS https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html AssumeRole obtains fresh credentials in 100–200ms, which is negligible compared to the multi-second delays of exponential backoff with jitter https://docs.aws.amazon.com/general/latest/gr/api-retries.html . Over a six-month production deployment across over 4,000 users, the N×M quota exploration where N is the number of models and M is the number of accounts absorbed traffic surges and reduced user-visible throttling during peak periods. The degree of improvement scales directly with the number of model-account combinations configured. Load testing using the Locust framework revealed that the system was able to support model requests from over 100 users concurrently for a sustained period without any failed requests. The following snapshot shows the load test details for over 100 user requests for a streaming response from Amazon Bedrock through the application. The key takeaway is that infrastructure did not need to scale. The quota space expanded instead. Real-time streaming evaluation The failover architecture now provides an uninterrupted token stream under quota pressure. However, a reliably delivered response that contains hallucinated revenue figures has only made the problem worse. Availability without accuracy is a liability. Our initial approach implemented quality checks as a sequential post-processing step. It would generate the complete response, run the evaluators, then deliver the validated output. While this achieved high accuracy, time-to-first-content TTFC exceeded a minute as users waited for the entire response to generate and evaluate before seeing the output. This tradeoff between accuracy and responsiveness led us to a key question: does the entire response need to be complete before validation can begin? This section presents the parallel evaluation architecture that answers that question and achieves near-zero validation overhead by using the producer-consumer concurrency pattern https://arxiv.org/pdf/cs/0210001 . Paragraph independence Validating output paragraph N does not require waiting for paragraph N+1 to be generated. This logical independence makes parallel execution possible. It dramatically reduces time-to-first-content TTFC , which is the elapsed time from request submission until the user sees the first delivered evaluated content. In a sequential pipeline, users wait for all paragraphs to generate and then all paragraphs to evaluate before seeing anything. Our parallel approach validates each paragraph as it’s produced, so users see the first content after only one paragraph generates and evaluates, not after the entire response completes. Only the very first paragraph incurs evaluation latency before the user sees content. Every subsequent paragraph is evaluated concurrently with generation of the next, so evaluation cost is absorbed within the generation window. For each subsequent paragraph, the delivery delay is simply whichever takes longer, either evaluating the current paragraph or generating the next one. In practice, paragraph generation takes on the order of a few seconds, while deterministic checks weasel words, emoji complete in tens of milliseconds, providing a wide stability margin that keeps evaluation effectively invisible to users in the common case, even with occasional invocation of expensive LLM-based evaluation. Producer-consumer architecture and performance model The implementation coordinates three asynchronous components. The producer task receives tokens from the Amazon Bedrock streaming API and accumulates them into paragraph-sized units detected through configurable heuristics double newlines for markdown boundaries . Complete paragraphs enqueue in a thread-safe bounded buffer with mutual blocking and First-In-First-Out FIFO ordering to preserve coherence. A sentinel value signals completion. The consumer task dequeues paragraphs, executes sequential validation checks such as data accuracy verification, and streams approved content as multi-word chunks for smooth perceived delivery. In practice, this steady-state behavior splits into two distinct regimes depending on which evaluators are triggered for a given paragraph. Define λ