Artificial Intelligence #
As enterprises build production agents with dozens of tools and large structured inputs, token consumption and latency become the primary blockers to a good user experience. When these agents process large JSON data, every additional tool invocation injects the full payload into the large language model (LLM) context window, causing token usage and response latency to spike. Over multi-turn conversations, this pressure compounds, driving up costs and degrading response quality. Effectively managing context-window pressure in agentic systems is key to reliability and responsiveness at scale. In this post, we share three composable strategies we used to build a production agent for Flightscape that handles 20 MB+ JSON payloads. You can apply them to reduce token costs, lower latency, and maintain full reasoning capability across extended sessions. Together, these optimizations reduced token consumption by up to 71% and cut average latency by up to 80% across our benchmark runs on a representative 23 MB payload.
Solution overview #
Flightscape is an airline operations platform that helps airlines plan and manage their crews, flights, and operations. As part of Flightscape’s crew management capabilities, we built a Training Schedule Optimization agent that verifies crew training schedule assignments against business rules. With 40+ tools and JSON inputs in megabytes, this agent could answer complex queries across crew assignments, scheduling, and resource allocation. Although the agent delivered the required functionality, average request latencies exceeded 15 seconds across most calls. Every tool call injected its full payload into the model context, inflating token counts and costs. On larger datasets, the context window filled up before the agent could finish reasoning. The following diagram (Figure 1) details the agentic architecture and data flow:
In Figure 1, the Optimizer receives a Request JSON and produces a Response JSON, which is then validated by a Business Rules Validator to generate a Validation Report JSON. All three JSON documents are ingested by an AI Agent that provides a conversational interface for users to query optimization results, understand validation errors, and explore scheduling assignments. The agent is built with LangGraph, an open source framework by LangChain, and runs on Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale, with any framework or model. Amazon Bedrock Guardrails can be applied to validate and filter the agent’s conversational responses, helping facilitate responsible AI practices in production deployments. We defined a custom agent state that holds the data needed by our tools: optimization request JSON, optimization response JSON, and validation report JSON. We also added basic statistics and cache data to state for rich queries. The agent uses an Amazon Bedrock model for reasoning and Amazon Bedrock AgentCore memory, a capability of Amazon Bedrock AgentCore, to persist chat interactions across turns, and each turn follows this flow (Figure 2):
This is what the state structure looks like. The state references TOPT (Training Optimizer) request and response objects, which represent the crew scheduling inputs and optimization results:
Note: This state originally held heavy payloads inline. Later sections explain how we moved large artifacts out of conversational context to reduce token usage.
We implemented three strategies to reduce the token costs, latency, and memory overhead caused by passing large JSON payloads through the model context. The following diagram (Figure 3) shows how the three strategies fit into a single agent turn. Strategy 1 runs at the start and end of each turn, and Strategies 2 and 3 run each time a tool returns a result.
At the start of a turn, the hydrate_state() function restores lightweight session state from AgentCore memory. When a tool returns an oversized list or dictionary, the agent stores the full result in Amazon Simple Storage Service (Amazon S3) and passes the model a compact summary with a handle. The model can then retrieve more detail with get_result_page_tool(). At the end of the turn, the dehydrate_state() function writes heavy payloads to the S3ArtifactStore and saves only references to AgentCore memory. The following sections describe each strategy in detail.
Prerequisites #
To implement these strategies, make sure the following are in place:
- An AWS account with access to Amazon Bedrock and a supported model.
- An Amazon S3 bucket for artifact storage.
- Amazon Bedrock AgentCore with Amazon Bedrock AgentCore memory enabled for session-state persistence.
- A LangGraph-based agent (or equivalent graph-orchestration framework) with tool-calling support.
- A Python 3.12+ environment with
boto3,langchain-core, andlanggraph.
Optimization strategies #
The companion code is available in this GitHub repository.
Strategy 1: Persist lightweight session state, offload heavy artifacts
Our first optimization was to change how we persist session state. Initially, large payloads were being stored too close to the conversational state. We therefore moved heavy artifacts (request JSON, response JSON, validation data, and other large derived payloads) to Amazon S3. AgentCore memory then persists only stable references to these objects, keeping the session state lightweight.
To support this, we introduced a dedicated persisted-state schema that keeps only lightweight fields and artifact references. The live graph state still includes parsed domain objects for tool execution, but persistence now writes a dehydrated form. At runtime, tools hydrate heavy payloads on demand using those references, instead of everything on every turn.
This PersistedState schema is what gets written to AgentCore Memory:
The updated GraphState mirrors these references alongside the live domain objects:
The dehydrate_state function strips heavy payloads from the live state and returns the lightweight persisted form along with the artifacts to upload:
Rehydration reverses this process. When a session resumes, we rebuild the live graph state: deserialize LangChain messages, re-parse raw payloads into domain objects, and restore every reference key so later turns can locate their artifacts:
Both dehydration and rehydration rely on the artifact store returning a consistent reference for each payload. Instead of minting a random key, we derive the reference from the payload itself. This approach, known as content addressing, serializes the JSON with sorted keys and compact separators, and uses the SHA-256 digest of those bytes as the reference.
The _content_hash helper produces a deterministic SHA-256 reference from a JSON-serializable payload:
Content addressing makes writes idempotent. Identical payloads map to the same key, so a turn that re-serializes an unchanged request JSON reuses the object already in the bucket instead of up it again. The S3-backed store exposes a put_json and get_json interface. It also keeps a small in-process cache, so payloads already read or written during the current invocation do not require a second S3 round trip.
The S3ArtifactStore class wraps the put_json/ get_json interface with an in-process cache and content-addressed keys:
Off heavy artifacts to S3 and persisting only lightweight references reduced per-turn latency and the volume of data stored in AgentCore memory. With heavy payloads no longer persisted inline, tool-call responses became the dominant consumer of context tokens, prompting us to introduce result paging.
Strategy 2: Page large result lists on demand
Even after moving heavy payload persistence to Amazon S3, we still had a major token bottleneck at tool-response time. Many tool results were too large, which created two problems. First, we were sending full lists to the model even when it only needed a small subset, making each turn slower and more expensive. Second, those full list payloads were injected into context, which quickly consumed the context window over multi-turn conversations.
In some cases, a single tool response exceeded 80,000 tokens, and the cost compounded as users continued interacting with the agent. We wanted to preserve full data access without sending everything by default. To solve this, we introduced on-demand result paging for large tool outputs. Instead of returning full lists, tools now return a compact payload with total count, a small sample, and a result handle.
When a tool result exceeds the sample-size threshold, the full list is stored in S3 and only a compact summary is returned to the model: