{"slug": "build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors", "title": "Build agent memory with NVIDIA NeMo Agent Toolkit and Amazon S3 Vectors", "summary": "NVIDIA NeMo Agent Toolkit (NAT) now supports Amazon S3 Vectors as a custom persistent memory provider, implemented through NAT's MemoryEditor interface with add_items(), search(), and remove_items() methods, according to an AWS blog post. The post details deploying the stack on Amazon Elastic Kubernetes Service (Amazon EKS) using a multi-agent investment research use case, and notes NAT's built-in memory providers are Mem0, MemMachine, Redis, and Zep. The approach targets production multi-agent systems needing elastic vector storage, strong write consistency, and cost-efficient scaling to billions of vectors.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Build agent memory with NVIDIA NeMo Agent Toolkit and Amazon S3 Vectors\n\nIn my previous post: [Building persistent memory for multi-agent AI systems with Amazon S3 Vectors](https://aws.amazon.com/blogs/storage/building-persistent-memory-for-multi-agent-ai-systems-with-amazon-s3-vectors/), we explored why memory engineering is the foundational discipline for production multi-agent systems. We showed how [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html), a capability of Amazon Simple Storage Service (Amazon S3), meets the architectural requirements for agent memory: semantic retrieval, rich metadata, strong consistency, and elastic scale.\n\nIn this post, we move from architecture to implementation. We show you how to use Amazon S3 Vectors as the persistent memory layer within the [NVIDIA NeMo Agent Toolkit (NAT)](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html), deployed on [Amazon Elastic Kubernetes Service (Amazon EKS)](https://aws.amazon.com/eks/) for full operational control.\n\nBy the end of this post, you will understand how NAT’s memory subsystem works and how to implement Amazon S3 Vectors as a custom memory provider. You will also learn how to deploy the stack on Amazon EKS, using a multi-agent investment research use case as the running example.\n\n## What is NVIDIA NeMo Agent Toolkit?\n\n[NVIDIA NeMo Agent Toolkit (NAT)](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) is an open source framework for building, profiling, and optimizing AI agents. It’s framework-agnostic, working with Strands Agents, LangChain, LlamaIndex, CrewAI, and custom implementations. NAT provides four capabilities relevant to production agent systems:\n\n1. **Agent orchestration** – Define agents as composable workflows with configurable large language models (LLMs), tools, and prompts. You run them locally with`nat run` or as persistent services with`nat serve` .\n2. **Profiling** – Track token usage, latency, throughput, and run times across agents and individual tools to identify bottlenecks in multi-agent workflows.\n3. **Evaluation** – Built-in evaluators for answer accuracy, context relevance, response groundedness, and agent trajectory, with support for custom evaluators.\n4. **Optimization** – Automated hyperparameter tuning (temperature,`top_p` ,`max_tokens` ) that maximizes quality while minimizing cost and latency.\n\n### NAT’s memory module\n\nNAT includes a dedicated memory subsystem designed to store and retrieve conversation history, user preferences, and long-term knowledge across agent invocations. The memory module is extensible: you create custom memory providers (backends) by implementing NAT’s plugin interface. Key components include:\n\n- **`MemoryEditor`** – The abstract interface that all memory backends must implement. It defines three methods:`add_items()` ,`search()` , and`remove_items()` .\n- **`MemoryItem`** – The data model representing a piece of memory, containing fields for conversation history, tags, metadata,`user_id` , and an optional textual memory string.\n- **`MemoryBaseConfig`** – A Pydantic base class that custom memory configurations extend. NAT discovers providers via the`_type` field in its YAML config.\n- **Automatic memory wrapper** – The`auto_memory_agent` workflow type that wraps agents with automatic memory capture and retrieval, without requiring the LLM to explicitly invoke memory tools.\n\nNAT currently has these built-in memory providers: Mem0, MemMachine, Redis, and Zep. These cover common use cases. However, for production multi-agent systems that require elastic vector storage, strong write consistency, and cost-efficient scaling to billions of vectors, a custom provider backed by Amazon S3 Vectors is the right fit.\n\n## Why Amazon S3 Vectors as the persistent memory backend\n\n[Building persistent memory for multi-agent AI systems with Amazon S3 Vectors](https://aws.amazon.com/blogs/storage/building-persistent-memory-for-multi-agent-ai-systems-with-amazon-s3-vectors/) covers the architectural rationale in depth. The following table summarizes the properties that make S3 Vectors the right fit for NAT’s memory layer:\n\n| **Requirement** | **S3 Vectors capability** | \n| Semantic retrieval | Vector similarity search with configurable distance metrics (cosine, euclidean) | \n| Scoped queries | Filterable metadata on each vector (strings, numbers, Booleans, lists) | \n| Multi-agent coordination | Strong write consistency. Memories are visible immediately after insertion | \n| Scale | Up to 2 billion vectors per index with no capacity planning | \n| Cost efficiency | Pay only for storage, writes, and queries with no idle compute | \n| Access control | AWS Identity and Access Management (IAM) policies per bucket and index. Per-tenant indexes for hard isolation | \n\n## Prerequisites\n\nTo follow along with the implementation in this post, you need:\n\n- An [AWS account](https://aws.amazon.com/account/) with permissions to create Amazon S3 Vectors resources and Amazon EKS clusters.\n- An existing Amazon EKS cluster (see [Getting started with Amazon EKS](https://aws.amazon.com/eks/getting-started/) ).\n- [NVIDIA NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) installed (tested with version 1.6).\n- Python 3.11 or 3.12 (NAT requires >=3.11, <3.14).\n- An embedding model for generating vector representations (this post uses [Amazon Titan Text Embeddings V2](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html) ).\n- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) and[Docker](https://docs.aws.amazon.com/codecatalyst/latest/userguide/deploy-tut-eks.html) for container and cluster operations.\n\n## Implementing S3 Vectors as a NAT memory provider\n\nThe implementation has three steps:\n\nStep 1: Create the S3 Vectors infrastructure\n\nStep 2: Implement a custom MemoryEditor plugin\n\nStep 3: Configure the agent workflow\n\n## Step 1. Create the Amazon S3 Vectors infrastructure\n\nThe following code creates a vector bucket and an index with a metadata schema designed for agent memory. The index uses 1024 dimensions to match the output of Amazon Titan Text Embeddings V2, and it marks the large content field as non-filterable metadata:\n\nAfter creating the bucket and index, you can implement the memory provider that reads from and writes to them.\n\n## Step 2. Implement a custom MemoryEditor plugin\n\nThe following code implements the `MemoryEditor` interface as an Amazon S3 Vectors backend and registers it so that NAT can discover it:\n\nThe plugin generates embeddings with Amazon Titan Text Embeddings V2, stores each memory as a vector with scoped metadata, and translates search filters into Amazon S3 Vectors metadata queries.\n\n## Step 3. Configure the NAT agent workflow\n\nWith the plugin defined, configure it in NAT’s YAML config and wire it into an agent workflow:\n\nWith the `auto_memory_agent` wrapper, you can capture and retrieve memory automatically. User messages and agent responses are stored, and relevant context is injected before each agent call. This design removes the need for the LLM to explicitly invoke memory tools.\n\n## Responsible AI and data handling considerations\n\nBecause this solution persists conversation history and user memory, plan for how that data is retained and accessed. Set a retention policy for stored memories, and use the memory consolidation and removal paths in this post to expire data you no longer need. Avoid storing personally identifiable information (PII) in memory metadata, and redact or tokenize sensitive fields before you embed them. Scope access with per-tenant indexes and least-privilege IAM policies so that each agent reads and writes only the memory it owns.\n\n## Putting Amazon S3 Vectors in practice: Multi-agent investment research\n\nThe following use case demonstrates this pattern in practice. Consider three specialized agents collaborating on an investment research tool:\n\n- **Research Agent** – Gathers market data, earnings reports, and news.\n- **Analysis Agent** – Performs quantitative analysis and identifies patterns.\n- **Synthesis Agent** – Produces reports combining findings from both agents.\n\nWith persistent memory, each agent builds on prior work. The Research Agent recalls previously collected data, avoiding redundant API calls. The Analysis Agent builds on patterns identified in prior sessions. The Synthesis Agent accesses cumulative findings to produce progressively richer reports.\n\n### Multi-agent memory coordination\n\nEach agent uses the same S3 Vectors index but writes with its own `agent_id` in the metadata. Agents retrieve shared knowledge using metadata filters:\n\nNAT’s built-in multi-tenant memory isolation uses `user_id` to scope memory per user. For multi-agent team coordination, the `team_id` metadata field provides an additional grouping dimension within the same S3 Vectors index.\n\n### Memory consolidation\n\nOver time, episodic memories accumulate. Periodically consolidating them into semantic memories (generalized knowledge) keeps retrieval precise and efficient. You can trigger consolidation through a scheduled cron job, a vector count threshold, or an agent-initiated signal after a configurable number of research cycles:\n\n## Deploying agent workloads on Amazon EKS\n\nAmazon EKS is a deliberate choice for teams that need full operational control over their agent workloads. With Amazon EKS, you control scaling, networking, and lifecycle management, and you integrate natively with AWS Identity and Access Management (IAM) for Amazon S3 Vectors access. If you prefer a fully managed runtime, Amazon Bedrock AgentCore provides a managed alternative that removes this operational overhead. You run NAT agents as containerized services using `nat serve`. Each agent type is a Kubernetes Deployment with IAM access granted through [IAM Roles for Service Accounts (IRSA).](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)\n\n### Step 1: Build the agent container\n\nThe following Dockerfile packages the agent with its configuration and memory plugin:\n\n### Step 2: Deploy with Kubernetes manifests\n\nThe following manifests deploy the Research Agent with autoscaling:\n\n### IAM policy for S3 Vectors access\n\nThe following IAM policy grants scoped access to the memory layer. Attach it to the IAM role associated with the `agent-sa` service account through IRSA:\n\n### Deploy and verify\n\nAfter building and pushing the container image, deploy the agents:\n\nS3 Vectors provides strong write consistency. Pods and agent types can immediately see memories that agent pods store, with no cache invalidation required. The `HorizontalPodAutoscaler` scales agent replicas based on CPU utilization. Each replica connects to the same S3 Vectors index through IRSA, supporting consistent memory access regardless of which pod handles a given request.\n\n## Evaluating memory impact\n\nWith NAT’s evaluation harness (`nat eval`), you can quantify the effect of memory on agent performance. Configure two evaluation runs: one with memory enabled and one without, using the same dataset:\n\nBased on the architectural properties of this design, you can expect the following qualitative outcomes when comparing these runs. These are directional expectations, not benchmarked measurements:\n\n- **Groundedness improves** – Recalled memories provide verifiable source context that the agent can cite rather than relying on parametric knowledge alone.\n- **Token usage decreases** – Agents skip re-deriving facts already established in prior sessions. The reduction scales with how much repeated context your workflows involve.\n- **Latency increases modestly** – Each memory recall adds an S3 Vectors query (subsecond at standard access patterns), typically a small fraction of overall LLM inference time.\n- **Duplicate work drops** – With shared memory, agents avoid repeating research or analysis that teammates have already completed.\n\nThe magnitude of these effects depends on your workload: how much context is reused across sessions, how many agents coordinate, and how large your retrieval budget (top-k) is. NAT’s hyperparameter optimizer can systematically tune `top_k` and similarity threshold to find the optimal cost-quality trade-off for your use case.\n\n## Clean up resources\n\nTo avoid ongoing charges from resources created in this walkthrough, delete the following:\n\n1. Delete the Amazon EKS Deployment and associated resources:\n2. Delete the S3 Vectors index and vector bucket:\n3. Remove the IAM role and policy created for IRSA if no longer needed.\n\n## Conclusion\n\nThis post showed how to implement a custom NAT memory provider backed by Amazon S3 Vectors and deploy it on Amazon EKS. The key building blocks:\n\n- **NAT’s `MemoryEditor` interface** provides the plugin contract for custom memory backends.\n- The `auto_memory_agent` wrapper handles capture and retrieval automatically.\n- **Amazon S3 Vectors** delivers the durable, semantically queryable backend with strong consistency for multi-agent coordination.\n- **Amazon EKS** gives you full operational control over agent lifecycle, scaling, and network isolation.\n\nThese patterns apply to domains where agents benefit from accumulated knowledge. They include classifying memory as episodic, semantic, or procedural, sharing memory across agents with metadata filtering, and consolidating memory with an LLM. Examples include customer support, DevOps automation, legal research, and scientific discovery.\n\nFor the memory architecture foundations referenced throughout this post, see [Building persistent memory for multi-agent AI systems with Amazon S3 Vectors](https://aws.amazon.com/blogs/storage/building-persistent-memory-for-multi-agent-ai-systems-with-amazon-s3-vectors/). To get started with NAT, see the [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/) and the [Adding a Memory Provider guide](https://docs.nvidia.com/nemo/agent-toolkit/latest/extend/memory.html).", "url": "https://wpnews.pro/news/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors/", "published_at": "2026-10-01 17:34:56+00:00", "updated_at": "2026-10-01 17:44:50.458532+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "artificial-intelligence", "large-language-models", "ai-tools"], "entities": ["NVIDIA NeMo Agent Toolkit", "Amazon S3 Vectors", "Amazon Web Services", "Amazon Elastic Kubernetes Service", "Mem0", "MemMachine", "Redis", "Zep"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors", "markdown": "https://wpnews.pro/news/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors.md", "text": "https://wpnews.pro/news/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors.txt", "jsonld": "https://wpnews.pro/news/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors.jsonld"}}