Integrating Context-Aware Video AI Agents Into Enterprise Workflows NVIDIA introduced NemoClaw, a collection of open blueprints for building autonomous agents, to integrate context-aware video AI agents into enterprise workflows. The system combines video analytics with organizational knowledge to generate structured reports and trigger downstream actions such as ticket creation and anomaly escalation. This approach moves video analysis from passive observation to proactive, coordinated action at scale. A video analytics AI agent https://www.nvidia.com/en-us/ai/ that can perceive, reason, and act based on massive amounts of video footage must be integrated with existing workflows and applications to be useful. These include content management systems, messaging platforms, databases, ticket queue, and escalation paths. This integration is challenging because video systems, enterprise knowledge bases, and operational tools are usually siloed. Developers need to capture user intent, retrieve the correct organizational context, generate structured reports, and route findings into downstream systems. In a previous post https://developer.nvidia.com/blog/make-sense-of-video-analytics-by-integrating-nvidia-ai-blueprints/ , we explained how to enrich video analysis with document knowledge using NVIDIA Blueprints. This post continues with the topic and explains how to unlock the ability to not just analyze video, but to programmatically act on those analyses by introducing NVIDIA NemoClaw. You’ll learn how to: - Extend VSS for guided, context-aware video analysis - Orchestrate the VSS and RAG blueprints as a composable service using NVIDIA NemoClaw - Generate structured reports enriched with organizational and reference knowledge - Build multistep workflows where video analysis feeds into other business processes - Deploy and scale this solution across enterprise environments This approach is the next step in context-aware video AI agents, which involves moving from “What does this video show?” to “What should we do about what this video shows, and how do we coordinate that action at scale?” What are NVIDIA NemoClaw and NVIDIA Blueprints? NVIDIA NemoClaw https://www.nvidia.com/en-us/ai/nemoclaw/ is a collection of open blueprints for building autonomous agents. It enables the ecosystem to build domain-specialized, always-on agents that are safer, faster, and operate more cost efficiently across digital and physical workflows. NVIDIA Blueprints https://build.nvidia.com/blueprints are customizable reference workflows for building agentic AI https://www.nvidia.com/en-us/glossary/ai-agents/ pipelines at enterprise scale. They combine specialized microservices, optimized models, and composable APIs to accelerate time-to-value while maintaining modularity. In addition to NemoClaw, the main blueprints used in this post are: NVIDIA Metropolis Blueprint for Video Search and Summarization VSS https://build.nvidia.com/nvidia/video-search-and-summarization ingests streaming or archival video, generates captions and visual metadata, and supports semantic search, interactive Q&A, and event summarization. NVIDIA AI Blueprint for Retrieval-Augmented Generation RAG https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline indexes proprietary enterprise documents—manuals, policies, regulations, SOPs, and reference data—into a GPU-accelerated vector store for fast semantic search. How does VSS capture intent, retrieve knowledge, and generate reports from video? VSS provides guided, context-aware video analysis through a set of tools built into the agent. Human-in-the-loop HITL prompts capture what the user wants before any processing begins. The agent retrieves the relevant organizational knowledge and produces a structured, timestamped report. When combined with NVIDIA NemoClaw https://www.nvidia.com/en-us/ai/nemoclaw/ blueprints for building autonomous agents, this system can go beyond simple video analysis to programmatically act on those analyses. This unlocks the ability to generate tickets, compare patterns across multiple sources, draft revised procedures, escalate anomalies, and feed results into downstream workflows. Three agent tools work together to make this happen: Long video summary LVS video understanding tool: Performs long video summarization with mandatory HITL parameter collection. Users interactively specify the scenario what the video is about , events of interest what to detect , objects of focus what to track , and an optional knowledge-retrieval query. Knowledge retrieval frag tool: Calls the RAG Blueprint to retrieve organization-specific context from documents, policies, reference data, and knowledge bases. The RAG Blueprint handles embedding, reranking, and vector search internally. Report generation tool: Produces a structured report combining the video analysis with the retrieved context, complete with timestamps, narrative analysis, and citations. It can use HITL to let the user confirm or edit the prompt before the report is generated. Together, these tools collect user intent through HITL, query the RAG Blueprint for contextual knowledge, process the video with that context, and hand off to the report generation tool for formatted output. Generating assessments and recommended actions from a video To demonstrate this process, we will create a “healthy eating coach” that analyzes food videos to assess a user’s eating habits and return concrete, tracked next steps they can act on. The process is detailed in the following sections. User uploads a video and specifies what to analyze To start, a user uploads a meal preparation video through the VSS interface Figure 1 . NemoClaw then begins the workflow. It reads the vss-generate-video-report-rag skill definition SKILL.md to learn which parameters the analysis needs and hands the request to the VSS agent, which walks the user through a short series of HITL prompts in their terminal Figure 2 . The prompts ask what to analyze, the scenario, the events of interest, the objects to track, and an optional RAG Blueprint query for the reference knowledge to retrieve, such as the nutritional or regulatory guidelines. Capturing this intent up front scopes the analysis to what the user actually cares about before any video is processed. For automated batch runs, these answers can be supplied programmatically instead of interactively. NemoClaw orchestrates VSS and the RAG Blueprint Next, with the parameters confirmed, NemoClaw orchestrates the pipeline. The LVS video understanding tool first queries the RAG Blueprint for the relevant nutritional guidelines, and the RAG Blueprint returns the matching reference documents, handling the vector search internally. It then passes those parameters, the video, and the retrieved context to the LVS service, which summarizes the video hierarchically and weaves the reference knowledge into its findings. The report generation tool combines the result into a structured, timestamped report that includes detected events with timestamps, a narrative analysis grounded in the reference material, citations to the relevant source documents, and concrete recommended actions. Figure 3 shows this orchestration in the NemoClaw terminal, including the agent’s reasoning and tool calls. NemoClaw creates a Jira ticket NemoClaw reads the finished report and turns it into coordinated action. It presents the completed analysis with links to the Markdown and PDF reports and the video playback, a summary of why the meal is healthy, and recommended next steps Figure 4 . It then automatically creates a Jira ticket that summarizes the findings and the recommended dietary adjustments, with an appropriate priority and assignment so the action items are tracked to completion Figure 5 . This downstream step generalizes well beyond Jira. Depending on what the report contains, NemoClaw can also: - Create tickets for the findings, with the right priority and assignment. - Escalate or summarize patterns that emerge across multiple runs. - Bundle supporting evidence for review or compliance. - Route gaps to the appropriate follow-up workflow. At this point, the report is no longer a static document. It becomes the trigger for coordinated action across the systems your team already uses. Figure 6 shows the architecture in four layers: Orchestration: The NemoClaw agent, the vss-generate-video-report-rag skill, and the HITL prompts VSS agent: Tools include video I/O, search, understanding, LVS, knowledge retrieval, and report generation. Knowledge retrieval is part of the agent, not a separate extension RAG Blueprint: NVIDIA RAG API, Milvus vector database, NVIDIA Nemotron reranking NIM https://build.nvidia.com/models?q=rerank , and the indexed reference and organizational documents- LLM fusion: Enrichment of the VSS-provided summary with the context retrieved through the RAG Blueprint Data flows downward through the system, with the agent’s tools orchestrating calls to the LVS service and the RAG Blueprint, both of which feed into the report generation tool for final output. How to deploy the VSS agent with knowledge retrieval Follow the steps below to implement the solution for your own workflow. Prerequisites - NVIDIA GPU s with at least 24 GB VRAM - Docker Engine plus Docker Compose v2 - NGC API key ngc.nvidia.com http://ngc.nvidia.com - NVIDIA Build API key build.nvidia.com http://build.nvidia.com - RAG Blueprint deployed and reachable from the agent its server URL accessible , and its collection name - NemoClaw installed for programmatic access Step 1: Clone the VSS repo and authenticate with NGC git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git ~/vss-public cd ~/vss-public echo "$NGC CLI API KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin Step 2: Configure the environment Edit the LVS profile .env file, deploy/docker/developer-profiles/dev-profile-lvs/.env . All the variables exist in that file, except the RAG values , which the agent’s RAG config reads and you add yourself. Deployment selection MODE=2d BP PROFILE=bp developer lvs HARDWARE PROFILE=H100 H100, L40S, RTXPRO4500BW, RTXPRO6000BW, DGX-SPARK, IGX-THOR, AGX-THOR, or OTHER LLM / VLM placement LLM MODE=local shared local shared runs LLM and VLM on one GPU; use 'local' for separate GPUs VLM MODE=local shared LLM DEVICE ID='0' VLM DEVICE ID='0' Paths you MUST set these VSS APPS DIR="