Okay so here’s what’s actually happening in robotics right now and it’s kind of wild. We’ve got humanoid robots that are walking, arm-swinging, task-doing robots, that are cloud dependent for their basic cognition. The robot sees a red cup, sends a JPEG to AWS, waits 800ms for a response, then decides to pick it up. By which point… the cup moved. Or the WiFi dropped. Or the API ratelimited. Game over.
This is genuinely broken. And here’s why it can’t work on a physics level, not just an engineering preference level.
“Reactive obstacle avoidance” the kind where the robot doesn’t walk into a wall or a toddler that needs to happen in under 50ms. That’s the hard budget. At a walking speed of 1 ms, 50ms is 5cm of travel without correction. A cloud round trip to us-east-1 costs you 20–50ms before any compute even starts. Add model inference (a GPT4 class VLM takes 500ms+ to first token), and you’re looking at 800ms to 2 seconds per decision. That’s literally impossible to build safe reactive behavior on. The math doesn’t work.
The latency problem is severe. The privacy problem is… honestly even more severe, just less obvious at first.
Think about what a humanoid robot in your home actually sees. It sees your medication on the counter. It sees your kid’s face. It sees your daily routine, your security vulnerabilities, what you bought, who you called, when you left for work. If all of that raw sensor data streams to a cloud server for processing every frame, every audio clip, every interaction you have built the most comprehensive domestic surveillance system in history. Inside your own house. That you paid for.
GDPR hates it. CCPA hates it. Your users will hate it the moment they find out. And honestly? They should.
So the question becomes: what if the robot just… carried its own brain? Full vector memory, local LLM reasoning, vision embeddings, speech processing and all that…… all running on a small computer strapped to its chassis. No WiFi needed. No privacy leak. No 800ms decision latency.
That’s what we’re building today. And it’s not theoretical. The hardware exists, the models are small enough, and Qdrant is the vector memory layer that ties it all together.
Turns out the neuroscience metaphor here is actually useful, not just cute.
The human hippocampus handles two kinds of memory. Episodic memory is autobiographical and specific events with full context.
“On Tuesday at 2pm I picked up a coffee mug from the kitchen counter, it was heavier than expected, and I set it on the wrong shelf.”
Semantic memory is general knowledge and decontextualized facts. “Mugs are cylindrical containers. They’re usually in kitchens. Handle is on the right side.”
Different systems, different update rates, different query patterns.
That’s exactly what we need for a robot. And Qdrant maps to this structure naturally.
The cognitive loop looks like this:
Each memory point in Qdrant carries up to three named vectors with one per modality. The same event, captured from every sensory angle:
from qdrant_client import QdrantClient, modelsclient = QdrantClient(":memory:") # or path="/opt/robot/memory" for persistenceclient.create_collection( collection_name="robot_memory", vectors_config={ "vision": models.VectorParams(size=512, distance=models.Distance.COSINE), "audio": models.VectorParams(size=384, distance=models.Distance.COSINE), "text": models.VectorParams(size=768, distance=models.Distance.COSINE), })
That’s it. One collection. Three vector spaces. The vision vector is a CLIP ViT-B/32 embedding of the camera frame (512D). The audio vector is an embedding of whatever Whisper heard (384D). The text vector is a sentence-transformer encoding of a natural language description of the event (768-dim). Same point_id, same payload with timestamp and location and task context, three different ways of encoding the same memory.
Now here’s where it gets interesting. The grounding loop is the thing that connects language to physical action and it works like this:
Robot hears “pick up the red cup.”
That text gets embedded. Qdrant searches the text vector space for similar past experiences. It finds 3 memories where the robot previously encountered cups, objects on counters, pick-and-place tasks. Those memories, including what worked and what didn't, what the visual context looked like - get handed to the local LLM as context. The LLM reasons with actual experience, not just training data. And then it acts.
This is RAG for robots. but the “documents” are the robot’s own lived experience and the “generation” is a motor plan.
Let’s get concrete about what each pipeline looks like on the hardware.
Vision: Camera frame hits OpenCV. You run it through CLIP ViT-B/32 in PyTorch with torch.no_grad() and L2-normalize the output. You get a 512-dimensional vector in about 18ms on a Jetson Orin GPU. That vector goes straight into Qdrant's vision space.
import clip, torch, cv2from PIL import Imagedevice = "cuda" if torch.cuda.is_available() else "cpu"model, preprocess = clip.load("ViT-B/32", device=device)model.eval()def embed_frame(frame): # OpenCV BGR frame image = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)) inp = preprocess(image).unsqueeze(0).to(device) with torch.no_grad(): emb = model.encode_image(inp) emb = emb / emb.norm(dim=-1, keepdim=True) return emb.cpu().numpy().squeeze().tolist() # 512 floats
18ms per frame. At 30Hz you obviously don’t store every frame, sample at 1Hz for background memory accumulation, or trigger writes on detected state changes.
Audio: Microphone stream feeds into Silero VAD (Voice Activity Detection) first then a tiny 1.8MB model that runs in under 1ms per 32ms chunk and tells you exactly when speech starts and stops. Once VAD says “speech detected,” you buffer audio and send it through faster-whisper. The whole thing takes about 80ms for a 3second voice clip. Then embed the transcript with a sentence-transformer and write to the audio vector space.
Fuzzy command matching is where voice interfaces usually fall apart. If you do exact string matching on ASR output you’re going to have a bad time. ASR is noisy. People have accents. They paraphrase. “Kill the lights” and “turn off the lights” mean exactly the same thing but won’t match on string comparison.
Semantic search doesn’t have this problem:
The embedding of “kill the lights” and “turn off the lights” are close in semantic space. Score over 0.85: execute confidently. Score 0.75–0.85: ask for confirmation. Score below 0.75: “sorry, can you repeat that?” Clean, robust, handles accents and paraphrases automatically.
The real power of this semantics kick in when the agent running locally needs to pull right parts of history & stats from online into context for better function calling or script coding/execution
Scene understanding with LLaVA: Sometimes the robot needs richer reasoning than CLIP gives you. When you need it to describe what it’s seeing rather than just embed it, route the frame through LLaVA via Ollama:
import ollama, base64, cv2def describe_scene(frame, question="What objects do you see? Are any relevant to the current task?"): _, buf = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, 85]) img_b64 = base64.b64encode(buf.tobytes()).decode() response = ollama.chat( model="llava:7b", messages=[{"role": "user", "content": question, "images": [img_b64]}] ) return response["message"]["content"]
Use that natural language description to build a richer text vector for the memory point. Now future queries can match on meaning, not just visual pixel similarity.
Speech output closes the loop. Piper TTS takes the LLM’s response text and synthesizes audio in about 30ms for a short sentence. No API. No network. The robot just… talks. Locally.
Every time something interesting happens, it gets written to Qdrant with full multimodal context and a rich payload:
import uuidfrom datetime import datetime, timezoneclient.upsert( collection_name="robot_memory", points=[ models.PointStruct( id=str(uuid.uuid4()), vector={ "vision": clip_embedding, # what it looked like "audio": audio_embedding, # what it sounded like "text": text_embedding, # natural language description }, payload={ "timestamp": datetime.now(timezone.utc).isoformat(), "location": {"room": "kitchen", "x": 1.2, "y": 0.5}, "task": "pick_cup", "outcome": "success", "confidence": 0.92, "scene_description": "Red ceramic mug on wooden counter, handle facing right", } ) ])
Before the robot attempts any task, it queries Qdrant for similar past situations. “I’m about to reach for something on a shelf what do I know about shelf interactions?” The retrieved memories become context for the LLM’s reasoning step. The robot isn’t starting from scratch every time it powers on. It’s actually learning from experience.
The forgetting problem is real though. A robot storing memories at 1Hz accumulates 86,400 points per day. After a year that’s 31 million points. You need a retention strategy. Importance-weighted retention tracks how often each memory gets retrieved and points that come up frequently are important, keep them. Points that were never retrieved in 30 days are probably routine, delete them:
Cross-modal recall is where it gets genuinely cool. The robot hears a sound it doesn’t recognize. It embeds that audio and queries the audio vector space. Finds past memories where it heard similar sounds. Then checks what the vision embeddings looked like at those moments - what was the robot seeing the last time it heard this?
Multi-modal memory association. The robot connects what it hears to what it’s seen in similar situations. That’s not a lookup table. That’s something closer to actual memory.
The hardware: NVIDIA Jetson Orin NX 16GB. About $499 for the module, ~$750 for the dev kit with carrier board. Critically the unified memory architecture, meaning CPU and GPU share the same 16GB pool. No PCIe bandwidth tax when moving data between CPU and GPU memory. This matters a lot for a stack that constantly shuffles tensors around.
All you would need is a Gemma kinda model that’s QAT (Quantization Aware Trained) to match better function calling + scription + context awareness with say something like Circular Attention or Linear Scaled Attention for large contexts like 1M tokens.
The latency comparison is pretty stark:
Cloud-dependent robot pipeline: hear command (10ms) + network (30ms) + cloud STT (400ms) + network (30ms) + cloud LLM (800ms) + network (30ms) + cloud TTS (300ms) = ~1.6 to 2.5 seconds
Local stack: hear command (10ms) + Whisper (80ms) + embed + Qdrant search (5ms) + Ollama first token (120ms) + Piper TTS (30ms) = ~250ms
That’s about 6–10x faster. But the numbers actually undersell the real win. The cloud pipeline has variance with sometimes 800ms, sometimes 2.5 seconds depending on API load. The local pipeline is predictable. Every time. No cold starts, no rate limits, no “degraded service” alerts at 3pm on a Tuesday.
For reactive behavior & obstacle avoidance, balance correction, collision response, the cloud isn’t even in the conversation. Sub100ms is the budget.
A CLIP embedding + Qdrant lookup + rule check takes 25–60ms on Jetson. A cloud round-trip alone is 50–200ms minimum. The local stack wins by physics.
On the vector DB options at the edge: Chroma runs in-process which is convenient, but has no quantization and no multi-vector support per point. Milvus Lite works but runs heavier. Qdrant is the only one with Turbo4 4bit vector compression…..which means you can fit 500K vectors in ~1GB instead of ~4GB. That compression is the difference between this working on 16GB hardware and not.
Here’s how to spin up the infrastructure:
Now the full demo. Runs completely locally, no API keys, no cloud:
"""Local robot brain demo: multimodal memory + LLM reasoning.Requirements: pip install qdrant-client ollama sentence-transformers numpy"""import uuidimport numpy as npimport ollamafrom qdrant_client import QdrantClient, modelsfrom sentence_transformers import SentenceTransformer# --- Setup ---client = QdrantClient(":memory:") # swap to path="/opt/robot/memory" for persistencetext_model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, fast, small (~90MB)client.create_collection( collection_name="robot_memory", vectors_config={ "vision": models.VectorParams(size=512, distance=models.Distance.COSINE), "text": models.VectorParams(size=384, distance=models.Distance.COSINE), })def embed_text(s): return text_model.encode(s).tolist()def fake_vision_embed(description): # Simulates a CLIP embedding - replace with real clip.encode_image() on hardware rng = np.random.default_rng(hash(description) % (2**32)) vec = rng.standard_normal(512).astype(np.float32) return (vec / np.linalg.norm(vec)).tolist()# --- Seed 5 past experiences ---memories = [ ("coffee mug on kitchen counter, ceramic, red, handle right", "kitchen", "pick_object", "success"), ("wooden chair blocking hallway, brown, four legs", "hallway", "navigate", "failure"), ("blue water bottle on desk, plastic, cap on", "office", "pick_object", "success"), ("person standing near doorway, arms crossed, casual clothes", "hallway", "avoid_human", "success"), ("small white bowl on kitchen table, ceramic, empty", "kitchen", "pick_object", "success"),]print("Seeding robot memory...")for desc, room, task, outcome in memories: client.upsert( collection_name="robot_memory", points=[models.PointStruct( id=str(uuid.uuid4()), vector={"vision": fake_vision_embed(desc), "text": embed_text(desc)}, payload={"description": desc, "room": room, "task": task, "outcome": outcome}, )] )# --- Simulate incoming voice command ---voice_command = "grab the mug from the counter" # swap with Whisper output on real hardwareprint(f"\nVoice command: '{voice_command}'")# --- Retrieve relevant past experiences ---results = client.query_points( collection_name="robot_memory", query=embed_text(voice_command), using="text", limit=3, with_payload=True, score_threshold=0.3,)context_lines = [ f"- {r.payload['description']} | task={r.payload['task']} | outcome={r.payload['outcome']}" for r in results.points]context = "\n".join(context_lines) if context_lines else "No relevant past experience found."print(f"\nRetrieved memories:\n{context}")# --- Ask the local LLM to reason with that context ---prompt = f"""You are a robot assistant. Use past experience to plan your next action.Past relevant experience:{context}Current command: "{voice_command}"Respond with ONE short action plan (2 sentences max)."""print("\nThinking (local LLM, no cloud)...")response = ollama.generate(model="llama3.2:3b", prompt=prompt, options={"temperature": 0.1})robot_speech = response["response"].strip()print(f"\nRobot says: {robot_speech}")
Run it and you’ll see something like (Hopefully 🤣):
Seeding robot memory...Voice command: 'grab the mug from the counter'Retrieved memories:- coffee mug on kitchen counter, ceramic, red, handle right | task=pick_object | outcome=success- small white bowl on kitchen table, ceramic, empty | task=pick_object | outcome=successThinking (local LLM, no cloud)...Robot says: Based on past experience, approach the kitchen counter and locate the red ceramicmug. Grip the handle from the right side for a successful pick.
Yup! The robot used its own memories to inform the action plan. No API key. No latency spike. No data leaving the machine.
Want to try this on a Raspberry Pi 5 (8GB)? The Qdrant + sentence-transformer text-only version runs fine on Pi. Just skip the vision embedding and run text-only queries. Ollama will be slow (~3–5 tokens/sec on Pi 5 CPU) but it works offline. For realtime vision and reasonable LLM speeds you need the Jetson.
The Qdrant docs cover quantization options for when you push this to production — INT8 scalar quantization cuts vector memory by 4x, letting you store 2 million memories in the same 1GB budget.
Where does this go next?
The Unitree G1 starts at $13.5K now a full humanoid, Jetson Orin onboard, 4-mic array, depth cameras, full SDK access. Figure’s Helix VLA model runs at 200Hz on embedded hardware. HuggingFace’s LeRobot just added G1 as a native hardware target. The pieces are all here.
The gap isn’t hardware and it isn’t models. It’s the memory layer and the thing that makes a robot accumulate experience instead of starting from scratch every power cycle. Episodic memory. Semantic memory. Cross-modal recall. Forgetting curves. All of it maps directly onto what Qdrant already does.
We’re building robot brains from off the shelf components. A $500 compute module, open-source models, a vector database that fits in 1GB of RAM. The question isn’t “can we do this” but it’s “how fast can we wire it up.”
Pretty wild right?
Want to stay up to date on what’s happening at the intersection of vector search and AI? Subscribe to the Qdrant newsletter and get practical tutorials and project deep-dives. And if you want to run Qdrant at scale without managing infrastructure, Qdrant Cloud has a generous free tier to start.
Enabled Cognition | Agentic Autonomous Humanoids| Embedded Edge Qdrant VectorDB was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.