{"slug": "vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer", "title": "Vector Search Latency Crisis: What I Learned Integrating RAGFlow's Memory Layer", "summary": "A developer integrating RAGFlow into an agent memory pipeline hit a latency wall at roughly 50 concurrent document insertions per second, with query latency spiking from 80ms to over 4 seconds and recall dropping 30%, traced to pgvector HNSW index lock contention and unnormalized embeddings. The fix used a dual-index schema with IVFFlat for writes and HNSW for reads, L2 normalization at write time, and hybrid vector plus full-text search fused via reciprocal rank fusion. After normalization, recall@10 rose from 0.68 to 0.91 on the team's test set.", "body_md": "While integrating RAGFlow into our agent memory pipeline, we hit a wall at around 50 concurrent document insertions per second. Query latency spiked from 80ms to 4+ seconds, and recall rates dropped by 30%. The culprit? HNSW index locks and improper embedding normalization.\n\nRAGFlow is a solid open-source RAG engine that fuses retrieval with agent capabilities, but like any vector search system, it exposes you to PostgreSQL's pgvector quirks under load. Here's what broke, and the exact fixes that stabilized our production deployment.\n\nPgvector's HNSW index is not optimized for high-concurrency writes. During bulk document ingestion, we saw INSERT queries blocking each other, creating a convoy effect.\n\n**The Fix: Separate Write and Read Indexes**\n\nWe created a dual-index schema with IVFFlat for writes and HNSW for reads:\n\n```\n-- Schema for agent memory chunks with dual indexing\nCREATE TABLE agent_memory_chunks (\n    id BIGSERIAL PRIMARY KEY,\n    agent_id VARCHAR(64) NOT NULL,\n    chunk_text TEXT NOT NULL,\n    embedding vector(1536) NOT NULL,\n    metadata JSONB DEFAULT '{}',\n    created_at TIMESTAMP DEFAULT NOW()\n);\n\n-- Fast writes: IVFFlat with low list count\nCREATE INDEX idx_memory_ivfflat ON agent_memory_chunks \nUSING ivfflat (embedding vector_cosine_ops) \nWITH (lists = 100);\n\n-- Fast reads: HNSW with tuned parameters\nCREATE INDEX idx_memory_hnsw ON agent_memory_chunks \nUSING hnsw (embedding vector_cosine_ops) \nWITH (m = 16, ef_construction = 64);\n\n-- Filtered search with agent_id partition\nCREATE INDEX idx_agent_embedding ON agent_memory_chunks (agent_id, ((embedding <=> '[...]'::vector)));\n```\n\nThe IVFFlat index handles inserts without blocking, while HNSW serves reads. Rebuild HNSW asynchronously during low-traffic windows:\n\n```\n-- Rebuild HNSW without blocking writes\nREINDEX INDEX CONCURRENTLY idx_memory_hnsw;\n```\n\n**Query with agent-specific filtering:**\n\n``` js\nSELECT id, chunk_text, metadata,\n       1 - (embedding <=> $1::vector) AS similarity\nFROM agent_memory_chunks\nWHERE agent_id = $2\n  AND metadata @> $3::jsonb  -- JSON filter for tags/context\nORDER BY embedding <=> $1::vector\nLIMIT 20;\n```\n\nWe initially stored raw embeddings from OpenAI's API without normalization. Cosine similarity in pgvector expects unit vectors, but our embeddings had varying magnitudes, causing semantically similar chunks to rank poorly.\n\n**The Fix: Normalize at Write Time**\n\n``` python\nimport numpy as np\nfrom typing import List\n\ndef normalize_embedding(embedding: List[float]) -> List[float]:\n    \"\"\"L2 normalization for cosine similarity.\"\"\"\n    vec = np.array(embedding)\n    norm = np.linalg.norm(vec)\n    if norm == 0:\n        return embedding\n    return (vec / norm).tolist()\n```\n\nAfter normalization, recall@10 jumped from 0.68 to 0.91 in our test set.\n\nVector search alone misses exact keyword matches. We implemented hybrid search combining pgvector with PostgreSQL's full-text search, fused via RRF:\n\n``` python\nimport psycopg2\nfrom typing import List, Tuple\n\ndef hybrid_search_rrf(\n    conn,\n    query_embedding: List[float],\n    query_text: str,\n    agent_id: str,\n    k: int = 20,\n    rrf_k: int = 60\n) -> List[Tuple[int, str, float]]:\n    \"\"\"\n    Hybrid search with reciprocal rank fusion.\n\n    Args:\n        rrf_k: RRF constant (default 60 per academic papers)\n\n    Returns:\n        List of (id, chunk_text, fused_score)\n    \"\"\"\n    cursor = conn.cursor()\n\n    # Vector search ranks\n    cursor.execute(\"\"\"\n        SELECT id, chunk_text, \n               ROW_NUMBER() OVER (ORDER BY embedding <=> %s::vector) as rank\n        FROM agent_memory_chunks\n        WHERE agent_id = %s\n        ORDER BY embedding <=> %s::vector\n        LIMIT %s\n    \"\"\", (query_embedding, agent_id, query_embedding, k))\n    vector_ranks = {row[0]: (row[1], row[2]) for row in cursor.fetchall()}\n\n    # Full-text search ranks\n    cursor.execute(\"\"\"\n        SELECT id, chunk_text,\n               ROW_NUMBER() OVER (ORDER BY ts_rank(to_tsvector('english', chunk_text), \n                                                    plainto_tsquery('english', %s)) DESC) as rank\n        FROM agent_memory_chunks\n        WHERE agent_id = %s\n          AND to_tsvector('english', chunk_text) @@ plainto_tsquery('english', %s)\n        LIMIT %s\n    \"\"\", (query_text, agent_id, query_text, k))\n    fts_ranks = {row[0]: (row[1], row[2]) for row in cursor.fetchall()}\n\n    # RRF fusion\n    fused_scores = {}\n    all_ids = set(vector_ranks.keys()) | set(fts_ranks.keys())\n\n    for doc_id in all_ids:\n        vector_rank = vector_ranks.get(doc_id, (None, float('inf')))[1]\n        fts_rank = fts_ranks.get(doc_id, (None, float('inf')))[1]\n\n        rrf_score = (1.0 / (rrf_k + vector_rank)) + (1.0 / (rrf_k + fts_rank))\n        text = vector_ranks.get(doc_id, fts_ranks.get(doc_id))[0]\n        fused_scores[doc_id] = (text, rrf_score)\n\n    # Sort by fused score descending\n    results = [(doc_id, text, score) \n               for doc_id, (text, score) in fused_scores.items()]\n    results.sort(key=lambda x: x[2], reverse=True)\n\n    return results[:k]\n```\n\nAfter these changes:\n\nWhen evaluating RAGFlow or any vector-backed RAG system, stress-test your memory layer early. Index strategy and embedding hygiene matter more than model selection.\n\nThis article reflects genuine production debugging. No affiliate links, no sponsored tools. Just PostgreSQL, Python, and the scars from a latency crisis.", "url": "https://wpnews.pro/news/vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer", "canonical_source": "https://dev.to/samuel_oakley/vector-search-latency-crisis-what-i-learned-integrating-ragflows-memory-layer-3nog", "published_at": "2026-10-17 22:28:54+00:00", "updated_at": "2026-10-17 22:37:31.323961+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["RAGFlow", "pgvector", "PostgreSQL", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer", "markdown": "https://wpnews.pro/news/vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer.md", "text": "https://wpnews.pro/news/vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer.txt", "jsonld": "https://wpnews.pro/news/vector-search-latency-crisis-what-i-learned-integrating-ragflow-s-memory-layer.jsonld"}}