{"slug": "the-qdrant-output-connector", "title": "The Qdrant Output Connector", "summary": "Qdrant, the Rust-based vector database, has been integrated into OpenCrawling's event-driven microservice architecture via a new Qdrant Output Connector that enables sub-millisecond ACL pre-filtering for enterprise RAG, ensuring users only retrieve documents they have explicit rights to see. The connector, which uses the gRPC Java SDK over port 6334, automatically provisions collections and creates KEYWORD payload indexes on security fields such as security_allowed_read and security_denied_read, and is configurable via Spring Boot YAML properties or environment variables.", "body_md": "## 01. Why Qdrant for Enterprise Vector Search?\n\nAs AI applications expand across enterprise ecosystems, vector storage demands extreme query throughput, low memory footprint, and strict security guarantees. While vector databases excel at similarity search, enterprise content repositories (like SharePoint, Alfresco, and Iceberg) require document-level Access Control Lists (ACLs) to ensure users only retrieve information they have explicit rights to see.\n\n**Qdrant** is engineered in Rust with SIMD hardware acceleration, making it one of the fastest vector engines in existence. But what truly sets Qdrant apart for enterprise RAG is its **Payload Index Engine**. By supporting `KEYWORD`\n\npayload indexes directly alongside vector points, Qdrant allows filtering by payload attributes (such as user security SIDs) *before* distance calculations occur, preventing security leaks without sacrificing search performance.\n\n**Sub-Millisecond ACL Pre-Filtering:** Qdrant's payload index allows OpenCrawling's `McpVectorServer`\n\nto inject document ACL filters directly into similarity queries, ensuring zero unauthorized vector returns.\n\n## 02. Architecture & Decoupled Data Flow\n\nThe `oc-qdrant-output-connector`\n\nintegrates directly into OpenCrawling's event-driven decoupled microservice architecture:\n\n```\nQdrant Output Connector (gRPC)\n```\n\n**Repository Ingestion:** Repository connectors (Filesystem, Alfresco, Flowable, Camunda, S3) discover documents and publish lightweight claim-check messages to Kafka.**Text Extraction & Chunking:**`IngestionConsumer`\n\nextracts clean text via Apache Tika and generates chunked payloads with associated document ACL security SIDs.**Scalable Embedding:**`oc-embedding-service`\n\ndynamically computes vector embeddings (via Ollama or OpenAI) and pushes embedded chunks to Kafka.**High-Speed Binary Upsert:**`QdrantStoreWriterConsumer`\n\nreceives the embedded chunks and uses the gRPC Java SDK (`io.qdrant:client`\n\n) to upsert`PointStruct`\n\nbatches into Qdrant over port`6334`\n\n.\n\n## 03. Automated Collection Provisioning & Payload ACL Indexing\n\nUpon startup, `QdrantCollectionInitializer`\n\nverifies whether the configured target collection (e.g. `enterprise_kb`\n\n) exists. If not, it automatically provisions the collection with the desired vector dimensions and distance metric (`COSINE`\n\n, `DOT`\n\n, or `EUCLID`\n\n).\n\nSimultaneously, the connector creates `KEYWORD`\n\npayload indexes on the security payload fields:\n\n`security_allowed_read`\n\n: List of user/group SIDs permitted to access the document.`security_denied_read`\n\n: Explicit deny SIDs taking precedence over allow rules.\n\n```\n// Example Qdrant Point Payload Structure generated by oc-qdrant-output-connector\n{\n  \"id\": \"c8a912e4-712d-4b92-b2fa-1092a9f1a091\",\n  \"vector\": [0.0142, -0.0521, 0.0892, ...],\n  \"payload\": {\n    \"text\": \"Enterprise quarterly financial report and forecasts...\",\n    \"uri\": \"file:///data/finance/q3_report.pdf\",\n    \"security_allowed_read\": [\"ROLE_FINANCE\", \"USER_S-1-5-21-3623811015\"],\n    \"security_denied_read\": [\"ROLE_EXTERN\"],\n    \"lastModified\": \"2026-07-29T18:00:00Z\"\n  }\n}\n```\n\n## 04. Configuration & Deployment\n\nThe Qdrant Output Connector is fully configurable via standard Spring Boot YAML properties or environment variables:\n\n| Property | Environment Variable | Default | Description |\n|---|---|---|---|\n`spring.opencrawling.output.qdrant.host` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_HOST` |\n`localhost` |\nHostname of self-hosted Qdrant or Qdrant Cloud instance |\n`spring.opencrawling.output.qdrant.port` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_PORT` |\n`6334` |\nHigh-performance gRPC port (default: 6334) |\n`spring.opencrawling.output.qdrant.api-key` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_API_KEY` |\n`\"\"` |\nOptional API key for Qdrant Cloud or Enterprise cluster |\n`spring.opencrawling.output.qdrant.collection-name` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_COLLECTION_NAME` |\n`enterprise_kb` |\nTarget Qdrant vector collection name |\n`spring.opencrawling.output.qdrant.dimensions` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_DIMENSIONS` |\n`1024` |\nVector dimension size (e.g. 1024 for `mxbai-embed-large` ) |\n`spring.opencrawling.output.qdrant.distance` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_DISTANCE` |\n`COSINE` |\nVector distance metric (`COSINE` , `DOT` , `EUCLID` ) |\n`spring.opencrawling.output.qdrant.quantization` |\n`SPRING_OPENCRAWLING_OUTPUT_QDRANT_QUANTIZATION` |\n`NONE` |\nQuantization mode (`NONE` , `SCALAR` , `BINARY` ) |\n\n**Automated Integration Test:** You can test the end-to-end Qdrant pipeline locally with one command: `./scripts/test-qdrant-decoupled.sh`\n\n. The script spins up Qdrant, Ollama, Kafka, and OpenCrawling microservices, injects a test document, and verifies point count in Qdrant.\n\n## 05. Get Started Today\n\nThe Qdrant Output Connector is available starting in OpenCrawling `1.0.0-SNAPSHOT`\n\n. Check out the documentation or run the Docker Compose environment to start building secure, low-latency RAG pipelines with Qdrant!\n\n### Ready to Supercharge Your RAG Architecture with Qdrant?\n\nExplore the source code on GitHub, read the Wiki configuration guide, or join our community discussions.", "url": "https://wpnews.pro/news/the-qdrant-output-connector", "canonical_source": "https://opencrawling.org/blog/introducing-qdrant-output-connector.html", "published_at": "2026-08-14 16:35:18+00:00", "updated_at": "2026-08-14 17:50:19.324022+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-products", "ai-tools"], "entities": ["Qdrant", "OpenCrawling", "Apache Tika", "Ollama", "OpenAI", "Kafka", "Spring Boot", "Qdrant Cloud"], "alternates": {"html": "https://wpnews.pro/news/the-qdrant-output-connector", "markdown": "https://wpnews.pro/news/the-qdrant-output-connector.md", "text": "https://wpnews.pro/news/the-qdrant-output-connector.txt", "jsonld": "https://wpnews.pro/news/the-qdrant-output-connector.jsonld"}}