Six months of operating an on-prem
If I were rebuilding today with the same privacy constraints:
RAGpipeline for internal document search taught me more than two years of prototypes ever did. The gap between "works in a notebook" and "serves 200 queries a minute at p99 < 800ms" is not a gap — it's a chasm.What we run: vLLM for LLM serving, Qdrant for vector search, Airflow for orchestration, and a custom evaluation loop with Langfuse for tracing. Workload is internal knowledge retrieval over ~2M legal documents, with a Llama-3.1-70B model. I'd recommend all of them if you have the ops capacity; if you don't, use managed versions and save your sanity.Self-host vs managed: We self-host inference and vector search, but moved observability to a managed service. Self-hosting the rest was a choice forced by data privacy rules, not cost savings.Tools we abandoned: We killed Redis-based semantic caching after two weeks. It kept returning stale embeddings after model updates, and debugging the invalidation logic cost more than the latency it saved. Also dropped Ray for batch inference — the cluster autoscaling was great until it wasn't; a simple argo-workflows queue did the job with 10x less complexity.Problems that only appear post-prototype: Memory leaks in vLLM under continuous batching. They don't show up in load tests under 30 minutes. We hit OOM after 6 hours and had to wrap the process with a watchdog that restarts on a memory threshold. Another one: embedding drift. The same prompt started returning different results after a minor embedding model update, and no one noticed until a user complained about "wrong answers" for two days. You need a golden dataset pinned to every model version, or you'll chase ghosts.Tool operations overhead: Observability is the silent killer. We spent three days instrumenting vLLM metrics into Prometheus, only to realize we couldn't trace a bad answer back to the exact prompt stack without Langfuse-style tracing. Now every model request carries a trace ID from ingest to output. Do that from day one.What I'd redo: I'd start with a managed vector DB instead of self-hosting Qdrant. Replication plus disk snapshots plus backup verification wasted a week. Also, I'd build a proper multi-tenant quota system from the start. Our "low priority" batch jobs started starving interactive requests at 3x concurrency. Turns out, unbounded queues in front of a shared inference endpoint is a problem you inherit, not build.
If I were rebuilding today with the same privacy constraints:
serving = vLLM
vector_db = managed_qdrant # or an API service
orchestration = argo_workflows
observability = langfuse + prometheus
eval = custom pytest-style harness with golden set
The single biggest advice: put evaluation and traceability into the CI/CD pipeline before you add any other feature. Production failures are mostly silent — embedding drift, memory leaks, stale caches. If you can't detect those automatically, you're not operating infrastructure, you're just babysitting it.
Story tracker · related coverage
Open-Weight Models Now Match Proprietary Titans 4d ago
Circular AI Deals: The Truth Behind the Intelligence Trade 5d ago
World AI Conference 2026: Key Strategic Takeaways 5d ago
AI Influence Strategies: Analyzing Model Narratives 5d ago
Cadence Money: A Budgeting Tool with an MCP Server 5d ago
Next AGs Order OpenAI to Preserve Hugging Face Hack Records →
Hands-on notes on AI tools and LLMs are collected in
a library of Claude prompt techniques, with plenty of directly applicable cases.## All Replies （3）
F
What did you end up using for vector search? We're struggling with latency at scale.
0
M
We learned the hard way that caching query results beats optimizing the model. Huge win for us.
0
N
Same here — my prototype fell apart as soon as real users, auth, and retries showed up.
0