{"slug": "how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline", "title": "How I bypassed Vercel Serverless timeouts to build a decoupled document ingestion pipeline", "summary": "A developer built a decoupled document ingestion pipeline to bypass Vercel Serverless timeout limitations for asynchronous processing tasks. The architecture uses Next.js API routes for request validation and file storage, BullMQ with TCP Redis for background job queuing, and a persistent Node.js worker on Railway to handle PDF parsing, semantic chunking, and batch embedding without serverless execution constraints. The system also implements a Stateless Pass-Through Mode for data privacy, processing documents and streaming results via webhook with zero data retention.", "body_md": "If you’ve ever tried to build an asynchronous document processing or RAG pipeline using Next.js API routes hosted on Vercel, you know that even with max duration configuration adjustments, keeping intensive computing tasks entirely inline on serverless routes can get messy.\n\nWhen a user uploads large PDFs or batch data sources, parsing the text layers, chunking them semantically, and running batch embedding requests consumes serious time. Relying on synchronous, API-facing function execution windows for deep I/O tasks often leaves you managing brittle state.\n\nTo make my application, I spent my time decoupling my stack into a clean, asynchronous background processing worker architecture. Here is a breakdown of how the data flows.\n\nThe Stack Architecture\n\nIngress Layer (Next.js): The API endpoints strictly handle incoming request validation, file storage organization, and API idempotency keys (managed via Upstash Redis).\n\nThe Worker Queue (BullMQ + TCP Redis): Instead of processing files inline, the Next.js route enqueues the task into a BullMQ background line. Because BullMQ requires a persistent, low-latency binary TCP connection, this Redis instance is hosted directly on Railway alongside our workers.\n\nPersistent Worker Instance: A standalone Node.js background process running on Railway listens to the BullMQ stream. Because it runs on a dedicated server environment, it completely removes the headache of managing serverless execution constraints. It streams files from Cloudflare R2, runs semantic paragraph chunking, and processes text embeddings in parallel batches.\n\nConcurrency and Data Privacy Safety\n\nHandling high-volume API requests means designing for deep multi-tenant safety. To prevent concurrent race conditions across quota meters during multi-file processing, the platform utilizes a Postgres SELECT FOR UPDATE block inside an explicit database transaction to lock and update user tokens safely at the database layer.\n\nFurthermore, to solve data privacy and compliance hurdles for teams that do not want third-party database lock in, I implemented a strict Stateless Pass-Through Mode. By sending a passthrough: true flag to the endpoint, the background worker processes the document, generates the raw 1,536 dimension float arrays using OpenAI, streams the payload back via an asynchronous webhook, and instantly flushes the server RAM. Zero data retention.\n\nOpen Beta & Feedback\n\nI've packaged this entire decoupled pipeline layout into a developer utility called ContextFlow AI.\n\nThe public beta is completely live, open, and free to try. You can check out the landing page, read the documentation, and inspect the JSON payload schemas directly at [https://usecontextflow.com](https://usecontextflow.com).\n\nI'd love to get your thoughts on the webhook event schemas or how you are structuring background queues for your own AI applications!", "url": "https://wpnews.pro/news/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline", "canonical_source": "https://dev.to/edwin_93a122d31bc978aa64a/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-ingestion-pipeline-2p0j", "published_at": "2026-05-30 20:17:55+00:00", "updated_at": "2026-05-30 20:41:37.763561+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "ai-products", "ai-startups"], "entities": ["Vercel", "Next.js", "BullMQ", "Redis", "Upstash", "Railway", "Cloudflare"], "alternates": {"html": "https://wpnews.pro/news/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline", "markdown": "https://wpnews.pro/news/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline.md", "text": "https://wpnews.pro/news/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline.txt", "jsonld": "https://wpnews.pro/news/how-i-bypassed-vercel-serverless-timeouts-to-build-a-decoupled-document-pipeline.jsonld"}}