{"slug": "day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai", "title": "Day 22 of 60: I Built a Production Background Task Pipeline That Processes AI Jobs Asynchronously", "summary": "A developer built a production background task pipeline using FastAPI and PostgreSQL that processes AI document jobs asynchronously, returning a job ID in milliseconds while Claude handles summarization, data extraction, and evaluation tasks in the background. The pipeline supports three task types—summarize, extract, and evaluate—with full audit trails stored as JSONB in PostgreSQL, and optional webhook support eliminates the need for client polling. The system is designed for real-world use cases including contract review, loan application screening, and insurance claims processing.", "body_md": "** The Problem**\n\nProcessing large documents through an AI model takes 30-60 seconds. A synchronous API makes the client wait the entire time. Browsers timeout. Users think it crashed. The experience feels broken.\n\nYesterday I learned why this happens. Today I built the fix.\n\n**## What I Built**\n\nA document processing pipeline with FastAPI and PostgreSQL that handles long-running AI tasks in the background. Three task types. Immediate response. Full audit trail.\n\nClient submits a document and gets a job ID in milliseconds. Claude processes it in the background. Client polls for the result when ready. Everything logged to PostgreSQL.\n\n**## Three Task Types**\n\nSummarise: submit a document and receive a structured summary with executive overview and key points. Focus area is configurable.\n\nExtract: specify which fields you want pulled out. Company name, net profit, revenue, key risks. Claude reads the document and returns exactly what you asked for as structured JSON.\n\nEvaluate: provide a list of criteria and Claude checks whether the document meets each one. Returns a pass or fail with reasoning for every criterion.\n\n**## How the Pipeline Works\n**\n\n```\nPOST /jobs/summarise\n↓\nJob ID returned in milliseconds\nstatus: pending\n↓\nClaude processes in background\nstatus: running\n↓\nGET /jobs/{job_id}\nstatus: completed\nfull result ready\n```\n\nOptional webhook support means the server can call your endpoint when the job completes and no polling needed.\n\n**## The Database Behind It**\n\nEvery job is stored in PostgreSQL with full audit trail:\n\nJob ID, status at every stage, task type, input data as JSONB, result as JSONB, error message if failed, webhook URL, created timestamp, completed timestamp.\n\nIf a job fails the error is recorded for debugging. Nothing disappears silently.\n\n**## Real Use Cases**\n\nContract review for law firms. Loan application screening for SACCOs. Insurance claims processing. Report summarisation for enterprises.\n\nThe same pipeline. Different documents. Different criteria. One system.\n\n**## What I Learned**\n\nFastAPI's BackgroundTasks is the simplest way to separate submission from processing. The endpoint returns immediately. The heavy work happens after the response is sent.\n\nJSONB in PostgreSQL stores the full input and result as structured data — not a string you have to parse later. Query it, index it, use it.\n\nWebhooks flip the polling model entirely. Instead of the client asking if the job is done, the server tells the client when it is ready.\n\n🔗 Full project on GitHub → [https://github.com/mbuguacessy-glitch](https://github.com/mbuguacessy-glitch)\n\n38 more to go.", "url": "https://wpnews.pro/news/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai", "canonical_source": "https://dev.to/mbugua_cessy/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai-jobs-asynchronously-3na3", "published_at": "2026-05-29 11:06:58+00:00", "updated_at": "2026-05-29 11:11:30.868635+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-products", "ai-tools", "large-language-models", "mlops"], "entities": ["FastAPI", "PostgreSQL", "Claude"], "alternates": {"html": "https://wpnews.pro/news/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai", "markdown": "https://wpnews.pro/news/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai.md", "text": "https://wpnews.pro/news/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai.txt", "jsonld": "https://wpnews.pro/news/day-22-of-60-i-built-a-production-background-task-pipeline-that-processes-ai.jsonld"}}