{"slug": "an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker", "title": "An event-driven AI pipeline using FastAPI, Redpanda, and Docker", "summary": "A developer released a demo AI pipeline using FastAPI, Redpanda (Kafka-compatible), and Docker, where an API gateway publishes events consumed by separate workers for extraction, summarization, and notification. The project illustrates decoupling AI pipeline stages for scalable processing.", "body_md": "A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain.\n\n```\nai-kafka-pipeline-demo/\n├── api-gateway/\n│   └── app/\n│       ├── main.py\n│       ├── routes/submit.py\n│       ├── services/publisher.py\n│       └── config.py\n├── workers/\n│   ├── extractor/\n│   ├── summarizer/\n│   └── notifier/\n├── shared/\n│   ├── kafka/\n│   ├── schemas/\n│   ├── config/\n│   └── utils/\n├── Dockerfile\n├── docker-compose.yml\n├── requirements.txt\n└── .env\n```\n\n- POST\n`/submit`\n\nto the API gateway - Gateway publishes\n`document.submitted`\n\n- Extractor consumes and publishes\n`text.extracted`\n\n- Summarizer consumes and publishes\n`summary.generated`\n\n- Notifier consumes and logs final completion\n\n```\ndocker compose up --build\n```\n\nOpen another terminal:\n\n```\ncurl -X POST http://localhost:8000/submit \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"user_id\": \"user-1\",\n    \"content\": \"Kafka helps decouple AI pipeline stages for scalable processing in production systems.\"\n  }'\n```\n\n- API returns\n`Processing started`\n\n- Extractor logs the incoming event\n- Summarizer logs the next event\n- Notifier logs the final pipeline completion\n\n- FastAPI handles intake, not heavy processing.\n- Kafka turns the request into an event.\n- Each worker owns one stage.\n- Shared schemas keep the contracts explicit.\n- This is the simplest form of a production-style AI pipeline.\n- I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here:\n[https://youtu.be/c2ijN2KAWXw](https://youtu.be/c2ijN2KAWXw) [https://youtu.be/KjvbABpajUs](https://youtu.be/KjvbABpajUs)", "url": "https://wpnews.pro/news/an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker", "canonical_source": "https://github.com/Infodatamatrix/AIKafkaPipelineDemo", "published_at": "2026-06-18 04:35:12+00:00", "updated_at": "2026-06-18 04:52:45.463115+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["FastAPI", "Redpanda", "Docker", "Kafka"], "alternates": {"html": "https://wpnews.pro/news/an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker", "markdown": "https://wpnews.pro/news/an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker.md", "text": "https://wpnews.pro/news/an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker.txt", "jsonld": "https://wpnews.pro/news/an-event-driven-ai-pipeline-using-fastapi-redpanda-and-docker.jsonld"}}