An event-driven AI pipeline using FastAPI, Redpanda, and Docker

A developer released a demo AI pipeline using FastAPI, Redpanda (Kafka-compatible), and Docker, where an API gateway publishes events consumed by separate workers for extraction, summarization, and notification. The project illustrates decoupling AI pipeline stages for scalable processing.

A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain. ai-kafka-pipeline-demo/ ├── api-gateway/ │ └── app/ │ ├── main.py │ ├── routes/submit.py │ ├── services/publisher.py │ └── config.py ├── workers/ │ ├── extractor/ │ ├── summarizer/ │ └── notifier/ ├── shared/ │ ├── kafka/ │ ├── schemas/ │ ├── config/ │ └── utils/ ├── Dockerfile ├── docker-compose.yml ├── requirements.txt └── .env - POST /submit to the API gateway - Gateway publishes document.submitted - Extractor consumes and publishes text.extracted - Summarizer consumes and publishes summary.generated - Notifier consumes and logs final completion docker compose up --build Open another terminal: curl -X POST http://localhost:8000/submit \ -H "Content-Type: application/json" \ -d '{ "user id": "user-1", "content": "Kafka helps decouple AI pipeline stages for scalable processing in production systems." }' - API returns Processing started - Extractor logs the incoming event - Summarizer logs the next event - Notifier logs the final pipeline completion - FastAPI handles intake, not heavy processing. - Kafka turns the request into an event. - Each worker owns one stage. - Shared schemas keep the contracts explicit. - This is the simplest form of a production-style AI pipeline. - I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here: https://youtu.be/c2ijN2KAWXw https://youtu.be/c2ijN2KAWXw https://youtu.be/KjvbABpajUs https://youtu.be/KjvbABpajUs