An event-driven AI pipeline using FastAPI, Redpanda, and Docker

A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain.

ai-kafka-pipeline-demo/
├── api-gateway/
│   └── app/
│       ├── main.py
│       ├── routes/submit.py
│       ├── services/publisher.py
│       └── config.py
├── workers/
│   ├── extractor/
│   ├── summarizer/
│   └── notifier/
├── shared/
│   ├── kafka/
│   ├── schemas/
│   ├── config/
│   └── utils/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env

• POST /submit

to the API gateway - Gateway publishes document.submitted

• Extractor consumes and publishes text.extracted

• Summarizer consumes and publishes summary.generated

• Notifier consumes and logs final completion

docker compose up --build

Open another terminal:

curl -X POST http://localhost:8000/submit \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": "user-1",
    "content": "Kafka helps decouple AI pipeline stages for scalable processing in production systems."
  }'

• API returns Processing started

• Extractor logs the incoming event

• Summarizer logs the next event

• Notifier logs the final pipeline completion

• FastAPI handles intake, not heavy processing.

• Kafka turns the request into an event.

• Each worker owns one stage.

• Shared schemas keep the contracts explicit.

• This is the simplest form of a production-style AI pipeline.

• I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here: https://youtu.be/c2ijN2KAWXw https://youtu.be/KjvbABpajUs