# An event-driven AI pipeline using FastAPI, Redpanda, and Docker

> Source: <https://github.com/Infodatamatrix/AIKafkaPipelineDemo>
> Published: 2026-06-18 04:35:12+00:00

A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain.

```
ai-kafka-pipeline-demo/
├── api-gateway/
│   └── app/
│       ├── main.py
│       ├── routes/submit.py
│       ├── services/publisher.py
│       └── config.py
├── workers/
│   ├── extractor/
│   ├── summarizer/
│   └── notifier/
├── shared/
│   ├── kafka/
│   ├── schemas/
│   ├── config/
│   └── utils/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env
```

- POST
`/submit`

to the API gateway - Gateway publishes
`document.submitted`

- Extractor consumes and publishes
`text.extracted`

- Summarizer consumes and publishes
`summary.generated`

- Notifier consumes and logs final completion

```
docker compose up --build
```

Open another terminal:

```
curl -X POST http://localhost:8000/submit \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": "user-1",
    "content": "Kafka helps decouple AI pipeline stages for scalable processing in production systems."
  }'
```

- API returns
`Processing started`

- Extractor logs the incoming event
- Summarizer logs the next event
- Notifier logs the final pipeline completion

- FastAPI handles intake, not heavy processing.
- Kafka turns the request into an event.
- Each worker owns one stage.
- Shared schemas keep the contracts explicit.
- This is the simplest form of a production-style AI pipeline.
- I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here:
[https://youtu.be/c2ijN2KAWXw](https://youtu.be/c2ijN2KAWXw) [https://youtu.be/KjvbABpajUs](https://youtu.be/KjvbABpajUs)