Visualizing how multimodal vector search works under the hood

A new open-source project demonstrates how multimodal vector search works by combining OpenAI's CLIP model with Qdrant's vector database to enable text-to-image and image-to-image search. The system embeds both text and images into a shared 512-dimensional vector space and includes a real-time UI animation that visualizes the AI processing pipeline from tokenization to ranked results.

Search images using natural language or image upload — powered by OpenAI CLIP and Qdrant vector search. This project was built from scratch to explore how modern image search works under the hood. Instead of relying on manual tags or filenames, this app "understands" visual content by embedding both text and images into a shared 512-dimensional vector space. Text-to-Image Search: Describe what you are looking for e.g., "sunset over mountains" . Image-to-Image Search: Upload an image to find visually similar ones. Pipeline Visualization: A real-time UI animation that shows the AI processing your query step-by-step: Tokenization → CLIP Encoder → Vector Embedding → Qdrant Search → Ranked Results. | Component | Technology | |---|---| Embedding Model | OpenAI CLIP ViT-B/32 | Vector Database | Qdrant HNSW indexing | Frontend/UI | HTML / CSS / Vanilla JS | Backend | FastAPI, Python PyTorch, Transformers | Follow these steps to run the project locally on your machine. Python 3.10+ Docker to run the local Qdrant database Open a terminal and start a local Qdrant instance using Docker: docker run -p 6333:6333 -p 6334:6334 \ -v $ pwd /qdrant storage:/qdrant/storage:z \ qdrant/qdrant Open a new terminal window in the project directory, create a virtual environment, and install the required packages: Create and activate virtual environment python3 -m venv .venv source .venv/bin/activate Install dependencies pip install -r requirements.txt The repository includes a pre-exported file of vectors data/vectors.json containing 541 sample images. Start the FastAPI server: uvicorn app.api:app --host 0.0.0.0 --port 8000 Note: On the very first run, the app will automatically read data/vectors.json and import the vectors into your local Qdrant database. It will also download the CLIP model weights. Open your browser and navigate to: http://localhost:8000 http://localhost:8000 Type a query or upload an image, and watch the pipeline animate as it searches app/api.py : FastAPI server routes and static file serving. ui/app.html : Custom HTML/JS frontend with the animated pipeline. app/embeddings.py : CLIP model initialization and inference code. app/vector store.py : Qdrant client connection and search logic. app/config.py : Configuration settings and environment variables. scripts/ : Utilities for downloading datasets and importing/exporting vectors. data/ : Contains the image dataset and the exported vectors JSON.