Semantic Image Search with Elasticsearch

A tutorial from SerpApi and Jina AI demonstrates building a semantic image search pipeline that retrieves Instagram photos by visual similarity rather than caption text, using SerpApi's Instagram Profile API to fetch images, Jina AI's embedding model to convert images and text into a shared vector space, and Elasticsearch to store and search vectors with cosine similarity. The pipeline enables queries like "a dog wearing sunglasses" to return matching images ranked by visual meaning, bypassing the need for keyword-based captions.

Searching Instagram by hashtag or caption text misses the point when the content is visual. A photo of a Samoyed in the snow doesn't always say " Samoyed " in the caption, but you'd still want to find it by typing "a white fluffy dog in the snow." This tutorial builds a pipeline that searches images by what they look like, not what the caption says. We'll fetch a public profile's feed with SerpApi's Instagram Profile API https://serpapi.com/instagram-profile-api , embed every cover image into a shared text-image vector space with Jina AI https://jina.ai/ , store those vectors in Elasticsearch https://www.elastic.co/ , and search with plain language. By the end, you'll have a working search interface where typing " a dog wearing sunglasses " returns the matching image ranked by visual similarity, no caption needed. The complete project, notebook and Streamlit app, is on GitHub: Prerequisites - Python 3.9+ SerpApi API key https://serpapi.com/manage-api-key free tier, 250 searches/month Jina AI API key https://jina.ai/api-dashboard/key-manager free tier, 10 million tokens - Elasticsearch cluster 9.0 or higher - ~2 GB disk space for downloaded images Structured Image Search at Scale The hard part of image search has never been the vector math. It's getting the images in the first place. Public Instagram profiles show thousands of posts, but there's no "download all" button and no official bulk API from Instagram. SerpApi Instagram Profile API https://serpapi.com/blog/how-to-scrape-instagram-profile-data-with-serpapi/ solves the access problem. One call with engine="instagram profile" and a username returns the profile's posts along with captions, like counts, comment counts, and a direct URL to each cover image. That means a few dozen API calls give us hundreds of high-quality images with their metadata, ready to embed. We'll use the @apple https://serpapi.com/playground?engine=instagram profile&profile id=apple profile as our example. The feed spans landscapes, portraits, animals, and architectural photos, which gives the search plenty of visual variety to work with. How It Works Four steps take us from a username to searchable images. Fetch: Page through the profile's feed with SerpApi, collecting cover images images + video thumbnails . Embed: Convert each image into a vector that captures its visual meaning, using Jina AI. Index: Store the vector plus the metadata in Elasticsearch dense vector field. Search: Type a query and get the ranked images by similarity. What Are Embeddings? An embedding https://www.ibm.com/think/topics/embedding is a list of numbers a vector that captures what an image means, its visual features, objects, colors, and composition. Similar images produce vectors close together in this high-dimensional space; different images produce vectors far apart. The key that makes text-to-image search possible is that models like Jina V5 Omni https://jina.ai/models/jina-embeddings-v5-omni-small/ are trained on image-text pairs, so they map both images and text into the same space. A photo of a sunset and the sentence " a person at sunset " end up near each other, not because of keywords but because the model learned their meaning is similar. How Vector Search Works Once all images are embedded and stored, the search flow is straightforward. - Jina embeds your text query " a dog wearing sunglasses " into a vector using the same model. - Elasticsearch compares that query vector against every stored image vector using cosine similarity https://www.ibm.com/think/topics/cosine-similarity , a measure of how close two vectors point in the same direction. - The k nearest neighbors are returned, ranked by similarity score. Scores closer to 1.0 mean more similar; scores closer to 0 mean unrelated. The query never touches the captions. The ranking is purely visual. Why These Three Tools? Each tool handles one stage of the pipeline. | Component | Role | Why this one | |---|---|---| | Jina v5 Omni https://jina.ai/models/jina-embeddings-v5-omni-small/ Elasticsearch https://www.elastic.co/ dense vector field with built-in kNN, exact and approximate search out of the box, scales from hundreds to millions of vectorsTwo design choices are worth calling out. One model for everything: Jina v5 Omni handles both images and text in a single pipeline. If you later want to add semantic search over captions or post metadata, the same model and the same index work. No second model, no separate pipeline. No raw images stored in the database: We store only the vector, a local file path, and metadata caption, post URL, likes . Images live in a local images/ folder, just long enough to embed and display results. What Is Elasticsearch? If you haven't used Elasticsearch https://www.elastic.co/elasticsearch before, it's a search engine the same technology behind site search on large websites that also handles vector search natively. You store documents; each document has fields, and you query against those fields. For our use case, one of those fields is a vector, a long list of numbers that represents the visual content of an image. The relevant feature is vector search via the knn retriever https://www.elastic.co/docs/solutions/search/vector/knn . Given a query vector, it finds the closest stored vectors and returns them ranked by similarity. That's our image search. Two Ways to Run It Two options for the Elasticsearch cluster. The notebook code is identical either way. Only ES URL and ES API KEY in .env change. Local Docker We start a local single-node cluster using the Elastic start-local script https://www.elastic.co/docs/deploy-manage/deploy/self-managed/local-development-installation-quickstart : curl -fsSL https://elastic.co/start-local | sh This prints an ES LOCAL API KEY . Put it in your .env as ES API KEY . The endpoint is http://localhost:9200 . Elastic Cloud Create a free project on Elastic Cloud https://www.elastic.co/cloud/elasticsearch-service/signup 14-day free trial . Copy the endpoint URL and API key into .env . Good if your machine is light on RAM or you want persistence without managing Docker. The Pipeline Step 1. Fetch the Profile We'll page through the profile feed using the SerpApi Python client https://serpapi.com/integrations/python : python import serpapi serp client = serpapi.Client api key=os.getenv "SERPAPI API KEY" , timeout=30 base = {"engine": "instagram profile", "profile id": "apple"} params = dict base for page in range 1, max pages + 1 : results = serp client.search params profile = results.get "profile results", {} posts = profile.get "posts", next token = results.get "serpapi pagination", {} .get "next page token" if not next token: break params = { base, "next page token": next token} Each request returns about 12 posts along with a next page token , which we send in the next call to get the next batch. When no token comes back, we've reached the end of the feed. So a few dozen calls are enough to pull hundreds of images. Step 2. Embed the Images Here we are sending the images downloaded from the Instagram API to the Jina Embedding API as base64, receiving one vector per image: JINA URL = "https://api.jina.ai/v1/embeddings" JINA MODEL = "jina-embeddings-v5-omni-small" def embed images images b64 : inputs = {"image": f"data:image/jpeg;base64,{b}"} for b in images b64 payload = {"model": JINA MODEL, "task": "retrieval.passage", "dimensions": 1024, "input": inputs} r = requests.post JINA URL, headers=jina headers, json=payload, timeout=120 r.raise for status data = sorted r.json "data" , key=lambda d: d "index" return d "embedding" for d in data The one detail worth calling out is task="retrieval.passage" . Jina uses different task modes for the vectors you store retrieval.passage and the vectors you search with retrieval.query . Pairing them this way is what Jina recommends https://jina.ai/models/jina-embeddings-v5-omni-small/ :~:text=Best%20Practice for retrieval. Step 4 handles the query side. Step 3. Create the Index We create an Elasticsearch index with a vector field to store the embeddings: es.indices.create index="instagram photos", mappings={"properties": { "embedding": {"type": "dense vector", "dims": 1024, "similarity": "cosine"}, "caption": {"type": "text"}, "shortcode": {"type": "keyword"}, "username": {"type": "keyword"}, ... post url, image url, is video, liked by count, comments count }} We use cosine similarity because it's the standard match for text and image embeddings, and 1024 dimensions matches Jina's output. See the notebook for the full mapping. Step 4. Search We embed the query with Jina this time with task="retrieval.query" and pass the vector to Elasticsearch's kNN retriever https://www.elastic.co/docs/solutions/search/vector/knn : python def search query, username, k=6 : query vector = embed query query resp = es.search index="instagram photos", retriever={"knn": { "field": "embedding", "query vector": query vector, "k": k, "filter": {"term": {"username": username}}, }}, size=k, source excludes= "embedding" , return resp "hits" "hits" The filter scopes results in a single profile even when the index holds many. source excludes keeps the response lean by dropping the 1024-float vector we don't need back. Results With 577 images indexed from the Apple profile, the search delivers exactly what you'd expect. The top result is a video thumbnail poster frame from Apple's feed, a dog in sunglasses, scored at 0.737 cosine similarity. No caption matching. No keyword overlap. The pipeline matched the visual content of the image to the meaning of the text query. A dog wearing sunglasses More examples from the same index: Snow-capped mountains Horseback Riding Someone holding a phone A close-up portrait Beyond Instagram The embedding and search code doesn't change. Only the SerpApi engine and the fields you extract differ. The same architecture applies to any visual data source SerpApi can reach. | Source | SerpApi engine | Image field | What you'd search | |---|---|---|---| | youtube thumbnail.static Google Images https://serpapi.com/images-results google images original Google Shopping https://serpapi.com/shopping-results google shopping thumbnail Google Lens https://serpapi.com/google-lens-api google lens thumbnail Amazon Search https://serpapi.com/amazon-search-api amazon thumbnail Where to Go from Here Concrete next steps to extend this pipeline: Hybrid search combining captions and embeddings: Add a text query over the caption field alongside the vector search using Elasticsearch's RRF Reciprocal Rank Fusion https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion . Posts where both the image and the caption match will rank higher. Index multiple profiles: The username field already supports multi-profile indexing. Fetch a second profile, embed its images, and use the filter to search within or across profiles. Try other SerpApi engines from the table above: Swap the fetch step for YouTube thumbnails or Google Shopping product photos. The embedding and search code stays identical. Conclusion This pipeline reads images the way people describe them, not the way someone happened to describe or tag them. That single capability changes what you can do with an image catalog. The Instagram example is just a demonstration. An e-commerce store can surface products by what they look like, not by category tags. A media archive can retrieve photos by scene, mood, or subject. An LLM agent can query the same index in plain language and reason over the matches, no captions or manual tagging required. Both sides of the search now carry meaning. The query is interpreted by what it means, and the images are stored by what they show. That opens a second layer of interaction between your users, or an LLM, and your content.