{"slug": "semantic-image-search-with-elasticsearch", "title": "Semantic Image Search with Elasticsearch", "summary": "A tutorial from SerpApi and Jina AI demonstrates building a semantic image search pipeline that retrieves Instagram photos by visual similarity rather than caption text, using SerpApi's Instagram Profile API to fetch images, Jina AI's embedding model to convert images and text into a shared vector space, and Elasticsearch to store and search vectors with cosine similarity. The pipeline enables queries like \"a dog wearing sunglasses\" to return matching images ranked by visual meaning, bypassing the need for keyword-based captions.", "body_md": "Searching Instagram by hashtag or caption text misses the point when the content is visual. A photo of a Samoyed in the snow doesn't always say \"*Samoyed*\" in the caption, but you'd still want to find it by typing \"a white fluffy dog in the snow.\" This tutorial builds a pipeline that searches images by what they look like, not what the caption says.\n\nWe'll fetch a public profile's feed with [SerpApi's Instagram Profile API](https://serpapi.com/instagram-profile-api), embed every cover image into a shared text-image vector space with [Jina AI](https://jina.ai/), store those vectors in [Elasticsearch](https://www.elastic.co/), and search with plain language. By the end, you'll have a working search interface where typing \"*a dog wearing sunglasses*\" returns the matching image ranked by visual similarity, no caption needed.\n\nThe complete project, notebook and Streamlit app, is on GitHub:\n\n## Prerequisites\n\n- Python 3.9+\n[SerpApi API key](https://serpapi.com/manage-api-key)(free tier, 250 searches/month)[Jina AI API key](https://jina.ai/api-dashboard/key-manager)(free tier, 10 million tokens)- Elasticsearch cluster (9.0 or higher)\n- ~2 GB disk space for downloaded images\n\n## Structured Image Search at Scale\n\nThe hard part of image search has never been the vector math. It's getting the images in the first place. Public Instagram profiles show thousands of posts, but there's no \"download all\" button and no official bulk API from Instagram.\n\n[SerpApi Instagram Profile API](https://serpapi.com/blog/how-to-scrape-instagram-profile-data-with-serpapi/) solves the access problem. One call with `engine=\"instagram_profile\"`\n\nand a username returns the profile's posts along with captions, like counts, comment counts, and a direct URL to each cover image.\n\nThat means a few dozen API calls give us hundreds of high-quality images with their metadata, ready to embed.\n\nWe'll use the [@apple](https://serpapi.com/playground?engine=instagram_profile&profile_id=apple) profile as our example. The feed spans landscapes, portraits, animals, and architectural photos, which gives the search plenty of visual variety to work with.\n\n## How It Works\n\nFour steps take us from a username to searchable images.\n\n**Fetch:** Page through the profile's feed with SerpApi, collecting cover images (images + video thumbnails).**Embed:** Convert each image into a vector that captures its visual meaning, using Jina AI.**Index:** Store the vector plus the metadata in Elasticsearch`dense_vector`\n\nfield.**Search:** Type a query and get the ranked images by similarity.\n\n### What Are Embeddings?\n\nAn [embedding](https://www.ibm.com/think/topics/embedding) is a list of numbers (a vector) that captures what an image means, its visual features, objects, colors, and composition. Similar images produce vectors close together in this high-dimensional space; different images produce vectors far apart.\n\nThe key that makes text-to-image search possible is that models like [Jina V5 Omni](https://jina.ai/models/jina-embeddings-v5-omni-small/) are trained on image-text pairs, so they map both images and text into the same space. A photo of a sunset and the sentence \"*a person at sunset*\" end up near each other, not because of keywords but because the model learned their meaning is similar.\n\n### How Vector Search Works\n\nOnce all images are embedded and stored, the search flow is straightforward.\n\n- Jina embeds your text query \"\n*a dog wearing sunglasses*\" into a vector using the same model. - Elasticsearch compares that query vector against every stored image vector using\n[cosine similarity](https://www.ibm.com/think/topics/cosine-similarity), a measure of how close two vectors point in the same direction. - The\n`k`\n\nnearest neighbors are returned, ranked by similarity score. Scores closer to 1.0 mean more similar; scores closer to 0 mean unrelated.\n\nThe query never touches the captions. The ranking is purely visual.\n\n### Why These Three Tools?\n\nEach tool handles one stage of the pipeline.\n\n| Component | Role | Why this one |\n|---|---|---|\n|\n\n[Jina v5 Omni](https://jina.ai/models/jina-embeddings-v5-omni-small/)[Elasticsearch](https://www.elastic.co/)`dense_vector`\n\nfield with built-in kNN, exact and approximate search out of the box, scales from hundreds to millions of vectorsTwo design choices are worth calling out.\n\n**One model for everything:** Jina v5 Omni handles both images and text in a single pipeline. If you later want to add semantic search over captions or post metadata, the same model and the same index work. No second model, no separate pipeline.**No raw images stored in the database:** We store only the vector, a local file path, and metadata (caption, post URL, likes). Images live in a local`images/`\n\nfolder, just long enough to embed and display results.\n\n## What Is Elasticsearch?\n\nIf you haven't used [Elasticsearch](https://www.elastic.co/elasticsearch) before, it's a search engine (the same technology behind site search on large websites) that also handles vector search natively. You store documents; each document has fields, and you query against those fields. For our use case, one of those fields is a vector, a long list of numbers that represents the visual content of an image.\n\nThe relevant feature is vector search via the [knn retriever](https://www.elastic.co/docs/solutions/search/vector/knn). Given a query vector, it finds the closest stored vectors and returns them ranked by similarity. That's our image search.\n\n### Two Ways to Run It\n\nTwo options for the Elasticsearch cluster. The notebook code is identical either way. Only `ES_URL`\n\nand `ES_API_KEY`\n\nin `.env`\n\nchange.\n\n#### Local Docker\n\nWe start a local single-node cluster using the [Elastic start-local script](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/local-development-installation-quickstart):\n\n```\ncurl -fsSL https://elastic.co/start-local | sh\n```\n\nThis prints an `ES_LOCAL_API_KEY`\n\n. Put it in your `.env`\n\nas `ES_API_KEY`\n\n. The endpoint is `http://localhost:9200`\n\n.\n\n#### Elastic Cloud\n\nCreate a free project on [Elastic Cloud](https://www.elastic.co/cloud/elasticsearch-service/signup) (14-day free trial). Copy the endpoint URL and API key into `.env`\n\n. Good if your machine is light on RAM or you want persistence without managing Docker.\n\n## The Pipeline\n\n### Step 1. Fetch the Profile\n\nWe'll page through the profile feed using the [SerpApi Python client](https://serpapi.com/integrations/python):\n\n``` python\nimport serpapi\n\nserp_client = serpapi.Client(api_key=os.getenv(\"SERPAPI_API_KEY\"), timeout=30)\n\nbase = {\"engine\": \"instagram_profile\", \"profile_id\": \"apple\"}\nparams = dict(base)\n\nfor page in range(1, max_pages + 1):\n    results = serp_client.search(params)\n    profile = results.get(\"profile_results\", {})\n    posts = profile.get(\"posts\", [])\n\n    next_token = results.get(\"serpapi_pagination\", {}).get(\"next_page_token\")\n    if not next_token:\n        break\n    params = {**base, \"next_page_token\": next_token}\n```\n\nEach request returns about 12 posts along with a `next_page_token`\n\n, which we send in the next call to get the next batch. When no token comes back, we've reached the end of the feed. So a few dozen calls are enough to pull hundreds of images.\n\n### Step 2. Embed the Images\n\nHere we are sending the images downloaded from the Instagram API to the Jina Embedding API as base64, receiving one vector per image:\n\n```\nJINA_URL = \"https://api.jina.ai/v1/embeddings\"\nJINA_MODEL = \"jina-embeddings-v5-omni-small\"\n\ndef embed_images(images_b64):\n    inputs = [{\"image\": f\"data:image/jpeg;base64,{b}\"} for b in images_b64]\n    payload = {\"model\": JINA_MODEL, \"task\": \"retrieval.passage\", \"dimensions\": 1024, \"input\": inputs}\n    r = requests.post(JINA_URL, headers=jina_headers, json=payload, timeout=120)\n    r.raise_for_status()\n    data = sorted(r.json()[\"data\"], key=lambda d: d[\"index\"])\n    return [d[\"embedding\"] for d in data]\n```\n\nThe one detail worth calling out is `task=\"retrieval.passage\"`\n\n. Jina uses different task modes for the vectors you store (`retrieval.passage`\n\n) and the vectors you search with (`retrieval.query`\n\n). Pairing them this way is what [Jina recommends](https://jina.ai/models/jina-embeddings-v5-omni-small/#:~:text=Best%20Practice) for retrieval. Step 4 handles the query side.\n\n### Step 3. Create the Index\n\nWe create an Elasticsearch index with a vector field to store the embeddings:\n\n```\nes.indices.create(\n    index=\"instagram_photos\",\n    mappings={\"properties\": {\n        \"embedding\": {\"type\": \"dense_vector\", \"dims\": 1024, \"similarity\": \"cosine\"},\n        \"caption\": {\"type\": \"text\"},\n        \"shortcode\": {\"type\": \"keyword\"},\n        \"username\": {\"type\": \"keyword\"},\n        # ... post_url, image_url, is_video, liked_by_count, comments_count\n    }}\n)\n```\n\nWe use cosine similarity because it's the standard match for text and image embeddings, and 1024 dimensions matches Jina's output. See the notebook for the full mapping.\n\n### Step 4. Search\n\nWe embed the query with Jina (this time with `task=\"retrieval.query\"`\n\n) and pass the vector to Elasticsearch's [kNN retriever](https://www.elastic.co/docs/solutions/search/vector/knn):\n\n``` python\ndef search(query, username, k=6):\n    query_vector = embed_query(query)\n    resp = es.search(\n        index=\"instagram_photos\",\n        retriever={\"knn\": {\n            \"field\": \"embedding\",\n            \"query_vector\": query_vector,\n            \"k\": k,\n            \"filter\": {\"term\": {\"username\": username}},\n        }},\n        size=k,\n        source_excludes=[\"embedding\"],\n    )\n    return resp[\"hits\"][\"hits\"]\n```\n\nThe `filter`\n\nscopes results in a single profile even when the index holds many. `source_excludes`\n\nkeeps the response lean by dropping the 1024-float vector we don't need back.\n\n## Results\n\nWith 577 images indexed from the Apple profile, the search delivers exactly what you'd expect.\n\nThe top result is a video thumbnail (poster frame) from Apple's feed, a dog in sunglasses, scored at 0.737 cosine similarity. No caption matching. No keyword overlap. The pipeline matched the visual content of the image to the meaning of the text query.\n\n#### A dog wearing sunglasses\n\n### More examples from the same index:\n\n#### Snow-capped mountains\n\n#### Horseback Riding\n\n#### Someone holding a phone\n\n#### A close-up portrait\n\n## Beyond Instagram\n\nThe embedding and search code doesn't change. Only the SerpApi engine and the fields you extract differ. The same architecture applies to any visual data source SerpApi can reach.\n\n| Source | SerpApi engine | Image field | What you'd search |\n|---|---|---|---|\n|\n\n`youtube`\n\n`thumbnail.static`\n\n[Google Images](https://serpapi.com/images-results)`google_images`\n\n`original`\n\n[Google Shopping](https://serpapi.com/shopping-results)`google_shopping`\n\n`thumbnail`\n\n[Google Lens](https://serpapi.com/google-lens-api)`google_lens`\n\n`thumbnail`\n\n[Amazon Search](https://serpapi.com/amazon-search-api)`amazon `\n\n`thumbnail`\n\n## Where to Go from Here\n\nConcrete next steps to extend this pipeline:\n\n**Hybrid search combining captions and embeddings:** Add a`text`\n\nquery over the`caption`\n\nfield alongside the vector search using Elasticsearch's[RRF (Reciprocal Rank Fusion)](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion). Posts where both the image and the caption match will rank higher.**Index multiple profiles:** The`username`\n\nfield already supports multi-profile indexing. Fetch a second profile, embed its images, and use the filter to search within or across profiles.**Try other SerpApi engines from the table above:** Swap the fetch step for YouTube thumbnails or Google Shopping product photos. The embedding and search code stays identical.\n\n## Conclusion\n\nThis pipeline reads images the way people describe them, not the way someone happened to describe or tag them. That single capability changes what you can do with an image catalog.\n\nThe Instagram example is just a demonstration. An e-commerce store can surface products by what they look like, not by category tags. A media archive can retrieve photos by scene, mood, or subject. An LLM agent can query the same index in plain language and reason over the matches, no captions or manual tagging required.\n\nBoth sides of the search now carry meaning. The query is interpreted by what it means, and the images are stored by what they show. That opens a second layer of interaction between your users, or an LLM, and your content.", "url": "https://wpnews.pro/news/semantic-image-search-with-elasticsearch", "canonical_source": "https://serpapi.com/blog/semantic-image-search-elasticsearch/", "published_at": "2026-07-03 17:53:02+00:00", "updated_at": "2026-07-21 18:20:15.968159+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision", "developer-tools", "ai-tools"], "entities": ["SerpApi", "Jina AI", "Elasticsearch", "SerpApi Instagram Profile API", "Jina V5 Omni", "Instagram"], "alternates": {"html": "https://wpnews.pro/news/semantic-image-search-with-elasticsearch", "markdown": "https://wpnews.pro/news/semantic-image-search-with-elasticsearch.md", "text": "https://wpnews.pro/news/semantic-image-search-with-elasticsearch.txt", "jsonld": "https://wpnews.pro/news/semantic-image-search-with-elasticsearch.jsonld"}}