{"slug": "built-my-first-local-embeddings-pipeline-files-a-live-url", "title": "Built My First Local Embeddings Pipeline (Files + a Live URL)", "summary": "A developer built a local embeddings pipeline using Ollama's nomic-embed-text model, ChromaDB, and Python, embedding local files and a live URL for semantic search. The pipeline encountered a context length error with a 13,985-character file, which was temporarily fixed by truncating to 6,000 characters. The developer notes that truncation is not the right long-term fix.", "body_md": "**Context:** An embedding model doesn’t generate text — it converts text into a list of numbers (a vector) that represents the *meaning* of that text. Two pieces of writing about similar topics end up with similar vectors, even if they don’t share any exact words, which is what makes semantic search possible: you can find “things that mean something like this,” not just “things that contain this exact word.” One detail that surprised me going in: the vector is always the same fixed length, no matter how long the original text is — a one-sentence note and a 10-page article both come out as the same-sized list of numbers, like a fingerprint that summarizes something much bigger into a fixed format. RAG (retrieval-augmented generation) is built on this: embed a bunch of documents, embed a question the same way, and find whichever documents are numerically closest to the question’s meaning.\n\n**Ran:** Built this up one command at a time instead of writing the full script upfront — easier to see what each piece actually does before combining them.\n\n**Step 1 — Pull the embedding model**\n\n```\nollama pull nomic-embed-text\n```\n\n**Step 2 — Test the embedding API directly, before writing any Python**\n\nOllama exposes embeddings over a local HTTP API, so you can sanity-check it works with a plain `curl` before involving any code:\n\n```\ncurl http://localhost:11434/api/embeddings -d '{\"model\": \"nomic-embed-text\", \"prompt\": \"hello world\"}'\n```\n\nThis returns a JSON object with a single `embedding` field — a list of 768 numbers. That’s the whole idea of an embedding in one command: text in, fixed-length list of numbers out.\n\n**Step 3 — Install the Python pieces**\n\n```\npip3 install ollama chromadb requests beautifulsoup4 --break-system-packages\n```\n\n**Step 4 — Embed one local file, interactively**\n\nRather than run a full script blind, this is small enough to do a few lines at a time in a Python shell (`python3`):\n\n``` python\nimport ollama\ntext = open(\"02-oc-cli-mentor-system-prompt.md\").read()\nresponse = ollama.embeddings(model=\"nomic-embed-text\", prompt=text)\nlen(response[\"embedding\"])   # → 768\n```\n\nHit `ModuleNotFoundError: No module named 'ollama'` on the first attempt here — Step 3 hadn’t been run yet. A reminder that even a five-step walkthrough has room to skip a step by accident.\n\n**Step 5 — Store it in Chroma**\n\n``` python\nimport chromadb\nclient = chromadb.PersistentClient(path=\"./chroma_db\")\ncollection = client.get_or_create_collection(name=\"today_i_ran_notes\")\ncollection.upsert(\n    ids=[\"02-oc-cli-mentor-system-prompt.md\"],\n    embeddings=[response[\"embedding\"]],\n    documents=[text],\n)\ncollection.count()   # → 1\n```\n\n**Step 6 — Add a longer file, and hit a real limit**\n\nRepeating Step 4 against a longer draft article failed differently:\n\n```\nollama._types.ResponseError: the input length exceeds the context length\n```\n\n`nomic-embed-text` has a 2048-token context window, and the longer draft (13,985 characters) blew past it. Quick fix for now — truncate to a safe length before embedding:\n\n```\ntext = text[:6000]\n```\n\nNot the right long-term fix (more on that below), but enough to keep moving.\n\n**Step 7 — Add a live URL as a source**\n\nPulling in a web page instead of a local file needs one extra step first: fetching the page and stripping out the HTML noise (nav bars, scripts, footers) so only the article text gets embedded.\n\n``` python\nimport requests\nfrom bs4 import BeautifulSoup\n\nresponse = requests.get(\"https://pipelineandprompts.com/posts/03-1b-vs-3b-memory-comparison/\",\n                         headers={\"User-Agent\": \"Mozilla/5.0\"})\nsoup = BeautifulSoup(response.text, \"html.parser\")\nfor tag in soup([\"script\", \"style\", \"nav\", \"footer\", \"header\", \"aside\"]):\n    tag.decompose()\narticle_text = soup.get_text(separator=\"\\n\", strip=True)\nlen(article_text)   # → 3359\n```\n\nThat gets fed into the same `ollama.embeddings()` call from Step 4.\n\n**Result:** Three sources ended up embedded across the two files plus the URL from the steps above:\n\n| Source | Type | Characters sent | Vector dimensions | \n|---|---|---|---|\n| `cloud-without-chaos-01.md` | local file | 6,000 (truncated from 13,985) | 768 | \n| `02-oc-cli-mentor-system-prompt.md` | local file | 2,941 | 768 | \n| Entry 03 (live URL) | web article | 3,359 | 768 | \n\nTwo things stood out. First, every vector came back at exactly 768 dimensions regardless of source length — confirms the “fixed-size fingerprint” idea from the context section wasn’t just theory. Second, and more important: the truncation fix meant **57% of the longest draft’s content (7,985 of 13,985 characters) never made it into its embedding at all.** That’s not a rounding error — over half the article is invisible to any future search against that vector. This is exactly why real RAG pipelines chunk long documents into smaller overlapping pieces instead of truncating: chunking keeps everything searchable, truncation just quietly throws away whatever didn’t fit.\n\nThe URL fetch also worked cleanly — 3,359 characters extracted from the live page is close to the article’s actual body length, suggesting the nav/footer/script stripping in the script did its job without pulling in much boilerplate.\n\n**Takeaway:** The pipeline works end-to-end — files and live URLs, embedded into a queryable local store — but truncation is a real data-loss bug, not just a technical footnote, once you’re working with anything longer than a short note. [Entry 06](https://pipelineandprompts.com/posts/06-querying-embeddings-store/) queries this store for the first time, and [Entry 07](https://pipelineandprompts.com/posts/07-chunking-retrieval-bias/) replaces truncation with real chunking. For the production-grade version of this — FastAPI, proper chunking, and a real API — see [Build a RAG Pipeline for Internal Runbooks](https://pipelineandprompts.com/posts/ai-in-the-stack-02-rag-runbooks/) in the AI in the Stack series.", "url": "https://wpnews.pro/news/built-my-first-local-embeddings-pipeline-files-a-live-url", "canonical_source": "https://pipelineandprompts.com/posts/05-local-embeddings-pipeline/", "published_at": "2026-08-21 00:00:00+00:00", "updated_at": "2026-09-07 17:30:30.693674+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "developer-tools"], "entities": ["Ollama", "nomic-embed-text", "ChromaDB", "Python", "BeautifulSoup", "requests"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/built-my-first-local-embeddings-pipeline-files-a-live-url", "markdown": "https://wpnews.pro/news/built-my-first-local-embeddings-pipeline-files-a-live-url.md", "text": "https://wpnews.pro/news/built-my-first-local-embeddings-pipeline-files-a-live-url.txt", "jsonld": "https://wpnews.pro/news/built-my-first-local-embeddings-pipeline-files-a-live-url.jsonld"}}