{"slug": "why-your-rag-system-is-only-as-good-as-its-translator-model", "title": "Why Your RAG System Is Only as Good as Its Translator Model", "summary": "GlobalFoundries CIO rebuilt its data foundation with a real-time, governed platform before deploying AI agents, and will discuss this approach on September 10. The article explains that RAG systems' accuracy depends on the embedding model, which translates text into vectors for retrieval, and that a poor embedding model can cause incorrect answers even with a strong language model.", "body_md": "[A CIO on the data foundation AI agents need (Sponsored)](https://go.bytebytego.com/Redpanda_090226)\n\nThe CIO of GlobalFoundries had a straightforward view of it: you don’t get AI agents until the data underneath them is real-time and governed.\n\nSo he rebuilt that layer first — one platform carrying data across fabs on three continents, with identity, permissions, and audit trails handled once instead of per project. The agents followed, across IT, procurement, and other business functions.\n\nJoin us on September 10 to hear how he did it, and get your questions answered live.\n\nRAG, or Retrieval-Augmented Generation, is helping a lot of companies build chatbots specific to their requirements. However, the success of any such RAG system depends on the quality of the embedding model (the model that helps translate words into numbers) that is used by the RAG system.\n\nConsider the example where a particular product’s documentation specifies that annual subscriptions can be refunded only within 30 days. A chatbot built using RAG is supposed to answer customer queries based on this support documentation. However, when a customer asks whether a subscription bought 45 days ago can be refunded, the chatbot confidently answers “yes”.\n\nWhy did the chatbot make a mess of this seemingly simple question?\n\nThe answer lies in the behaviour of the embedding model, which is kind of a translator for AI. This model controls the answer-searching process and is different from the language model that generates the actual answer. No matter how good the language model might be, it cannot give correct answers if the embedding model doesn’t do its job properly.\n\nIn this article, we’re going to look at how this embedding model works in an RAG setup and what makes it such a critical part of the system. Here’s what we will cover:\n\nWhy a RAG system searches before it answers\n\nHow embeddings make it possible to search by meaning\n\nWhy related information is not always the right information\n\nWhy a better language model cannot repair bad retrieval\n\nWhat makes an embedding model suitable for a RAG system\n\nHow to compare embedding models without trusting benchmark scores blindly\n\nChoosing between commercial APIs and locally run models\n\nWhy changing the embedding model later becomes expensive\n\nHow Matryoshka embeddings offer more control over vector size\n\n## Why a RAG System Searches Before Answering\n\nA generic language model’s knowledge depends on what it was taught during training. You can’t expect such a model to know about a company’s private documents, policies, or internal source code. It won’t even know about the company’s latest information that might have been published after the model was trained. Technically, it’s quite costly to retrain a language model every time there’s a change in the information.\n\nThis is where RAG helps. It separates the ability to generate language from finding relevant details from a pool of stored knowledge. While the language model handles the understanding and writing parts, the external knowledge collection contains the actual information that should be used for writing the answers.\n\nRAG works in two different phases: indexing and retrieval.\n\nDuring the indexing phase, the system prepares the documents:\n\nIt collects documents from various sources (files, websites, databases, etc.).\n\nIt extracts their text.\n\nIt divides the text into smaller passages known as chunks.\n\nIt sends each chunk into an embedding model.\n\nIt stores the resulting vector with the original text and metadata.\n\nThe metadata may contain details such as the document title, publication date, language, version, and so on.\n\nDuring the retrieval phase, the system follows the following steps:\n\nIt sends the question through the same embedding model.\n\nIt searches for document vectors that are close to the question vectors.\n\nIt retrieves a small number of chunks. For example, the best 5 chunks or something like that.\n\nIt filters and reranks those chunks, placing them inside the language model’s prompt.\n\nFinally, the language model writes an answer using the prompt that was provided.\n\nThe main takeaway from all this is that RAG doesn’t place the entire document collection into the model’s prompt. This is because doing so would exceed the model’s context limit and fill it with irrelevant information. It also increases costs and latency. The retrieval phase, led by the RAG’s embedding model, serves as the selection step, reducing thousands of passages to a small set for the language model to check.\n\n## How Embeddings Make it Possible to Search by Meaning\n\nAn embedding is basically just a list of numbers that represents a piece of text. A real embedding might contain numbers such as 384, 768, 1024, or several thousand numbers.\n\nThese individual numbers don’t have simple labels describing their meanings. You cannot look at a particular value and say that this means “subscription” while another means “refund”. The overall meaning is distributed across the complete vector. Think of the vector as the coordinates of a point in a mathematical space. An embedding model is trained to place text with related meaning near each other within this mathematical vector space.\n\nThis is how the system can connect a term like “yearly plan” with “annual subscription”. It can also connect complex phrases like “get my money back” with “receive a refund”. In contrast, a keyword search struggles when a question and the document use different words to explain the same concept. But an embedding model tries to compare the ideas present within the text rather than just the vocabulary. Common techniques behind this are cosine similarity, dot product, and Euclidean distance. In a nutshell, the goal of an embedding model is to produce a mathematical score that indicates how close two vectors are to each other.\n\nThe embedding model often returns the first k results. This is known as top-k retrieval. For example, if k is 5, the retrieval returns the top 5 highest-ranked chunks.\n\nEmbedding models also support asymmetric retrieval where the query and document have different forms. The query may be a short question. But the matching document is a longer explanatory passage. For example, the query could be something like: “Can an annual subscription be refunded after 6 weeks?” The answer passage is a statement like: “Annual subscriptions can be refunded within 30 days.”\n\nAn embedding model trained to compare similar sentences may not perform as well as a model that is trained to connect questions with the relevant passages. To summarize, the embedding model defines what the RAG system considers similar.\n\n## Why Related Information is not Always the Right Information\n\nAn embedding model is trained to identify semantic similarity. But a RAG system requires something a bit more strict. It needs to find passages that might contain the information needed to answer a specific question. The problem is that a passage can be related to the question without clearly answering it.\n\nFor example, a customer asking how long a refund takes might receive a passage explaining who qualifies for the refund. Both these pieces of information concern refunds, but only one talks about the actual processing time. See the diagram below that shows the concept of semantic search space.\n\nMultiple failure modes can pop up in this situation:\n\n**Similar Subject, Different Question:** The query asks “How long will an approved refund take to arrive?” The retrieved passage says: “Purchases can be refunded within 30 days.” The passage is relevant to refunds but does not answer the timing question.**Same Words, Different Entity:** The query asks “how can the billing address be changed.” The retrieved passage explains how to change the account’s email address. Both talk about changing account details. But they refer to completely different fields.**Negation:** Consider the two passages: “Administrators can delete archived projects”, and “Administrators cannot delete archived projects”. Most of the words are identical. This means their embeddings may be close. But as you can see, they have opposite meanings.**Versions and Dates:** A knowledge base may contain an old policy and its replacement. The text may be almost identical except for a date, limit, or price. Embeddings cannot automatically know which document is authoritative. Metadata filters or version management rules may need to exclude outdated content explicitly.**Numerical Identifiers:** The two sentences “Annual subscriptions can be refunded within 30 days” and “Annual subscriptions can be refunded within 60 days” are semantically very similar. But there is a difference in one number, which determines the final answer.**Domain-specific Meanings:** General models can misunderstand special vocabulary. For example, “capture” has one meaning in an ordinary language and a specific meaning in payment processing. Similarly, the word “port” can refer to networking, hardware, or moving software between platforms. The best model for general web text may not be the best model for legal contracts, medical reports, financial documents, or source code.**Multi-part questions:** A customer may ask multi-part questions. For example, there could be a question like “Can I cancel the subscription, and how long will the refund take?” The answer for this requires at least two passages. One might explain eligibility for cancellation. Another might explain processing time. The model that retrieves only one subject may produce an incomplete answer.\n\n## Why a Better Language Model Cannot Repair Bad Retrieval\n\nIn an RAG system, the language model just sees the user’s question and the selected passages. It doesn’t see every document that might have been stored in the vector database.\n\nFor example, if the annual refund policy document is not retrieved, the language model won’t have any idea about the missing information. A more capable language model may recognize that the retrieved information doesn’t contain the answer. This is useful because it can at least choose not to respond with invalid information. However, this failed retrieval scenario also creates several possible outcomes:\n\nThe model answers from its general training knowledge.\n\nIt incorrectly applies a related passage.\n\nIt might invent a plausible rule.\n\nIt says that the available information is insufficient.\n\nIt combines conflicting passages incorrectly.\n\nA prompt such as “answer only from the supplied documents” can reduce unsupported answers. But it cannot make the correct document magically appear.\n\nThis is why testing and debugging are so critical during the development of an RAG system. Developers need to inspect retrieved chunks before changing prompts or swapping language models. It takes a lot more time to solve a retrieval problem in the generation phase.\n\n## What Makes an Embedding Model Suitable For a RAG System\n\nThe most important concern when choosing an embedding model should be retrieval performance. The model should be really good at connecting short questions with longer sentences that might contain the answers.\n\nThe following characteristics matter:\n\n**Domain and Vocabulary:** The model should understand the language used by the documents and users. Testing should include the use of abbreviations, internal product names, technical terms, and so on. If a company calls annual subscriptions “yearly plans” in the UI, but “annual contracts” in their legal documents, the model should be able to connect those expressions.**Language Support:** A multilingual model is critical when documents, queries, or both may use different languages. The model should be able to connect a query in one language with an answer found in another language.**Embedding Dimensions:** Embedding dimension is the number of values in each stored vector. Larger vectors preserve more information. But size does not guarantee better retrieval. Also, dimensions have a direct impact on raw storage costs.**Maximum Input Length:** A model with a limit of 8192 tokens can embed much longer text than one limited to 512 tokens. But it doesn’t mean each document should become one big chunk. A long chunk might discuss multiple things like refunds, cancellations, billing addresses, and so on. A focused question may match it less precisely than it would match a smaller passage specific to the relevant policy.**Model and Query Speed:** Indexing speed determines how much time it takes to embed the document collection. Query speed affects every user request. Some important metrics to consider are the number of chunks embedded per second, query-embedding latency, CPU or GPU requirements, memory use, and the cost of running the model. A model that gains a bit of retrieval quality at the cost of a substantial increase in latency may not be the right choice.**Deployment Requirements:** Embedding dimensions determine the size of each output vector. Model size determines how much memory and computation are needed to run the embedding model. These are key details required to plan for the model’s deployment.\n\n## Why Changing the Embedding Model Later Is Expensive\n\nEach embedding model creates its own vector space, which is unique to the particular model. This means that the embeddings produced by Model A have no dependable relationship with the embeddings produced by Model B.\n\nConsider an example where all document chunks are embedded using Model A. If we suddenly start to embed the incoming questions using Model B, the system will end up comparing vectors from incompatible spaces. It will be like an apples-to-oranges comparison. Even if both models output the same number of dimensions, the numbers can mean very different things.\n\nThis means that the entire corpus of documents needs to be re-embedded using Model B. In other words, migrating an embedding model needs a lot more than just running the new model. It involves the following steps:\n\nGenerate new vectors for every chunk.\n\nBuild a new vector index.\n\nApply the correct metadata and access permissions.\n\nKeep new or modified documents synchronized during the migration.\n\nTest retrieval quality.\n\nMove production searches to the new index.\n\nRetain a rollback path if the new system performs poorly.\n\nThe costs of the migration process include embedding API charges or GPU time, index-building time, paying for temporary duplicate storage, data transfer costs, evaluation work, and operational risk.\n\nThe migration also changes rankings. For example, a model with a higher benchmark score may perform worse on the application’s specific domain terminology. Therefore, application-specific testing is extremely important to make sure that the answer quality has not gone down.\n\nA few design choices can make it safer to implement future changes:\n\nThe original chunks should remain the source of truth. A vector database should not be the only place where processed text exists.\n\nEach chunk should have a stable identifier and a content hash. The hash helps determine whether the text changed and needs a new embedding\n\nEmbedding records should include:\n\nModel name\n\nModel version or revision\n\nEmbedding dimension\n\nQuery and passage format\n\nNormalization method\n\nChunking version\n\nCreation time\n\nA new model should receive a new index or vector field. Its embeddings should not be mixed with old embeddings.\n\nTo summarize, a safe migration resembles the classic blue-green deployment approach where the old index continues to serve production traffic even as a new index is built in parallel.\n\nTo be clear about things, changing the embedding model is just one of the operations that requires a rebuild. Changing the chunking strategy, parser, cleaning rules, prefixes, or stored dimensions may also require the documents to be rebuilt.\n\n## How Matryoshka Embeddings Offer More Control Over Vector Size\n\nA basic embedding model produces a fixed-size vector. Its dimensions are meant to work together. We cannot delete most of these dimensions without severely reducing the quality of retrieval.\n\nIn contrast, a Matryoshka model is trained differently. During training, such a model is configured to produce useful representations at different prefix lengths, such as the first 256, 512, 1024, and full dimensions.\n\nWhile the initial dimensions contain a useful coarse representation, the later dimensions pack more detailed information. For example, in a 1024-dimensional embedding, the first 256 dimensions can contain a useful small representation. Similarly, the first 512 dimensions contain a more detailed representation. At the end, all 1024 dimensions provide a complete representation.\n\nThere are three practical storage designs to support Matryoshka models:\n\n**Storing the smaller vector:** The system creates an embedding. It then keeps the first 256 dimensions and normalizes the shortened vector if needed. Only these dimensions are stored. This reduces the index size and overall search cost. However, the discarded dimensions are gone forever. If we want to migrate the system to a larger dimension, we need to generate the embeddings again.**Storing the full vector with a smaller search representation:** The system stores the complete vector in secondary storage. But it places a smaller prefix in the fast vector index. This supports future flexibility. We can build a larger index from the stored vectors without the need to rerun the model.**Storing smaller and full vectors for two-stage retrieval:** The system makes a search on the entire corpus using just 256-dimensional vectors (the reduced dimension). It then retrieves the full vectors for the best candidates and compares them precisely. This brings down the cost of the large initial search. But it retains full precision for the shortlisted candidates.\n\nTo be clear, Matryoshka embeddings don’t solve incompatibility between different models. They mainly help support different usable sizes within a model’s vector space. Switching to another embedding model still requires the entire re-embedding process we talked about earlier.\n\n## Conclusion\n\nAs we discussed, an RAG system can’t generate a reliably grounded answer unless it first retrieves the details required for that answer. This makes an embedding model the most important part of an RAG setup.\n\nThis doesn’t mean that other things are not important. Retrieval quality of the RAG system also depends on various factors such as:\n\nThe existence of the correct documents\n\nHow well the parsing process keeps their content intact\n\nHow the documents are divided into chunks\n\nHow versioning is managed\n\nMetadata filters and their working\n\nHow a reranker improves the candidate list\n\nNevertheless, the embedding model remains central because it helps decide which information enters the language model’s context. It acts as the first major relevance decision in a typical RAG pipeline. The key lesson is to choose an embedding model that retrieves the correct evidence for the questions at an acceptable cost and speed.", "url": "https://wpnews.pro/news/why-your-rag-system-is-only-as-good-as-its-translator-model", "canonical_source": "https://blog.bytebytego.com/p/how-to-shrink-a-language-model-without", "published_at": "2026-09-02 15:31:12+00:00", "updated_at": "2026-09-02 15:54:24.583631+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-infrastructure"], "entities": ["GlobalFoundries", "RAG", "embedding model"], "alternates": {"html": "https://wpnews.pro/news/why-your-rag-system-is-only-as-good-as-its-translator-model", "markdown": "https://wpnews.pro/news/why-your-rag-system-is-only-as-good-as-its-translator-model.md", "text": "https://wpnews.pro/news/why-your-rag-system-is-only-as-good-as-its-translator-model.txt", "jsonld": "https://wpnews.pro/news/why-your-rag-system-is-only-as-good-as-its-translator-model.jsonld"}}