{"slug": "from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts", "title": "From Software Engineer to AI Engineer - Part 4: RAG-ing the facts", "summary": "A developer's blog post explains how to implement Retrieval-Augmented Generation (RAG) for AI applications, focusing on the agentic variant where the model decides what to search. The post covers using embeddings and vector databases to improve search over full-text methods, and emphasizes chunking documents to manage context and token usage.", "body_md": "Models are trained on a massive amount of data. That doesn't mean that they know all the details of your specific situation. The training data might contain all chargeback fee schedules for Mastercard up until 2025, but you live in 2026 and actually ask about chargebacks on Wero. The model's knowledge is broad, generic and frozen at training time. You cannot just retrain Claude to include your data (except if you own Anthropic). So how can we pass knowledge that is specific to your situation or company? Well, we can use a **tool**. As we learned in [Part 3](https://dev.to/bjornvdlaan/from-software-engineer-to-ai-engineer-part-3-2o9k), tools are simply functions that the model can request to invoke with the output of the tool being pasted back into the model's context window.\n\nImagine that we have a collection of documents and we create a search engine tool. The model can request the tool with a specific question like \"what are the chargeback fees for Wero in 2026?\" and our Python code browses the document collection to return the relevant sections. This pattern is so common that it even got a dedicated AI name: Retrieval-Augmented Generation, or RAG. This article shows you the basics.\n\nNote: this article is about the agentic variant of RAG, in which the model decides what it wants to search to answer the question at hand. This is different from traditional RAG where relevant (parts of) documents are retrieved and included into the first prompt to the model.\n\nNote 2: the examples in this article show processing fees of different payment methods. The fees and all other information are purely fictional. These are not the actual numbers.\n\nThe simplest way to implement this search engine tool is to do a full-text search over all documents based on the search query and then copy-paste matching documents into the context window. However, this might not even yield the best results. RAG uses two concepts to do better:\n\nFull-text search is not only slow, it can also easily miss relevant sections. Synonyms, jargon words, implicit descriptions. Instead of comparing the text itself, we convert the text into so-called **embeddings**. An embedding is a fancy word for a vector (yes, from mathematics) that represents the meaning of that text. These embedding vectors are calculated in such a way that text with similar meaning maps to similar vectors (think similar as in cosine similarity or Euclidian distance), even if the words are not exactly the same. \"Initiating a chargeback\" and \"How to get my money back?\" have different words but similar meaning. Their vectors should therefore be more similar. Calculating similarity between vectors is also a lot faster than full-text comparison. Mapping text to an embedding is done by an **embedding model**, which itself is an AI model that was trained on millions of text pairs that do or do not belong together. The embeddings are stored in a **vector database** that can perform the similarity calculations.\n\nA document is rarely about just one topic. A single article might explain processing fees, refund rules, chargeback procedures, settlement times and more. The embedding of that multi-topic document will also be less similar to text about any of the topics. The different topics all steer the vector in a different direction and it ends up no where close to the individual topics. And even if the document is still selected, you'd risk pasting a 30-page document into the model's context where you only need three sentences. This will rot your context and cost your tokens. That is why documents are split into **chunks** by a **splitter** before the embeddings are calculated. Many splitters exist: some split every 500 tokens, while others try to be smart and split according to the document type's structure. Later in this article, we see a Markdown-specific chunking approach.\n\nIf the above sounds a bit abstract and mathematical, then you are completely right. Embedding models and splitters are academic topics. They are to AI engineers what databases and search algorithms are to software engineers. You should understand fundamentals and be able to apply them in practice, but there's no need to understand their full academic depth. Therefore, let's leave the theory and look at a concrete example document:\n\n```\n# Payment Operations Notes\n\n## Card Processing Fees\n\nMastercard transactions incur a 2.2% processing fee plus €0.30, while Wero charges €0.32 flat.\n\n## Refund Policy\n\nCustomers may request a refund within 30 days of purchase.\n\n## Chargebacks\n\nChargeback disputes must be answered within 7 days.\n```\n\nWe want to calculate its embedding so that the model can search for it later. First, the splitter splits the document into three chunks:\n\n```\n| Chunk | Content                                                               |\n|:------|:----------------------------------------------------------------------|\n| C1    | Card Processing Fees + the paragraph describing processing fees       |\n| C2    | Refund Policy + the refund rules                                      |\n| C3    | Chargebacks + the chargeback policy                                   |\n```\n\nEach chunk is passed to the embedding model which converts it into a vector. The embedding model that we'll use in the code example later (`all-MiniLM-L6-v2`\n\n) produces a vector of 384 numbers for every chunk, which is still small compared to production models that go over a thousand numbers per vector. For our example here, let's pretend the model produces only vectors of 2. The embedding model produces the following vectors:\n\n```\n| Chunk | Embedding vector |\n|:------|:-----------------|\n| C1    | (0.91, 0.15)     |\n| C2    | (0.23, 0.84)     |\n| C3    | (0.88, 0.11)     |\n```\n\nThe numbers themselves are meaningless to us humans. We cannot look at `(0.91, 0.15)`\n\nand conclude that it represents payment fees. What matters is the similarity of the vectors. Chunks with similar meaning end up close together in the vector space, while unrelated chunks end up further apart.\n\nSuppose the model uses the tool and asks: \"What are the card processing fees?\". The search engine tool first embeds the query using the same embedding model:\n\n```\n| Query                                | Embedding vector |\n|:-------------------------------------|:-----------------|\n| \"What are the card processing fees?\" | (0.90, 0.14)     |\n```\n\nThe query's embedding is passed to the vector database and compared with every other embedding. One common similarity measure is cosine similarity, which measures the extend to which two vectors point in the same direction.\n\n```\n| Chunk | Cosine similarity | Most similar vector |\n|:------|:------------------|:--------------------|\n| C1    | 0.99              | ✅                 |\n| C2    | 0.28              |                    |\n| C3    | 0.74              |                    |\n```\n\nBecause Chunk 1 has the highest similarity score, it is retrieved and returned to the model via it's context window and used to answer the user's question.\n\nNow lets build this in code. Create `data/payment_ops_notes.md`\n\nand fill it with Markdown:\n\n```\n# Payment Operations Notes - Fees, Refunds & Disputes\n\n## Card Processing Fees\nEuropean consumer cards cost 1.8% + €0.25 per transaction. Non-European and\ncommercial cards run higher, around 2.9% + €0.25. Fees are charged on the\noriginal transaction and are NOT returned when you refund. This means that\na refunded sale still costs you the full processing fee.\n\n## Refunds\nProcessing fees from the original charge are not returned, and the processor\ncharges a €0.25 admin fee per refund on top. Refunds are possible up to 180\ndays after the original charge; after that, use a manual bank transfer.\nCard refunds take 5-10 business days to reach the customer, which is the\nsingle most common cause of \"where is my refund\" tickets. Set that\nexpectation in the first reply.\n\n## Chargebacks (cards)\nThe processor charges a €15 dispute fee per chargeback, win or lose. Respond\nbefore the deadline in the dispute notification - typically 20-30 days\ndepending on the card network; a missed deadline is an automatic loss.\nCommon reason codes: 10.4 (fraud, card-absent), 13.1 (goods not received),\n13.6 (credit not processed). Evidence that wins fraud disputes: a 3-D Secure\nauthentication record, AVS/CVV match, delivery confirmation to the\ncardholder's billing address, and prior undisputed orders from the same\ncustomer.\n\n## Dispute Economics Rule\nOnly fight a chargeback when (disputed amount × realistic win probability)\nclearly exceeds the dispute fee plus internal handling cost (budget ~€25 of\nstaff time per response). Below roughly €50 disputed, accepting is almost\nalways cheaper than fighting.\n\n## Chargeback-Rate Monitoring\nCard networks track your chargeback rate, not just your losses. Above\nroughly 0.9% of transactions, merchants enter monitoring programs with\nmonthly fines and, eventually, termination of card acceptance. This is why\nrefunding a plausible fraud complaint fast is often cheaper than \"winning\"\nthe dispute: a refund does not count toward the chargeback rate, a\nchargeback does - even one you win.\n```\n\nAnd also create the RAG tool in `app/rag.py`\n\n:\n\n``` python\nfrom pathlib import Path\n\nfrom langchain_core.documents import Document\nfrom langchain_core.tools import tool\nfrom langchain_core.vectorstores import InMemoryVectorStore\nfrom langchain_huggingface import HuggingFaceEmbeddings\nfrom langchain_text_splitters import MarkdownHeaderTextSplitter\n\nDATA_PATH = Path(__file__).parent.parent / \"data\" / \"payment_ops_notes.md\"\n\ndef build_vectorstore() -> InMemoryVectorStore:\n    text = DATA_PATH.read_text()\n\n    # the splitter\n    splitter = MarkdownHeaderTextSplitter(\n        headers_to_split_on=[(\"##\", \"section\")]\n    )\n    chunks: list[Document] = splitter.split_text(text)\n\n    # the embedding model\n    embeddings = HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\")\n\n    return InMemoryVectorStore.from_documents(chunks, embeddings)\n\n_vectorstore: InMemoryVectorStore | None = None\n\ndef get_vectorstore() -> InMemoryVectorStore:\n    global _vectorstore\n    if _vectorstore is None:\n        _vectorstore = build_vectorstore()\n    return _vectorstore\n\n@tool\ndef search_payments_knowledge_base(query: str) -> str:\n    \"\"\"Search internal notes on processing fees, refund policy, and disputes.\n\n    Use this before quoting fee figures or policy deadlines yourself, and\n    before calling calculate_refund_cost if the user hasn't already given you\n    the payment method's fee structure. Ground your answer in these notes\n    rather than guessing.\n\n    Args:\n        query: What you need to know. Examples: \"card processing fees\" or\n            \"chargeback response deadline\".\n    \"\"\"\n    results = get_vectorstore().similarity_search(query, k=2)\n    if not results:\n        return \"No relevant notes found.\"\n    return \"\\n\\n---\\n\\n\".join(doc.page_content for doc in results)\n```\n\nSome notes about the above:\n\n`##`\n\nheaders, so every chunk is one coherent section with its heading attached. Such structural splitting can be applied if you know that your documents are written in Markdown. As mentioned before, more general splitters also exist.`all-MiniLM-L6-v2`\n\nis fine for examples and small applications. Its small enough to fit in your laptops memory. Use it for developing your agent. Once you reach serious scale, you can always switch to hosted embedding APIs from OpenAI and others. Although, do note that the embedding model used on the query must be the same as the model you used to fill the vector store. Compare it to hashing: if all passwords in the database are hashed with SHA256 and you has the given password with SHA3 then it will never match.`InMemoryVectorStore`\n\nis really the sqlite of AI engineering: a simple in-memory database for tutorials and local development only. Beyond that, I recommend the `pgvector`\n\nextension for Postgresql. Now we just need to import our tool and run the application. Create `04_tool_rag.py`\n\n:\n\n``` python\nfrom dotenv import load_dotenv\nfrom langchain.chat_models import init_chat_model\nfrom langchain_core.messages import ToolMessage\n\nfrom app.rag import search_payments_knowledge_base\n\nload_dotenv()\n\nmodel = init_chat_model(\"anthropic:claude-sonnet-5\")\nmodel_with_tools = model.bind_tools([search_payments_knowledge_base])\n\nquestion = \"What is the transaction fee for European consumer cards? Can they do chargebacks?\"\nmsg = model_with_tools.invoke(question)\nprint(msg.tool_calls)\n\ntool_map = {\n    \"search_payments_knowledge_base\": search_payments_knowledge_base\n}\n\ntool_messages = []\nfor tool_call in msg.tool_calls:\n    # retrieve function by name\n    tool = tool_map[tool_call[\"name\"]]\n    # call the function\n    result = tool.invoke(tool_call[\"args\"])\n    # store result\n    tool_messages.append(\n        ToolMessage(content=str(result), tool_call_id=tool_call[\"id\"])\n    )\n\nnext_msg = model_with_tools.invoke(\n    [{\"role\": \"user\", \"content\": question}, msg, *tool_messages]\n)\nprint(next_msg.content)\n```\n\nRun it with `python 04_tool_rag.py`\n\nand you'll see something like:\n\n``` bash\n$ python 04_tool_rag.py\n\n[\n    {'name': 'search_payments_knowledge_base', 'args': {'query': 'European consumer card transaction fee'}, 'id': 'toolu_01JzAV1GyQEyuu6zUoEne4EM', 'type': 'tool_call'}, \n    {'name': 'search_payments_knowledge_base', 'args': {'query': 'chargeback eligibility European consumer cards'}, 'id': 'toolu_01B4kA41umG1UHUV3XRmefTX', 'type': 'tool_call'}\n]\n\n**Transaction fee:** European consumer cards cost **1.8% + €0.25** per transaction. (For comparison, non-European or commercial cards run higher, around 2.9% + €0.25.) Note that this fee is charged on the original sale and is **not refunded** even if you later issue a refund — so a refunded transaction still costs you the full processing fee.\n\n**Chargebacks:** Yes, European consumer cardholders can file chargebacks like any other card network transaction. A couple of important points from a risk-management perspective:\n\n- Card networks monitor your **chargeback rate** (not just dollar losses). If it exceeds roughly **0.9% of transactions**, you can be placed into a monitoring program with monthly fines, and eventually risk losing card acceptance entirely.\n- Chargebacks count against that rate **even if you win the dispute** — but refunds do **not** count against it.\n- So for plausible fraud/dispute complaints, it's often cheaper overall to issue a refund proactively rather than fight (and \"win\") a chargeback, since winning still hurts your chargeback ratio while a refund doesn't.\n\nLet me know if you'd like me to calculate the actual cost of a refund vs. a chargeback for a specific transaction amount.\n```\n\nAbove you see the tool calls and then the final answer. What I find pretty exciting is that the model decides by itself to perform two separate queries. After using the RAG tool, the model answers based on our document instead of from its training data. It flagged where the figures came from because the docstring told it to. Nice, right? Nothing in our code explicitly enforced any of that.\n\nThis article showed how we can make our own internal documents available to the agent. Of course, there is much more to explore with RAG. One advanced concept worth looking up is reranking, where the chunks from the vector store are 'reranked' so that only the most relevant subset is passed to the agent.\n\nAs you might have realised, tools make the model pretty powerful. Real use cases often involve many tools and, like libraries and frameworks, you do not want to hand-roll that for every project. The next article shows how to publish tool catalogs through a protocol called MCP for others to use. Then, the article after that ([Part 6](https://dev.to/bjornvdlaan/from-software-engineer-to-ai-engineer-part-6-closing-the-loop-3gba)) finally ties everything together when we build a full-fledged agent like the ones we're used to from Claude Code or Codex.\n\nFind all code samples in the companion repo here:\n\n[https://github.com/BjornvdLaan/ai-engineering-articles-code-samples]", "url": "https://wpnews.pro/news/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts", "canonical_source": "https://dev.to/bjornvdlaan/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts-1m99", "published_at": "2026-08-31 19:12:51+00:00", "updated_at": "2026-08-31 19:53:34.923414+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Anthropic", "Claude", "Mastercard", "Wero"], "alternates": {"html": "https://wpnews.pro/news/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts", "markdown": "https://wpnews.pro/news/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts.md", "text": "https://wpnews.pro/news/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts.txt", "jsonld": "https://wpnews.pro/news/from-software-engineer-to-ai-engineer-part-4-rag-ing-the-facts.jsonld"}}