# Automating Data Ingestion with a RAG Flywheel > Source: > Published: 2026-10-11 22:00:00+00:00 # Automating Data Ingestion with a RAG Flywheel Automating schema mapping for diverse supplier data using a RAG Flywheel and LLMs. ## The Problem: The Schema Mapping Bottleneck[¶](#the-problem-the-schema-mapping-bottleneck) At Zalando, our Supply Operations team manages a vast array of supplier offers. Each supplier provides their data in different formats, structures, and schemas. This diversity creates a significant challenge for the data ingestion processes, as they need to map these varying schemas into a unified format that our systems can understand and process efficiently. For every uploaded file, operations specialists must manually map the highly diverse schemas (source headers/columns), ranging from 10-60 columns per sheet per workbook, with unpredictable variations in header naming conventions like "Art.-Nr.", "Article Number", "SKU", or "Farbe" (color in German) to our strictly defined internal target schema of over 55 columns (for example, `SKU`, `Color Code`, and `Color Name`). Traditionally, automating this type of schema matching relied heavily on hardcoded rules, regular expressions, or Levenshtein distance-based fuzzy matching. However, these methods often fall short when dealing with the high variability and complexity of real-world data. The semantic meaning of a column such as `Retail Price` might appear under different names like `UVP`, `Price`, or `MSRP`. Since these legacy engines lack semantic understanding of the data, they tend not to scale with the growing diversity of supplier data and require continuous manual maintenance from engineers every time a new supplier is onboarded or an existing supplier changes its data format. To eliminate this operational bottleneck, we needed a system capable of understanding the semantic meaning of the data and generating smart mapping suggestions. In this post, we explore how we built an LLM-powered continuous learning loop using a Retrieval-Augmented Generation (RAG) Flywheel to automate the schema mapping process. We also explain how we constrained the model output and integrated `pgvector` to produce safer and more historically consistent results. ## Context Segregation & Controlled Output[¶](#context-segregation-controlled-output) To build the system within our infrastructure constraints, we rely on AWS Bedrock to host our external AI dependencies. We split the workload across 2 distinct components: `Amazon Titan Text Embeddings V2` to generate 1024-dimensional vectors for semantic search, and a generative model (like `Anthropic Claude Sonnet` or `Gemini`) to execute the logical reasoning and deductions. The API acts as the central orchestrator, isolating the static internal target schemas from the dynamic, historically embedded supplier data and mappings before sending them to the LLM. ## Enforcing Safety with Tool Use[¶](#enforcing-safety-with-tool-use) LLMs are not deterministic: they can produce different outputs and hallucinate. Our goal is therefore to constrain their behavior and keep the API as the final decision boundary. We implement two primary constraints at the API level: 1. **Zero Temperature**: We hardcode the temperature parameter to `0.0`. Schema matching benefits from repeatable reference lookups, and a higher temperature can introduce unnecessary variation. 2. **Tool Use**: We require the model to return a fixed structure containing `sourceColumn`, `targetColumn`, `reasoning`, and `confidenceScore`. Tool use constrains the response shape, but it does not guarantee that the semantic mapping is correct; that is why every suggestion is validated and reviewed by a human. ## API as the Gatekeeper[¶](#api-as-the-gatekeeper) Even with the above constraints, we treat the LLM output as untrusted until verified. Before any LLM-generated suggestion reaches users or the database, the API performs synchronous validation checks that catch model misbehavior. These checks include: - **Unknown Source Column**: Rejecting any source column that is not present in the uploaded file. - **Unknown Target Column**: Rejecting any target column that is not part of the internal target schema. - **Missing Source Column**: Guaranteeing full coverage of all source columns from the uploaded file. ### Validation in Practice[¶](#validation-in-practice) The validation process applies a predictable sequence of checks to the raw mappings generated by the LLM. 1. **Prepare Verification Sets**: The API constructs sets of known source and target columns from the uploaded file and the internal schema, respectively. 2. **Iterate and Validate**: Loop through each mapping suggestion: - If a source column is not in the known source set, reject the suggestion. - If a target column is not in the known target set, reject the suggestion. 3. **Verify Coverage and Stable Ordering**: Finally, iterate through the known source columns to ensure that each one has a corresponding mapping, and reorder the suggestions to match the original file structure. Here is a simplified TypeScript implementation of the validation logic: ``` function validateGeneratedMappings( mappings: MappingItem[], sourceColumns: SourceColumn[], targetColumns: TargetColumnDefinition[] ): MappingItem[] { const expectedSet = new Set(sourceColumns.map(sc => sc.name)); const allowedTargets = new Set(targetColumns.map(tc => tc.name)); const bySource = new Map(); const errors: string[] = []; for (const m of mappings) { if (!expectedSet.has(m.sourceColumn)) { errors.push(`Unexpected source column: "${m.sourceColumn}"`); } if (m.targetColumn !== null && !allowedTargets.has(m.targetColumn)) { errors.push(`Unknown target column: "${m.targetColumn}"`); } if (!Number.isFinite(m.confidenceScore) || m.confidenceScore < 0 || m.confidenceScore > 100) { errors.push(`Invalid confidence score for "${m.sourceColumn}": ${m.confidenceScore}`); } bySource.set(m.sourceColumn, m); } const ordered: MappingItem[] = []; for (const sc of sourceColumns) { const m = bySource.get(sc.name); if (!m) { errors.push(`Missing mapping for source column: "${sc.name}"`); } else { ordered.push(m); } } if (errors.length > 0) { throw new Error(`LLM output validation failed with ${errors.length} issues:\n- ${errors.join('\n- ')}`); } return ordered; } ``` ## The RAG Flywheel: From Zero-Shot to Continuous Learning[¶](#the-rag-flywheel-from-zero-shot-to-continuous-learning) Our goal was not only to automate the schema mapping process, but also to create a system that continuously improves over time with human feedback. We therefore architected the mapping lifecycle around 2 phases: an initial zero-shot prediction for previously unseen data (`cold start`), and a retrieval-driven learning loop (` warm start`) that leverages historical mappings to improve future predictions. However, before getting to either of these phases, we must establish the strict constraints and operational boundaries. ### The Foundation[¶](#the-foundation) Regardless of the phase, each request to the LLM is governed by a foundational `System Prompt` and a strict Tool Calling configuration. For the AWS Bedrock Mantle endpoint, which is compatible with the OpenAI SDK, we can use the standard OpenAI Node SDK to interface with the models. This gives us access to standard `Function Calling` (Tool Use) features for constraining the output schema. The system prompt sets the model's role, injects the internal target schema definitions (``), and enforces mapping rules such as falling back to `null` when no mapping can be found and how confidence scores should be calculated. #### Target Column Schema:[¶](#target-column-schema) ``` interface TargetColumnDefinition { name: string; label: string; description: string; data_type: string; is_key: boolean; is_required: boolean; samples: Array; } ``` #### System Prompt[¶](#system-prompt) ``` js const getSystemPrompt = ( targetSchema: Array, ) => `You are an expert data architect responsible for mapping external Excel data to a strict internal database schema. Your task is to evaluate a list of source columns and their sample data, and map them to the correct target column. You must use the provided tool to return your final decisions. ${JSON.stringify(targetSchema)} RULES FOR MAPPING: 1. Strict Naming: You may only map to a target column if it exactly matches the "name" property in the . 2. Null Fallback: If a source column does not logically match ANY of the target columns, you MUST return JSON null for "targetColumn". Do not force a bad match. 3. Reasoning: Provide a brief, logical explanation in the "reasoning" field [...] 4. Confidence Scoring: 95-100 for an exact historical or identical-name match, down to 1-49 for a low-confidence guess [...]`; [...] ``` To constrain the JSON response, we configure the OpenAI SDK with a tool definition mirroring the `MappingItem` interface. We set `tool_choice` to force the model to use the specific function and hardcode `temperature` to `0.0` to reduce variation. The API still validates the returned arguments because tool use does not verify the correctness of the mapping itself. ``` python import OpenAI from 'openai'; // Initialize OpenAI SDK pointing to AWS Bedrock Mantle endpoint const openai = new OpenAI({ baseURL: 'https://bedrock-mantle..api.aws/v1', apiKey: process.env.API_TOKEN, }); const columnMappingTool = { type: "function", function: { name: "submit_column_mappings", description: "Submit the final deduced column mappings.", parameters: { type: "object", properties: { mappings: { type: "array", items: { type: "object", properties: { sourceColumn: { type: "string" }, targetColumn: { type: "string", nullable: true }, reasoning: { type: "string" }, confidenceScore: { type: "number" } }, required: ["sourceColumn", "targetColumn", "reasoning", "confidenceScore"] } } }, required: ["mappings"] } } }; // Example execution wrapper async function generateSuggestions(systemPrompt: string, userPrompt: string) { const response = await openai.chat.completions.create({ model: "anthropic.claude-3-sonnet", // or any other generative model temperature: 0.0, // Reduce variation in model output messages: [ { role: "system", content: systemPrompt }, { role: "user", content: userPrompt } ], tools: [columnMappingTool], tool_choice: { type: "function", function: { name: "submit_column_mappings" } } }); // The tool constrains the response shape; the API validates mapping values separately. return JSON.parse(response.choices[0].message.tool_calls[0].function.arguments).mappings; } ``` ### The Baseline: Zero-Shot Mapping (Cold Start)[¶](#the-baseline-zero-shot-mapping-cold-start) When a new file is uploaded for the first time, the system has no historical context to draw upon. In this `cold start` scenario, the LLM relies solely on logical deductions. We provide the model with the static internal target schema definitions and the source columns from the uploaded file, and ask it to deduce the relationships based on the general knowledge of data types and semantic meanings. #### Source Column Schema:[¶](#source-column-schema) ``` interface SourceColumn { name: string; dataType: "string" | "number" | "boolean" | "null"; samples: Array; } ``` Here is the simplified prompt we send to the LLM for zero-shot mapping: ``` js const getColdStartPrompt = ( sourceColumns: Array, ) => ` NO HISTORICAL CONTEXT AVAILABLE. This is a cold start. You must rely purely on logical deduction, the provided definitions, and your general knowledge of data types to determine the mappings. Please map the following incoming Excel columns: ${JSON.stringify(sourceColumns)} Return the final mappings as structured JSON only.`; ``` ### Generating Semantic Embeddings (Vectors)[¶](#generating-semantic-embeddings-vectors) To move from a zero-shot approach to a learning loop, we need a way to mathematically compare new source columns with historically approved mappings. Exact string matching is insufficient here: `UVP` and `rrp_de` may be semantically equivalent, but string matching would fail. For each source column, the system constructs an embedding string that captures the semantic meaning of the column name, inferred data type, and representative sample values. ``` js // Constructing the embedding payload const inputText = `${col.name} | ${col.dataType} | samples: ${col.samples.slice(0, 5).join(', ')}`; ``` This concatenated string is then sent to the `Amazon Titan Text Embeddings V2` model to generate a 1024-dimensional vector. This process essentially transforms the semantic meaning of the source column into a numerical representation. ### The Warm Start: Retrieval-Augmented Generation (RAG)[¶](#the-warm-start-retrieval-augmented-generation-rag) With historical vectors stored along with their corresponding human-approved mappings, we can now close the RAG Flywheel loop. We utilize the `pgvector` extension in PostgreSQL and an HNSW (Hierarchical Navigable Small World) index to store and query these vectors efficiently. When a new file is uploaded, the system generates embeddings for its source columns and performs a cosine similarity search against historical vectors. We enforce a minimum cosine similarity threshold of `0.8` (80%) so that only sufficiently similar matches are retrieved. At the same time, to reduce historical drift, the similarity query ranks results by an approval count. A mapping confirmed multiple times across different uploads naturally outweighs a single anomaly. Because the system is human-in-the-loop, every suggestion must be reviewed by a operations specialist before it is accepted or used as historical knowledge. 1. **Fetch Historical Matches** : ``` -- Query to retrieve top historical matches for an incoming column embedding SELECT cm.source_column, cm.target_field AS target_column, COUNT(DISTINCT cm.mapping_config_id) AS approval_count, MAX(1 - (cm.embedding <=> $1::vector)) AS similarity FROM column_mappings cm WHERE cm.embedding IS NOT NULL -- Filter by minimum 0.8 cosine similarity (cosine distance <= 0.2) AND (1 - (cm.embedding <=> $1::vector)) >= 0.8 GROUP BY cm.source_column, cm.target_field ORDER BY approval_count DESC, similarity DESC LIMIT 3; ``` This query is executed once for each incoming source column. The `LIMIT 3` therefore caps the retrieved history for that column at three matches; it is not a limit across the entire uploaded file. #### Query Mechanics[¶](#query-mechanics) - `<=>` operator:`pgvector` 's cosine distance operator, similarity is calculated as`1 - cosine_distance` . - `$1::vector` is the embedding vector of the incoming source column. - `approval_count` : Consensus over noise. Grouping by (`source_column` ,`target_field` ) and ordering by`approval_count DESC` ensures that the most historically validated mappings are prioritized. - **Top-3 Limit** : We cap the results to top 3 high-confidence matches per column to keep the prompt concise and focused and avoid context pollution. - **Inject Historical Context into LLM Prompt** : high-confidence historical mappings are injected into the LLM prompt as`` : ``` js const getWarmStartPrompt = ( sourceColumns: Array, retrievedHistoryRows: Array, ) => ` The following mappings have been previously verified and approved by human users. Use these as your primary guide for similar columns. ${JSON.stringify(retrievedHistoryRows)} Please map the following incoming Excel columns: ${JSON.stringify(sourceColumns)} Return the final mappings as structured JSON only. Prioritize historical mappings. Assign a high confidence score (>90) for multi-upload matches (approvalCount > 1), and >80 for single-upload matches. Otherwise, fall back to logical deduction.`; ``` Every mapping suggestion goes through human review. When a user updates or approves a mapping, the approved decision is asynchronously embedded and stored in `column_mappings`, where it can inform future predictions. The historical embeddings are deliberately rebuildable: we can reset the historical context by deleting embeddings for old mappings from the database. Once deleted, those embeddings are no longer considered by future historical checks. This provides a simple way to remove stale or incorrect history without changing the rest of the mapping workflow. ### Early Signs From Production[¶](#early-signs-from-production) The suggestions are designed to accelerate the creation of mappings by leveraging continuous learning from previously approved mappings, rather than treating each upload in isolation. Since launch, we have tracked how often operations specialists accept suggestions without changes. The current acceptance rate is 85% and has increased as the historical index has grown with approved mappings — an encouraging signal that suggestion accuracy is improving on its own as data accumulates, and that the flywheel is helping. This reflects human judgments in live workflows rather than a formal evaluation against a labeled dataset. We plan to gather more data and build out full analytics for these metrics later in the year. ### Graceful Degradation When the LLM Is Unavailable[¶](#graceful-degradation-when-the-llm-is-unavailable) The initial mapping suggestion depends on the availability of the embedding and generative-model services. However, the system can continue to provide useful suggestions when the LLM is temporarily unavailable. As the number of reviewed mappings grows, the historical database becomes a valuable source of approved decisions. For a new source column, the system can query `pgvector` and return a previously approved mapping when the similarity score and approval history meet the configured thresholds. This allows the workflow to fall back to retrieval-based suggestions instead of failing completely when an LLM call cannot be completed. This fallback does not bypass human review. A retrieved mapping remains a suggestion until an operations specialist confirms it, and columns without a sufficiently similar historical match remain unresolved rather than being mapped by force. The quality of this fallback therefore depends on the availability of representative approved mappings, the embedding model, and the similarity threshold. Because historical mappings can become stale or incorrect, the embedding index is rebuildable. We can remove embeddings for old mappings, after which they are excluded from future retrievals. This makes it possible to reset the historical context without changing the rest of the mapping workflow. **NOTE** The API uses camelCase names such as `sourceColumn` and `confidenceScore`, while the database example uses snake_case names such as `source_column` and `target_field`. They represent the same concepts at different boundaries. ## Security and Operational Boundaries[¶](#security-and-operational-boundaries) Supplier column names and sample values are external input. They should be treated as untrusted data, kept clearly separated from system instructions, and limited to the minimum necessary content before being sent to a model. In particular, sample values can contain personal, commercial, or otherwise sensitive information, and a value that looks like an instruction must remain data rather than becoming part of the prompt's instructions. The same boundary applies to retrieved historical mappings. Access controls, retention rules, logging, and deletion procedures should be defined for both the original supplier data and its embeddings. Embeddings are not a substitute for anonymisation: they can still encode information about the source data. Human review, API validation, and the ability to delete historical embeddings together provide control over what enters and remains in the learning loop. ## Conclusion[¶](#conclusion) Automating schema mapping is not simply a matter of asking an LLM to guess the meaning of a column. The reliable part of this system comes from combining several bounded responsibilities: the model proposes a mapping, tool use constrains the response shape, and the API verifies that every result is valid before it reaches users or downstream systems. The RAG flywheel adds a learning mechanism to that foundation. A cold-start prediction can become useful historical knowledge only after a human has reviewed and approved it. Future uploads can then retrieve those approved mappings, rank them by similarity and past usage, and present better suggestions. Human feedback is therefore not just a correction step; it is the data that improves the next prediction. In production, operations specialists currently accept 85% of mapping suggestions without changes. Acceptance has increased as the history of approved mappings has grown—an encouraging operational signal, rather than a formal benchmark of mapping accuracy. The system is designed to reduce repetitive work, not to remove human judgment. Ambiguous or low-confidence mappings remain visible for review, while clear and repeatedly confirmed mappings can require less investigation. The historical index can also be reset when mappings become stale or incorrect, keeping the learning loop under engineering control. The broader lesson is that LLMs are most useful in data ingestion when they operate inside a controlled workflow. Retrieval provides relevant context, validation protects the contract, and human approval keeps the learning loop grounded in operational reality. *We're hiring! Do you like working in an ever evolving organization such as Zalando? Consider joining our teams as a [Machine Learning Engineer](https://jobs.zalando.com/en/jobs?category=Software+Engineering+-+Machine+Learning&utm_source=eng_blog&utm_content=structure-automating-data-inge)!*