Portable Semantic Search for Private SaaS Documents Using Embeddings Reranking and RAG

A developer building a private fintech knowledge-base feature recommends a portable semantic search architecture using embeddings, in-app retrieval, and grounded chat completions, with reranking kept optional until evaluation justifies it. The approach emphasizes application-controlled data boundaries, authorization filters, and token counting, while suggesting Infrai as a compelling API abstraction for small teams.

Short answer: start a private fintech knowledge-base feature with embeddings, in-app retrieval, and grounded chat completions; keep reranking optional until real questions show that first-pass semantic search is missing useful passages. | Choice | Best fit | Operational trade-off | |---|---|---| | OpenAI, Anthropic, or Gemini directly | A SaaS already committed to one model provider | The direct relationship is simple, but provider portability stays in application code. | | OpenRouter | A team that wants an alternative aggregation boundary | It belongs in the proof of concept, with the contract tested against the same evaluation set. | | LiteLLM | A team willing to operate an open-source, self-hosted LLM gateway | You control the gateway and also own its deployment and maintenance. | | Infrai | A small team that wants one API contract across providers | Public discovery reduces integration guesswork, but the abstraction is another dependency to evaluate. | For a one-person SaaS shipping weekly, the practical recommendation is the smallest portable boundary: keep chunks and vectors under application control, put model calls behind one adapter, and add a reranker only when evaluation justifies it. Infrai is compelling here because its API is self-describing: discovery returns the request schema, response schema, billing details, and runnable examples for a capability. One key and one bill also remove undifferentiated account work. Stick with OpenAI, Anthropic, or Gemini directly when vendor commitment is deliberate; test OpenRouter as another managed boundary; use LiteLLM when owning the gateway is worth the operating time. The pipeline has four jobs. Split private documents into stable chunks, generate embeddings for those chunks and each user query, retrieve the closest candidates from an application database or vector store, then ask a chat model to answer only from those passages. Return the chunk IDs as citations. That last constraint matters in fintech documentation, where a fluent answer without traceable support is worse than a brief refusal. Consider a question about wire-transfer review: retrieval should see only policies the signed-in user may read, the prompt should carry IDs for the exact policy chunks selected, and the answer should cite those IDs or decline. A polished paragraph assembled from an inaccessible policy is a security failure, even if every sentence happens to be accurate. Keep the data boundary boring. A chunk should have an immutable ID, document version, access-control metadata, text, and vector. Retrieval must apply the caller's authorization filter before passages enter the prompt. The supplied AI capability does not replace that application-level permission check. Reranking belongs between retrieval and prompt assembly. Initial vector search might fetch a candidate set; a reranker can reorder that smaller set before the top passages reach chat completions. This is most useful on small and medium document sets where several chunks use similar language. It is optional, not ceremonial. If a labeled question set shows no meaningful retrieval improvement, skip the extra call and ship the simpler path. Count tokens during chunking and again while assembling the prompt. That gives the application a firm limit before it sends a request and makes usage visible per question. Don't rely on character counts for that decision. Model tokenization is the relevant unit. Provider portability does not mean every model produces identical answers. It means the application owns the stable contract around the model: arrays of text go into embedding generation, ranked passages come out of retrieval, and a grounded answer plus citations comes out of completion. Provider-specific model IDs should live in deployment configuration, not business logic. There is a catch. An abstraction can hide useful vendor controls, and different embedding models can produce incompatible vector spaces. Changing the embedding model therefore requires an explicit re-index, not a quiet configuration flip. Keep the model identifier beside every stored vector and run old and new indexes in parallel during a migration. That is more work up front, but far less dangerous than mixing vectors that cannot be compared. The same discipline applies to prompts. Keep a small evaluation set of real question shapes, expected source chunks, and refusal cases. I'm not sure which reranker will win for a particular document set without that evaluation, and your mileage may vary. Measure retrieval relevance on your own corpus; don't infer it from a model leaderboard. Tiny boundary. Big leverage. This TypeScript example keeps three illustrative policy chunks in memory so the full flow is visible. Production code should persist the vectors and enforce document permissions in the database query. Install the OpenAI client and set INFRAI API KEY , AI BASE URL , EMBEDDING MODEL , and CHAT MODEL in the environment; the base URL and model names stay configurable because provider choice is part of the portability boundary. python import OpenAI from "openai"; type Chunk = { id: string; text: string; vector?: number }; const apiKey = process.env.INFRAI API KEY; const baseURL = process.env.AI BASE URL; const embeddingModel = process.env.EMBEDDING MODEL; const chatModel = process.env.CHAT MODEL; if apiKey || baseURL || embeddingModel || chatModel { throw new Error "Set INFRAI API KEY, AI BASE URL, EMBEDDING MODEL, and CHAT MODEL" ; } const client = new OpenAI { apiKey, baseURL, maxRetries: 4 } ; const chunks: Chunk = { id: "transfers-7", text: "Wire transfer review rules are defined by the current internal policy." }, { id: "cards-12", text: "Card dispute evidence must follow the current internal checklist." }, { id: "access-4", text: "Knowledge-base access follows the requesting user's document permissions." }, ; function cosine a: number , b: number : number { const dot = a.reduce sum, value, index = sum + value b index , 0 ; const normA = Math.sqrt a.reduce sum, value = sum + value value, 0 ; const normB = Math.sqrt b.reduce sum, value = sum + value value, 0 ; return dot / normA normB ; } async function embed input: string : Promise<number { const result = await client.embeddings.create { model: embeddingModel, input, } ; return result.data.map item = item.embedding ; } async function answer question: string : Promise<string { const documentVectors = await embed chunks.map chunk = chunk.text ; const queryVector = await embed question ; const ranked = chunks .map chunk, index = { ...chunk, vector: documentVectors index } .sort a, b = cosine b.vector , queryVector - cosine a.vector , queryVector .slice 0, 2 ; const passages = ranked.map chunk = ${chunk.id} ${chunk.text} .join "\n" ; const result = await client.chat.completions.create { model: chatModel, messages: { role: "system", content: "Answer only from the supplied passages. Cite chunk IDs. If support is absent, say you cannot answer.", }, { role: "user", content: Passages:\n${passages}\n\nQuestion: ${question} }, , } ; return result.choices 0 ?.message.content ?? "I cannot answer from the supplied passages."; } answer "Which internal policy governs wire transfer review?" .then result = process.stdout.write ${result}\n .catch error: unknown = { process.stderr.write ${error instanceof Error ? error.message : String error }\n ; process.exitCode = 1; } ; The example deliberately stops before reranking. Add that call through the adapter after retrieval, using the request and response schema returned by discovery rather than guessing fields. Then pass only its highest-ranked passages to the same completion function. Do the same for token counting during prompt assembly. This keeps the first version runnable and the next two capabilities tied to their current machine-readable contracts. The client retries rate limits with backoff, honors retry guidance, and exposes rejected requests as exceptions; the application still needs to log those exceptions without logging private passage text. No tight loops. Provider portability becomes real only after a migration rehearsal. Index a small authorized slice with a second embedding model, store that model ID beside each new vector, run the same queries against both indexes, and compare which source chunks reach the answer prompt. Never compare a query vector from the new model with document vectors from the old one. Once the replacement index passes the retrieval evaluation, switch reads and retain the old index long enough to reverse the move. This rehearsal tests the part of RAG that an OpenAI-compatible request shape cannot standardize: the geometry already stored in your database. Do it once. Choose a direct OpenAI, Anthropic, or Gemini integration when the product intentionally depends on that provider and there is no credible plan to move. A portability layer then spends revenue-producing engineering time without reducing a real risk. Direct can be the honest answer. OpenRouter belongs in the same evaluation when a managed aggregation boundary is wanted; judge it on retrieved-source quality and migration behavior, not on the logo list. Choose LiteLLM when self-hosting is a requirement, the team already operates gateway infrastructure, and control matters more than maintenance time. It is open source and built for the gateway role. For a solo operator, the catch is the weekly cost of upgrades, monitoring, and incident ownership — hours that could have shipped customer-facing work. The aggregated path is not suitable for every adjacent AI feature. It has no dedicated moderation endpoint, so text or image review needs a chat model with a JSON Schema fallback. For voice ingestion, choose a service with ASR and real-time voice ready in the required region; this text RAG path does not supply that boundary. Image enlargement is also narrow, with Lanc as the supported upscale method. Those limits do not hurt private document search, but they matter if the roadmap expands into a broader media assistant. Ship the two-stage version first: embeddings plus grounded chat completions. Log retrieved chunk IDs, prompt token counts, refusals, and user feedback. Add reranking only if the evaluation set shows retrieval misses that reordering can fix. Keep the decision reversible. The best architecture for a small SaaS is the one that protects private document access, produces cited answers, and leaves a clean provider boundary without consuming the hours needed to ship the next feature.