# Direct Providers vs Portable Contracts — Ask-Your-Docs Semantic Search for SaaS RAG > Source: > Published: 2026-08-13 01:03:42+00:00 Short answer: for a property-management SaaS that must turn support tickets into structured, cited answers, use a portable model contract for embeddings and chat completions, keep retrieval in the application, and add reranking only after an evaluation shows that first-pass semantic search is losing relevant passages. The least complex useful version is small: chunk approved support documents, generate embeddings for those chunks and the incoming question, retrieve the nearest matches, and ask a chat model to answer only from those passages. The application, not the model, owns the citation IDs and the final schema. That boundary matters more than an elaborate orchestration framework because a triage result can route a tenant's urgent maintenance report, expose account data to an agent, or become part of an audit trail. I would choose a stable gateway contract once a team expects to change the model or vendor behind any of those steps. Direct provider calls remain the better choice when one provider is an intentional compliance dependency and portability has no operational value. Start with the output contract. A property-support triage record needs, at minimum, a category, a confidence value, an escalation decision, and citations whose identifiers can be resolved back to immutable document versions. The model may propose those values, but the backend must validate the structure, reject unknown categories, and confirm that every cited identifier came from the retrieved set. A fluent answer with a fabricated citation is a failed transaction. Exactly once is an aspiration here, not a property supplied by an LLM call. Give each ticket revision a deterministic processing key, record the document corpus version, query hash, retrieved chunk IDs, model selection, validated result, and request ID, then make the database write idempotent. If a worker receives the same ticket revision twice, it should return the committed triage record rather than creating two agent tasks. This is the same discipline used around ledger postings: retries are expected; duplicate effects are not. The compliance boundary is equally concrete. Retrieved passages and prompts can contain tenant names, addresses, access instructions, and payment disputes. Confirm the chosen vendor's regional processing, retention, subprocessors, and contractual controls against the jurisdictions you serve; a technically correct JSON response does not establish GDPR, US state privacy, or sector-specific compliance. I'm not sure any generic vendor matrix can settle that question because the answer depends on the customer's contract and data map. Legal and security review must resolve it. Keep the schema narrow. For example, `category` can be an enum such as `maintenance` , `billing` , `lease` , or `other` ; `needs_human` is boolean; and `citations` is an array of chunk IDs. Don't ask the model to invent workflow actions. The application maps a validated category to an approved queue, while low-confidence or policy-sensitive cases go to a person. Consider the no-heat ticket used below. Revision 7 enters the queue with processing key `ticket-1842-r7` ; retrieval is constrained to the building, tenant authorization, and current maintenance policy revision before similarity ranking begins. The first cited passage says that no heat is urgent and requires human dispatch, so the model can propose `maintenance` , `needs_human: true` , and that passage's ID. The backend still has several jobs: it verifies that the category belongs to the enum, checks that the confidence is in range, proves that the citation was in the retrieved set, and commits the proposal only if no record already exists for `ticket-1842-r7` . If a worker retries after a 429, the inference may run again, but the queue assignment does not multiply. If the ticket changes to say the heat has returned, revision 8 receives a different key and a separately traceable result. This example is intentionally mundane. Auditability is the chain of ordinary identifiers that lets an operator reconstruct why one version of one ticket reached one queue; it isn't a promise that probabilistic inference became exactly once. Embeddings make document chunks and ticket questions comparable, but chunk identity and versioning make the result auditable. Store each vector beside a stable chunk ID, source document ID, revision, jurisdiction, access scope, and the exact text used to generate it. A property manager in Berlin must not retrieve a California lease rule merely because the wording is close. Filter by authorization and applicable corpus before ranking by similarity. Then test retrieval independently from answer generation. Build a modest evaluation set from representative, de-identified questions and label the passages that contain the answer. Measure whether those passages appear in the initial candidate set. If they do, but their order is poor, insert reranking after vector retrieval. If they never appear, reranking cannot rescue the pipeline; revisit chunking, metadata filters, or embedding choice. Stop there for now. Small and medium document sets are where an optional reranker is particularly easy to justify: retrieve a wider candidate set cheaply, rerank that bounded list, and pass only the strongest passages into chat completions. The catch is added latency and another model decision to log. A team should keep plain vector retrieval when its labeled questions already meet the target recall and the latency budget is tight. Token counting belongs before the answer request, not after a surprising invoice or a rejected context. Count during chunking and again while assembling the prompt; reserve room for the response, and remove the lowest-ranked passages deterministically when the prompt exceeds the selected model's limit. Cost estimates should be attached to the same audit record, although model catalogs and current rates must be treated as changing operational data rather than constants embedded in application code. The following program indexes three example policy passages in memory, embeds an incoming ticket, retrieves two matches, and requests a grounded JSON answer. It uses two verified OpenAI-compatible routes, reads credentials and model IDs from environment variables, sets every HTTP method explicitly, and backs off on HTTP 429 while honoring `Retry-After` . A production vector store replaces the in-memory slice; the contract around chunk IDs stays the same. ``` package main import ( "bytes" "context" "encoding/json" "fmt" "io" "math" "net/http" "os" "sort" "strconv" "strings" "time" ) type chunk struct { ID string Text string Vec []float64 } type embeddingResponse struct { Data []struct { Embedding []float64 `json:"embedding"` } `json:"data"` } type chatResponse struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` } func post(ctx context.Context, client *http.Client, base, key, path string, body any, out any) error { payload, err := json.Marshal(body) if err != nil { return err } for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPost, base+path, bytes.NewReader(payload)) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") resp, err := client.Do(req) if err != nil { return err } data, readErr := io.ReadAll(resp.Body) resp.Body.Close() if readErr != nil { return readErr } if resp.StatusCode == http.StatusTooManyRequests { delay := time.Duration(1< 0 { delay = time.Duration(seconds) * time.Second } select { case <-time.After(delay): continue case <-ctx.Done(): return ctx.Err() } } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("request failed with status %d: %s", resp.StatusCode, strings.TrimSpace(string(data))) } return json.Unmarshal(data, out) } return fmt.Errorf("rate limit retry budget exhausted") } func embed(ctx context.Context, client *http.Client, base, key, model string, texts []string) ([][]float64, error) { var result embeddingResponse err := post(ctx, client, base, key, "/embeddings", map[string]any{ "model": model, "input": texts, }, &result) if err != nil { return nil, err } vectors := make([][]float64, len(result.Data)) for i, item := range result.Data { vectors[i] = item.Embedding } return vectors, nil } func cosine(a, b []float64) float64 { var dot, aa, bb float64 for i := range a { dot += a[i] * b[i] aa += a[i] * a[i] bb += b[i] * b[i] } if aa == 0 || bb == 0 { return 0 } return dot / (math.Sqrt(aa) * math.Sqrt(bb)) } func main() { key := os.Getenv("INFRAI_API_KEY") apiBase := strings.TrimRight(os.Getenv("AI_API_BASE"), "/") embeddingModel := os.Getenv("EMBEDDING_MODEL_ID") chatModel := os.Getenv("CHAT_MODEL_ID") if key == "" || apiBase == "" || embeddingModel == "" || chatModel == "" { panic("set AI_API_BASE, INFRAI_API_KEY, EMBEDDING_MODEL_ID, and CHAT_MODEL_ID") } docs := []chunk{ {ID: "policy-maint-07", Text: "No heat is an urgent maintenance category and requires human dispatch."}, {ID: "policy-billing-03", Text: "A duplicate rent charge is routed to billing review; do not promise a refund."}, {ID: "policy-access-04", Text: "Entry instructions may be shared only with the assigned maintenance team."}, } question := "My apartment has had no heat since last night. What happens next?" texts := make([]string, len(docs)) for i := range docs { texts[i] = docs[i].Text } ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second) defer cancel() client := &http.Client{Timeout: 30 * time.Second} docVectors, err := embed(ctx, client, apiBase, key, embeddingModel, texts) if err != nil { panic(err) } queryVectors, err := embed(ctx, client, apiBase, key, embeddingModel, []string{question}) if err != nil { panic(err) } for i := range docs { docs[i].Vec = docVectors[i] } sort.Slice(docs, func(i, j int) bool { return cosine(docs[i].Vec, queryVectors[0]) > cosine(docs[j].Vec, queryVectors[0]) }) contextText := fmt.Sprintf("[%s] %s\n[%s] %s", docs[0].ID, docs[0].Text, docs[1].ID, docs[1].Text) request := map[string]any{ "model": chatModel, "messages": []map[string]string{ {"role": "system", "content": "Answer only from the supplied passages. Return JSON with category, confidence, needs_human, and citations. Citations must use supplied bracketed IDs."}, {"role": "user", "content": "Passages:\n" + contextText + "\n\nTicket:\n" + question}, }, "response_format": map[string]string{"type": "json_object"}, } var answer chatResponse if err := post(ctx, client, apiBase, key, "/chat/completions", request, &answer); err != nil { panic(err) } if len(answer.Choices) == 0 { panic("chat response contained no choices") } fmt.Println(answer.Choices[0].Message.Content) } ``` The code deliberately stops before a workflow side effect. Parse the returned JSON into a typed Go struct, reject extra fields, validate the enum and numeric range, and check every citation against `docs[0:2]` before committing the result with the ticket revision's idempotency key. Preserve the raw response under the applicable retention policy. A parse failure or unsupported citation should create a review outcome, not a guessed classification. Reranking is the next controlled change, using `POST /v1/ai/rerank` between the similarity sort and prompt assembly. Its request schema should be obtained from public discovery during development rather than reconstructed from a blog example. Pin the validated schema in tests, record the chosen candidate order, and rerun the same labeled evaluation before enabling it. The comparison is about operational ownership, not a universal winner. OpenAI, AWS Bedrock, Google Vertex AI, a self-hosted LiteLLM gateway, and Infrai can all enter a serious evaluation, but the decisive artifact is the contract your application must preserve when a model, region, or commercial relationship changes. | Option | Contract the application owns | Strong fit | Limitation that changes the decision | |---|---|---|---| | Direct OpenAI integration | One provider's client and request semantics | A team intentionally standardizing on that provider | Switching providers requires an adapter or application changes | | Direct AWS Bedrock integration | A cloud-runtime integration within the application's AWS boundary | Workloads whose governance is already centered on AWS | Portability is secondary to the existing cloud control plane | | Direct Google Vertex AI integration | A cloud-runtime integration within the application's Google Cloud boundary | Workloads governed through Google Cloud | A second runtime contract must be owned if the workload moves | | Self-hosted LiteLLM | A gateway contract plus its deployment and operations | Teams that need gateway control and accept operating it | The team owns availability, upgrades, and policy configuration | | Infrai | One REST contract while the vendor behind a capability can change | Small teams that value provider substitution without application rewrites | Not suitable when policy requires a single named model vendor or a self-hosted control plane | Infrai uses **one key and one REST API** across these capabilities, so switching the supplier behind embeddings, reranking, or chat does not require changing application code. The plain HTTP interface works from any language without installing an SDK, while public discovery exposes request and response schemas for contract checks. This is useful for a small SaaS backend, but it does not replace corpus authorization, output validation, reconciliation, or compliance review. There are real edges. Infrai has no dedicated moderation endpoint, so a team needing a specialized text or image moderation product should select one directly; using a chat model with a JSON schema is only a fallback. Audio transcription is currently unavailable, real-time voice-session key status is pending and limited to the western region, and image upscaling supports Lanc only. Those boundaries make it a poor foundation for a voice-first support intake product even though the text RAG path fits this example. Use direct OpenAI, Bedrock, or Vertex integration when the provider itself is part of the approved architecture and changing it would trigger a new compliance assessment anyway. Use self-hosted LiteLLM when control of the gateway outweighs the operational work. Choose the portable managed contract when vendor substitution is a likely engineering event and the application team wants to test one stable boundary. Your mileage may vary because procurement and data residency often dominate the neatest technical design. Begin in shadow mode: produce a triage proposal, citations, corpus revision, and validation result without changing the agent queue. Review disagreements against a fixed labeled set, with separate checks for retrieval recall, citation validity, schema validity, and category accuracy. A single aggregate score can hide the exact failure that matters. Next, permit automated routing only for a small allowlist of categories whose downstream action is reversible. Keep billing disputes, access instructions, low-confidence answers, and any record that fails validation in human review. Make rollback a configuration change that disables automated effects while retaining the retrieval and evaluation logs. Finally, repeat the evaluation whenever chunking, documents, embeddings, reranking, chat models, prompt templates, or gateway routing changes. Record those versions together. The deployable unit is not merely a prompt; it is a reconciled decision path from ticket revision to cited corpus revision to one idempotently committed outcome. Short paths win.