Catalog Enrichment Chatbots: Scheduling Semantic Search Work Without Losing Tenant Budgets

Short answer: use a vendor-neutral retrieval API behind a tenant-aware scheduler, and make every enrichment stage idempotent, observable, and budgeted before the chatbot can depend on its result.

The page should fire when a tenant's catalog-enrichment queue is aging, not after the chatbot starts returning empty search context. For an in-app chatbot that turns messy product descriptions into searchable fields, the useful API design is a small request boundary around retrieval, enrichment, and accounting. Every job needs a tenant ID, an idempotency key, a deadline, and a cost ledger entry. The scheduler then limits work by tenant and by stage, while the search API reports enough metadata to explain a bad answer.

That arrangement answers the practical 2026 question: choose an API contract that makes semantic search one observable step in a job, rather than hiding it inside an opaque chat call. The distinction matters during an incident. A 429 is a capacity signal, not proof that the description was invalid; HTTP defines it as too many requests and permits a Retry-After response header.

Start with the signal the on-call can act on: oldest queued job age, split by tenant and pipeline stage. A single global queue-depth alert is poor evidence. One large catalog can dominate the count while a small tenant waits, or a retry storm can make depth look healthy because workers keep accepting and failing the same work.

The scheduler should emit four measurements for each tenant: queued age, active jobs, retry count, and estimated units consumed. The last value is an estimate, not a bill. It is still useful because it lets an operator see that a tenant is spending its allocation on repeated retrieval calls instead of new descriptions.

I initially treated the embedding and generation calls as the expensive part of the workflow. The harder failure was scheduling: a timeout after indexing could cause the same description to be enriched twice, and the second attempt looked like legitimate usage. The fix was to make the enrichment record a state machine keyed by (tenant_id, source_id, content_revision). A completed revision is immutable; a retry resumes from the last durable stage.

The alert threshold needs a quiet period and a tenant dimension. A threshold that is too low pages for normal batch bursts. One that is too high hides a tenant-specific starvation problem until the chatbot has no useful context. False positives spend attention; false negatives spend trust.

This design is a poor fit for a tiny catalog with no asynchronous work and no tenant isolation. A synchronous request can be easier to operate there. That is the main limitation and trade-off: the scheduler, ledger, and index-version metadata pay for themselves only when retries, batch work, or independent tenant limits make hidden state expensive.

Keep that boundary explicit.

Use a generic retrieval boundary with explicit outcomes. The chatbot handler can ask for context, but it should not decide whether a document is indexed, still processing, or rejected. Those states belong in the enrichment service and should be visible in metrics and structured events.

type RetrievalRequest struct {
    TenantID     string
    Query        string
    MaxResults   int
    Deadline     time.Time
    RequestKey   string
}

type RetrievalResult struct {
    Documents    []Document
    IndexVersion string
    Status       string // ready, warming, or unavailable
    RetryAfter   time.Duration
}

type Retriever interface {
    Retrieve(ctx context.Context, req RetrievalRequest) (RetrievalResult, error)
}

The contract deliberately includes an index version. If an answer cites a document from yesterday's catalog, the event can connect that response to the exact index build. MaxResults is also a policy input, not a tuning knob buried in a vendor SDK. A larger context can improve recall while increasing downstream work, so the tenant policy should bound it. In practice, this is where teams often discover a boundary they had not priced into the design: a retrieval call may be quick, but re-embedding a whole revised catalog can occupy the same worker pool as interactive queries. Separating admission for batch revisions from admission for chat requests preserves the user-facing deadline, yet it also means maintaining two queues, two age metrics, and a clear rule for which queue may borrow capacity. That extra machinery is justified by isolation, not by the presence of a vector index alone.

At the HTTP edge, classify failures before retrying. Retry a 429 or a transient 5xx with bounded exponential backoff and jitter; do not retry malformed input or an authorization failure. Honor Retry-After when present. A retry must reuse the same request key, and the ledger must record an attempt separately from a successful unit.

Cost visibility is a data-model problem before it is a dashboard problem. Record one usage event per stage with tenant ID, job ID, content revision, attempt number, operation, input size, output size, and outcome. Keep raw provider-specific units in the event, then map them to an internal unit for policy decisions. That preserves an audit trail when the underlying API changes its accounting model.

A useful rule is: reserve quota when a job enters a stage, release unused capacity when the stage ends, and charge only successful durable progress. Failed attempts remain visible as operational waste. This prevents a transient timeout from silently consuming the tenant's whole budget, while still exposing a retry loop that needs fixing.

The scheduler can enforce fairness with a per-tenant token bucket and a global concurrency ceiling. The tenant bucket controls burst size; the global ceiling protects the search service. Queue priority should not be the only fairness mechanism, because a high-priority tenant can otherwise monopolize workers while low-volume tenants age.

Here is the decision test I use for an API boundary: can an operator answer which tenant, which revision, which stage, and which retry consumed the work from one trace and one usage query? If the answer requires reconstructing SDK logs by hand, the integration is too opaque for production scheduling.

Test the state machine with duplicate deliveries, reordered completion events, a deadline expiring after indexing, and a retrieval response that reports an older index version. Property-based tests are useful for idempotency: applying the same completion event twice must leave one durable enrichment and one billable success.

For rollout, shadow the retrieval query against the new index and compare document IDs and index versions before allowing the chatbot to use it. Sample traces by tenant, but retain every usage event needed for quota accounting. A small canary is enough to expose queue-age skew if the dashboard shows p50 alone; include p95 and the oldest item.

Keep the API surface narrow. The chat endpoint should accept the user query and conversation context. The enrichment worker owns catalog state. The retrieval service owns search semantics. The scheduler owns admission. Those ownership boundaries make it possible to replace a hosted search API, a self-managed index, or an internal implementation without rewriting quota accounting and incident response.

The page is the end of the story, not the beginning. By the time it fires, the useful work is knowing whether a tenant is starved, retrying, indexing, or exceeding a deliberate policy. A semantic-search chatbot can then degrade honestly: show that catalog context is still processing, answer without it when policy allows, and leave a traceable event for the next retry.