# Repetitive Chatbot API Queries: 4 Gates That Minimize Token Cost in CRM

> Source: <https://dev.to/rasmusberg6592/repetitive-chatbot-api-queries-4-gates-that-minimize-token-cost-in-crm-2old>
> Published: 2026-11-01 03:36:03+00:00

Short answer: do not choose a chatbot API from its advertised token rate. For repetitive in-app user queries, the lowest reliable cost comes from a cache contract around the whole request: normalize the input, retrieve a stable answer when the evidence is unchanged, route only misses to generation, and validate the structured result before writing CRM actions. For a B2B SaaS sales-call summarizer, correctness is the boundary that matters. A cheap malformed action is still an incident.

The API is only one part of the bill.

Correctness wins.

The practical decision rule is four gates: cache eligibility, evidence identity, schema validation, and an explicit miss budget. If a proposed runtime cannot expose enough signals to enforce those gates, its apparent savings are not operational savings. The tradeoff is deliberate: a stricter cache creates more misses, but a permissive cache can attach the wrong follow-up to the wrong account.

Misses are normal.

Repetition is rarely byte-for-byte repetition. Account executives ask “what did Acme agree to next?” with different punctuation, transcript excerpts, and time ranges. A naive key treats each request as new; an over-aggressive key merges two calls whose commitments differ. The failure mode is quiet: a plausible old answer becomes a new CRM task.

I separate the request into stable and changing material. The stable part contains the output schema, tenant policy, and instruction version. The changing part contains the transcript slice and the question. Hash the canonical representation of both. A cache hit is valid only when both hashes, the tenant identifier, and the source revision match.

This also gives capacity planning a useful shape. Let `N` be requests per hour, `H` the verified hit ratio, and `T` the average input tokens for a miss. The generation budget is approximately `(N * (1 - H) * T)` input tokens, plus output tokens for misses. At a hypothetical 95% hit ratio, five of every 100 requests still consume the miss path, so reserve capacity for bursts rather than treating the average as a guarantee. Measure `H` by tenant and by route; a single fleet-wide average hides a noisy account that is repeatedly missing the cache.

The honest answer is the one that lets the application prove when a query is reusable. An API with a lower nominal token cost can lose that advantage if it cannot expose usage, request identity, or enough response metadata for the cache and SLO. Conversely, a higher-cost miss path can be acceptable when a verified hit avoids generation for most stable user queries.

The limitation is important: this method is a poor fit for rapidly changing information, open-ended research, or requests whose answer depends on hidden session state. Those workloads need fresher retrieval or generation, and forcing them through a repetitive-query cache creates stale output rather than meaningful cost control.

Cache the decision, not an unexamined model response. The cached value should contain the normalized question, source revision, schema version, expiration policy, and the validated action object. It should not be accepted merely because it is syntactically valid JSON.

For example, the action schema might require an owner, due date, and evidence span. A response that says “follow up next week” can parse successfully while still lacking a deterministic due date. Treat that as a miss or a review state, according to the product contract. Do not silently fill it with the current date.

The safe implementation can stay small. The important detail is that cache lookup happens after canonicalization and validation, while the write happens only after the generated object passes the same checks.

```
package runtime

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "strings"
)

type Action struct {
    Owner       string `json:"owner"`
    DueDate     string `json:"due_date"`
    EvidenceRef string `json:"evidence_ref"`
}

type Cache interface {
    Get(ctx context.Context, key string) ([]byte, bool, error)
    Put(ctx context.Context, key string, value []byte) error
}

func cacheKey(tenant, schema, policy, sourceRevision, question string) string {
    canonical := strings.Join([]string{
        tenant, schema, policy, sourceRevision,
        strings.Join(strings.Fields(question), " "),
    }, "\x1f")
    sum := sha256.Sum256([]byte(canonical))
    return hex.EncodeToString(sum[:])
}

func validAction(raw []byte) (Action, error) {
    var action Action
    if err := json.Unmarshal(raw, &action); err != nil {
        return Action{}, err
    }
    if action.Owner == "" || action.DueDate == "" || action.EvidenceRef == "" {
        return Action{}, errors.New("required CRM action field is missing")
    }
    return action, nil
}
```

The example deliberately does not call a particular provider. The interface is the useful boundary: a managed service can implement the generator, or a self-hosted component can do so, while the application retains ownership of cache keys, validation, and audit data. I would keep that ownership even when the provider offers a built-in cache, because portability is useful only if the correctness checks remain portable too.

The choice is not “managed is cheap” versus “self-hosted is cheap.” It is a trade among request variance, on-call capacity, data controls, and lock-in. A platform team should own the controls that protect the SLO even when the generation backend changes.

| Boundary | Buy or managed path | Build or self-host path | 
|---|---|---|
| Generation | Faster integration and less model-serving work; usage limits and provider behavior become dependencies. | More control over placement and tuning; serving capacity, upgrades, and incident response become team work. | 
| Repetition control | Use only if request identity, cache behavior, and usage metrics are observable enough to enforce policy. | Implement a cache and key contract beside the application; correctness tests become your responsibility. | 
| Structured actions | Keep schema validation and rejection in the application. | Same rule; owning the generator does not make invalid actions safe. | 
| SLO | Define latency and correctness targets independent of backend choice. | Carry the same targets, plus capacity headroom for failover and maintenance. | 

For this workload, the generation backend is replaceable; the action contract is not. That is the useful lock-in test. If changing a backend requires rewriting CRM validation, the abstraction is in the wrong place.

Verification needs more than a hit counter. Record cache outcome, schema version, source revision, validation result, latency, input and output token counts, and the reason for a miss. Keep transcript content out of ordinary operational logs unless the data policy explicitly permits it; identifiers and hashes are usually enough to correlate a request.

Set an SLO for valid action production, such as the proportion of accepted outputs that contain all required fields and a traceable evidence reference. Set a separate latency SLO. A high hit ratio can conceal stale answers, so alert on source-revision mismatches and post-write corrections as well.

Rollback should be boring. Version the key with the prompt or schema revision, stop writes for a bad revision, and fall back to a review queue or a fresh generation path. Do not delete the prior cache indiscriminately: an older, validated result may still be correct for its original source revision. Expire it by policy, then compare miss rate and validation failures before restoring writes.

The runbook is therefore straightforward: sample hits and misses, replay a fixed set of redacted questions, verify that changed transcript evidence changes the key, and test that malformed actions cannot reach the CRM writer. Four gates. No pricing table can replace them.