Low-Cost Summarization Using OpenAI API Token Reduction in Node.js (and a Portable Choice) An SRE engineer outlined a provider-neutral approach to low-cost code-review summarization with the OpenAI API, cutting spend by trimming input tokens, chunking oversized diffs, and capping output tokens rather than polishing prompts. The engineer recommends budgeting an eight-chunk diff as eight calls plus a merge call, measuring the tail of the completion distribution instead of the median, and preserving a fallback contract when a provider rejects context. A Go request path sends the key only to the API host, checks non-success responses, and retries rate limits with exponential backoff. Short answer: Low-cost summarization using the OpenAI API comes from reducing input tokens, capping output tokens, chunking long diffs, and keeping the Node.js or Go caller behind a provider-neutral contract; that combination lowers API cost without making one vendor the architecture. The practical answer is to reduce the text before it reaches the model, cap the output deliberately, and keep a provider-neutral summary contract around the call. For a logistics platform that turns code-review changes into structured findings, that usually means counting tokens, chunking oversized diffs, asking for a short JSON-shaped result, and retaining a fallback path when a provider rejects the context. The largest portability mistake is optimizing the prompt around one vendor's client library and then discovering that the actual workflow is inseparable from it. I care about this as an SRE concern, not a prompt-polishing exercise. A shorter request lowers variable API consumption, but it also changes latency, retry volume, and the amount of evidence an auditor can inspect. The right target is therefore a bounded SLO for useful findings, not the smallest possible completion. Before choosing a model, define the output that downstream systems can tolerate. In this example, a review worker receives a pull request diff and returns findings with a severity, a file, a line, a concise explanation, and a confidence value. The worker does not need a prose essay. It needs stable records that can be stored, displayed, and reprocessed without a human guessing where one finding ends. That distinction matters for token reduction. Removing repeated instructions and irrelevant files is usually safer than aggressively shortening the diff itself. A changed route-planning function may look verbose, yet a deleted validation branch can be the most important evidence. I would keep the diff hunks, strip generated artifacts and duplicated repository context, then cap the completion at the number of findings the review product can actually show. I initially treated a generous output limit as harmless because most summaries would be short. The capacity-planning test changed my mind: the tail of the distribution, not the median review, drives queue pressure and retry cost. A bounded completion gives the worker a predictable upper limit, while chunking makes the input limit an explicit workflow decision instead of an exception discovered in production. For a concrete policy, an eight-chunk diff must be budgeted as eight calls plus a merge call, and every retry belongs in that estimate. Measure the tail. Use a three-stage path: The count is a planning signal, not a promise that every provider tokenizes text identically. A local tokenizer library such as tiktoken is useful for estimating a known encoding, while a token-counting endpoint can be used when the selected runtime exposes one. The estimate should feed admission control: accept, chunk, or fall back. It should not be used as a billing assertion. The fallback should preserve the contract. If a call is rejected for a token limit, retrying the identical payload is wasteful; reduce the chunk, remove low-value context, and record that the result was produced from a reduced view. If the final merge cannot fit, concatenate findings, deduplicate by file and line, and mark the merge as incomplete rather than silently returning an empty review. Here is the core request path in Go. It uses the OpenAI-compatible chat route, sends the key only to the API host, sets an explicit method, checks non-success responses, and retries rate limiting with exponential backoff. The max tokens value is a policy knob: it is intentionally small enough to protect the worker's queue SLO, but it must be tested against the finding schema and language of the repository. package main import "bytes" "encoding/json" "fmt" "io" "net/http" "os" "strconv" "time" type message struct { Role string json:"role" Content string json:"content" } type requestBody struct { Model string json:"model" Messages message json:"messages" MaxTokens int json:"max tokens" Temperature float64 json:"temperature" } func summarize chunk, model string byte, error { body, err := json.Marshal requestBody{ Model: model, Messages: message{ {Role: "system", Content: "Return only JSON findings with severity, file, line, explanation, and confidence."}, {Role: "user", Content: "Review this logistics code diff for correctness risks:\n" + chunk}, }, MaxTokens: 300, Temperature: 0, } if err = nil { return nil, err } apiKey := os.Getenv "INFRAI API KEY" if apiKey == "" { return nil, fmt.Errorf "INFRAI API KEY is required" } for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequest http.MethodPost, "https://placeholder.invalid", bytes.NewReader body if err = nil { return nil, err } req.Header.Set "Authorization", "Bearer "+apiKey req.Header.Set "Content-Type", "application/json" baseURL := os.Getenv "OPENAI BASE URL" if baseURL == "" { return nil, fmt.Errorf "OPENAI BASE URL is required" } req.URL, err = req.URL.Parse baseURL + "/chat/completions" if err = nil { return nil, err } resp, err := http.DefaultClient.Do req if err = nil { return nil, err } response, readErr := io.ReadAll resp.Body resp.Body.Close if readErr = nil { return nil, readErr } if resp.StatusCode == http.StatusTooManyRequests { delay := time.Duration 1<