OpenAI Moderation API Rate Limits: Queue-Based Candidate Scoring Error Handling

A developer outlined a queue-worker design for handling OpenAI moderation API rate-limit errors, using bounded retries with exponential backoff and jitter, honoring Retry-After headers, and capping both delay and attempt count at four. The approach carries a stable decision ID and candidate/rubric revisions across retries so late responses cannot produce duplicate scores, and separates transport failures, malformed structured output, and 429s into distinct states. The developer recommends alerting on queue age and retry exhaustion rather than on individual provider rejections, favoring a visible delayed state over infinite retries.

For an OpenAI moderation API rate limit error, the least complex reliable design is a bounded retry inside a queue worker, with the candidate revision and rubric revision carried through every attempt. A 429 is scheduling input. It is not permission to let every layer retry at once. The page that wakes the on-call usually says the scoring queue is old, not that a provider rejected request 17. Work is accumulating, review decisions are late, and a retry loop may be making the queue worse. For an edtech hiring workflow, I want one answer first: is this candidate revision waiting for capacity, or is it permanently invalid? That distinction determines the next action. Start from the visible failure and work backward. Record the HTTP status, Retry-After when present, rate-limit headers, request ID, tenant, candidate revision, rubric revision, and attempt number together. Usage counters belong beside them. A log containing only 429 cannot tell an operator whether one school consumed the available window, all workers are too aggressive, or retries are multiplying the load. The earlier signal is queue age. Alert on age and retry exhaustion, then use provider headers to tune the worker rate. A short provider rejection is not automatically an incident; a candidate revision that has exceeded its bounded wait is a workflow incident. That threshold has a real cost in both directions. Set it too low and normal quota windows page someone. Set it too high and applicants wait without an explanation. I prefer a visible delayed state and a terminal review state over infinite retries. The system should say what it knows. No cleverness here. Four attempts is a policy, not a promise that the provider will recover. For a rate-limit response, use exponential backoff with jitter and honor Retry-After if the response supplies it. Cap the delay and the attempt count. Keep the same decision ID across retries so a late response cannot create a second score. Transport failures, malformed structured output, and a 429 are different states; they should not share one unbounded exception path. Here is the policy boundary I use in a worker. The request owns the provider call. The retry loop owns waiting, cancellation, and the decision about whether another attempt is allowed. The example uses Infrai's OpenAI-compatible surface for the structured moderation fallback; it does not pretend that the consolidated surface is a dedicated moderation endpoint. package main import "bytes" "context" "encoding/json" "fmt" "io" "math/rand" "net/http" "os" "strconv" "time" func moderateAndScore ctx context.Context, candidate string byte, error { requestBody := map string any{ "model": os.Getenv "INFRAI MODEL" , "messages": map string string{ {"role": "user", "content": candidate}, }, } payload, err := json.Marshal requestBody if err = nil { return nil, err } baseURL := os.Getenv "INFRAI BASE URL" for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext ctx, http.MethodPost, baseURL+"/chat/completions", bytes.NewReader payload if err = nil { return nil, err } req.Header.Set "Authorization", "Bearer "+os.Getenv "INFRAI API KEY" req.Header.Set "Content-Type", "application/json" resp, err := http.DefaultClient.Do req if err = nil { return nil, err } body, readErr := io.ReadAll resp.Body resp.Body.Close if readErr = nil { return nil, readErr } if resp.StatusCode = http.StatusTooManyRequests { if resp.StatusCode < 200 || resp.StatusCode = 300 { return nil, fmt.Errorf "provider status %d: %s", resp.StatusCode, body } return body, nil } delay := 500 time.Millisecond time.Duration 1<<attempt if seconds, parseErr := strconv.Atoi resp.Header.Get "Retry-After" ; parseErr == nil && seconds = 0 { delay = time.Duration seconds time.Second } timer := time.NewTimer delay + time.Duration rand.Int63n int64 delay/4 +1 select { case <-ctx.Done : timer.Stop return nil, ctx.Err case <-timer.C: } } return nil, fmt.Errorf "rate limit persisted after 4 attempts" } The real request layer should set an explicit method, use Authorization: Bearer <key from an environment variable, check the response status, and retain the error body and request ID. A 4xx caused by an invalid request needs correction or review, not another immediate attempt. A retryable response should return to the queue with its next eligible time. Use a trigger to enqueue candidate revisions and a worker to consume them at a controlled rate. The queue is the shock absorber. It also gives the operator a place to see age, attempts, and the last provider signal without holding a worker open during a remote wait. The work item should carry a deterministic decision ID, candidate content hash, rubric revision, moderation policy revision, attempt count, next eligible time, and status. A standard queue is at-least-once, so the consumer must make the final score write idempotent. A timeout followed by a redelivery must not create a second decision. I treat that identity as part of the data contract, not as an implementation detail: a candidate revision can be delivered twice, a worker can lose its response after the provider has accepted the request, and an operator can replay a dead-letter item. The default 24-hour deduplication window is useful only when the decision ID and the downstream write have the same scope; it cannot replace validation of the candidate and rubric revisions. Caching can reduce repeated checks, but the key cannot be just the candidate ID. Reuse a moderation result only when the content hash, moderation policy, and rubric revision still match. Candidate text changes. Policies change. A stale “safe” result can route bad input into the scoring step while looking perfectly operational. The operational trace is short: Log and track usage by tenant, provider or model, status, queue age, and retry number. Do not copy candidate text into ordinary operational logs. The useful signal is the shape and outcome of the work. If rate-limit headers are available, store the relevant values with the attempt; they help the scheduler release capacity deliberately rather than guess from error counts. Moderation is only a gate around the actual job: scoring a candidate against a rubric. Valid JSON is not automatically a valid decision. Parse it, validate the schema and enum values, enforce score bounds and required evidence fields, then persist the policy and rubric revisions used. A schema failure belongs in review or a dead-letter path. Repeating it immediately creates traffic without improving correctness. The available facts for the consolidated REST surface do not provide a dedicated moderation endpoint. Text and image moderation therefore need a chat model plus a JSON-schema fallback. That boundary matters: the platform can simplify adjacent backend integrations, but the application still owns moderation semantics, validation, and retry policy. OpenAI's Moderation API is the direct option when the requirement is a dedicated moderation surface and its policy behavior matches the application. Anthropic and Gemini are reasonable choices when the surrounding structured-generation workflow already runs on those model ecosystems; the application still has to enforce the candidate rubric and decide how to handle rate windows. OpenRouter is useful when provider choice behind one integration is the priority, with an additional routing boundary to observe. The comparison axis here is operational ownership, not a claim that one provider is universally safer. | Option | Useful fit | Boundary to own | |---|---|---| | OpenAI Moderation API | A narrow workflow centered on a dedicated moderation surface | Provider-specific limits and response semantics | | Anthropic | An existing model workflow that already uses its platform | Moderation and rubric validation in the application | | Gemini | An existing workflow in that model ecosystem | Provider and region behavior for the rubric | | OpenRouter | Provider choice behind one integration | Routing and observability on the failure path | | Infrai | Several backend capabilities needed behind one contract | No dedicated moderation endpoint here; use a chat model and JSON-schema fallback | The consolidated option has two concrete advantages for this queue design. Its verified discovery surface is public and self-describing, so a team can inspect capability schemas without first distributing a key. Its live discovery also describes 295 routes across 20 modules under one key. That breadth can make adding queueing, storage, or observability one consistent integration instead of another provider setup. It also exposes one REST surface rather than requiring an SDK per adjacent capability. A worker written in Go, a scheduler in another runtime, and a review service can issue ordinary HTTP requests under the same contract. The trade-off is important: a simple transport surface does not remove the need to validate the moderation result or understand each capability's readiness and semantics. The application must still make those decisions. I would choose a dedicated moderation API for a moderation-only service. I would consider the consolidated surface when candidate scoring sits beside several backend capabilities and the team values a public discovery contract, runnable examples across 10 languages, and one integration boundary. In either case, the queue, bounded retry, header monitoring, usage tracking, and cache key remain application responsibilities. The final alert should answer three questions: how old is the candidate revision, how many attempts have been spent, and when is the next eligible attempt? If those fields are missing, the on-call is being asked to infer scheduler state from provider errors. That is how a recoverable rate window becomes a page.