Tracing 4 Missing Feature Flag Keys After Delete-Recreate Across Logistics Cost Cohorts

Short answer: treat a deleted or recreated feature flag as an identity change, return the caller's typed default when its key can't be resolved, and emit one evaluation event that separates a 404 lookup from an intentional fallback. For a logistics experiment, join that event to tenant, cohort, model usage, and shipment outcome; otherwise a quiet default can make the new cohort look cheaper simply because it stopped receiving the feature.

The least complex reliable design is an append-only evaluation log plus a tiny replay test. The production evaluator may live in Node.js, while a Python notebook checks the resulting JSON before a rollout advances. This keeps the notebook-to-production path honest: the same event contract answers both the incident question (why was this tenant assigned the default?) and the experiment question (which cohort actually incurred the AI cost?).

Don't collapse those questions into one boolean.

A lookup has at least four outcomes worth preserving: an explicit variant, a missing key, a control-plane 404, and a locally unavailable snapshot. The application may choose the same fallback value for the last three, but observability must not label them as the same event. value=false describes behavior. reason=KEY_NOT_FOUND or reason=REMOTE_NOT_FOUND describes why that behavior happened.

That distinction matters after delete and recreate. A human-readable key can be reused while the underlying flag definition has a new lifecycle. If analysis groups only by key, evaluations before deletion and after recreation become one misleading time series. Record a stable definition identifier or revision when the flag system provides one. When it doesn't, record the configuration revision and deployment revision, then treat the recreate timestamp as an analysis boundary. I'm not sure every flag provider exposes a definition identifier; the provider's evaluation metadata is what resolves that uncertainty.

The application contract should be boring: callers always supply a typed default, evaluation always returns a typed value, and every result carries a reason. A 404 belongs in metadata, not in business control flow. In other words, don't let an HTTP status decide whether a shipment gets an AI-generated exception summary. Let the declared default decide, while the status remains searchable evidence.

This is the first checkpoint.

For the logistics experiment, assume ai_exception_summary is enabled for selected tenant cohorts. Each summary consumes model tokens, so the decision record needs enough context to attribute spend without putting prompt text or sensitive shipment data into the flag event. A practical event contains tenant ID, experiment cohort, flag key, definition revision, evaluated value, reason, evaluation timestamp, request correlation ID, and model usage identifiers. The usage ledger remains a separate dataset; correlation IDs and tenant IDs connect the two.

The following standard-library script is deliberately small enough for a notebook and strict enough for CI. Its data is illustrative, not a benchmark. It replays four evaluation outcomes, applies the declared default, and rejects the dangerous case where a fallback has no reason. The same JSON shape can be emitted by a Node.js service even though the analysis harness is Python.

from __future__ import annotations

from dataclasses import asdict, dataclass
from enum import StrEnum
from typing import Any

class Reason(StrEnum):
    TARGET_MATCH = "TARGET_MATCH"
    KEY_NOT_FOUND = "KEY_NOT_FOUND"
    REMOTE_NOT_FOUND = "REMOTE_NOT_FOUND"
    SNAPSHOT_UNAVAILABLE = "SNAPSHOT_UNAVAILABLE"

@dataclass(frozen=True)
class Lookup:
    tenant_id: str
    cohort: str
    flag_key: str
    definition_revision: str | None
    configured_value: bool | None
    reason: Reason
    correlation_id: str

@dataclass(frozen=True)
class Evaluation:
    tenant_id: str
    cohort: str
    flag_key: str
    definition_revision: str | None
    value: bool
    default_used: bool
    reason: str
    correlation_id: str

def evaluate(lookup: Lookup, default: bool) -> Evaluation:
    resolved = lookup.configured_value
    default_used = resolved is None

    if default_used and lookup.reason == Reason.TARGET_MATCH:
        raise ValueError("fallback evaluation requires a fallback reason")
    if not default_used and lookup.reason != Reason.TARGET_MATCH:
        raise ValueError("configured value requires TARGET_MATCH")

    return Evaluation(
        tenant_id=lookup.tenant_id,
        cohort=lookup.cohort,
        flag_key=lookup.flag_key,
        definition_revision=lookup.definition_revision,
        value=default if default_used else resolved,
        default_used=default_used,
        reason=lookup.reason.value,
        correlation_id=lookup.correlation_id,
    )

def replay(rows: list[Lookup], default: bool) -> list[dict[str, Any]]:
    return [asdict(evaluate(row, default)) for row in rows]

rows = [
    Lookup(
        "tenant-north", "control", "ai_exception_summary",
        "rev-17", False, Reason.TARGET_MATCH, "req-701"
    ),
    Lookup(
        "tenant-east", "treatment", "ai_exception_summary",
        None, None, Reason.KEY_NOT_FOUND, "req-702"
    ),
    Lookup(
        "tenant-west", "treatment", "ai_exception_summary",
        None, None, Reason.REMOTE_NOT_FOUND, "req-703"
    ),
    Lookup(
        "tenant-south", "holdout", "ai_exception_summary",
        "rev-17", None, Reason.SNAPSHOT_UNAVAILABLE, "req-704"
    ),
]

events = replay(rows, default=False)
assert events[0]["default_used"] is False
assert events[1]["reason"] == "KEY_NOT_FOUND"
assert events[2]["reason"] == "REMOTE_NOT_FOUND"
assert events[3]["value"] is False

Notice what the test refuses to infer. It doesn't assume that a missing key means deletion, because a typo, environment mismatch, stale configuration reference, or a recreated definition can produce the same application-level symptom. It also doesn't retry a 404 as if repetition could restore identity. The diagnostic path checks the requested environment, the exact key, the definition or configuration revision, and the event timestamp. Only then can an operator classify the lifecycle transition.

Keep one fixture from before deletion and one from after recreation. The pre-delete fixture should resolve against the old revision. The interval with no definition should use the declared default and report a missing reason. The post-recreate fixture should resolve against the new revision only after targeting is configured. This three-window test is more useful than a unit test that mocks every lookup as a successful boolean, because it exercises the boundary that changes during the incident.

One sharp edge remains: defaults must be owned by the application contract, not copied casually into dashboards. If a Node.js caller uses false while an offline evaluator assumes true, the replay will confidently reconstruct behavior that never happened. Store the actual evaluated value and default_used; don't recompute either field during analysis.

A cohort label says what the experiment intended. An evaluation event says what the request received. Cost attribution must use the latter. Join model usage to successful flag exposure through the correlation ID, then aggregate by tenant, cohort, definition revision, evaluated value, and reason. Requests that defaulted should remain visible as their own slice instead of being silently counted as treatment.

A compact decision table makes the rule auditable:

Suppose the treatment cohort has 1,000 shipment exceptions assigned, but only 820 evaluation events contain the expected treatment value and revision. Dividing token cost by 1,000 answers cost per assigned exception; dividing by 820 answers cost per observed exposure. Both can be useful, but swapping one for the other changes the decision. These numbers are an example, not a measured result. The report should name its denominator every time.

Raw totals aren't enough. Tenant workloads differ in route count, document length, exception type, and model usage. Report at least cost per observed exposure and cost per completed business outcome, with tenant-level slices so a large account doesn't hide the rest of the cohort. For a distribution with a long tail, inspect percentiles as well as averages. The Core Web Vitals guidance uses the 75th percentile to assess user experiences; that doesn't make p75 automatically correct for model cost, but it is a useful reminder to choose and document a cohort threshold instead of letting an average make the decision by accident.

Prompt-cost awareness belongs in the same trace. Keep input tokens, output tokens, model identifier, and prompt revision in the usage ledger. Then a flag revision and a prompt revision can be analyzed independently. Without both dimensions, a prompt expansion deployed during the flag experiment may be blamed on targeting, while a fallback wave may falsely appear to reduce cost.

Short version: assignment is intent; exposure is evidence.

For an AI-generated shipment summary, false is usually the conservative example because the system can continue with the existing non-AI workflow. It isn't a universal rule. If the flag gates a safety check, compliance control, or data-isolation boundary, a permissive default may be unacceptable. The owning team must define the safe behavior per call site and test it as part of the application, not inherit a global default because it was convenient in an SDK sample.

There is a real trade-off here. Local snapshot evaluation reduces dependence on a request-time control-plane lookup, but it introduces freshness and revision questions. Remote evaluation can centralize targeting decisions, but the application still needs deterministic behavior when no value is available. Neither design removes the need for typed defaults, reason codes, and revision metadata. Stick with the simpler local contract when targeting can be represented in a validated snapshot; choose remote evaluation when policy genuinely requires centralized context, then budget for its latency and failure semantics in the application design.

A delete-and-recreate workflow is also not suitable when the goal is an ordinary . If lifecycle identity matters to experiments, disabling targeting or archiving a definition preserves a cleaner analytical boundary than reusing a key. Where deletion is required by policy, close the old experiment window first, retain the evaluation ledger under the applicable retention rules, and begin a new revision segment after recreation.

Your mileage may vary on the exact metadata names. The invariant shouldn't: the event must reveal the evaluated value, whether a default was used, why it was used, and which definition or configuration produced the decision.

The operational checklist is prose because the sequence matters. Before rollout, freeze representative control, treatment, missing-key, 404, and snapshot-unavailable fixtures; replay them in CI; and confirm that the Node.js producer and Python consumer agree on booleans, nullability, reason strings, and timestamps. During rollout, watch the fallback rate by tenant and revision alongside token cost per observed exposure. After any delete or recreate operation, mark the analysis boundary, verify targeting on the new revision with a synthetic tenant that cannot affect real shipments, and advance the cohort only when evaluation and usage events join cleanly.

Then stop. A dashboard with twenty derived ratios won't repair an event that omitted its reason.