{"slug": "reflex-rust-library-for-control-loops-that-act-on-observability-data", "title": "Reflex: Rust library for control loops that act on observability data", "summary": "Datadog Labs released Reflex, a Rust library for control loops that act on observability data, turning Datadog metrics and Toto forecasts into typed state that a model such as TypeSafe AI's Jev can act on. Reflex commits a recommended action only if it passes declared guards and invariants, and the repository includes a simulator that runs Reflex against simulated services for circuit breaking, scheduling, autoscaling, and retries. Guards reject recommendations when the circuit changed mid-decision, Datadog evidence is missing or stale, or the recommendation exceeds its 10-second execution deadline.", "body_md": "Reflex is a Rust library for control loops that act on observability data. Metrics queried from [**Datadog**](https://www.datadoghq.com/) and forecasts from [** Toto**](https://github.com/DataDog/toto) become typed state. A model, such as TypeSafe AI's **Jev**, recommends an action, and Reflex commits it only if it passes the guards and invariants you declared.\n\n```\nApplication supplies typed state\n              ↓\nJudge recommends an action\n              ↓\nExecutor checks legality, guards, and invariants\n              ↓\nVerified state transition\n              ↓\nOptional effects perform external work\n```\n\nReflex fits control loops where the right action depends on changing conditions, but execution must obey fixed constraints.\n\n- **Circuit breaking.** Open, probe, or close based on service health, while enforcing cooldowns and probe limits.\n- **Scheduling.** Choose which request gets capacity next, while enforcing resource limits.\n- **Autoscaling.** Add or remove capacity based on demand and forecasts, while enforcing capacity bounds.\n- **Retries.** Decide whether to retry a failed request and how long to back off, while enforcing retry budgets and a maximum backoff.\n\nThe repository includes a simulator that runs Reflex against simulated services. The sections below walk through its circuit breaker, starting with the state machine and then the running simulation.\n\nA gateway routes client traffic to three services, Catalog, Payments, and Search, each behind its own circuit. Jev decides when a circuit should open or probe. Reflex decides whether that decision may take effect.\n\nEach recommendation is a typed action. Besides Jev's choice, it carries the Datadog evidence the decision was based on, when that evidence was observed, and the circuit revision it applies to.\n\n```\nstruct Action {\n    choice: Choice,                          // Open, Probe, or NoChange\n    telemetry: Option<TelemetryEvidence>,    // Datadog observations Jev received\n    observed_at: f64,\n    revision: u64,\n}\n```\n\nThese are the machine's main rules, abridged from [`jev.rs`](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-sim/src/jev.rs).\n\n``` js\nlet definition = state_machine! {\n    phase: CircuitPhase,\n    data: Data,\n    action: Action,\n    event: Event,\n    invariants: [valid_phase],\n    transitions: [\n        // Jev's recommendations\n        CircuitPhase::Closed + action(Action { choice: Choice::Open, .. }) => CircuitPhase::Open {\n            guard: can_open, update: open,\n        },\n        CircuitPhase::Open + action(Action { choice: Choice::Probe, .. }) => CircuitPhase::HalfOpen {\n            guard: can_probe, update: probe,\n        },\n        CircuitPhase::Closed + action(Action { choice: Choice::NoChange, .. }) => CircuitPhase::Closed {\n            guard: fresh,\n        },\n        // Probe results\n        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::HalfOpen {\n            guard: current_probe, update: probe_succeeded,\n        },\n        CircuitPhase::HalfOpen + event(Event::FinalProbe(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::Closed {\n            guard: final_probe, update: close,\n        },\n        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Error | ClientOutcome::Timeout, .. })) => CircuitPhase::Open {\n            guard: current_probe, update: reopen,\n        },\n        // Evaluation failed (for example, an inference error): leave the circuit as it is\n        CircuitPhase::Closed + evaluation_error(_) => unchanged {},\n        // ... clock ticks, response recording, and the remaining no-change rules\n    ],\n};\n```\n\nThe guards are where observability data meets fixed rules.\n\n| Guard | Rejects a recommendation when | \n|---|---|\n| `fresh` | The circuit changed while Jev was deciding, the Datadog evidence is missing or stale, or the recommendation is past its 10-second execution deadline | \n| `can_open` | The evidence does not include ten responses from a complete Datadog window collected after the circuit's last transition | \n| `can_probe` | The circuit's 3-second cooldown has not elapsed | \n| `current_probe` | A response does not belong to the probe currently in flight | \n| `final_probe` | Fewer than five consecutive probes have succeeded | \n\nThe `valid_phase` invariant checks that the phase, cooldown, and probe reservation always agree. Guards run at execution time against current state, so a recommendation based on delayed telemetry or an inaccurate forecast cannot open, probe, or close a circuit on its own.\n\nDuring a traffic surge, Jev opens the Payments circuit. After the cooldown, it probes, and the circuit closes once traffic recovers and five probes succeed. The panel on the right compares a Toto forecast of Payments request rate with the Datadog observations that followed.\n\n**Where the state comes from.** The simulated services publish metrics to Datadog. The simulator queries them back to build each service's state, such as its error rate, latency, and queue depth.\n\n**How it is forecast.** A local [Toto](https://github.com/DataDog/toto) service receives the observed history of request rate, queue depth, and utilization, and forecasts the next 120 seconds as p10/p50/p90 values in ten-second buckets. It needs 320 seconds of history before the first forecast. Select a service to compare the frozen forecast with what was later observed.\n\n**What happens to a decision.** Every 10 seconds, Jev evaluates each service and returns Open, Probe, or NoChange. Reflex applies the recommendation only if the matching transition's guard passes. Open **Activity** and inspect a decision to see the Datadog values and timestamps Jev received, any forecast, and whether the transition was applied or rejected, with the rejection reason.\n\n**The scenario shown.** In **Cyclical load · Toto**, Payments repeats a two-minute pattern for ten minutes. Traffic rises 4× at +30s, service time rises 6× at +60s, and both recover at +90s. Jev chooses when to open and probe. Reflex enforces the cooldown and the five-probe close.\n\nReflex also emits its own OpenTelemetry counters and spans (`reflex.evaluations`, `reflex.transitions`). The simulator exports these to Datadog with the application metrics, and the supplied [dashboards](https://github.com/datadog-labs/reflex/blob/main/dashboards/README.md) show both. The simulator also includes a **resource scheduler** that follows the same pattern; see the [simulator guide](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-sim/README.md).\n\n| Requirement | Used for | \n|---|---|\n| Rust 1.92+ and Node.js | The simulator and its UI | \n| `DD_API_KEY` ,`DD_APP_KEY` ,`DD_SITE` | Publishing and querying metrics. The application key needs `timeseries_query` permission. | \n| `TYPESAFE_API_KEY` | Live Jev recommendations | \n| [uv](https://docs.astral.sh/uv/) | The local Toto service. Toto needs no API key. | \n\nBuild the UI once.\n\n```\nnpm ci --prefix crates/reflex-sim/ui\nnpm run build --prefix crates/reflex-sim/ui\n```\n\nIn one terminal, start the Toto service.\n\n```\nuv sync --project integrations/toto --python 3.12 --locked\nuv run --project integrations/toto --locked reflex-toto\n```\n\nWait for `Ready: http://127.0.0.1:8765`. The first start downloads a pinned **Toto-2.0-22m** checkpoint, which runs on CPU by default.\n\nIn a second terminal, copy [`.env.example`](https://github.com/datadog-labs/reflex/blob/main/.env.example) to `.env.local`, fill in your keys, and start the simulator.\n\n```\nset -a\n. ./.env.local\nset +a\ncargo run -p reflex-sim --features datadog --locked -- \\\n  --playground --policy jev --datadog --datadog-evidence \\\n  --toto-url http://127.0.0.1:8765\n```\n\nSelect **Cyclical load · Toto → Run**, then select **Payments**. Use the decision's `simulation_run` to filter the [Datadog dashboards](https://github.com/datadog-labs/reflex/blob/main/dashboards/README.md). Press **Ctrl+C** to stop and flush telemetry.\n\nThere are other ways to run it.\n\n- To run **without Datadog** , drop`--features datadog` ,`--datadog` , and`--datadog-evidence` . State and forecasts then come from the simulator's local observations, and only`TYPESAFE_API_KEY` is required.\n- To **publish to Datadog but decide on local state** , use`--datadog` without`--datadog-evidence` . Publishing needs only`DD_API_KEY` .\n- To run **without credentials** , use`cargo run -p reflex --example circuit_breaker` , a smaller circuit breaker with a deterministic judge.\n\nYour application gathers the evidence, whether that is a Datadog query, a forecast, or local measurements, and builds the typed state passed to the judge.\n\n| Concept | What it does | \n|---|---|\n| **State** | A Rust value your application prepares for evaluation, such as local observations, queried metrics, or forecasts. | \n| **Judge** | Implements `Judge<S, A>` and returns a typed action with optional confidence. It can call Jev or use an ordinary algorithm. | \n| **Controller** | Calls the judge with an inference deadline and returns a proposed decision or an evaluation error. It does not change system state. | \n| **State machine** | Declares phases and transitions. Guards check whether an action is allowed; invariants check properties every committed state must satisfy. | \n| **Executor** | Checks current state, prepares a candidate, validates it, and commits it. Declared asynchronous effects run after commit and return events to the machine. | \n\nYour application calls the controller, then the executor.\n\n```\n// Gather evidence (for example, Datadog observations and a Toto forecast) into `state`.\nlet evaluation = controller.evaluate(&state).await;\nlet outcome = executor.execute(evaluation).await?;\n```\n\nPass the entire evaluation result to the executor; the machine can define transitions for inference failures as well as recommendations. State transitions commit in memory, and external effects run afterward. See [execution semantics](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md#guarantees-and-current-scope) for persistence and failure handling.\n\nFor a smaller, self-contained machine that runs without credentials, see the [circuit-breaker example](https://github.com/datadog-labs/reflex/blob/main/crates/reflex/examples/circuit_breaker.rs) and the [SDK guide](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md).\n\nThe crates are **not yet published to crates.io**. Add path dependencies on your checkout.\n\n```\n[dependencies]\nreflex = { path = \"../reflex/crates/reflex\" }\ntokio = { version = \"1\", features = [\"rt-multi-thread\", \"macros\"] }\n```\n\nTo use Jev, also add the `typesafe-ai` and `reflex-typesafe` crates and inject a `TypeSafeClient` into `TypeSafeJudge`. The [live Jev example](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-typesafe/examples/jev.rs) shows the setup (`cargo run -p reflex-typesafe --example jev`, with `TYPESAFE_API_KEY` set). See the [SDK guide](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md) for hook signatures, executor construction, and the [TypeSafe integration](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md#typesafe-integration).\n\n**Datadog and Toto**\n\n**Reflex SDK**\n\nRun the workspace tests with these commands.\n\n```\nnpm ci --prefix crates/reflex-sim/ui\nnpm run build --prefix crates/reflex-sim/ui\ncargo test --workspace --all-targets --locked\ncargo test --workspace --doc --locked\n```\n\nReflex is licensed under [Apache-2.0](https://github.com/datadog-labs/reflex/blob/main/LICENSE). See [NOTICE](https://github.com/datadog-labs/reflex/blob/main/NOTICE) for attribution. Third-party dependencies and assets retain their own licenses.\n\nThird-party components are listed in [LICENSE-3rdparty.csv](https://github.com/datadog-labs/reflex/blob/main/LICENSE-3rdparty.csv). See the [inventory notes](https://github.com/datadog-labs/reflex/blob/main/third_party/README.md) for coverage, sources, and unresolved entries.\n\nOwned by Datadog, Inc. See [maintenance responsibilities](https://github.com/datadog-labs/reflex/blob/main/MAINTAINERS.md).", "url": "https://wpnews.pro/news/reflex-rust-library-for-control-loops-that-act-on-observability-data", "canonical_source": "https://github.com/datadog-labs/reflex", "published_at": "2026-09-27 02:41:55+00:00", "updated_at": "2026-09-27 03:01:22.288180+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "developer-tools", "ai-tools"], "entities": ["Datadog", "Datadog Labs", "Reflex", "Toto", "TypeSafe AI", "Jev", "Catalog", "Payments"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/reflex-rust-library-for-control-loops-that-act-on-observability-data", "markdown": "https://wpnews.pro/news/reflex-rust-library-for-control-loops-that-act-on-observability-data.md", "text": "https://wpnews.pro/news/reflex-rust-library-for-control-loops-that-act-on-observability-data.txt", "jsonld": "https://wpnews.pro/news/reflex-rust-library-for-control-loops-that-act-on-observability-data.jsonld"}}