# Reflex: Rust library for control loops that act on observability data

> Source: <https://github.com/datadog-labs/reflex>
> Published: 2026-09-27 02:41:55+00:00

Reflex is a Rust library for control loops that act on observability data. Metrics queried from [**Datadog**](https://www.datadoghq.com/) and forecasts from [** Toto**](https://github.com/DataDog/toto) become typed state. A model, such as TypeSafe AI's **Jev**, recommends an action, and Reflex commits it only if it passes the guards and invariants you declared.

```
Application supplies typed state
              ↓
Judge recommends an action
              ↓
Executor checks legality, guards, and invariants
              ↓
Verified state transition
              ↓
Optional effects perform external work
```

Reflex fits control loops where the right action depends on changing conditions, but execution must obey fixed constraints.

- **Circuit breaking.** Open, probe, or close based on service health, while enforcing cooldowns and probe limits.
- **Scheduling.** Choose which request gets capacity next, while enforcing resource limits.
- **Autoscaling.** Add or remove capacity based on demand and forecasts, while enforcing capacity bounds.
- **Retries.** Decide whether to retry a failed request and how long to back off, while enforcing retry budgets and a maximum backoff.

The repository includes a simulator that runs Reflex against simulated services. The sections below walk through its circuit breaker, starting with the state machine and then the running simulation.

A gateway routes client traffic to three services, Catalog, Payments, and Search, each behind its own circuit. Jev decides when a circuit should open or probe. Reflex decides whether that decision may take effect.

Each recommendation is a typed action. Besides Jev's choice, it carries the Datadog evidence the decision was based on, when that evidence was observed, and the circuit revision it applies to.

```
struct Action {
    choice: Choice,                          // Open, Probe, or NoChange
    telemetry: Option<TelemetryEvidence>,    // Datadog observations Jev received
    observed_at: f64,
    revision: u64,
}
```

These are the machine's main rules, abridged from [`jev.rs`](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-sim/src/jev.rs).

``` js
let definition = state_machine! {
    phase: CircuitPhase,
    data: Data,
    action: Action,
    event: Event,
    invariants: [valid_phase],
    transitions: [
        // Jev's recommendations
        CircuitPhase::Closed + action(Action { choice: Choice::Open, .. }) => CircuitPhase::Open {
            guard: can_open, update: open,
        },
        CircuitPhase::Open + action(Action { choice: Choice::Probe, .. }) => CircuitPhase::HalfOpen {
            guard: can_probe, update: probe,
        },
        CircuitPhase::Closed + action(Action { choice: Choice::NoChange, .. }) => CircuitPhase::Closed {
            guard: fresh,
        },
        // Probe results
        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::HalfOpen {
            guard: current_probe, update: probe_succeeded,
        },
        CircuitPhase::HalfOpen + event(Event::FinalProbe(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::Closed {
            guard: final_probe, update: close,
        },
        CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Error | ClientOutcome::Timeout, .. })) => CircuitPhase::Open {
            guard: current_probe, update: reopen,
        },
        // Evaluation failed (for example, an inference error): leave the circuit as it is
        CircuitPhase::Closed + evaluation_error(_) => unchanged {},
        // ... clock ticks, response recording, and the remaining no-change rules
    ],
};
```

The guards are where observability data meets fixed rules.

| Guard | Rejects a recommendation when | 
|---|---|
| `fresh` | The circuit changed while Jev was deciding, the Datadog evidence is missing or stale, or the recommendation is past its 10-second execution deadline | 
| `can_open` | The evidence does not include ten responses from a complete Datadog window collected after the circuit's last transition | 
| `can_probe` | The circuit's 3-second cooldown has not elapsed | 
| `current_probe` | A response does not belong to the probe currently in flight | 
| `final_probe` | Fewer than five consecutive probes have succeeded | 

The `valid_phase` invariant checks that the phase, cooldown, and probe reservation always agree. Guards run at execution time against current state, so a recommendation based on delayed telemetry or an inaccurate forecast cannot open, probe, or close a circuit on its own.

During a traffic surge, Jev opens the Payments circuit. After the cooldown, it probes, and the circuit closes once traffic recovers and five probes succeed. The panel on the right compares a Toto forecast of Payments request rate with the Datadog observations that followed.

**Where the state comes from.** The simulated services publish metrics to Datadog. The simulator queries them back to build each service's state, such as its error rate, latency, and queue depth.

**How it is forecast.** A local [Toto](https://github.com/DataDog/toto) service receives the observed history of request rate, queue depth, and utilization, and forecasts the next 120 seconds as p10/p50/p90 values in ten-second buckets. It needs 320 seconds of history before the first forecast. Select a service to compare the frozen forecast with what was later observed.

**What happens to a decision.** Every 10 seconds, Jev evaluates each service and returns Open, Probe, or NoChange. Reflex applies the recommendation only if the matching transition's guard passes. Open **Activity** and inspect a decision to see the Datadog values and timestamps Jev received, any forecast, and whether the transition was applied or rejected, with the rejection reason.

**The scenario shown.** In **Cyclical load · Toto**, Payments repeats a two-minute pattern for ten minutes. Traffic rises 4× at +30s, service time rises 6× at +60s, and both recover at +90s. Jev chooses when to open and probe. Reflex enforces the cooldown and the five-probe close.

Reflex also emits its own OpenTelemetry counters and spans (`reflex.evaluations`, `reflex.transitions`). The simulator exports these to Datadog with the application metrics, and the supplied [dashboards](https://github.com/datadog-labs/reflex/blob/main/dashboards/README.md) show both. The simulator also includes a **resource scheduler** that follows the same pattern; see the [simulator guide](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-sim/README.md).

| Requirement | Used for | 
|---|---|
| Rust 1.92+ and Node.js | The simulator and its UI | 
| `DD_API_KEY` ,`DD_APP_KEY` ,`DD_SITE` | Publishing and querying metrics. The application key needs `timeseries_query` permission. | 
| `TYPESAFE_API_KEY` | Live Jev recommendations | 
| [uv](https://docs.astral.sh/uv/) | The local Toto service. Toto needs no API key. | 

Build the UI once.

```
npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui
```

In one terminal, start the Toto service.

```
uv sync --project integrations/toto --python 3.12 --locked
uv run --project integrations/toto --locked reflex-toto
```

Wait for `Ready: http://127.0.0.1:8765`. The first start downloads a pinned **Toto-2.0-22m** checkpoint, which runs on CPU by default.

In a second terminal, copy [`.env.example`](https://github.com/datadog-labs/reflex/blob/main/.env.example) to `.env.local`, fill in your keys, and start the simulator.

```
set -a
. ./.env.local
set +a
cargo run -p reflex-sim --features datadog --locked -- \
  --playground --policy jev --datadog --datadog-evidence \
  --toto-url http://127.0.0.1:8765
```

Select **Cyclical load · Toto → Run**, then select **Payments**. Use the decision's `simulation_run` to filter the [Datadog dashboards](https://github.com/datadog-labs/reflex/blob/main/dashboards/README.md). Press **Ctrl+C** to stop and flush telemetry.

There are other ways to run it.

- To run **without Datadog** , drop`--features datadog` ,`--datadog` , and`--datadog-evidence` . State and forecasts then come from the simulator's local observations, and only`TYPESAFE_API_KEY` is required.
- To **publish to Datadog but decide on local state** , use`--datadog` without`--datadog-evidence` . Publishing needs only`DD_API_KEY` .
- To run **without credentials** , use`cargo run -p reflex --example circuit_breaker` , a smaller circuit breaker with a deterministic judge.

Your application gathers the evidence, whether that is a Datadog query, a forecast, or local measurements, and builds the typed state passed to the judge.

| Concept | What it does | 
|---|---|
| **State** | A Rust value your application prepares for evaluation, such as local observations, queried metrics, or forecasts. | 
| **Judge** | Implements `Judge<S, A>` and returns a typed action with optional confidence. It can call Jev or use an ordinary algorithm. | 
| **Controller** | Calls the judge with an inference deadline and returns a proposed decision or an evaluation error. It does not change system state. | 
| **State machine** | Declares phases and transitions. Guards check whether an action is allowed; invariants check properties every committed state must satisfy. | 
| **Executor** | Checks current state, prepares a candidate, validates it, and commits it. Declared asynchronous effects run after commit and return events to the machine. | 

Your application calls the controller, then the executor.

```
// Gather evidence (for example, Datadog observations and a Toto forecast) into `state`.
let evaluation = controller.evaluate(&state).await;
let outcome = executor.execute(evaluation).await?;
```

Pass the entire evaluation result to the executor; the machine can define transitions for inference failures as well as recommendations. State transitions commit in memory, and external effects run afterward. See [execution semantics](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md#guarantees-and-current-scope) for persistence and failure handling.

For a smaller, self-contained machine that runs without credentials, see the [circuit-breaker example](https://github.com/datadog-labs/reflex/blob/main/crates/reflex/examples/circuit_breaker.rs) and the [SDK guide](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md).

The crates are **not yet published to crates.io**. Add path dependencies on your checkout.

```
[dependencies]
reflex = { path = "../reflex/crates/reflex" }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
```

To use Jev, also add the `typesafe-ai` and `reflex-typesafe` crates and inject a `TypeSafeClient` into `TypeSafeJudge`. The [live Jev example](https://github.com/datadog-labs/reflex/blob/main/crates/reflex-typesafe/examples/jev.rs) shows the setup (`cargo run -p reflex-typesafe --example jev`, with `TYPESAFE_API_KEY` set). See the [SDK guide](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md) for hook signatures, executor construction, and the [TypeSafe integration](https://github.com/datadog-labs/reflex/blob/main/SDK_README.md#typesafe-integration).

**Datadog and Toto**

**Reflex SDK**

Run the workspace tests with these commands.

```
npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui
cargo test --workspace --all-targets --locked
cargo test --workspace --doc --locked
```

Reflex is licensed under [Apache-2.0](https://github.com/datadog-labs/reflex/blob/main/LICENSE). See [NOTICE](https://github.com/datadog-labs/reflex/blob/main/NOTICE) for attribution. Third-party dependencies and assets retain their own licenses.

Third-party components are listed in [LICENSE-3rdparty.csv](https://github.com/datadog-labs/reflex/blob/main/LICENSE-3rdparty.csv). See the [inventory notes](https://github.com/datadog-labs/reflex/blob/main/third_party/README.md) for coverage, sources, and unresolved entries.

Owned by Datadog, Inc. See [maintenance responsibilities](https://github.com/datadog-labs/reflex/blob/main/MAINTAINERS.md).