Reflex is a Rust library for control loops that act on observability data. Metrics queried from Datadog and forecasts from ** Toto** become typed state. A model, such as TypeSafe AI's Jev, recommends an action, and Reflex commits it only if it passes the guards and invariants you declared.
Application supplies typed state
↓
Judge recommends an action
↓
Executor checks legality, guards, and invariants
↓
Verified state transition
↓
Optional effects perform external work
Reflex fits control loops where the right action depends on changing conditions, but execution must obey fixed constraints.
- Circuit breaking. Open, probe, or close based on service health, while enforcing cooldowns and probe limits.
- Scheduling. Choose which request gets capacity next, while enforcing resource limits.
- Autoscaling. Add or remove capacity based on demand and forecasts, while enforcing capacity bounds.
- Retries. Decide whether to retry a failed request and how long to back off, while enforcing retry budgets and a maximum backoff.
The repository includes a simulator that runs Reflex against simulated services. The sections below walk through its circuit breaker, starting with the state machine and then the running simulation.
A gateway routes client traffic to three services, Catalog, Payments, and Search, each behind its own circuit. Jev decides when a circuit should open or probe. Reflex decides whether that decision may take effect.
Each recommendation is a typed action. Besides Jev's choice, it carries the Datadog evidence the decision was based on, when that evidence was observed, and the circuit revision it applies to.
struct Action {
choice: Choice, // Open, Probe, or NoChange
telemetry: Option<TelemetryEvidence>, // Datadog observations Jev received
observed_at: f64,
revision: u64,
}
These are the machine's main rules, abridged from jev.rs.
let definition = state_machine! {
phase: CircuitPhase,
data: Data,
action: Action,
event: Event,
invariants: [valid_phase],
transitions: [
// Jev's recommendations
CircuitPhase::Closed + action(Action { choice: Choice::Open, .. }) => CircuitPhase::Open {
guard: can_open, update: open,
},
CircuitPhase::Open + action(Action { choice: Choice::Probe, .. }) => CircuitPhase::HalfOpen {
guard: can_probe, update: probe,
},
CircuitPhase::Closed + action(Action { choice: Choice::NoChange, .. }) => CircuitPhase::Closed {
guard: fresh,
},
// Probe results
CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::HalfOpen {
guard: current_probe, update: probe_succeeded,
},
CircuitPhase::HalfOpen + event(Event::FinalProbe(Observation { outcome: ClientOutcome::Success, .. })) => CircuitPhase::Closed {
guard: final_probe, update: close,
},
CircuitPhase::HalfOpen + event(Event::Response(Observation { outcome: ClientOutcome::Error | ClientOutcome::Timeout, .. })) => CircuitPhase::Open {
guard: current_probe, update: reopen,
},
// Evaluation failed (for example, an inference error): leave the circuit as it is
CircuitPhase::Closed + evaluation_error(_) => unchanged {},
// ... clock ticks, response recording, and the remaining no-change rules
],
};
The guards are where observability data meets fixed rules.
| Guard | Rejects a recommendation when |
|---|---|
fresh |
The circuit changed while Jev was deciding, the Datadog evidence is missing or stale, or the recommendation is past its 10-second execution deadline |
can_open |
The evidence does not include ten responses from a complete Datadog window collected after the circuit's last transition |
can_probe |
The circuit's 3-second cooldown has not elapsed |
current_probe |
A response does not belong to the probe currently in flight |
final_probe |
Fewer than five consecutive probes have succeeded |
The valid_phase invariant checks that the phase, cooldown, and probe reservation always agree. Guards run at execution time against current state, so a recommendation based on delayed telemetry or an inaccurate forecast cannot open, probe, or close a circuit on its own.
During a traffic surge, Jev opens the Payments circuit. After the cooldown, it probes, and the circuit closes once traffic recovers and five probes succeed. The panel on the right compares a Toto forecast of Payments request rate with the Datadog observations that followed.
Where the state comes from. The simulated services publish metrics to Datadog. The simulator queries them back to build each service's state, such as its error rate, latency, and queue depth.
How it is forecast. A local Toto service receives the observed history of request rate, queue depth, and utilization, and forecasts the next 120 seconds as p10/p50/p90 values in ten-second buckets. It needs 320 seconds of history before the first forecast. Select a service to compare the frozen forecast with what was later observed.
What happens to a decision. Every 10 seconds, Jev evaluates each service and returns Open, Probe, or NoChange. Reflex applies the recommendation only if the matching transition's guard passes. Open Activity and inspect a decision to see the Datadog values and timestamps Jev received, any forecast, and whether the transition was applied or rejected, with the rejection reason.
The scenario shown. In Cyclical load · Toto, Payments repeats a two-minute pattern for ten minutes. Traffic rises 4× at +30s, service time rises 6× at +60s, and both recover at +90s. Jev chooses when to open and probe. Reflex enforces the cooldown and the five-probe close.
Reflex also emits its own OpenTelemetry counters and spans (reflex.evaluations, reflex.transitions). The simulator exports these to Datadog with the application metrics, and the supplied dashboards show both. The simulator also includes a resource scheduler that follows the same pattern; see the simulator guide.
| Requirement | Used for |
|---|---|
| Rust 1.92+ and Node.js | The simulator and its UI |
DD_API_KEY ,DD_APP_KEY ,DD_SITE |
Publishing and querying metrics. The application key needs timeseries_query permission. |
TYPESAFE_API_KEY |
Live Jev recommendations |
| uv | The local Toto service. Toto needs no API key. |
Build the UI once.
npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui
In one terminal, start the Toto service.
uv sync --project integrations/toto --python 3.12 --locked
uv run --project integrations/toto --locked reflex-toto
Wait for Ready: http://127.0.0.1:8765. The first start downloads a pinned Toto-2.0-22m checkpoint, which runs on CPU by default.
In a second terminal, copy .env.example to .env.local, fill in your keys, and start the simulator.
set -a
. ./.env.local
set +a
cargo run -p reflex-sim --features datadog --locked -- \
--playground --policy jev --datadog --datadog-evidence \
--toto-url http://127.0.0.1:8765
Select Cyclical load · Toto → Run, then select Payments. Use the decision's simulation_run to filter the Datadog dashboards. Press Ctrl+C to stop and flush telemetry.
There are other ways to run it.
- To run without Datadog , drop
--features datadog,--datadog, and--datadog-evidence. State and forecasts then come from the simulator's local observations, and onlyTYPESAFE_API_KEYis required. - To publish to Datadog but decide on local state , use
--datadogwithout--datadog-evidence. Publishing needs onlyDD_API_KEY. - To run without credentials , use
cargo run -p reflex --example circuit_breaker, a smaller circuit breaker with a deterministic judge.
Your application gathers the evidence, whether that is a Datadog query, a forecast, or local measurements, and builds the typed state passed to the judge.
| Concept | What it does |
|---|---|
| State | A Rust value your application prepares for evaluation, such as local observations, queried metrics, or forecasts. |
| Judge | Implements Judge<S, A> and returns a typed action with optional confidence. It can call Jev or use an ordinary algorithm. |
| Controller | Calls the judge with an inference deadline and returns a proposed decision or an evaluation error. It does not change system state. |
| State machine | Declares phases and transitions. Guards check whether an action is allowed; invariants check properties every committed state must satisfy. |
| Executor | Checks current state, prepares a candidate, validates it, and commits it. Declared asynchronous effects run after commit and return events to the machine. |
Your application calls the controller, then the executor.
// Gather evidence (for example, Datadog observations and a Toto forecast) into `state`.
let evaluation = controller.evaluate(&state).await;
let outcome = executor.execute(evaluation).await?;
Pass the entire evaluation result to the executor; the machine can define transitions for inference failures as well as recommendations. State transitions commit in memory, and external effects run afterward. See execution semantics for persistence and failure handling.
For a smaller, self-contained machine that runs without credentials, see the circuit-breaker example and the SDK guide.
The crates are not yet published to crates.io. Add path dependencies on your checkout.
[dependencies]
reflex = { path = "../reflex/crates/reflex" }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
To use Jev, also add the typesafe-ai and reflex-typesafe crates and inject a TypeSafeClient into TypeSafeJudge. The live Jev example shows the setup (cargo run -p reflex-typesafe --example jev, with TYPESAFE_API_KEY set). See the SDK guide for hook signatures, executor construction, and the TypeSafe integration.
Datadog and Toto
Reflex SDK
Run the workspace tests with these commands.
npm ci --prefix crates/reflex-sim/ui
npm run build --prefix crates/reflex-sim/ui
cargo test --workspace --all-targets --locked
cargo test --workspace --doc --locked
Reflex is licensed under Apache-2.0. See NOTICE for attribution. Third-party dependencies and assets retain their own licenses.
Third-party components are listed in LICENSE-3rdparty.csv. See the inventory notes for coverage, sources, and unresolved entries.
Owned by Datadog, Inc. See maintenance responsibilities.