# Why Learning Causality Doesn’t Automatically Make RL Better — Teaching RL Agents Cause and Effect > Source: > Published: 2026-07-07 17:01:03+00:00 How do you know when your RL agent is lying to you? What happens when RL agent trusts a broken causal map of its world? Causal RL Isn’t Just About Learning a Graph — It’s About Knowing When It’s Right! **Who should read this?** This article is written for three kinds of readers: 1. Practitioners building agents in non-stationary environments who want to know when causal models help versus hurt. 2. RL researchers interested in the formal sample complexity of causal discovery under policy-corrupted data. 3. CS / AI enthusiasts who have seen basic RL and want a concrete bridge into causal reasoning with runnable code. You will need familiarity with Q-learning, basic probability, and directed graphs. Everything else is built up from scratch. GitHub repository: Standard reinforcement learning agents are, at their core, sophisticated correlation machines. They learn that *when I push this button and this light is on, I tend to get a reward, *without ever understanding *why*. This is Rung 1 on Judea Pearl’s famous Ladder of Causation: pure association. Causal RL promises something far more powerful. By learning the *structural causal mechanisms* that govern an environment, that is, who causes what, independent of how data happened to be collected, an agent should be able to plan counterfactually, generalise across distribution shifts, and intervene intelligently rather than just correlate blindly. It sounds like everything we want from a truly intelligent agent. “A causal RL agent with an imperfect causal graph can perform worse than a model-free agent that knows nothing about causality at all.” The Core Problem is that correlation measures whether two variables move together, but it’s mathematically agnostic about causation. It can’t tell you which direction the arrow points or whether a hidden third variable is driving both. Ice cream sales and drowning deaths correlate, but temperature causes both. Sicker patients get medication and have worse outcomes, but sickness — not the drug — causes the harm. The gap between “these variables move together” and “changing one will change the other” is where most critical mistakes happen. **The Formal Solution.** Structural Causal Models (SCMs) let you encode your beliefs about how a system actually works using directed graphs and equations, making causation explicit and testable. Once you draw your causal diagram showing which variables cause which, you can use algorithms like d-separation and Pearl’s do-calculus to identify which variables you must adjust for to estimate true causal effects from observational data. The “backdoor adjustment formula” does the work: by comparing outcomes within groups (e.g., by age, removing confounding), you can convert correlation into causation. The catch is that your causal diagram must be correct; if you miss a confounder or mistakenly adjust for a collider variable, you’ll reach the opposite conclusion. Modern tools (DoWhy, DAGitty) automate this process, but the human responsibility remains: think hard about your assumptions, draw your causal story, and check whether your answer is fragile or robust to uncertainty. That promise from causal algorithms for reinforcement learning is real, but it comes with a trap that this repository exposes with brutal clarity: **learning a causal graph from RL interaction data is deeply corrupted by the very policy that collected the data.** An agent that always plays the same action generates no variation in the variables that action controls, making the causal edges from that action statistically invisible. The causal discovery algorithm cannot see what it never varied. The result is that a causal agent armed with a wrong graph doesn’t just fail to improve — it actively makes things *worse*. This work asks: how many samples does an RL agent need before it can trust its own causal model enough to act on it? Judea Pearl’s Ladder of Causation organises the kinds of questions an agent can ask into three ascending rungs with each strictly more powerful than the last, and each requiring a richer model of the world. Model-free RL lives permanently on Rung 1. It asks: *given what I have seen, what action tends to work?* This is powerful. Q-learning can beat world champions at Go, but it is brittle. The association it learns is valid only in the distribution it was trained on. Change the environment slightly and the correlations that the agent relied on may evaporate. Rung 2 introduces the do(·) operator — the mathematical tool for reasoning about *interventions* rather than observations. Instead of asking "what happens when I observe the light is on?", it asks "what happens when I force the light to be on, regardless of everything else?" This is what causal RL agents do: they build a structural model and plan by simulating interventions. The payoff is genuine generalisation across distribution shifts, because causal mechanisms tend to be more stable than correlations. In Rung 3, counterfactuals go further still: “what would have happened if I had taken a different action, given that I observed this outcome?” This is the territory of causal policy gradients and multi-agent counterfactual reasoning. The repo lives squarely at the Rung 1 → Rung 2 transition. The question it asks is not “how do you use a causal model?” but “how do you know when your causal model is good enough to use?” A Structural Causal Model, formalised by [Pearl (2000)](https://bayes.cs.ucla.edu/BOOK-2K/), is a mathematical machine for generating the world. It has four ingredients: The key idea is the **causal Markov factorisation**. Once you know a variable’s causal parents, you know everything causally relevant to it. The joint distribution factorises cleanly: This is the engine of causal efficiency. Instead of a full joint over all variables (exponential in n), you only need each variable’s local conditional model. A system with 20 variables where each has at most 3 parents becomes dramatically cheaper to model and plan with. The repo works with Causal MDPs — a framework that lifts the standard MDP tuple with an explicit causal graph: The causal transition factorisation allows compact world models: Planning becomes tractable because the causal graph partitions state variables into causal modules, which are groups reachable from exactly one action node. Actions can be planned within their module without influencing other modules. This is the computational payoff of knowing the causal structure. Standard causal discovery algorithms like the PC algorithm, NOTEARS, and their kin assume data comes from a distribution with sufficient variation in all variables. They test conditional independence between pairs of variables to reconstruct the graph. This works beautifully on observational datasets collected from nature but fails with RL. The Fundamental Trap A policy that always plays action a=(1,1) generateszero variationin action-correlated state variables when the environment deterministically responds to those actions. The causal discovery algorithm cannot distinguish "no edge" from "an edge that was never varied." Both look identical: near-zero variance in the child variable after conditioning on everything else. In RL, the data-collection policy controls which actions are taken. A suboptimal policy will fail to vary certain action dimensions, making those causal edges statistically invisible. This is the bootstrapping nightmare: you need good data to learn a good causal graph, but you need a good causal graph to design exploration that generates good data. Before reaching the theorem, it is worth building a genuine mental model of why the MDP’s *mixing time* makes causal discovery so expensive. The formal definition of mixing time is the number of steps an MDP needs to “forget” its initial state and reach a stationary distribution. While mathematically precise, this definition does not fully capture its practical consequence: mixing time determines how many truly independent observations an agent can collect. The implications for sample efficiency are more visceral than that definition suggests. Imagine you are trying to estimate the bias of a coin by flipping it. If each flip is independent, 1,000 flips give you 1,000 data points. But what if consecutive flips are correlated — what if the coin tends to stay on the same face for 10 flips at a time before switching? Now your 1,000 flips give you only about 100 independent observations. The rest are redundant. Your estimate of the bias is no better than if you had flipped 100 times with a fair coin. The same phenomenon occurs in reinforcement learning. When an MDP has a long mixing time, consecutive states remain highly correlated because the environment evolves only gradually. Even if an agent collects millions of transitions, the number of effectively independent samples can be dramatically smaller. Since causal discovery relies on identifying subtle statistical dependencies, this reduction in effective sample size directly increases the amount of interaction required before the correct causal graph can be recovered. The Mixing Time Intuition In an MDP with mixing time τ_mix, consecutive state observations are statistically correlated for τ_mix steps. Feeding all N raw observations to a conditional independence test is like feeding correlated coin flips — the testthinksit has N data points but it effectively has N/τ_mix. The correlation inflates false confidence in discovered edges and suppresses detection of real ones. Fix provided in the repo: **thin the time-series by τ_mix** before running the CI test, subsampling every τ_mix-th observation to get approximately i.i.d. samples. This is statistically honest, but it means you need τ_mix times as many raw steps to get the same effective sample size. Now consider what happens when the policy is *also* non-exploratory. Not only are your samples correlated in time, but they cluster in a small region of state-action space. An agent that always plays (1,1) never explores the (0,0) or (0,1) corners of action space. The causal CI tests for X3’s effects receive near-zero variation. They cannot distinguish a missing edge from a present-but-never-exercised one. These two problems: temporal correlation and insufficient exploration, multiply. The sample complexity lower bound is their product. Let *M* be a Causal MDP with *d* state variables and mixing time τ_mix. For any causal discovery algorithm *A* and confidence δ ∈ (0,1), there exists an MDP instance such that *A* requires at least: The proof combines two ingredients: - **Mixing time lower bound.** In a τ_mix-mixing MDP, consecutive observations are statistically dependent for τ_mix steps. The effective sample size from N raw observations is N/τ_mix. This directly multiplies the raw sample requirement. - **Dimension lower bound.** Each of the d(d−1) potential directed edges requires an independent conditional independence test. Each CI test needs at least log(1/δ) effective samples to achieve power δ. Multiplying: N/τ_mix ≥ d² · log(1/δ) → N ≥ d² · τ_mix · log(1/δ). For the repo’s setup — d=6, τ_mix=10, δ=0.05: This number is hardcoded in AdaptiveSwitchPolicy as a mandatory gate — the agent simply refuses to consider switching until the exploration buffer contains at least 1,078 samples. Knowing how many samples to collect is only half the problem. The other half: *how accurate does your graph need to be before the causal planner is worth using?* **Structural Hamming Distance (SHD)** is an option that can be used as the accuracy metric. It is the minimum number of edge edits (additions, deletions, reversals) to transform Ĝ into G*: The repo defines exclusive causal action modules — groups of state variables reachable from exactly one action node. In the fork graph: Theorem 3.3 — Phase Transition & Adaptive SwitchingLet e* = ⌈√K⌉ where K is the number of exclusive causal action modules. An adaptive agent that switches from model-free to causal planning when SHD(Ĝ, G*) ≤ e* achieves: The theoretical threshold is e*=2 for this graph. The empirical reality is sharper: because the fork graph has a *chain structure* within each module, any single missing edge (SHD=1) completely severs that module’s path to reward. There is a cliff at SHD=0→1 that the theory does not fully capture for this specific topology. Drop X3→X4 (one edge, SHD=1) and X3’s entire module goes dark. The planner commits to X3=1, expecting reward that no longer arrives. Q-learning, with no graph to mislead it, empirically discovers the switch and adapts. The wrong causal model is worse than no model at all. The repo implements a linear-Gaussian causal MDP — a fork DAG with 6 state variables and one binary reward: Before seeing the full 6-node implementation, let’s walk the PC algorithm on a tiny 3-node graph to build genuine intuition. This is the hardest-to-grasp part of the repo, and a concrete example makes all the difference. Suppose we have three variables — call them **A**, **B**, **C** — and the true (hidden) causal structure is the chain A → B → C. We observe data and want to recover this structure. **4. Test A ⊥ C | B (conditioning on B)** This is the key test. In a chain A→B→C, once you know B, A gives you no additional information about C. The path is 5. **Check unshielded triple A — B — C** A and C are not adjacent (we removed that edge). B is between them. Is B in sep_set(A,C)? Yes — sep_set(A,C) = {B}. So B is 6. **Meek rules + causal order tiebreak** Using causal order (A precedes B precedes C), orient A→B and B→C. The recovered graph is the true chain. The Key Insight from This ExampleThe PC algorithm identifies causal structure by findingwhich variable, when conditioned on, makes two others independent. In a chain A→B→C, conditioning on the middle node B blocks the path. In a v-structure A→B←C, conditioning on the collider Bopensa path — A and C become dependent given B. This “opening” is what lets PC distinguish v-structures (colliders) from chains and forks. In the 6-node fork graph, separating X1 from X4 requires conditioning on {X2, X5, X6} — three variables — because there are multiple paths between them through the collider X6. This is why the repo uses max_cond_size=3. Standard CI tests assume i.i.d. data. RL data is a correlated time series. The repo implements a **mixing-aware CI test** that handles this explicitly: causal_rl/models/discovery.py — mixing_aware_ci_test() ``` python def mixing_aware_ci_test(data, xi, xj, conditioning, tau_mix=10, alpha=0.05): # Step 1: Thin the time series by tau_mix # This converts a Markov chain into approx i.i.d. samples idx = np.arange(0, data.shape[0], max(1, tau_mix)) d = data[idx] # Step 2: Binarise each column at its median # PC CI tests work on discrete data; this avoids distribution assumptions xi_b = (d[:, xi] > np.median(d[:, xi])).astype(int) xj_b = (d[:, xj] > np.median(d[:, xj])).astype(int) # Step 3: For each stratum of conditioning variables, # run a 2×2 chi-squared test def safe_chi2(av, bv): ct = np.zeros((2, 2), dtype=int) for a, b in zip(av, bv): ct[a, b] += 1 if ct.sum() < 4 or ct.min() == 0: return 0.5 # sparse stratum → agnostic _, p, _, _ = chi2_contingency(ct) return float(p) # Step 4: Combine stratum p-values via Fisher's method # T = -2 Σ ln(pᵢ) ~ chi²(df = 2k) # This pools evidence across strata without assuming independence stat = -2.0 * np.sum(np.log(np.clip(pvals, 1e-300, 1.0))) p = 1.0 - chi2_dist.cdf(stat, df=2 * k) return p > alpha # True = independent (remove edge) ``` Causal RL is not a new idea. Multiple research threads have explored the intersection of causality and reinforcement learning over the past decade. It is worth being clear about where this repo’s contribution sits in that landscape. **Comparison to Related Approaches** Causal Bandits (Lattimore et al., 2016; Varici et al., 2023) studies intervention selection in bandit settings where the causal graph is *known*. The question is “which variable to intervene on?” not “what is the graph?” This repo tackles the prior, harder question: learning the graph itself from policy-collected data. Knowing the graph is precisely what is assumed away here. NOTEARS (Zheng et al., 2018) presents a continuous optimisation approach to structure learning that avoids the exponential conditioning sets of PC. It works beautifully on i.i.d. data. The repo’s mixing-aware CI test is a response to the specific problem of Markov-correlated observations, something NOTEARS in its standard form is not designed for. NOTEARS is listed as a natural extension. Causal World Models (Zhu et al., 2022) uses causal structure for offline RL planning. Assumes a good causal model is available or assumes away the data-collection corruption problem. The repo’s core contribution is precisely the question Zhu et al. bracket: when is your offline-learned causal model reliable enough to trust? Active Causal Structure Learning (Squires et al., 2020) designs interventional experiments to learn causal graphs efficiently. This is complementary: the repo shows how much data you need, while Squires et al. show how to collect it efficiently via designed interventions. In RL, designed interventions correspond to purposeful exploration policies — the 33% random exploration in AdaptiveSwitchPolicy is a crude version of this idea. Sample Complexity of Causal Representation Learning (Acartürk et al., 2024) is the closest in spirit — formal sample complexity bounds for causal learning. That work focuses on interventional data; this repo focuses on the policy-corrupted observational regime specific to RL. The repo’s unique contribution is the combination of: (1) a formal lower bound for the RL-specific regime (policy-corrupted, Markov-correlated data), (2) an empirically validated phase transition showing exactly when causal planning pays off, and (3) a gated adaptive agent that uses (1) and (2) operationally. None of the above works deliver all three together. The repo benchmarks four distinct agents, each representing a different bet about causal knowledge. The AdaptiveSwitchPolicy is the centrepiece. It solves the bootstrapping problem by treating causal discovery as an *online process* with two formal quality gates before ever committing to causal planning. Algorithm — AdaptiveSwitchPolicy (Full) Initialise:ModelFreePolicy as default · exploration buffer · Theorem 3.1 gatemin_N = D² · τ_mix · log(1/δ) = 1,078With prob 1/3: random action → push to explore_buffer Each step: Else: follow ModelFreePolicy Update ModelFreePolicy with (s, a, r, s’)Every 1,000 steps (if not yet switched):① Check: explore_buffer.size ≥ min_N? If not → skip ② Run PC algorithm on explore_buffer[-12,000:] ③ Re-inject prior edges if PC removed them ④ Apply oracle-assisted orientation correction ⑤ Compute FN-only SHD vs G*_learned ⑥ If SHD ≤ ε̂ AND both actions have path to R: → Permanently switch to CausalPlannerPolicy → Record switch time and SHD causal_rl/agents/adaptive_switch.py — update() core logic ``` python def update(self, state, action, reward, next_state): self.t += 1 self.behaviour_buffer.append(state.copy()) if self._explore_this_step: self.explore_buffer.append(state.copy()) if hasattr(self.policy, "update"): self.policy.update(state, action, reward, next_state) if self.t % self.explore_steps == 0 and not self.switched: # ── Gate 1: Theorem 3.1 minimum-sample requirement ─────────── if len(self.explore_buffer) < self.min_samples_to_switch: print(f"[AdaptiveSwitch] Gate 1 not met: {len(self.explore_buffer)} < {self.min_samples_to_switch}") return # ── PC on the exploration buffer ────────────────────────────── data = np.array(self.explore_buffer[-self.window:]) skel, ssets, tna = pc_skeleton(data, self.state_nodes, self.tau_mix, max_cond_size=3) self._reseed_priors(skel, ssets, tna) G_raw = orient_skeleton(skel, self.state_nodes, ssets, tna) # ── Oracle-assisted orientation correction (see Section 12) ── G_corrected = correct_orientations(G_raw, self.G_true, self.PRIOR_EDGES) # ── Gate 2: FN-only SHD quality check ──────────────────────── shd = structural_hamming_distance(G_corrected, self.G_true_learned) if shd <= self.eps_hat: candidate = CausalPlannerPolicy(G_final, self.nodes, required_actions=self.REQUIRED_ACTIONS) if candidate.plan_is_valid(): self.policy = candidate self.switched = True self.shd_at_switch = shd # From now on: all act() calls go to the causal planner ``` The single biggest limitation of the repo deserves its own section, not a table row. The correct_orientations() function uses the *ground-truth graph G** to fix edge directions and drop false positives. Without this function, the PC skeleton phase recovers which pairs of variables are causally related, but struggles to orient all edges correctly with observational data alone. Oracle-Assisted Orientation — What It Actually DoesAfter the PC algorithm produces a partially oriented graph, correct_orientations(G_est, G_true): For each edge (u→v) in G_est: if G_true has (u→v), keep it. If G_true has (v→u), reverse it. If neither, drop it (false positive). For each prior edge in PRIOR_EDGES: re-inject it unconditionally. Check the result is a DAG (raises if not). The resulting graph has zero false positives and zero reversed edges — only false negatives. The SHD gate checks only false negatives.This is not fully unsupervised causal discovery.It simulates an agent with domain knowledge of causal ordering but uncertainty about which edges actually exist. There are two reasons for the design choice: First, it *isolates the hardest problem:* edge existence versus edge orientation, and focuses the theoretical analysis on the former. The sample complexity bound (Theorem 3.1) is about how many samples you need to recover which edges exist. Edge orientation in observational data requires additional assumptions (faithfulness, Markov condition, and sometimes functional form assumptions like linear non-Gaussian). Second, in many real-world settings, causal ordering *is* available as prior knowledge: actions causally precede observations, early pipeline stages precede later ones, known temporal ordering constrains the graph. Domain experts often know “X causes Y” but not “does X cause Y at all?” What would it take to remove the oracle? Two paths are known: The repo lists fully unsupervised orientation as explicit future work. Until then, treat all results as conditional on the oracle-corrected orientation assumption. The experiments run for 25,000 steps with a mechanism shift at t=15,000. Here is not just what the panels show, but *why each result comes out the way it does*. **Why does the SHD=5 planner score only 0.22 — worse than even random?** The wrong graph routes X3 through a false path that leads nowhere near R. The planner confidently executes a plan that is structurally incapable of generating reward. It gets 0 for actions that route through dead ends, and only occasional reward when X6 happens to be high by chance. Q-learning, by contrast, gets at least 0.64 by blundering its way into (1,0) through trial-and-error. **Why does the Adaptive Switch outperform both the model-free agent and the Adaptive Oracle?** It discovers the true causal graph between t=5,000–7,000 and immediately commits to the causal planner. This gives it ~8,000 steps of near-perfect pre-shift performance before the shift. After the shift, it reverts to Q-learning behaviour (since the planner remains fixed with the pre-shift graph). Its high cumulative reward is a consequence of front-loading performance during the reliable pre-shift window. **Why is the cliff so sharp?** The fork graph uses chain structures within each module: X3→X4→X5→X6→R. Causal paths in a chain are fragile — breaking any single link severs the entire path. On a graph with redundant paths (multiple routes from an action to reward), the cliff would be softer. The repo’s chosen graph is deliberately minimal to exhibit the worst case. The Theorem 3.1 bound scales as Ω(d² · τ_mix · log(1/δ)). As τ_mix grows, the agent needs linearly more exploration steps before any causal discovery attempt is theoretically justified. The gate in AdaptiveSwitchPolicy enforces this — and experiments confirm that attempting causal discovery below the gate leads to unreliable graphs and premature switching failures. With 33% random exploration interspersed with Q-learning, the FN-only SHD (after oracle correction) typically reaches 0 between steps 5,000–7,000. The PC algorithm sees enough variation in all state variables because random actions ensure all four action combinations are exercised. Once SHD=0 and both action paths are validated, the agent permanently commits to the causal planner — no looking back. ``` # 1. Clone the repogit clone https://github.com/AditiSSNR/causal-rl.gitcd causal-rl# 2. Virtual environmentpython -m venv .venvsource .venv/bin/activate # Windows: .venv\Scripts\activate# 3. Installpip install -r requirements.txtpip install -e .# 4. Verifypython -c "import causal_rl; print('OK')"# 5. Run full experiments (produces causal_rl_results.png, ~10–20 min)python scripts/run_experiment.py# 6. Phase-transition sweep onlypython scripts/shd_sweep.py# 7. Run testspytest tests/ -v python from causal_rl.envs import CausalGridworldfrom causal_rl.models.causal_graph import true_dag, structural_hamming_distanceimport numpy as np# Build the ground-truth causal graph and inspect itG = true_dag(d=6)print("True edges:", list(G.edges()))# → [('X1','X2'), ('X3','X4'), ('X4','X5'), ('X2','X6'), ('X5','X6'), ('X6','R')]# Test expected rewards per action (no shift, 5000 steps each)for action, label in [([1,1], "(1,1)"), ([1,0], "(1,0)"), ([0,1], "(0,1)"), ([0,0], "(0,0)")]: env = CausalGridworld(d=6) env.reset() rewards = [env.step(action)[1] for _ in range(5000)] print(f"Action {label}: E[R] = {np.mean(rewards):.3f}")# → (1,1): 1.00 (1,0): 0.64 (0,1): 0.22 (0,0): 0.00# Simulate the mechanism shift manuallyenv_shift = CausalGridworld(d=6, shift_step=100) # shift at step 100env_shift.reset()pre_rewards = [env_shift.step([1,1])[1] for _ in range(50)]post_rewards = [env_shift.step([1,1])[1] for _ in range(200)]print(f"Pre-shift E[R]: {np.mean(pre_rewards):.3f}")print(f"Post-shift E[R]: {np.mean(post_rewards):.3f}")# → Pre: ~1.0 Post: ~0.64 (X3→X4 severed → X3 loses effect) ``` run_pc.py ``` python from causal_rl.envs import CausalGridworldfrom causal_rl.models.causal_graph import true_dag, structural_hamming_distancefrom causal_rl.models.discovery import pc_skeleton, orient_skeletonimport numpy as np# Collect purely random data (best case for discovery)env = CausalGridworld(d=6)env.reset()data = []for _ in range(5000): a = np.random.randint(0, 2, 2) s, _, _, _ = env.step(a) data.append(s)data = np.array(data)# Run PC algorithmstate_nodes = [f"X{i}" for i in range(1, 7)]skel, sep_sets, non_adj = pc_skeleton(data, state_nodes, tau_mix=10, max_cond_size=3)G_est = orient_skeleton(skel, state_nodes, sep_sets, non_adj)# Compare to ground truthG_true = true_dag(d=6)shd = structural_hamming_distance(G_est, G_true)print(f"Estimated edges: {list(G_est.edges())}")print(f"True edges: {list(G_true.edges())}")print(f"SHD (before oracle correction): {shd}")# With 5000 random samples, typically SHD ∈ {0, 1, 2} ``` custom_agent.py ``` python from causal_rl.envs import CausalGridworldfrom causal_rl.agents import AdaptiveSwitchPolicyfrom causal_rl.models.causal_graph import true_dagimport numpy as npG_true = true_dag(d=6)nodes = [f"X{i}" for i in range(1, 7)] + ["R"]agent = AdaptiveSwitchPolicy( G_true=G_true, nodes=nodes, eps_hat=0.0, # only switch at SHD=0 (conservative) explore_steps=1000, # run PC every 1000 steps window=12000, # use last 12k explore samples tau_mix=10, # estimated mixing time force_explore_every=3,# 1/3 of steps are random)env = CausalGridworld(d=6, shift_step=15000)state = env.reset()cumulative_reward = 0for t in range(25000): action = agent.act(state) next_state, reward, _, _ = env.step(action) agent.update(state, action, reward, next_state) cumulative_reward += reward state = next_stateprint(f"\n=== Results ===")print(f"Total reward: {cumulative_reward:.0f}")print(f"Switched to causal planner: {agent.switched}")if agent.switched: print(f"Switch step: {agent._switched_at_t}") print(f"SHD at switch: {agent.shd_at_switch}") ``` my_env.py — subclass CausalGridworld ``` python from causal_rl.envs.causal_gridworld import CausalGridworldimport numpy as npclass MyColliderEnv(CausalGridworld): """ A different causal structure: X1 → X3 ← X2 (collider at X3). Demonstrates how collider structure changes discovery difficulty. """ def _structural_equations(self, action, eps): x = np.zeros(self.d) x[0] = float(action[0]) # X1 x[1] = float(action[1]) # X2 # X3 is a collider: caused by BOTH X1 and X2 x[2] = np.clip(0.5*x[0] + 0.5*x[1] + eps[2], 0, 1) reward = float(x[2] > 0.5) return x, reward ``` The most exciting open problem remains fully unsupervised orientation under policy-corrupted data. The PC skeleton recovery is already unsupervised and works well here. Combining it with interventional exploration — designing policies that force specific state variables to fixed values to break confounding — is the natural next step toward a deployable causal RL agent that truly earns its trust in its own model. 5 main takeaways are: *📌 **Repository:** **github.com/AditiSSNR/causal-rl** — MIT Licensed · Python 3.9+ · NumPy, SciPy, NetworkX, Matplotlib, pandas* *Article written as a deep study of the repository source code, README, and theoretical documentation. All equations, algorithms, and code reproduced from the codebase for educational purposes.* [Why Learning Causality Doesn’t Automatically Make RL Better — Teaching RL Agents Cause and Effect](https://pub.towardsai.net/why-learning-causality-doesnt-automatically-make-rl-better-teaching-rl-agents-cause-and-effect-99b20d429f79) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.