We’ve all been there: staring at a Fitbit or Apple Health dashboard, trying to figure out if that 4 PM espresso is the reason we're tossing and turning at 2 AM. In the world of Quantified Self and Predictive Medicine, we often fall into the trap of "correlation equals causation." We see a downward trend in sleep quality as caffeine intake rises and assume one causes the other. But what if it’s actually work stress causing both the extra coffee consumption and the poor sleep?
To move beyond simple statistics, we need Causal Inference. By using Microsoft’s DoWhy library and CausalML, we can build a structural model to isolate the "treatment effect" of caffeine on sleep. This article explores how to use Python Data Analysis and advanced Data Engineering techniques to quantify the truth behind your daily habits.
Before we dive into the code, we need to understand the Directed Acyclic Graph (DAG). In causal inference, we don't just look at $X$ and $Y$. We look at "Confounders"—variables that influence both the cause and the effect.
The following diagram illustrates how stress and age might "confound" the relationship between caffeine and sleep.
graph TD
A[Age] --> S[Sleep Quality]
W[Work Stress] --> C[Caffeine Intake]
W --> S[Sleep Quality]
C --> S
H[Health Consciousness] --> C
H --> S
style C fill:#f96,stroke:#333,stroke-width:2px
style S fill:#69f,stroke:#333,stroke-width:2px
To follow this tutorial, you'll need a standard Python environment. We will use DoWhy
for the causal modeling framework and Pandas
for data manipulation.
pip install dowhy causalml pandas matplotlib numpy
Since sharing raw medical data is tricky, let’s generate a synthetic dataset that mimics a real-world scenario where Stress is a major confounder.
import pandas as pd
import numpy as np
import dowhy
from dowhy import CausalModel
np.random.seed(42)
num_days = 1000
stress = np.random.normal(5, 2, num_days)
caffeine = 100 + 20 * stress + np.random.normal(0, 10, num_days)
sleep_quality = 90 - 0.05 * caffeine - 3.0 * stress + np.random.normal(0, 5, num_days)
df = pd.DataFrame({
'caffeine': caffeine,
'sleep_quality': sleep_quality,
'stress': stress
})
print(df.head())
With DoWhy
, the workflow follows four distinct steps: Model, Identify, Estimate, and Refute. This structure forces us to be explicit about our assumptions.
model = CausalModel(
data=df,
treatment='caffeine',
outcome='sleep_quality',
common_causes=['stress'] # This is our confounder
)
identified_estimand = model.identify_effect(proceed_when_unidentified=True)
print(identified_estimand)
Now we use a linear regression estimator to find the "Causal Effect." This represents the change in sleep quality for every additional mg of caffeine, adjusting for stress.
estimate = model.estimate_effect(
identified_estimand,
method_name="backdoor.linear_regression"
)
print(f"Causal Estimate (Effect of Caffeine): {estimate.value}")
In our simulation, the "True" effect was -0.05. If you ran a simple correlation, you would likely see a much larger negative number because it would include the negative impact of the stress that caused you to drink the coffee!
💡
Advanced Patterns for Data EngineeringWhile this example uses synthetic data, production-grade causal pipelines require robust data validation and drift detection. For more production-ready examples and advanced architectural patterns in predictive medicine, check out the detailed guides at
. They cover everything from high-throughput data ingestion to deploying ML models in regulated environments.[WellAlly Tech Blog]
This is the most critical step in Causal Inference. We try to disprove our own model using "Refutation" tests, such as adding a random common cause or replacing the treatment with a placebo.
refutation = model.refute_estimate(
identified_estimand,
estimate,
method_name="placebo_treatment_refuter"
)
print(refutation)
If the New Effect
after adding a placebo treatment is close to zero, it means our original model is robust. If it's still high, your model is capturing noise!
By using DoWhy and CausalML, we move from descriptive analytics ("I sleep worse when I drink coffee") to prescriptive insights ("If I reduce my caffeine by 100mg, my sleep quality score will improve by 5 points, regardless of my stress level").
This approach is the backbone of modern Predictive Medicine and personalized health tech. Instead of following generic advice, you can use your own data to find what actually works for your body.
What's next?
Happy hacking, and sleep well! 🥑💻