Basic IRM Models

init_notebook_modetrusted

ATE Coverage

The simulations are based on the the make_irm_data-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

Metadata

DoubleML Version                   0.10.dev0
Script                   irm_ate_coverage.py
Date                     2025-02-17 13:38:59
Total Runtime (seconds)          3587.198893
Python Version                        3.12.9
Number of observations                   500
Number of repetitions                   1000

Table 1: Coverage for 95.0%-Confidence Interval over 1000 Repetitions

Learner g	Learner m	Bias	CI Length	Coverage
Lasso	Logistic Regression	0.123	0.557	0.935
Lasso	Random Forest	0.149	0.724	0.960
Random Forest	Logistic Regression	0.150	0.617	0.886
Random Forest	Random Forest	0.152	0.747	0.951

Table 2: Coverage for 90.0%-Confidence Interval over 1000 Repetitions

Learner g	Learner m	Bias	CI Length	Coverage
Lasso	Logistic Regression	0.123	0.468	0.875
Lasso	Random Forest	0.149	0.608	0.918
Random Forest	Logistic Regression	0.150	0.518	0.794
Random Forest	Random Forest	0.152	0.627	0.887

ATTE Coverage

As for the ATE, the simulations are based on the the make_irm_data-DGP with \(500\) observations.

Metadata

DoubleML Version                    0.10.dev0
Script                   irm_atte_coverage.py
Date                      2025-02-17 13:39:11
Total Runtime (seconds)           3599.870577
Python Version                         3.12.9
Number of observations                    500
Number of repetitions                    1000

Table 3: Coverage for 95.0%-Confidence Interval over 1000 Repetitions

Learner g	Learner m	Bias	CI Length	Coverage
Lasso	Logistic Regression	0.135	0.635	0.937
Lasso	Random Forest	0.182	0.864	0.955
Random Forest	Logistic Regression	0.150	0.655	0.919
Random Forest	Random Forest	0.188	0.899	0.944

Table 4: Coverage for 90.0%-Confidence Interval over 1000 Repetitions

Learner g	Learner m	Bias	CI Length	Coverage
Lasso	Logistic Regression	0.135	0.533	0.889
Lasso	Random Forest	0.182	0.725	0.900
Random Forest	Logistic Regression	0.150	0.550	0.869
Random Forest	Random Forest	0.188	0.754	0.900

Sensitivity

The simulations are based on the the ADD-DGP with \(10,000\) observations. As the DGP is nonlinear, we will only use corresponding learners. Since the DGP includes an unobserved confounder, we would expect a bias in the ATE estimates, leading to low coverage of the true parameter.

The confounding is set such that both sensitivity parameters are approximately \(cf_y=cf_d=0.1\), such that the robustness value \(RV\) should be approximately \(10\%\). Further, the corresponding confidence intervals are one-sided (since the direction of the bias is unkown), such that only one side should approximate the corresponding coverage level (here only the lower coverage is relevant since the bias is positive). Remark that for the coverage level the value of \(\rho\) has to be correctly specified, such that the coverage level will be generally (significantly) larger than the nominal level under the conservative choice of \(|\rho|=1\).

init_notebook_modetrusted

ATE

Metadata

DoubleML Version                      0.10.dev0
Script                   irm_ate_sensitivity.py
Date                        2025-02-17 13:05:22
Total Runtime (seconds)             1572.634208
Python Version                           3.12.9
Number of observations                    10000
Number of repetitions                       100

Table 5: Coverage for 95.0%-Confidence Interval over 100 Repetitions

Learner l	Learner m	Bias	Bias (Lower)	Bias (Upper)	Coverage	Coverage (Lower)	Coverage (Upper)	RV	RVa
LGBM	LGBM	0.173	0.031	0.318	0.000	0.990	1.000	0.120	0.058
LGBM	Logistic Regr.	0.151	0.022	0.296	0.040	1.000	1.000	0.104	0.043
Linear Reg.	LGBM	0.174	0.032	0.318	0.000	1.000	1.000	0.120	0.058
Linear Reg.	Logistic Regr.	0.089	0.057	0.235	0.720	1.000	1.000	0.063	0.008

Table 6: Coverage for 90.0%-Confidence Interval over 100 Repetitions

Learner l	Learner m	Bias	Bias (Lower)	Bias (Upper)	Coverage	Coverage (Lower)	Coverage (Upper)	RV	RVa
LGBM	LGBM	0.173	0.031	0.318	0.000	0.950	1.000	0.120	0.072
LGBM	Logistic Regr.	0.151	0.022	0.296	0.000	0.980	1.000	0.104	0.057
Linear Reg.	LGBM	0.174	0.032	0.318	0.000	0.940	1.000	0.120	0.072
Linear Reg.	Logistic Regr.	0.089	0.057	0.235	0.570	1.000	1.000	0.063	0.016

ATTE

Metadata

DoubleML Version                       0.10.dev0
Script                   irm_atte_sensitivity.py
Date                         2025-02-17 13:05:01
Total Runtime (seconds)              1552.786642
Python Version                            3.12.9
Number of observations                     10000
Number of repetitions                        100

Table 7: Coverage for 95.0%-Confidence Interval over 100 Repetitions

Learner l	Learner m	Bias	Bias (Lower)	Bias (Upper)	Coverage	Coverage (Lower)	Coverage (Upper)	RV	RVa
LGBM	LGBM	0.121	0.041	0.251	0.670	1.000	1.000	0.093	0.016
LGBM	Logistic Regr.	0.119	0.042	0.248	0.660	1.000	1.000	0.092	0.015
Linear Reg.	LGBM	0.118	0.042	0.248	0.710	1.000	1.000	0.091	0.014
Linear Reg.	Logistic Regr.	0.051	0.093	0.166	0.990	1.000	1.000	0.040	0.000

Table 8: Coverage for 90.0%-Confidence Interval over 100 Repetitions

Learner l	Learner m	Bias	Bias (Lower)	Bias (Upper)	Coverage	Coverage (Lower)	Coverage (Upper)	RV	RVa
LGBM	LGBM	0.121	0.041	0.251	0.490	0.990	1.000	0.093	0.028
LGBM	Logistic Regr.	0.119	0.042	0.248	0.530	0.970	1.000	0.092	0.027
Linear Reg.	LGBM	0.118	0.042	0.248	0.510	0.980	1.000	0.091	0.026
Linear Reg.	Logistic Regr.	0.051	0.093	0.166	0.950	1.000	1.000	0.040	0.002