Basic IRM Models

Coverage

The simulations are based on the the make_irm_data-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

DoubleML Version                           0.11.5.dev55
Script                         IRMATECoverageSimulation
Date                                   2026-08-10 12:26
Total Runtime (minutes)                      113.261698
Python Version                                   3.12.3
Config File              scripts/irm/irm_ate_config.yml

ATE

ⓘLearner g Learner m Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0.283 1.456 0.978
LGBM Regr. Logistic 0.176 0.927 0.974
LassoCV LGBM Clas. 0.251 1.298 0.970
LassoCV Logistic 0.158 0.797 0.963
LassoCV RF Clas. 0.134 0.693 0.959
RF Regr. Logistic 0.168 0.882 0.970
RF Regr. RF Clas. 0.143 0.741 0.951
Coverage for 95%-Confidence Interval over 1000 Repetitions
ⓘLearner g Learner m Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0.283 1.222 0.945
LGBM Regr. Logistic 0.176 0.778 0.937
LassoCV LGBM Clas. 0.251 1.089 0.940
LassoCV Logistic 0.158 0.669 0.922
LassoCV RF Clas. 0.134 0.582 0.916
RF Regr. Logistic 0.168 0.740 0.928
RF Regr. RF Clas. 0.143 0.622 0.911
Coverage for 90%-Confidence Interval over 1000 Repetitions

ATTE

As for the ATE, the simulations are based on the the make_irm_data-DGP with \(500\) observations.

DoubleML Version                            0.11.5.dev55
Script                         IRMATTECoverageSimulation
Date                                    2026-08-10 12:23
Total Runtime (minutes)                       110.565889
Python Version                                    3.12.3
Config File              scripts/irm/irm_atte_config.yml
ⓘLearner g Learner m Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0.341 1.780 0.976
LGBM Regr. Logistic 0.206 1.005 0.959
LassoCV LGBM Clas. 0.336 1.668 0.982
LassoCV Logistic 0.197 0.940 0.969
LassoCV RF Clas. 0.141 0.691 0.955
RF Regr. Logistic 0.198 0.968 0.954
RF Regr. RF Clas. 0.147 0.714 0.957
Coverage for 95%-Confidence Interval over 1000 Repetitions
ⓘLearner g Learner m Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0.341 1.494 0.934
LGBM Regr. Logistic 0.206 0.844 0.916
LassoCV LGBM Clas. 0.336 1.400 0.930
LassoCV Logistic 0.197 0.789 0.918
LassoCV RF Clas. 0.141 0.580 0.907
RF Regr. Logistic 0.198 0.813 0.907
RF Regr. RF Clas. 0.147 0.599 0.904
Coverage for 90%-Confidence Interval over 1000 Repetitions

Sensitivity

The simulations are based on the the make_confounded_irm_data-DGP with \(5,000\) observations. Since the DGP includes an unobserved confounder, we would expect a bias in the ATE estimates, leading to low coverage of the true parameter.

The confounding is set such that both sensitivity parameters are approximately \(cf_y=cf_d=0.1\), such that the robustness value \(RV\) should be approximately \(10\%\). Further, the corresponding confidence intervals are one-sided (since the direction of the bias is unkown), such that only one side should approximate the corresponding coverage level (here only the lower coverage is relevant since the bias is positive). Remark that for the coverage level the value of \(\rho\) has to be correctly specified, such that the coverage level will be generally (significantly) larger than the nominal level under the conservative choice of \(|\rho|=1\).

ATE

DoubleML Version                                       0.11.5.dev55
Script                          IRMATESensitivityCoverageSimulation
Date                                               2026-08-10 11:11
Total Runtime (minutes)                                   37.944184
Python Version                                               3.12.3
Config File              scripts/irm/irm_ate_sensitivity_config.yml
ⓘLearner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM Regr. LGBM Clas. 0.185 0.046 0.328 0.226 0.994 1.000 0.128 0.038
LGBM Regr. Logistic 0.152 0.027 0.301 0.500 1.000 1.000 0.102 0.020
Linear LGBM Clas. 0.183 0.046 0.323 0.244 0.992 1.000 0.129 0.037
Linear Logistic 0.092 0.056 0.238 0.960 1.000 1.000 0.065 0.002
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM Regr. LGBM Clas. 0.185 0.046 0.328 0.078 0.944 1.000 0.128 0.058
LGBM Regr. Logistic 0.152 0.027 0.301 0.248 0.994 1.000 0.102 0.037
Linear LGBM Clas. 0.183 0.046 0.323 0.074 0.954 1.000 0.129 0.058
Linear Logistic 0.092 0.056 0.238 0.848 1.000 1.000 0.065 0.007
Coverage for 90%-Confidence Interval over 500 Repetitions

ATTE

DoubleML Version                                        0.11.5.dev55
Script                          IRMATTESensitivityCoverageSimulation
Date                                                2026-08-10 11:11
Total Runtime (minutes)                                    38.095549
Python Version                                                3.12.3
Config File              scripts/irm/irm_atte_sensitivity_config.yml
ⓘLearner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM Regr. LGBM Clas. 0.138 0.062 0.263 0.840 0.980 1.000 0.107 0.012
LGBM Regr. Logistic 0.134 0.060 0.264 0.838 0.988 1.000 0.101 0.011
Linear LGBM Clas. 0.128 0.060 0.250 0.870 0.986 1.000 0.102 0.010
Linear Logistic 0.074 0.092 0.180 0.974 1.000 1.000 0.058 0.002
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM Regr. LGBM Clas. 0.138 0.062 0.263 0.702 0.946 1.000 0.107 0.024
LGBM Regr. Logistic 0.134 0.060 0.264 0.720 0.946 1.000 0.101 0.022
Linear LGBM Clas. 0.128 0.060 0.250 0.756 0.956 1.000 0.102 0.020
Linear Logistic 0.074 0.092 0.180 0.944 1.000 1.000 0.058 0.005
Coverage for 90%-Confidence Interval over 500 Repetitions

Tuning

The simulations are based on the the make_irm_data-DGP with \(500\) observations. This is only an example as the untuned version just relies on the default configuration.

DoubleML Version                                   0.12.dev0
Script                        IRMATETuningCoverageSimulation
Date                                        2025-12-01 12:02
Total Runtime (minutes)                            27.278139
Python Version                                        3.12.9
Config File              scripts/irm/irm_ate_tune_config.yml

ATE

ⓘLearner g Learner m Tuned Bias CI Length Coverage Loss g0 Loss g1 Loss m
LGBM Regr. LGBM Clas. False 0.645 3.150 0.985 1.113 1.136 0.667
LGBM Regr. LGBM Clas. True 0.129 0.613 0.935 0.999 1.061 0.519
Coverage for 95%-Confidence Interval over 200 Repetitions
ⓘLearner g Learner m Tuned Bias CI Length Coverage Loss g0 Loss g1 Loss m
LGBM Regr. LGBM Clas. False 0.645 2.643 0.920 1.113 1.136 0.667
LGBM Regr. LGBM Clas. True 0.129 0.515 0.890 0.999 1.061 0.519
Coverage for 90%-Confidence Interval over 200 Repetitions