Basic IRM Models

ATE Coverage

The simulations are based on the the make_irm_data-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

DoubleML Version                   0.10.dev0
Script                   irm_ate_coverage.py
Date                     2025-01-08 13:14:33
Total Runtime (seconds)          3543.555382
Python Version                        3.12.8
Table 1: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m Bias CI Length Coverage
Lasso Logistic Regression 0.123 0.557 0.935
Lasso Random Forest 0.148 0.721 0.959
Random Forest Logistic Regression 0.150 0.617 0.883
Random Forest Random Forest 0.154 0.754 0.959
Table 2: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m Bias CI Length Coverage
Lasso Logistic Regression 0.123 0.468 0.875
Lasso Random Forest 0.148 0.605 0.906
Random Forest Logistic Regression 0.150 0.518 0.803
Random Forest Random Forest 0.154 0.633 0.899

ATTE Coverage

As for the ATE, the simulations are based on the the make_irm_data-DGP with \(500\) observations.

DoubleML Version                    0.10.dev0
Script                   irm_atte_coverage.py
Date                      2025-01-08 13:14:10
Total Runtime (seconds)           3538.650552
Python Version                         3.12.8
Table 3: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m Bias CI Length Coverage
Lasso Logistic Regression 0.134 0.635 0.942
Lasso Random Forest 0.181 0.872 0.948
Random Forest Logistic Regression 0.150 0.656 0.928
Random Forest Random Forest 0.183 0.894 0.948
Table 4: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m Bias CI Length Coverage
Lasso Logistic Regression 0.134 0.533 0.887
Lasso Random Forest 0.181 0.732 0.895
Random Forest Logistic Regression 0.150 0.551 0.871
Random Forest Random Forest 0.183 0.750 0.901

Sensitivity

The simulations are based on the the ADD-DGP with \(10,000\) observations. As the DGP is nonlinear, we will only use corresponding learners. Since the DGP includes an unobserved confounder, we would expect a bias in the ATE estimates, leading to low coverage of the true parameter.

The confounding is set such that both sensitivity parameters are approximately \(cf_y=cf_d=0.1\), such that the robustness value \(RV\) should be approximately \(10\%\). Further, the corresponding confidence intervals are one-sided (since the direction of the bias is unkown), such that only one side should approximate the corresponding coverage level (here only the lower coverage is relevant since the bias is positive). Remark that for the coverage level the value of \(\rho\) has to be correctly specified, such that the coverage level will be generally (significantly) larger than the nominal level under the conservative choice of \(|\rho|=1\).

ATE

DoubleML Version                      0.10.dev0
Script                   irm_ate_sensitivity.py
Date                        2025-01-08 14:51:20
Total Runtime (seconds)             9351.407656
Python Version                           3.12.8
Table 5: Coverage for 95.0%-Confidence Interval over 500 Repetitions
Learner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM LGBM 0.179 0.043 0.322 0.318 0.998 1.000 0.124 0.034
LGBM Logistic Regr. 0.149 0.029 0.298 0.548 1.000 1.000 0.101 0.019
Linear Reg. LGBM 0.179 0.045 0.319 0.314 0.998 1.000 0.126 0.035
Linear Reg. Logistic Regr. 0.090 0.057 0.235 0.974 1.000 1.000 0.063 0.001
Table 6: Coverage for 90.0%-Confidence Interval over 500 Repetitions
Learner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM LGBM 0.179 0.043 0.322 0.112 0.962 1.000 0.124 0.054
LGBM Logistic Regr. 0.149 0.029 0.298 0.292 1.000 1.000 0.101 0.035
Linear Reg. LGBM 0.179 0.045 0.319 0.122 0.964 1.000 0.126 0.055
Linear Reg. Logistic Regr. 0.090 0.057 0.235 0.860 1.000 1.000 0.063 0.007

ATTE

DoubleML Version                       0.10.dev0
Script                   irm_atte_sensitivity.py
Date                         2025-01-08 15:14:53
Total Runtime (seconds)              10782.06734
Python Version                            3.12.8
Table 7: Coverage for 95.0%-Confidence Interval over 500 Repetitions
Learner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM LGBM 0.135 0.065 0.259 0.826 0.982 1.000 0.105 0.012
LGBM Logistic Regr. 0.131 0.065 0.259 0.834 0.984 1.000 0.098 0.011
Linear Reg. LGBM 0.125 0.065 0.244 0.858 0.986 1.000 0.099 0.010
Linear Reg. Logistic Regr. 0.074 0.095 0.175 0.976 0.998 1.000 0.058 0.002
Table 8: Coverage for 90.0%-Confidence Interval over 500 Repetitions
Learner l Learner m Bias Bias (Lower) Bias (Upper) Coverage Coverage (Lower) Coverage (Upper) RV RVa
LGBM LGBM 0.135 0.065 0.259 0.702 0.950 1.000 0.105 0.024
LGBM Logistic Regr. 0.131 0.065 0.259 0.714 0.964 1.000 0.098 0.022
Linear Reg. LGBM 0.125 0.065 0.244 0.754 0.962 1.000 0.099 0.020
Linear Reg. Logistic Regr. 0.074 0.095 0.175 0.948 0.996 1.000 0.058 0.004