This is the init_notebook_mode
cell from ITables v2.1.4
(you should not see this message - is your notebook trusted?)
ATE Coverage
The simulations are based on the the make_irm_data-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.
DoubleML Version 0.8.2
Script irm_ate_coverage.py
Date 2024-08-13 16:44:37
Total Runtime (seconds) 1115.591309
Python Version 3.12.4
Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Lasso |
Logistic Regression |
0.123 |
0.557 |
0.935 |
Lasso |
Random Forest |
0.147 |
0.720 |
0.956 |
Random Forest |
Logistic Regression |
0.150 |
0.616 |
0.877 |
Random Forest |
Random Forest |
0.152 |
0.750 |
0.947 |
Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Lasso |
Logistic Regression |
0.123 |
0.468 |
0.875 |
Lasso |
Random Forest |
0.147 |
0.604 |
0.906 |
Random Forest |
Logistic Regression |
0.150 |
0.517 |
0.796 |
Random Forest |
Random Forest |
0.152 |
0.629 |
0.902 |
ATTE Coverage
As for the ATE, the simulations are based on the the make_irm_data-DGP with \(500\) observations.
DoubleML Version 0.8.2
Script irm_atte_coverage.py
Date 2024-08-13 17:15:42
Total Runtime (seconds) 1171.313816
Python Version 3.12.4
Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Lasso |
Logistic Regression |
0.135 |
0.635 |
0.940 |
Lasso |
Random Forest |
0.181 |
0.868 |
0.956 |
Random Forest |
Logistic Regression |
0.150 |
0.656 |
0.924 |
Random Forest |
Random Forest |
0.180 |
0.882 |
0.947 |
Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Lasso |
Logistic Regression |
0.135 |
0.533 |
0.888 |
Lasso |
Random Forest |
0.181 |
0.729 |
0.895 |
Random Forest |
Logistic Regression |
0.150 |
0.551 |
0.874 |
Random Forest |
Random Forest |
0.180 |
0.740 |
0.892 |
Sensitivity
The simulations are based on the the ADD-DGP with \(10,000\) observations. As the DGP is nonlinear, we will only use corresponding learners. Since the DGP includes an unobserved confounder, we would expect a bias in the ATE estimates, leading to low coverage of the true parameter.
The confounding is set such that both sensitivity parameters are approximately \(cf_y=cf_d=0.1\), such that the robustness value \(RV\) should be approximately \(10\%\). Further, the corresponding confidence intervals are one-sided (since the direction of the bias is unkown), such that only one side should approximate the corresponding coverage level (here only the lower coverage is relevant since the bias is positive). Remark that for the coverage level the value of \(\rho\) has to be correctly specified, such that the coverage level will be generally (significantly) larger than the nominal level under the conservative choice of \(|\rho|=1\).
This is the init_notebook_mode
cell from ITables v2.1.4
(you should not see this message - is your notebook trusted?)
ATE
DoubleML Version 0.8.2
Script irm_ate_sensitivity.py
Date 2024-08-14 22:52:08
Total Runtime (seconds) 9943.911803
Python Version 3.12.4
Coverage for 95.0%-Confidence Interval over 500 Repetitions
LGBM |
LGBM |
0.179 |
0.043 |
0.322 |
0.318 |
0.998 |
1.000 |
0.124 |
0.034 |
LGBM |
Logistic Regr. |
0.149 |
0.029 |
0.298 |
0.548 |
1.000 |
1.000 |
0.101 |
0.019 |
Linear Reg. |
LGBM |
0.179 |
0.045 |
0.319 |
0.314 |
0.998 |
1.000 |
0.126 |
0.035 |
Linear Reg. |
Logistic Regr. |
0.090 |
0.057 |
0.235 |
0.974 |
1.000 |
1.000 |
0.063 |
0.001 |
Coverage for 90.0%-Confidence Interval over 500 Repetitions
LGBM |
LGBM |
0.179 |
0.043 |
0.322 |
0.112 |
0.962 |
1.000 |
0.124 |
0.054 |
LGBM |
Logistic Regr. |
0.149 |
0.029 |
0.298 |
0.292 |
1.000 |
1.000 |
0.101 |
0.035 |
Linear Reg. |
LGBM |
0.179 |
0.045 |
0.319 |
0.122 |
0.964 |
1.000 |
0.126 |
0.055 |
Linear Reg. |
Logistic Regr. |
0.090 |
0.057 |
0.235 |
0.860 |
1.000 |
1.000 |
0.063 |
0.007 |
ATTE
DoubleML Version 0.8.2
Script irm_atte_sensitivity.py
Date 2024-08-15 17:57:40
Total Runtime (seconds) 5941.642393
Python Version 3.12.4
Coverage for 95.0%-Confidence Interval over 500 Repetitions
LGBM |
LGBM |
0.135 |
0.074 |
0.219 |
0.826 |
0.946 |
1.000 |
0.148 |
0.018 |
LGBM |
Logistic Regr. |
0.131 |
0.071 |
0.218 |
0.834 |
0.962 |
1.000 |
0.139 |
0.016 |
Linear Reg. |
LGBM |
0.125 |
0.070 |
0.208 |
0.858 |
0.964 |
1.000 |
0.135 |
0.014 |
Linear Reg. |
Logistic Regr. |
0.074 |
0.074 |
0.140 |
0.976 |
0.996 |
1.000 |
0.081 |
0.002 |
Coverage for 90.0%-Confidence Interval over 500 Repetitions
LGBM |
LGBM |
0.135 |
0.074 |
0.219 |
0.702 |
0.866 |
1.000 |
0.148 |
0.035 |
LGBM |
Logistic Regr. |
0.131 |
0.071 |
0.218 |
0.714 |
0.892 |
1.000 |
0.139 |
0.032 |
Linear Reg. |
LGBM |
0.125 |
0.070 |
0.208 |
0.754 |
0.906 |
1.000 |
0.135 |
0.028 |
Linear Reg. |
Logistic Regr. |
0.074 |
0.074 |
0.140 |
0.948 |
0.988 |
1.000 |
0.081 |
0.006 |