APO Models

Coverage

APO Pointwise Coverage

The simulations are based on the the make_irm_data_discrete_treatments-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

DoubleML Version                          0.12.dev0
Script                        APOCoverageSimulation
Date                               2025-12-04 18:24
Total Runtime (minutes)                    75.00107
Python Version                               3.12.3
Config File              scripts/irm/apo_config.yml
ⓘLearner g Learner m Treatment Level Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0 1.947 9.970 0.959
LGBM Regr. LGBM Clas. 1 8.086 40.398 0.963
LGBM Regr. LGBM Clas. 2 8.353 39.660 0.963
LGBM Regr. Logistic 0 1.314 6.717 0.955
LGBM Regr. Logistic 1 1.620 8.487 0.969
LGBM Regr. Logistic 2 1.558 8.433 0.959
Linear LGBM Clas. 0 1.276 6.509 0.949
Linear LGBM Clas. 1 2.086 11.687 0.977
Linear LGBM Clas. 2 1.565 8.551 0.975
Linear Logistic 0 1.257 6.373 0.953
Linear Logistic 1 1.284 6.465 0.954
Linear Logistic 2 1.247 6.408 0.952
Coverage for 95%-Confidence Interval over 1000 Repetitions
ⓘLearner g Learner m Treatment Level Bias CI Length Coverage
LGBM Regr. LGBM Clas. 0 1.947 8.367 0.911
LGBM Regr. LGBM Clas. 1 8.086 33.903 0.927
LGBM Regr. LGBM Clas. 2 8.353 33.284 0.912
LGBM Regr. Logistic 0 1.314 5.637 0.912
LGBM Regr. Logistic 1 1.620 7.123 0.920
LGBM Regr. Logistic 2 1.558 7.077 0.924
Linear LGBM Clas. 0 1.276 5.463 0.906
Linear LGBM Clas. 1 2.086 9.808 0.939
Linear LGBM Clas. 2 1.565 7.176 0.934
Linear Logistic 0 1.257 5.348 0.906
Linear Logistic 1 1.284 5.426 0.900
Linear Logistic 2 1.247 5.377 0.914
Coverage for 90%-Confidence Interval over 1000 Repetitions

APOS Coverage

The simulations are based on the the make_irm_data_discrete_treatments-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

The non-uniform results (coverage, ci length and bias) refer to averaged values over all levels (point-wise confidende intervals).

DoubleML Version                           0.12.dev0
Script                        APOSCoverageSimulation
Date                                2025-12-04 18:24
Total Runtime (minutes)                    74.426826
Python Version                                3.12.3
Config File              scripts/irm/apos_config.yml
ⓘLearner g Learner m Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM Regr. LGBM Clas. 6.363 30.187 0.961 36.702 0.975
LGBM Regr. Logistic 1.573 7.946 0.950 9.386 0.954
Linear LGBM Clas. 1.707 8.948 0.963 10.611 0.965
Linear Logistic 1.320 6.429 0.948 6.849 0.945
Coverage for 95%-Confidence Interval over 1000 Repetitions
ⓘLearner g Learner m Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM Regr. LGBM Clas. 6.363 25.334 0.911 32.468 0.937
LGBM Regr. Logistic 1.573 6.668 0.902 8.202 0.905
Linear LGBM Clas. 1.707 7.510 0.916 9.290 0.928
Linear Logistic 1.320 5.395 0.895 5.817 0.887
Coverage for 90%-Confidence Interval over 1000 Repetitions

Causal Contrast Coverage

The simulations are based on the the make_irm_data_discrete_treatments-DGP with \(500\) observations. Due to the linearity of the DGP, Lasso and Logit Regression are nearly optimal choices for the nuisance estimation.

The non-uniform results (coverage, ci length and bias) refer to averaged values over all levels (point-wise confidende intervals).

DoubleML Version                           0.12.dev0
Script                        APOSCoverageSimulation
Date                                2025-12-04 18:24
Total Runtime (minutes)                    74.426826
Python Version                                3.12.3
Config File              scripts/irm/apos_config.yml
ⓘLearner g Learner m Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM Regr. LGBM Clas. 8.670 39.947 0.956 45.539 0.963
LGBM Regr. Logistic 1.118 6.456 0.975 7.360 0.979
Linear LGBM Clas. 1.307 7.919 0.985 9.033 0.989
Linear Logistic 0.285 1.368 0.944 1.558 0.932
Coverage for 95%-Confidence Interval over 1000 Repetitions
ⓘLearner g Learner m Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM Regr. LGBM Clas. 8.670 33.525 0.908 39.690 0.912
LGBM Regr. Logistic 1.118 5.418 0.942 6.414 0.951
Linear LGBM Clas. 1.307 6.646 0.958 7.873 0.968
Linear Logistic 0.285 1.148 0.896 1.357 0.885
Coverage for 90%-Confidence Interval over 1000 Repetitions

Tuning

The simulations are based on the the make_irm_data_discrete_treatments-DGP with \(500\) observations. This is only an example as the untuned version just relies on the default configuration.

APOS Coverage

The non-uniform results (coverage, ci length and bias) refer to averaged values over all levels (point-wise confidende intervals). The same holds for the loss values which are averaged over all treatment levels.

DoubleML Version                                0.12.dev0
Script                       APOSTuningCoverageSimulation
Date                                     2025-12-01 13:09
Total Runtime (minutes)                         38.631183
Python Version                                     3.12.9
Config File              scripts/irm/apos_tune_config.yml
ⓘLearner g Learner m Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. False 7.055 33.427 0.977 40.557 0.980 10.232 13.632 0.798
LGBM Regr. LGBM Clas. True 1.525 7.314 0.945 8.538 0.940 9.749 11.553 0.604
Coverage for 95%-Confidence Interval over 200 Repetitions
ⓘLearner g Learner m Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. False 7.055 28.053 0.913 35.926 0.945 10.232 13.632 0.798
LGBM Regr. LGBM Clas. True 1.525 6.138 0.887 7.418 0.865 9.749 11.553 0.604
Coverage for 90%-Confidence Interval over 200 Repetitions

Causal Contrast Coverage

The non-uniform results (coverage, ci length and bias) refer to averaged values over all quantiles (point-wise confidende intervals). The same holds for the loss values which are averaged over all treatment levels.

DoubleML Version                                0.12.dev0
Script                       APOSTuningCoverageSimulation
Date                                     2025-12-01 13:09
Total Runtime (minutes)                         38.631183
Python Version                                     3.12.9
Config File              scripts/irm/apos_tune_config.yml
ⓘLearner g Learner m Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. False 9.703 44.753 0.963 51.032 0.975 10.232 13.632 0.798
LGBM Regr. LGBM Clas. True 1.146 5.102 0.945 5.809 0.950 9.749 11.553 0.604
Coverage for 95%-Confidence Interval over 200 Repetitions
ⓘLearner g Learner m Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. False 9.703 37.558 0.905 44.436 0.930 10.232 13.632 0.798
LGBM Regr. LGBM Clas. True 1.146 4.282 0.863 5.059 0.885 9.749 11.553 0.604
Coverage for 90%-Confidence Interval over 200 Repetitions