DiD for Cross-Sectional Data over Multiple Periods

Coverage

The simulations are based on the make_did_cs_CS2021-DGP with \(1000\) observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:

  • Type 1: Linear outcome model and treatment assignment
  • Type 4: Nonlinear outcome model and treatment assignment
  • Type 6: Randomized treatment assignment and nonlinear outcome model

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidence intervals).

DoubleML Version                                   0.12.dev0
Script                          DIDCSMultiCoverageSimulation
Date                                        2025-12-04 22:25
Total Runtime (minutes)                           315.672405
Python Version                                        3.12.3
Config File              scripts/did/did_cs_multi_config.yml

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.218 16.515 0.975 23.956 0.992 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 3.865 20.295 0.980 29.404 0.994 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.987 15.491 0.978 22.511 0.996 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.559 7.974 0.967 11.562 0.988 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.819 9.507 0.966 13.797 0.980 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.562 7.935 0.969 11.534 0.980 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.131 0.749 0.976 1.074 0.992 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.091 5.670 0.960 8.139 0.970 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.818 5.253 0.990 7.556 0.998 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.131 0.745 0.976 1.069 0.996 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.081 5.649 0.958 8.113 0.970 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.817 5.220 0.989 7.510 0.996 7.559 12.019 7.577 12.096 0.694
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.218 13.860 0.935 21.935 0.974 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 3.865 17.032 0.943 26.926 0.984 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.987 13.001 0.939 20.613 0.982 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.559 6.692 0.924 10.588 0.962 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.819 7.978 0.927 12.629 0.954 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.562 6.659 0.926 10.562 0.962 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.131 0.628 0.943 0.977 0.968 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.091 4.758 0.916 7.419 0.924 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.818 4.408 0.968 6.887 0.996 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.131 0.625 0.941 0.973 0.980 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.081 4.741 0.921 7.393 0.940 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.817 4.381 0.968 6.850 0.988 7.559 12.019 7.577 12.096 0.694
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.783 3.947 0.949 5.688 0.924 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.695 3.389 0.944 4.894 0.934 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.555 3.423 0.987 4.945 0.992 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.788 3.948 0.949 5.693 0.930 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.694 3.390 0.944 4.897 0.918 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.561 3.426 0.983 4.945 0.994 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.130 0.707 0.968 1.015 0.970 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.045 4.830 0.927 6.957 0.882 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.776 4.982 0.989 7.172 0.998 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.130 0.707 0.968 1.016 0.968 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.047 4.833 0.924 6.960 0.896 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.775 4.983 0.989 7.179 1.000 7.557 12.027 7.578 12.090 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.783 3.313 0.900 5.191 0.874 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.695 2.844 0.892 4.474 0.850 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.555 2.872 0.958 4.517 0.984 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.788 3.313 0.897 5.192 0.868 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.694 2.845 0.895 4.472 0.850 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.561 2.875 0.957 4.516 0.984 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.130 0.594 0.925 0.925 0.950 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.045 4.054 0.871 6.342 0.804 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.776 4.181 0.967 6.539 0.992 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.130 0.594 0.927 0.926 0.944 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.047 4.056 0.871 6.346 0.800 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.775 4.182 0.966 6.544 0.996 7.557 12.027 7.578 12.090 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Aggregated Effects

These simulations test different types of aggregation, as described in DiD User Guide.

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidence intervals).

Group Effects

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.467 17.339 0.971 21.077 0.980 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 4.014 20.894 0.975 25.400 0.980 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 3.336 16.788 0.968 20.440 0.980 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.703 8.500 0.958 10.334 0.970 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.940 9.898 0.968 12.035 0.962 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.690 8.500 0.965 10.333 0.962 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.116 0.667 0.978 0.813 0.988 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.266 6.205 0.917 7.550 0.904 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.911 5.855 0.985 7.113 0.994 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.115 0.664 0.979 0.808 0.988 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.246 6.171 0.912 7.504 0.900 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.902 5.820 0.985 7.079 0.988 7.559 12.019 7.577 12.096 0.694
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.467 14.552 0.933 18.637 0.944 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 4.014 17.535 0.927 22.506 0.956 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 3.336 14.089 0.917 18.107 0.944 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.703 7.133 0.909 9.143 0.932 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.940 8.306 0.923 10.651 0.922 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.690 7.133 0.914 9.153 0.920 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.116 0.560 0.944 0.720 0.962 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.266 5.207 0.867 6.681 0.836 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.911 4.914 0.963 6.301 0.976 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.115 0.557 0.945 0.716 0.962 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.246 5.179 0.863 6.640 0.834 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.902 4.884 0.957 6.262 0.976 7.559 12.019 7.577 12.096 0.694
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.872 4.359 0.929 5.302 0.914 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.783 3.721 0.909 4.526 0.866 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.622 3.749 0.982 4.569 0.988 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.886 4.358 0.933 5.295 0.906 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.789 3.720 0.910 4.520 0.862 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.624 3.754 0.983 4.565 0.988 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.116 0.632 0.965 0.769 0.964 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.221 5.391 0.864 6.562 0.792 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.866 5.549 0.986 6.754 0.992 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.116 0.632 0.965 0.771 0.966 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.219 5.395 0.863 6.564 0.796 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.867 5.550 0.982 6.754 0.994 7.557 12.027 7.578 12.090 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.872 3.659 0.868 4.693 0.840 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.783 3.123 0.847 4.009 0.792 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.622 3.146 0.949 4.039 0.974 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.886 3.658 0.873 4.693 0.846 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.789 3.122 0.847 4.009 0.794 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.624 3.150 0.949 4.041 0.976 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.116 0.530 0.929 0.682 0.924 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.221 4.524 0.806 5.801 0.684 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.866 4.657 0.957 5.974 0.970 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.116 0.531 0.928 0.681 0.918 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.219 4.528 0.807 5.806 0.684 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.867 4.657 0.956 5.976 0.972 7.557 12.027 7.578 12.090 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Time Effects

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.007 15.935 0.973 19.371 0.976 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 3.868 20.718 0.987 25.197 0.996 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.687 14.009 0.979 17.050 0.986 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.447 7.360 0.960 8.959 0.968 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.692 9.181 0.967 11.164 0.968 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.377 7.093 0.966 8.624 0.976 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.112 0.636 0.973 0.775 0.984 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.198 5.831 0.962 7.100 0.960 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.778 5.064 0.987 6.164 0.998 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.111 0.633 0.975 0.769 0.982 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.193 5.807 0.952 7.066 0.960 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.776 5.034 0.986 6.132 0.996 7.559 12.019 7.577 12.096 0.694
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 3.007 13.373 0.929 17.170 0.952 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 3.868 17.388 0.939 22.298 0.978 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.687 11.757 0.935 15.096 0.964 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.447 6.177 0.921 7.930 0.922 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.692 7.705 0.927 9.881 0.930 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.377 5.952 0.915 7.647 0.942 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.112 0.534 0.939 0.685 0.948 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 1.198 4.893 0.893 6.282 0.928 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.778 4.249 0.959 5.452 0.986 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.111 0.532 0.937 0.682 0.950 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 1.193 4.874 0.901 6.260 0.906 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.776 4.225 0.960 5.434 0.978 7.559 12.019 7.577 12.096 0.694
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.752 3.738 0.947 4.552 0.958 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.702 3.165 0.941 3.851 0.936 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.521 3.222 0.991 3.922 0.992 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.760 3.737 0.947 4.544 0.948 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.698 3.165 0.940 3.851 0.922 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.532 3.225 0.987 3.930 0.986 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.112 0.586 0.963 0.712 0.960 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.125 4.595 0.913 5.591 0.932 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.743 4.780 0.982 5.815 0.994 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.112 0.586 0.963 0.712 0.962 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.130 4.597 0.913 5.591 0.918 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.742 4.782 0.984 5.825 0.996 7.557 12.027 7.578 12.090 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.752 3.137 0.901 4.029 0.912 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.702 2.656 0.879 3.414 0.868 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.521 2.704 0.961 3.475 0.986 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.760 3.136 0.900 4.026 0.902 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.698 2.656 0.880 3.411 0.862 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.532 2.707 0.967 3.476 0.970 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.112 0.492 0.916 0.631 0.916 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 1.125 3.856 0.842 4.955 0.846 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.743 4.012 0.956 5.152 0.980 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.112 0.492 0.914 0.630 0.918 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 1.130 3.858 0.841 4.958 0.832 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.742 4.013 0.957 5.155 0.972 7.557 12.027 7.578 12.090 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Event Study Aggregation

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 2.373 12.567 0.975 16.806 0.984 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 2.981 15.504 0.977 20.761 0.992 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.277 11.555 0.973 15.472 0.982 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.136 5.982 0.970 8.012 0.984 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.328 7.077 0.968 9.480 0.972 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.167 5.886 0.969 7.880 0.980 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.093 0.531 0.979 0.707 0.986 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 0.965 4.020 0.929 5.363 0.928 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.605 3.697 0.986 4.933 0.996 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.093 0.529 0.977 0.703 0.982 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 0.954 4.001 0.928 5.336 0.942 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.603 3.669 0.985 4.897 0.992 7.559 12.019 7.577 12.096 0.694
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 2.373 10.547 0.934 15.181 0.956 6.614 10.442 5.989 9.402 0.622
LGBM Regr. LGBM Clas. 4 False 2.981 13.011 0.935 18.724 0.966 5.484 8.574 5.545 8.686 0.621
LGBM Regr. LGBM Clas. 6 False 2.277 9.698 0.925 13.970 0.970 5.556 8.697 5.566 8.725 0.643
LGBM Regr. LGBM Clas. 1 True 1.136 5.020 0.924 7.238 0.950 6.611 10.431 5.988 9.398 0.623
LGBM Regr. LGBM Clas. 4 True 1.328 5.939 0.932 8.561 0.940 5.482 8.586 5.550 8.692 0.620
LGBM Regr. LGBM Clas. 6 True 1.167 4.940 0.926 7.122 0.960 5.556 8.697 5.569 8.730 0.642
Linear Logistic 1 False 0.093 0.446 0.947 0.636 0.964 1.433 1.432 1.431 1.432 0.673
Linear Logistic 4 False 0.965 3.374 0.867 4.832 0.862 7.317 11.670 7.363 11.707 0.673
Linear Logistic 6 False 0.605 3.103 0.961 4.443 0.982 7.556 12.018 7.577 12.092 0.694
Linear Logistic 1 True 0.093 0.444 0.943 0.634 0.972 1.433 1.433 1.431 1.432 0.673
Linear Logistic 4 True 0.954 3.357 0.867 4.811 0.882 7.318 11.671 7.363 11.711 0.673
Linear Logistic 6 True 0.603 3.079 0.963 4.412 0.976 7.559 12.019 7.577 12.096 0.694
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.643 2.778 0.926 3.709 0.888 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.578 2.405 0.927 3.218 0.900 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.418 2.431 0.982 3.248 0.992 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.638 2.778 0.928 3.710 0.900 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.578 2.406 0.927 3.217 0.916 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.423 2.434 0.980 3.252 0.992 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.095 0.503 0.967 0.670 0.980 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 0.937 3.395 0.889 4.534 0.852 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.579 3.507 0.986 4.681 0.994 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.095 0.504 0.968 0.670 0.974 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 0.939 3.397 0.885 4.534 0.854 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.578 3.508 0.988 4.680 0.998 7.557 12.027 7.578 12.090 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_d0_t0 Loss g_d0_t1 Loss g_d1_t0 Loss g_d1_t1 Loss m
LGBM Regr. LGBM Clas. 1 False 0.643 2.332 0.863 3.345 0.816 6.617 10.434 5.988 9.402 nan
LGBM Regr. LGBM Clas. 4 False 0.578 2.018 0.865 2.899 0.832 5.479 8.579 5.547 8.696 nan
LGBM Regr. LGBM Clas. 6 False 0.418 2.040 0.950 2.932 0.980 5.558 8.695 5.565 8.730 nan
LGBM Regr. LGBM Clas. 1 True 0.638 2.332 0.868 3.349 0.836 6.614 10.429 5.987 9.402 nan
LGBM Regr. LGBM Clas. 4 True 0.578 2.019 0.867 2.901 0.842 5.484 8.588 5.546 8.685 nan
LGBM Regr. LGBM Clas. 6 True 0.423 2.042 0.947 2.936 0.978 5.556 8.697 5.567 8.732 nan
Linear Logistic 1 False 0.095 0.422 0.923 0.603 0.946 1.433 1.433 1.431 1.432 nan
Linear Logistic 4 False 0.937 2.849 0.813 4.085 0.758 7.313 11.668 7.365 11.710 nan
Linear Logistic 6 False 0.579 2.943 0.962 4.213 0.980 7.558 12.021 7.579 12.100 nan
Linear Logistic 1 True 0.095 0.423 0.923 0.603 0.942 1.433 1.432 1.431 1.432 nan
Linear Logistic 4 True 0.939 2.851 0.813 4.086 0.758 7.317 11.673 7.363 11.711 nan
Linear Logistic 6 True 0.578 2.944 0.962 4.223 0.976 7.557 12.027 7.578 12.090 nan
Coverage for 90%-Confidence Interval over 500 Repetitions