DiD for Panel Data over Multiple Periods

Coverage

The simulations are based on the the make_did_CS2021-DGP with \(1000\) observations. Learners are both set to either boosting or a linear (logistic) model. Due to time constraints we only consider the following DGPs:

  • Type 1: Linear outcome model and treatment assignment
  • Type 4: Nonlinear outcome model and treatment assignment
  • Type 6: Randomized treatment assignment and nonlinear outcome model

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidende intervals).

DoubleML Version                                   0.12.dev0
Script                            DIDMultiCoverageSimulation
Date                                        2025-12-04 19:08
Total Runtime (minutes)                           118.965039
Python Version                                        3.12.3
Config File              scripts/did/did_pa_multi_config.yml

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.050 5.056 0.973 7.220 0.996 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.318 6.185 0.981 8.781 0.998 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.917 4.609 0.974 6.585 0.990 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.416 2.130 0.971 3.048 0.972 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.498 2.543 0.960 3.621 0.954 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.430 2.099 0.959 3.008 0.976 3.667 3.673 0.882
Linear Logistic 1 False 0.110 0.535 0.947 0.768 0.950 1.431 1.433 0.677
Linear Logistic 4 False 0.682 2.058 0.795 2.766 0.682 4.632 4.675 0.677
Linear Logistic 6 False 0.389 1.855 0.953 2.507 0.956 4.788 4.816 0.697
Linear Logistic 1 True 0.110 0.529 0.945 0.759 0.958 1.431 1.434 0.677
Linear Logistic 4 True 0.684 2.043 0.792 2.750 0.668 4.633 4.672 0.677
Linear Logistic 6 True 0.391 1.841 0.948 2.488 0.936 4.789 4.818 0.697
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.050 4.243 0.930 6.572 0.976 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.318 5.191 0.933 7.975 0.982 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.917 3.868 0.929 5.991 0.976 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.416 1.787 0.925 2.779 0.932 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.498 2.134 0.915 3.291 0.914 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.430 1.762 0.904 2.740 0.948 3.667 3.673 0.882
Linear Logistic 1 False 0.110 0.449 0.896 0.699 0.894 1.431 1.433 0.677
Linear Logistic 4 False 0.682 1.727 0.702 2.476 0.568 4.632 4.675 0.677
Linear Logistic 6 False 0.389 1.557 0.903 2.245 0.894 4.788 4.816 0.697
Linear Logistic 1 True 0.110 0.444 0.890 0.692 0.904 1.431 1.434 0.677
Linear Logistic 4 True 0.684 1.715 0.696 2.463 0.574 4.633 4.672 0.677
Linear Logistic 6 True 0.391 1.545 0.895 2.228 0.872 4.789 4.818 0.697
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.442 1.356 0.808 1.879 0.702 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.435 1.189 0.762 1.677 0.600 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.248 1.194 0.946 1.682 0.936 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.443 1.357 0.805 1.883 0.702 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.435 1.189 0.758 1.676 0.602 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.249 1.194 0.947 1.680 0.938 3.667 3.673 nan
Linear Logistic 1 False 0.108 0.499 0.932 0.716 0.922 1.431 1.434 nan
Linear Logistic 4 False 0.727 1.655 0.666 2.245 0.460 4.630 4.676 nan
Linear Logistic 6 False 0.338 1.666 0.946 2.255 0.952 4.790 4.820 nan
Linear Logistic 1 True 0.108 0.499 0.932 0.716 0.922 1.431 1.434 nan
Linear Logistic 4 True 0.727 1.655 0.667 2.243 0.446 4.632 4.675 nan
Linear Logistic 6 True 0.338 1.666 0.948 2.254 0.942 4.786 4.816 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.442 1.138 0.712 1.694 0.580 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.435 0.998 0.666 1.518 0.484 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.248 1.002 0.892 1.522 0.890 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.443 1.139 0.716 1.696 0.586 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.435 0.998 0.665 1.516 0.494 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.249 1.002 0.896 1.521 0.896 3.667 3.673 nan
Linear Logistic 1 False 0.108 0.419 0.878 0.652 0.860 1.431 1.434 nan
Linear Logistic 4 False 0.727 1.389 0.571 2.013 0.326 4.630 4.676 nan
Linear Logistic 6 False 0.338 1.398 0.900 2.021 0.890 4.790 4.820 nan
Linear Logistic 1 True 0.108 0.419 0.876 0.653 0.862 1.431 1.434 nan
Linear Logistic 4 True 0.727 1.389 0.571 2.010 0.322 4.632 4.675 nan
Linear Logistic 6 True 0.338 1.398 0.902 2.020 0.900 4.786 4.816 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Aggregated Effects

These simulations test different types of aggregation, as described in DiD User Guide.

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidende intervals).

Group Effects

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.010 5.016 0.981 6.051 0.986 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.310 6.248 0.985 7.511 0.986 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.880 4.589 0.979 5.527 0.984 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.397 2.120 0.975 2.562 0.982 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.475 2.583 0.972 3.115 0.960 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.411 2.106 0.965 2.551 0.970 3.667 3.673 0.882
Linear Logistic 1 False 0.098 0.477 0.953 0.581 0.962 1.431 1.433 0.677
Linear Logistic 4 False 0.773 2.283 0.792 2.732 0.708 4.632 4.675 0.677
Linear Logistic 6 False 0.426 2.034 0.952 2.436 0.960 4.788 4.816 0.697
Linear Logistic 1 True 0.097 0.472 0.953 0.574 0.966 1.431 1.434 0.677
Linear Logistic 4 True 0.779 2.273 0.791 2.719 0.692 4.633 4.672 0.677
Linear Logistic 6 True 0.430 2.019 0.954 2.416 0.952 4.789 4.818 0.697
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.010 4.209 0.942 5.323 0.978 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.310 5.244 0.935 6.609 0.970 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.880 3.852 0.937 4.872 0.968 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.397 1.779 0.930 2.261 0.966 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.475 2.168 0.927 2.747 0.934 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.411 1.767 0.910 2.248 0.932 3.667 3.673 0.882
Linear Logistic 1 False 0.098 0.401 0.889 0.514 0.924 1.431 1.433 0.677
Linear Logistic 4 False 0.773 1.916 0.691 2.397 0.588 4.632 4.675 0.677
Linear Logistic 6 False 0.426 1.707 0.905 2.137 0.910 4.788 4.816 0.697
Linear Logistic 1 True 0.097 0.396 0.886 0.509 0.918 1.431 1.434 0.677
Linear Logistic 4 True 0.779 1.907 0.686 2.389 0.596 4.633 4.672 0.677
Linear Logistic 6 True 0.430 1.694 0.893 2.120 0.906 4.789 4.818 0.697
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.496 1.457 0.790 1.743 0.708 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.486 1.265 0.727 1.527 0.558 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.266 1.270 0.942 1.532 0.944 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.494 1.459 0.778 1.745 0.684 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.486 1.265 0.728 1.529 0.570 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.266 1.270 0.944 1.532 0.942 3.667 3.673 nan
Linear Logistic 1 False 0.098 0.447 0.929 0.544 0.934 1.431 1.434 nan
Linear Logistic 4 False 0.824 1.828 0.657 2.196 0.440 4.630 4.676 nan
Linear Logistic 6 False 0.374 1.839 0.946 2.209 0.950 4.790 4.820 nan
Linear Logistic 1 True 0.098 0.447 0.931 0.544 0.932 1.431 1.434 nan
Linear Logistic 4 True 0.823 1.828 0.661 2.196 0.444 4.632 4.675 nan
Linear Logistic 6 True 0.373 1.838 0.949 2.209 0.954 4.786 4.816 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.496 1.223 0.680 1.530 0.584 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.486 1.062 0.633 1.348 0.466 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.266 1.066 0.888 1.352 0.880 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.494 1.224 0.686 1.534 0.574 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.486 1.062 0.629 1.348 0.456 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.266 1.066 0.893 1.352 0.904 3.667 3.673 nan
Linear Logistic 1 False 0.098 0.375 0.873 0.482 0.858 1.431 1.434 nan
Linear Logistic 4 False 0.824 1.534 0.555 1.933 0.336 4.630 4.676 nan
Linear Logistic 6 False 0.374 1.543 0.900 1.942 0.898 4.790 4.820 nan
Linear Logistic 1 True 0.098 0.375 0.868 0.482 0.854 1.431 1.434 nan
Linear Logistic 4 True 0.823 1.534 0.555 1.935 0.342 4.632 4.675 nan
Linear Logistic 6 True 0.373 1.543 0.901 1.942 0.900 4.786 4.816 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Time Effects

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.025 5.146 0.982 6.104 0.984 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.408 6.656 0.983 7.818 0.990 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.810 4.325 0.970 5.145 0.974 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.385 2.043 0.975 2.423 0.984 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.500 2.595 0.956 3.050 0.956 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.391 1.931 0.959 2.294 0.968 3.667 3.673 0.882
Linear Logistic 1 False 0.095 0.456 0.943 0.554 0.938 1.431 1.433 0.677
Linear Logistic 4 False 0.868 2.192 0.641 2.435 0.612 4.632 4.675 0.677
Linear Logistic 6 False 0.391 1.835 0.947 2.058 0.940 4.788 4.816 0.697
Linear Logistic 1 True 0.096 0.451 0.935 0.548 0.936 1.431 1.434 0.677
Linear Logistic 4 True 0.870 2.180 0.640 2.419 0.616 4.633 4.672 0.677
Linear Logistic 6 True 0.389 1.814 0.943 2.037 0.940 4.789 4.818 0.697
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.025 4.319 0.945 5.331 0.960 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.408 5.586 0.937 6.816 0.970 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.810 3.629 0.939 4.499 0.936 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.385 1.715 0.945 2.122 0.952 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.500 2.178 0.908 2.662 0.904 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.391 1.621 0.905 2.008 0.916 3.667 3.673 0.882
Linear Logistic 1 False 0.095 0.383 0.895 0.490 0.878 1.431 1.433 0.677
Linear Logistic 4 False 0.868 1.840 0.529 2.088 0.484 4.632 4.675 0.677
Linear Logistic 6 False 0.391 1.540 0.893 1.774 0.894 4.788 4.816 0.697
Linear Logistic 1 True 0.096 0.379 0.889 0.485 0.866 1.431 1.434 0.677
Linear Logistic 4 True 0.870 1.830 0.521 2.077 0.484 4.633 4.672 0.677
Linear Logistic 6 True 0.389 1.522 0.892 1.757 0.886 4.789 4.818 0.697
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.502 1.343 0.719 1.534 0.690 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.531 1.126 0.565 1.305 0.540 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.244 1.130 0.935 1.308 0.922 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.503 1.344 0.710 1.539 0.680 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.532 1.125 0.569 1.300 0.516 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.243 1.130 0.939 1.309 0.944 3.667 3.673 nan
Linear Logistic 1 False 0.096 0.414 0.912 0.504 0.904 1.431 1.434 nan
Linear Logistic 4 False 0.928 1.633 0.405 1.827 0.372 4.630 4.676 nan
Linear Logistic 6 False 0.334 1.631 0.939 1.832 0.944 4.790 4.820 nan
Linear Logistic 1 True 0.096 0.414 0.913 0.503 0.906 1.431 1.434 nan
Linear Logistic 4 True 0.928 1.633 0.404 1.830 0.378 4.632 4.675 nan
Linear Logistic 6 True 0.334 1.631 0.940 1.832 0.940 4.786 4.816 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.502 1.127 0.598 1.330 0.570 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.531 0.945 0.450 1.132 0.410 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.244 0.949 0.879 1.137 0.872 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.503 1.128 0.602 1.329 0.558 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.532 0.944 0.448 1.131 0.424 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.243 0.948 0.881 1.136 0.878 3.667 3.673 nan
Linear Logistic 1 False 0.096 0.348 0.855 0.445 0.818 1.431 1.434 nan
Linear Logistic 4 False 0.928 1.370 0.307 1.574 0.284 4.630 4.676 nan
Linear Logistic 6 False 0.334 1.369 0.897 1.577 0.882 4.790 4.820 nan
Linear Logistic 1 True 0.096 0.348 0.854 0.446 0.820 1.431 1.434 nan
Linear Logistic 4 True 0.928 1.371 0.310 1.575 0.280 4.632 4.675 nan
Linear Logistic 6 True 0.334 1.369 0.901 1.575 0.902 4.786 4.816 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Event Study Aggregation

Observational Score

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.022 4.963 0.977 6.456 0.988 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.309 6.241 0.978 8.044 0.986 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.841 4.412 0.980 5.736 0.992 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.379 2.019 0.973 2.634 0.986 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.476 2.502 0.964 3.236 0.960 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.397 1.984 0.968 2.588 0.978 3.667 3.673 0.882
Linear Logistic 1 False 0.079 0.380 0.952 0.505 0.946 1.431 1.433 0.677
Linear Logistic 4 False 0.771 2.127 0.760 2.598 0.674 4.632 4.675 0.677
Linear Logistic 6 False 0.391 1.877 0.956 2.300 0.954 4.788 4.816 0.697
Linear Logistic 1 True 0.079 0.377 0.949 0.501 0.946 1.431 1.434 0.677
Linear Logistic 4 True 0.770 2.115 0.754 2.583 0.672 4.633 4.672 0.677
Linear Logistic 6 True 0.396 1.861 0.951 2.280 0.946 4.789 4.818 0.697
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 1.022 4.165 0.931 5.764 0.970 4.253 3.906 0.859
LGBM Regr. LGBM Clas. 4 False 1.309 5.238 0.937 7.163 0.964 3.603 3.657 0.855
LGBM Regr. LGBM Clas. 6 False 0.841 3.703 0.940 5.120 0.956 3.669 3.674 0.883
LGBM Regr. LGBM Clas. 1 True 0.379 1.694 0.931 2.351 0.958 4.253 3.909 0.860
LGBM Regr. LGBM Clas. 4 True 0.476 2.100 0.920 2.885 0.916 3.602 3.656 0.854
LGBM Regr. LGBM Clas. 6 True 0.397 1.665 0.918 2.310 0.940 3.667 3.673 0.882
Linear Logistic 1 False 0.079 0.319 0.897 0.455 0.888 1.431 1.433 0.677
Linear Logistic 4 False 0.771 1.785 0.657 2.278 0.564 4.632 4.675 0.677
Linear Logistic 6 False 0.391 1.575 0.906 2.019 0.902 4.788 4.816 0.697
Linear Logistic 1 True 0.079 0.316 0.891 0.451 0.896 1.431 1.434 0.677
Linear Logistic 4 True 0.770 1.775 0.648 2.268 0.572 4.633 4.672 0.677
Linear Logistic 6 True 0.396 1.562 0.900 2.005 0.894 4.789 4.818 0.697
Coverage for 90%-Confidence Interval over 500 Repetitions

Experimental Score

The results are only valid for the DGP 6, as the experimental score assumes a randomized treatment assignment.

ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.473 1.347 0.777 1.681 0.706 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.482 1.135 0.685 1.444 0.568 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.239 1.146 0.943 1.454 0.934 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.475 1.348 0.775 1.682 0.708 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.484 1.135 0.683 1.445 0.574 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.241 1.146 0.941 1.455 0.946 3.667 3.673 nan
Linear Logistic 1 False 0.080 0.356 0.928 0.473 0.908 1.431 1.434 nan
Linear Logistic 4 False 0.829 1.651 0.575 2.035 0.432 4.630 4.676 nan
Linear Logistic 6 False 0.335 1.666 0.951 2.050 0.946 4.790 4.820 nan
Linear Logistic 1 True 0.080 0.356 0.927 0.473 0.920 1.431 1.434 nan
Linear Logistic 4 True 0.829 1.651 0.577 2.035 0.454 4.632 4.675 nan
Linear Logistic 6 True 0.335 1.665 0.952 2.049 0.944 4.786 4.816 nan
Coverage for 95%-Confidence Interval over 500 Repetitions
ⓘLearner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.473 1.130 0.675 1.482 0.578 4.251 3.902 nan
LGBM Regr. LGBM Clas. 4 False 0.482 0.953 0.588 1.279 0.466 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 False 0.239 0.961 0.891 1.288 0.874 3.665 3.673 nan
LGBM Regr. LGBM Clas. 1 True 0.475 1.131 0.672 1.483 0.582 4.255 3.908 nan
LGBM Regr. LGBM Clas. 4 True 0.484 0.953 0.587 1.279 0.462 3.604 3.656 nan
LGBM Regr. LGBM Clas. 6 True 0.241 0.961 0.890 1.289 0.886 3.667 3.673 nan
Linear Logistic 1 False 0.080 0.299 0.867 0.426 0.858 1.431 1.434 nan
Linear Logistic 4 False 0.829 1.385 0.469 1.788 0.332 4.630 4.676 nan
Linear Logistic 6 False 0.335 1.398 0.905 1.800 0.896 4.790 4.820 nan
Linear Logistic 1 True 0.080 0.299 0.869 0.426 0.844 1.431 1.434 nan
Linear Logistic 4 True 0.829 1.386 0.474 1.788 0.342 4.632 4.675 nan
Linear Logistic 6 True 0.335 1.397 0.910 1.800 0.896 4.786 4.816 nan
Coverage for 90%-Confidence Interval over 500 Repetitions

Tuning

The simulations are based on the the make_did_CS2021-DGP with \(1000\) observations. Due to time constraints we only consider one learner, use in-sample normalization and the following DGPs:

  • Type 1: Linear outcome model and treatment assignment
  • Type 4: Nonlinear outcome model and treatment assignment

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.

DoubleML Version                                        0.12.dev0
Script                           DIDMultiTuningCoverageSimulation
Date                                             2025-12-03 19:48
Total Runtime (minutes)                                 38.914859
Python Version                                             3.12.9
Config File              scripts/did/did_pa_multi_tune_config.yml

Observational Score

ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.435 2.216 0.976 3.158 1.000 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.256 1.297 0.948 1.844 0.970 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.448 2.154 0.968 3.078 0.980 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.262 1.262 0.944 1.798 0.960 3.514 3.527 0.693
Coverage for 95%-Confidence Interval over 100 Repetitions
ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.435 1.859 0.927 2.875 0.970 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.256 1.089 0.907 1.674 0.930 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.448 1.808 0.911 2.802 0.960 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.262 1.059 0.893 1.635 0.890 3.514 3.527 0.693
Coverage for 90%-Confidence Interval over 100 Repetitions

Tuning Aggregated Effects

These simulations test different types of aggregation, as described in DiD User Guide.

As before, we only consider one learner, use in-sample normalization and the following DGPs:

  • Type 1: Linear outcome model and treatment assignment
  • Type 4: Nonlinear outcome model and treatment assignment

The non-uniform results (coverage, ci length and bias) refer to averaged values over all \(ATTs\) (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.

Group Effects

Observational Score

ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.408 2.249 0.987 2.717 0.990 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.259 1.341 0.960 1.615 0.960 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.428 2.149 0.980 2.599 0.990 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.265 1.296 0.953 1.570 0.960 3.514 3.527 0.693
Coverage for 95%-Confidence Interval over 100 Repetitions
ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.408 1.888 0.937 2.392 0.970 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.259 1.125 0.913 1.429 0.900 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.428 1.804 0.937 2.288 0.970 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.265 1.088 0.903 1.379 0.930 3.514 3.527 0.693
Coverage for 90%-Confidence Interval over 100 Repetitions

Time Effects

Observational Score

ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.368 2.040 0.977 2.407 0.980 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.226 1.212 0.963 1.425 0.980 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.378 1.965 0.980 2.325 0.970 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.252 1.176 0.953 1.388 0.960 3.514 3.527 0.693
Coverage for 95%-Confidence Interval over 100 Repetitions
ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.368 1.712 0.943 2.109 0.950 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.226 1.017 0.923 1.241 0.920 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.378 1.649 0.927 2.036 0.950 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.252 0.987 0.883 1.212 0.920 3.514 3.527 0.693
Coverage for 90%-Confidence Interval over 100 Repetitions

Event Study Aggregation

Observational Score

ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.393 2.104 0.973 2.737 1.000 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.240 1.228 0.960 1.581 0.960 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.396 2.039 0.977 2.658 0.990 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.251 1.191 0.950 1.541 0.910 3.514 3.527 0.693
Coverage for 95%-Confidence Interval over 100 Repetitions
ⓘLearner g Learner m DGP Tuned Bias CI Length Coverage Uniform CI Length Uniform Coverage Loss g_control Loss g_treated Loss m
LGBM Regr. LGBM Clas. 1 False 0.393 1.766 0.942 2.442 0.930 4.152 4.102 0.880
LGBM Regr. LGBM Clas. 1 True 0.240 1.031 0.908 1.407 0.930 3.601 3.610 0.694
LGBM Regr. LGBM Clas. 4 False 0.396 1.711 0.937 2.368 0.980 3.652 3.671 0.883
LGBM Regr. LGBM Clas. 4 True 0.251 0.999 0.900 1.371 0.880 3.514 3.527 0.693
Coverage for 90%-Confidence Interval over 100 Repetitions