DiD for Panel Data over Multiple Periods

ATTE Coverage

The simulations are based on the the make_did_SZ2020-DGP with \(1000\) observations. Learners are only set to boosting, due to time constraints (and the nonlinearity of some of the DGPs).

DoubleML Version                                   0.10.dev0
Script                            DIDMultiCoverageSimulation
Date                                        2025-03-22 13:07
Total Runtime (minutes)                           161.557258
Python Version                                        3.12.9
Config File              scripts/did/did_pa_multi_config.yml

Observational Score

Table 1: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.703 3.257 0.967 4.665 0.989
LGBM LGBM 4 False 0.958 4.174 0.966 5.938 0.991
LGBM LGBM 6 False 0.508 2.588 0.972 3.719 0.986
LGBM LGBM 1 True 0.269 1.325 0.960 1.902 0.979
LGBM LGBM 4 True 0.329 1.686 0.964 2.406 0.964
LGBM LGBM 6 True 0.248 1.220 0.959 1.757 0.961
Linear Linear 1 False 0.078 0.380 0.949 0.545 0.946
Linear Linear 4 False 0.801 1.473 0.516 1.975 0.274
Linear Linear 6 False 0.253 1.223 0.948 1.654 0.958
Linear Linear 1 True 0.078 0.377 0.947 0.541 0.941
Linear Linear 4 True 0.801 1.471 0.514 1.974 0.282
Linear Linear 6 True 0.252 1.216 0.947 1.644 0.955
Table 2: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.703 2.734 0.913 4.250 0.970
LGBM LGBM 4 False 0.958 3.503 0.908 5.396 0.970
LGBM LGBM 6 False 0.508 2.172 0.928 3.390 0.966
LGBM LGBM 1 True 0.269 1.112 0.914 1.734 0.940
LGBM LGBM 4 True 0.329 1.415 0.920 2.188 0.923
LGBM LGBM 6 True 0.248 1.024 0.907 1.602 0.917
Linear Linear 1 False 0.078 0.319 0.897 0.495 0.890
Linear Linear 4 False 0.801 1.236 0.417 1.766 0.182
Linear Linear 6 False 0.253 1.027 0.899 1.483 0.903
Linear Linear 1 True 0.078 0.317 0.895 0.493 0.884
Linear Linear 4 True 0.801 1.234 0.410 1.765 0.186
Linear Linear 6 True 0.252 1.020 0.895 1.474 0.890

Experimental Score

Table 3: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.457 0.797 0.490 1.111 0.131
LGBM LGBM 4 False 0.329 0.694 0.621 0.987 0.298
LGBM LGBM 6 False 0.142 0.691 0.951 0.983 0.940
LGBM LGBM 1 True 0.456 0.797 0.491 1.112 0.140
LGBM LGBM 4 True 0.329 0.694 0.620 0.988 0.286
LGBM LGBM 6 True 0.141 0.691 0.953 0.982 0.946
Linear Linear 1 False 0.081 0.351 0.917 0.505 0.861
Linear Linear 4 False 0.819 1.163 0.382 1.576 0.067
Linear Linear 6 False 0.240 1.171 0.949 1.583 0.951
Linear Linear 1 True 0.081 0.351 0.917 0.504 0.864
Linear Linear 4 True 0.820 1.163 0.384 1.576 0.065
Linear Linear 6 True 0.240 1.171 0.950 1.585 0.956
Table 4: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.457 0.669 0.393 1.002 0.066
LGBM LGBM 4 False 0.329 0.583 0.531 0.897 0.206
LGBM LGBM 6 False 0.142 0.580 0.899 0.893 0.902
LGBM LGBM 1 True 0.456 0.669 0.399 1.003 0.073
LGBM LGBM 4 True 0.329 0.583 0.529 0.898 0.210
LGBM LGBM 6 True 0.141 0.580 0.901 0.892 0.903
Linear Linear 1 False 0.081 0.295 0.852 0.459 0.765
Linear Linear 4 False 0.819 0.976 0.301 1.414 0.036
Linear Linear 6 False 0.240 0.983 0.898 1.420 0.910
Linear Linear 1 True 0.081 0.295 0.851 0.459 0.762
Linear Linear 4 True 0.820 0.976 0.302 1.413 0.039
Linear Linear 6 True 0.240 0.983 0.897 1.420 0.908

Aggregated Effects

Group Effects

Observational Score

Table 5: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.692 3.191 0.965 3.846 0.981
LGBM LGBM 4 False 0.975 4.198 0.967 5.041 0.987
LGBM LGBM 6 False 0.470 2.528 0.981 3.057 0.989
LGBM LGBM 1 True 0.254 1.303 0.969 1.572 0.979
LGBM LGBM 4 True 0.314 1.702 0.970 2.050 0.976
LGBM LGBM 6 True 0.235 1.207 0.969 1.461 0.976
Linear Linear 1 False 0.070 0.339 0.950 0.412 0.948
Linear Linear 4 False 0.912 1.639 0.505 1.961 0.288
Linear Linear 6 False 0.278 1.348 0.946 1.617 0.950
Linear Linear 1 True 0.070 0.337 0.950 0.410 0.946
Linear Linear 4 True 0.912 1.638 0.501 1.960 0.292
Linear Linear 6 True 0.277 1.340 0.945 1.609 0.948
Table 6: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.692 2.678 0.908 3.389 0.946
LGBM LGBM 4 False 0.975 3.523 0.909 4.436 0.937
LGBM LGBM 6 False 0.470 2.122 0.941 2.694 0.965
LGBM LGBM 1 True 0.254 1.093 0.925 1.388 0.931
LGBM LGBM 4 True 0.314 1.428 0.932 1.806 0.944
LGBM LGBM 6 True 0.235 1.013 0.919 1.290 0.938
Linear Linear 1 False 0.070 0.284 0.901 0.365 0.898
Linear Linear 4 False 0.912 1.376 0.407 1.724 0.201
Linear Linear 6 False 0.278 1.131 0.896 1.421 0.900
Linear Linear 1 True 0.070 0.282 0.899 0.363 0.895
Linear Linear 4 True 0.912 1.375 0.404 1.722 0.190
Linear Linear 6 True 0.277 1.125 0.895 1.413 0.896

Experimental Score

Table 7: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.514 0.846 0.444 1.007 0.125
LGBM LGBM 4 False 0.365 0.725 0.586 0.877 0.293
LGBM LGBM 6 False 0.148 0.720 0.952 0.871 0.958
LGBM LGBM 1 True 0.511 0.846 0.452 1.008 0.124
LGBM LGBM 4 True 0.365 0.725 0.580 0.876 0.299
LGBM LGBM 6 True 0.148 0.720 0.955 0.871 0.951
Linear Linear 1 False 0.077 0.315 0.894 0.383 0.856
Linear Linear 4 False 0.925 1.285 0.380 1.546 0.072
Linear Linear 6 False 0.266 1.293 0.947 1.552 0.951
Linear Linear 1 True 0.077 0.315 0.894 0.383 0.859
Linear Linear 4 True 0.924 1.285 0.383 1.546 0.072
Linear Linear 6 True 0.266 1.293 0.948 1.553 0.951
Table 8: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.514 0.710 0.363 0.884 0.063
LGBM LGBM 4 False 0.365 0.608 0.500 0.774 0.208
LGBM LGBM 6 False 0.148 0.604 0.892 0.768 0.905
LGBM LGBM 1 True 0.511 0.710 0.366 0.884 0.072
LGBM LGBM 4 True 0.365 0.608 0.493 0.773 0.213
LGBM LGBM 6 True 0.148 0.604 0.896 0.768 0.902
Linear Linear 1 False 0.077 0.264 0.827 0.339 0.774
Linear Linear 4 False 0.925 1.079 0.308 1.361 0.041
Linear Linear 6 False 0.266 1.085 0.898 1.367 0.897
Linear Linear 1 True 0.077 0.264 0.827 0.339 0.770
Linear Linear 4 True 0.924 1.079 0.311 1.359 0.042
Linear Linear 6 True 0.266 1.085 0.897 1.367 0.897

Time Effects

Observational Score

Table 9: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.765 3.463 0.956 4.100 0.975
LGBM LGBM 4 False 1.119 4.665 0.963 5.487 0.982
LGBM LGBM 6 False 0.442 2.395 0.982 2.862 0.984
LGBM LGBM 1 True 0.259 1.308 0.969 1.552 0.978
LGBM LGBM 4 True 0.321 1.754 0.974 2.066 0.980
LGBM LGBM 6 True 0.219 1.107 0.963 1.319 0.960
Linear Linear 1 False 0.068 0.327 0.940 0.397 0.927
Linear Linear 4 False 1.070 1.603 0.255 1.768 0.227
Linear Linear 6 False 0.250 1.205 0.947 1.352 0.945
Linear Linear 1 True 0.068 0.325 0.933 0.394 0.922
Linear Linear 4 True 1.070 1.603 0.249 1.769 0.221
Linear Linear 6 True 0.249 1.195 0.943 1.341 0.939
Table 10: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.765 2.906 0.894 3.589 0.920
LGBM LGBM 4 False 1.119 3.915 0.888 4.791 0.920
LGBM LGBM 6 False 0.442 2.010 0.942 2.510 0.958
LGBM LGBM 1 True 0.259 1.097 0.930 1.361 0.938
LGBM LGBM 4 True 0.321 1.472 0.942 1.801 0.940
LGBM LGBM 6 True 0.219 0.929 0.915 1.157 0.920
Linear Linear 1 False 0.068 0.275 0.891 0.352 0.877
Linear Linear 4 False 1.070 1.345 0.172 1.516 0.148
Linear Linear 6 False 0.250 1.012 0.888 1.165 0.885
Linear Linear 1 True 0.068 0.272 0.887 0.349 0.870
Linear Linear 4 True 1.070 1.345 0.161 1.516 0.131
Linear Linear 6 True 0.249 1.003 0.886 1.154 0.882

Experimental Score

Table 11: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.588 0.804 0.174 0.924 0.118
LGBM LGBM 4 False 0.418 0.648 0.320 0.759 0.244
LGBM LGBM 6 False 0.130 0.643 0.954 0.752 0.951
LGBM LGBM 1 True 0.586 0.804 0.171 0.924 0.123
LGBM LGBM 4 True 0.417 0.648 0.320 0.758 0.245
LGBM LGBM 6 True 0.129 0.643 0.952 0.753 0.955
Linear Linear 1 False 0.076 0.291 0.872 0.354 0.838
Linear Linear 4 False 1.083 1.151 0.062 1.288 0.050
Linear Linear 6 False 0.238 1.148 0.948 1.286 0.946
Linear Linear 1 True 0.076 0.291 0.870 0.354 0.839
Linear Linear 4 True 1.083 1.151 0.063 1.287 0.047
Linear Linear 6 True 0.238 1.148 0.948 1.288 0.949
Table 12: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.588 0.675 0.101 0.801 0.066
LGBM LGBM 4 False 0.418 0.544 0.224 0.660 0.169
LGBM LGBM 6 False 0.130 0.540 0.908 0.656 0.901
LGBM LGBM 1 True 0.586 0.674 0.107 0.800 0.066
LGBM LGBM 4 True 0.417 0.544 0.232 0.661 0.170
LGBM LGBM 6 True 0.129 0.539 0.906 0.657 0.903
Linear Linear 1 False 0.076 0.244 0.799 0.313 0.741
Linear Linear 4 False 1.083 0.966 0.036 1.108 0.031
Linear Linear 6 False 0.238 0.963 0.890 1.108 0.887
Linear Linear 1 True 0.076 0.244 0.800 0.313 0.732
Linear Linear 4 True 1.083 0.966 0.036 1.108 0.031
Linear Linear 6 True 0.238 0.963 0.888 1.107 0.885

Event Study Aggregation

Observational Score

Table 13: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.697 3.167 0.961 4.130 0.980
LGBM LGBM 4 False 1.006 4.219 0.965 5.451 0.981
LGBM LGBM 6 False 0.460 2.427 0.973 3.174 0.989
LGBM LGBM 1 True 0.250 1.251 0.967 1.636 0.965
LGBM LGBM 4 True 0.312 1.660 0.969 2.151 0.973
LGBM LGBM 6 True 0.224 1.126 0.963 1.477 0.969
Linear Linear 1 False 0.056 0.270 0.946 0.358 0.937
Linear Linear 4 False 0.935 1.533 0.410 1.868 0.262
Linear Linear 6 False 0.255 1.230 0.950 1.509 0.952
Linear Linear 1 True 0.056 0.268 0.946 0.357 0.941
Linear Linear 4 True 0.934 1.532 0.408 1.867 0.262
Linear Linear 6 True 0.253 1.221 0.949 1.501 0.946
Table 14: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.697 2.658 0.903 3.687 0.935
LGBM LGBM 4 False 1.006 3.541 0.898 4.850 0.948
LGBM LGBM 6 False 0.460 2.037 0.933 2.840 0.964
LGBM LGBM 1 True 0.250 1.050 0.921 1.463 0.935
LGBM LGBM 4 True 0.312 1.393 0.932 1.916 0.931
LGBM LGBM 6 True 0.224 0.945 0.916 1.321 0.924
Linear Linear 1 False 0.056 0.226 0.894 0.323 0.892
Linear Linear 4 False 0.935 1.286 0.314 1.637 0.176
Linear Linear 6 False 0.255 1.032 0.898 1.325 0.899
Linear Linear 1 True 0.056 0.225 0.892 0.321 0.893
Linear Linear 4 True 0.934 1.285 0.306 1.636 0.170
Linear Linear 6 True 0.253 1.025 0.897 1.319 0.890

Experimental Score

Table 15: Coverage for 95.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.529 0.791 0.353 0.989 0.127
LGBM LGBM 4 False 0.376 0.645 0.477 0.830 0.267
LGBM LGBM 6 False 0.132 0.643 0.950 0.827 0.948
LGBM LGBM 1 True 0.527 0.791 0.352 0.989 0.131
LGBM LGBM 4 True 0.376 0.645 0.473 0.830 0.271
LGBM LGBM 6 True 0.131 0.643 0.952 0.826 0.962
Linear Linear 1 False 0.064 0.250 0.888 0.333 0.824
Linear Linear 4 False 0.955 1.160 0.245 1.427 0.062
Linear Linear 6 False 0.241 1.172 0.949 1.441 0.954
Linear Linear 1 True 0.064 0.250 0.889 0.333 0.822
Linear Linear 4 True 0.956 1.160 0.244 1.429 0.063
Linear Linear 6 True 0.241 1.172 0.951 1.439 0.953
Table 16: Coverage for 90.0%-Confidence Interval over 1000 Repetitions
Learner g Learner m DGP In-sample-norm. Bias CI Length Coverage Uniform CI Length Uniform Coverage
LGBM LGBM 1 False 0.529 0.664 0.263 0.872 0.070
LGBM LGBM 4 False 0.376 0.542 0.383 0.738 0.184
LGBM LGBM 6 False 0.132 0.540 0.899 0.734 0.898
LGBM LGBM 1 True 0.527 0.664 0.261 0.872 0.076
LGBM LGBM 4 True 0.376 0.541 0.387 0.737 0.183
LGBM LGBM 6 True 0.131 0.540 0.900 0.734 0.903
Linear Linear 1 False 0.064 0.210 0.817 0.300 0.734
Linear Linear 4 False 0.955 0.973 0.182 1.255 0.035
Linear Linear 6 False 0.241 0.983 0.899 1.265 0.896
Linear Linear 1 True 0.064 0.210 0.818 0.300 0.735
Linear Linear 4 True 0.956 0.973 0.183 1.256 0.036
Linear Linear 6 True 0.241 0.983 0.898 1.265 0.897