Donohue and Levitt on Abortion, How can they miss having used fixed effects?
Quote from Foote and Goetz's paper, p. 8.
Sorry for making such a long post here, but I wanted to give non-statisticians at least a rough idea of what was required for john Donohue and Steve Levitt to miss seeing whether they had included fixed effects in their regressions on abortion. The two thoughtful authors from the Boston Fed mentioned the "programming oversight" done by Donohue and Levitt in their abortion research. I still believe that saying that is "being much too nice," but I will leave it to readers to decide what level of sloppiness is involved in this research. Here is an abortion regression with fixed effects (though I have three types of fixed effects rather than the two used by the Boston Fed people). It is hard for me to see how this could have been missed. If one looks at these regressions, it is just hard to see how someone could miss having included them. The state-year interactions discussed by Foote and Goetz would add up to ((51*number of years)-1) to the list of control variables. I only wish that Donohue and Levitt had provided me with their regressions and all their data when I first asked for this six years ago. Their refusal to provide this information is at best extremely unfortunate. The obvious thing is that they should be ashamed for not providing this information in a timely manner when it was asked for.
Here are two simple regressions. Obviously more control variables can be included, but I decided to just show the simplest case from my paper with John Whitley. Notice how in this case fixed effects change the coefficient sign on the abortionpop variable. Including an absorb statement can eliminate one set of these fixed effects from being reported and thus make this point less obvious, but the general point is still the same.
. xi:xtpois murders1 abortionpop Unktrnd- year_23, i(FIPSSTAT) irr pa robust
note: year_2 dropped due to collinearity
note: year_3 dropped due to collinearity
note: year_4 dropped due to collinearity
note: year_22 dropped due to collinearity
Iteration 1: tolerance = .09532036
Iteration 2: tolerance = .10020875
Iteration 3: tolerance = .03855165
Iteration 4: tolerance = .008876
Iteration 5: tolerance = .00160909
Iteration 6: tolerance = .00028531
Iteration 7: tolerance = .00005063
Iteration 8: tolerance = 8.966e-06
Iteration 9: tolerance = 1.588e-06
Iteration 10: tolerance = 2.813e-07
GEE population-averaged model Number of obs = 21756
Group variable: FIPSSTAT Number of groups = 51
Link: log Obs per group: min = 299
Family: Poisson avg = 426.6
Correlation: exchangeable max = 437
Wald chi2(40) = 196144.51
Scale parameter: 1 Prob > chi2 = 0.0000
(standard errors adjusted for clustering on FIPSSTAT)
------------------------------------------------------------------------------
Semi-robust
murders1 IRR Std. Err. z P>z [95% Conf. Interval]
-------------+----------------------------------------------------------------
abortionpop 1.405027 .2128516 2.24 0.025 1.044079 1.890756
Unktrnd 1.024132 .0046993 5.20 0.000 1.014963 1.033384
aFIPS_2 .0989696 .0001332 -1718.30 0.000 .0987088 .099231
aFIPS_4 .5898942 .0001919 -1622.84 0.000 .5895183 .5902703
aFIPS_5 .5880759 .0002303 -1355.74 0.000 .5876248 .5885275
aFIPS_6 5.794082 .0147365 690.75 0.000 5.765271 5.823037
aFIPS_8 .3988566 .000475 -771.84 0.000 .3979267 .3997886
aFIPS_9 .2799524 .0003301 -1079.79 0.000 .2793062 .2806001
aFIPS_10 .0661884 .0000789 -2277.49 0.000 .0660339 .0663433
aFIPS_11 .1034435 .0557807 -4.21 0.000 .0359503 .2976485
aFIPS_12 2.402414 .0112149 187.76 0.000 2.380533 2.424496
aFIPS_13 1.407365 .0012569 382.62 0.000 1.404904 1.409831
aFIPS_15 .1101832 .0002945 -825.28 0.000 .1096075 .1107619
aFIPS_16 .0940332 .0000456 -4871.68 0.000 .0939438 .0941227
aFIPS_17 1.614205 .0023658 326.72 0.000 1.609575 1.618849
aFIPS_18 .5732324 .0001949 -1636.57 0.000 .5728505 .5736146
aFIPS_19 .1128952 .0002638 -933.54 0.000 .1123794 .1134134
aFIPS_20 .2369441 .0024803 -137.56 0.000 .2321324 .2418555
aFIPS_21 .6326708 .0011611 -249.45 0.000 .6303992 .6349507
aFIPS_22 1.016173 .0002205 73.94 0.000 1.015741 1.016605
aFIPS_23 .0579125 .0002006 -822.27 0.000 .0575206 .0583071
aFIPS_24 .7755058 .0015016 -131.30 0.000 .7725683 .7784546
aFIPS_25 .3166921 .0007269 -500.97 0.000 .3152706 .3181199
aFIPS_26 1.645792 .0018611 440.58 0.000 1.642149 1.649444
aFIPS_27 .241033 .0002058 -1666.02 0.000 .2406298 .2414367
aFIPS_28 .5054685 .0002175 -1585.63 0.000 .5050424 .505895
aFIPS_29 .7520471 .000341 -628.41 0.000 .751379 .7527158
aFIPS_30 .0546092 .0002382 -666.66 0.000 .0541444 .0550781
aFIPS_31 .1011488 .0000618 -3747.60 0.000 .1010277 .1012701
aFIPS_32 .2792081 .0000737 -4831.47 0.000 .2790636 .2793526
aFIPS_33 .0504129 .0001025 -1469.58 0.000 .0502124 .0506142
aFIPS_34 .7724087 .0009178 -217.32 0.000 .7706118 .7742097
aFIPS_35 .2183319 .0004742 -700.57 0.000 .2174044 .2192634
aFIPS_36 2.17522 .0369114 45.80 0.000 2.104065 2.248782
aFIPS_37 1.589501 .0011554 637.51 0.000 1.587238 1.591767
aFIPS_38 .0182674 .0000229 -3195.96 0.000 .0182226 .0183123
aFIPS_39 1.225201 .0010425 238.71 0.000 1.223159 1.227246
aFIPS_40 .6485567 .0002798 -1003.57 0.000 .6480084 .6491054
aFIPS_41 .3239948 .0007478 -488.27 0.000 .3225324 .3254639
aFIPS_42 1.387713 .0019498 233.20 0.000 1.383897 1.39154
aFIPS_44 .0755543 .0000414 -4713.95 0.000 .0754732 .0756354
aFIPS_45 1.081743 .0003085 275.49 0.000 1.081138 1.082348
aFIPS_46 .02442 .0000246 -3684.40 0.000 .0243718 .0244683
aFIPS_47 .991302 .0002734 -31.67 0.000 .9907662 .991838
aFIPS_48 4.273158 .0012148 5108.87 0.000 4.270777 4.275539
aFIPS_49 .1169221 .0000596 -4211.61 0.000 .1168053 .1170389
aFIPS_50 .0294125 .0000315 -3295.45 0.000 .0293509 .0294743
aFIPS_51 1.147393 .0006312 249.93 0.000 1.146157 1.148631
aFIPS_53 .4719951 .0010989 -322.49 0.000 .4698463 .4741538
aFIPS_54 .3279638 .0001477 -2475.52 0.000 .3276744 .3282534
aFIPS_55 .3507432 .0007353 -499.74 0.000 .349305 .3521874
aFIPS_56 .0536214 .000039 -4025.53 0.000 .053545 .0536978
age1_11 1.966192 .3025431 4.39 0.000 1.454285 2.658291
age1_12 4.734692 .5743331 12.82 0.000 3.732831 6.005445
age1_13 13.53745 1.632393 21.61 0.000 10.68799 17.14659
age1_14 38.02512 5.282721 26.19 0.000 28.96113 49.92586
age1_15 87.06121 12.9882 29.94 0.000 64.98878 116.6302
age1_16 152.4797 23.9504 32.00 0.000 112.0757 207.4496
age1_17 220.7705 32.41546 36.76 0.000 165.5618 294.3892
age1_18 264.9041 39.6383 37.29 0.000 197.5701 355.1863
age1_19 274.0828 40.61976 37.88 0.000 204.9894 366.4645
age1_20 284.7132 47.03216 34.21 0.000 205.9664 393.5671
age1_21 234.3185 32.71881 39.08 0.000 178.2174 308.0798
age1_22 226.4202 32.14569 38.19 0.000 171.4222 299.0634
age1_23 214.5435 30.03564 38.35 0.000 163.0606 282.2811
age1_24 199.3217 28.51133 37.02 0.000 150.5903 263.8225
age1_25 212.4668 33.23983 34.25 0.000 156.3591 288.7082
age1_26 173.2493 24.38667 36.62 0.000 131.4788 228.2901
age1_27 166.417 23.04193 36.94 0.000 126.8648 218.3003
age1_28 159.6084 22.47962 36.02 0.000 121.1074 210.3493
age1_29 147.8315 19.87613 37.16 0.000 113.5853 192.4031
age1_30 161.0707 24.55669 33.33 0.000 119.4657 217.165
age1_31 1759.683 250.7702 52.44 0.000 1330.855 2326.688
age1_99 4.23e-18 3.85e-17 -4.39 0.000 7.45e-26 2.40e-10
year_5 1.532679 .118351 5.53 0.000 1.317416 1.783116
year_6 1.334756 .0968619 3.98 0.000 1.157794 1.538767
year_7 1.290539 .0928728 3.54 0.000 1.120766 1.486029
year_8 1.226638 .1079416 2.32 0.020 1.032316 1.45754
year_9 1.126 .0994771 1.34 0.179 .9469754 1.33887
year_10 1.120204 .0910245 1.40 0.162 .9552807 1.313601
year_11 1.223229 .0872979 2.82 0.005 1.063556 1.406874
year_12 1.135106 .0966264 1.49 0.137 .9606778 1.341206
year_13 1.234025 .0961375 2.70 0.007 1.05928 1.437598
year_14 1.271295 .0877407 3.48 0.000 1.11045 1.455437
year_15 1.378969 .1031067 4.30 0.000 1.190994 1.596613
year_16 1.483771 .1128602 5.19 0.000 1.278268 1.722313
year_17 1.410172 .0995151 4.87 0.000 1.228013 1.61935
year_18 1.433584 .0916863 5.63 0.000 1.264689 1.625035
year_19 1.367123 .0674563 6.34 0.000 1.241103 1.505939
year_20 1.20845 .0494248 4.63 0.000 1.11536 1.309309
year_21 1.048189 .0542946 0.91 0.364 .9469969 1.160194
year_23 .9175812 .0253189 -3.12 0.002 .869275 .9685718
------------------------------------------------------------------------------
Here is what it would look like without fixed effects:
. poisson murders1 abortionpop Unktrnd, irr robust
Iteration 0: log likelihood = -1248609.3
Iteration 1: log likelihood = -658235.69
Iteration 2: log likelihood = -544299.3
Iteration 3: log likelihood = -531248.22
Iteration 4: log likelihood = -531238.71
Iteration 5: log likelihood = -531238.71
Poisson regression Number of obs = 21756
Wald chi2(2) = 1419.07
Prob > chi2 = 0.0000
Log likelihood = -531238.71 Pseudo R2 = 0.2365
------------------------------------------------------------------------------
Robust
murders1 IRR Std. Err. z P>z [95% Conf. Interval]
-------------+----------------------------------------------------------------
abortionpop .3488533 .0530301 -6.93 0.000 .2589701 .4699332
Unktrnd 1.001123 .0000327 34.38 0.000 1.001059 1.001187
------------------------------------------------------------------------------
[Update: An email from Carl Moody indicates that Donohue is giving out a regression file that accounts for fixed effects in at least one of the sets of estimates. The way that this is done indicates that they do not surpress the print out of the estimates and thus one would see whether fixed effects had been estimated. In any case, while I am very glad to see this now being made available, it would have been useful 4, 5, or 6 years ago when I was asking for the information.]
Update 2: It should also be made clear that despite whatever sloppiness occurred Donohue and Levitt have acknowledged the coding error and agree that correcting the error and using arrest rates suggests no effect of legalizing abortion on crime.
posted by John Lott at 12:59 PM
0 Comments:
Post a Comment
<< Home