Some notes on John Roman's "Race, Justifiable Homicide, and Stand Your Ground Laws: Analysis of FBI Supplementary Homicide Report Data," Urban Institute

A copy of Roman's report is available here.  Apparently, the report is going to be used in testimony tomorrow before the Senate.

1) If you were to do this study right, you would look at the change in justifiable homicides in each state before and after the Stand Your Ground law went into effect.  You want to see whether the rate of justifiable homicides in a state changed after that state changed its law.  This paper doesn't do that.  Instead, it looks across 10 regions, but some of the states in those regions are changing their laws and others aren't.  Roman uses what are called year fixed effects, but he isn't using proper geographic fixed effects and that raises a red flag.

2) Roman doesn't know the proper statistical test.  Take his Table 3.

The issue isn't whether the coefficients for "white on black" are different from zero.  The question is whether the rate in non-stand your ground states is different from stand your ground states.  Thus, while 41.14 and 44.71 are statistically different from zero, they are extremely unlikely to be statistically different from each other.  Again, the real comparison should be did the rate of justifiable homicides go up in a category in a state after that state changed its law.  Making comparisons across states in a region could simply be picking up differences in those states.

Indeed, it is very likely that the coefficients for black on black (10.24 and 9.94) as well as black on white (7.69 and 11.10) are not different between non-stand your ground states and stand your ground states.

Take Roman's discussion of this table: "Racial disparities are much larger, as white-on-black homicides have justifiable findings 33 percentage points more often than black-on-white homicides.  Stand Your Ground laws appear to exacerbate those differences, as cases overall are significantly more likely to be ruled justified in SYG states than in non-SYG states (p = 0.02)." While it is true that his results show that whites defend themselves against blacks more frequently than the reverse is true, Roman ignores the fact that Stand Your Ground laws actually reduce this imbalance.  And the presumed question for this paper was presumably how Stand Your Ground laws altered this imbalance. 

In non-Stand Your Ground states, White on black versus Black on White coefficient ratio = 5.34
In Stand Your Ground states,  White on black versus Black on White coefficient ratio = 4.03 

So, if you believe Roman's results, Stand Your Ground laws actually improved the ability of Blacks to defend themselves against Whites relative to the ability of Whites to defend themselves against Blacks by causing the ration to fall by about 25 percent. 

3) Roman should be properly acknowledged for noting that unlike the data from the Tampa Bay Tribune, "The data here cannot completely address this problem because the setting of the incident cannot be observed" (p. 11).

4) The data in this paper uses justifiable homicides and Stand Your Ground cases are likely to only be a subset of those cases.  Ignoring the prior concerns, the question then becomes are you picking changes in justifiable homicides or changes due to Stand Your Ground cases.  To further make that point clear, there are huge changes that occur in terms of how the justifiable homicide data are reported across different states over time.  While often about 35 states report this data, a large percentage of the jurisdictions in even those states don’t report the data. What states and what jurisdictions within those statesreport this data changes dramatically over time. The implication is any changes over time might simply arise from changes in the states or portions of states that are reporting this data. 

The biggest problem involves how this data is collected. Police initially report the cases as criminal homicides. If it’s later determined to be justifiable, they don’t frequently don’t go back and recode the data. The problem is greatest for those deaths where the greatest amount of time elapses between the death and it is determined to be justifiable. There is also some evidence that recoding is less likely to occur in the larger urban areas where you are likely to have a greater percentage of crime involving blacks. If so, correcting this bias would produce even larger stronger evidence that blacks are more likely to engage in justifiable homicides. John Barnes, “Justified to kill: Why there are moreself-defense killings in Michigan than anyone knows,” MLive, June 12, 2012.  

Since many of the data isn't even collected on many justifiable homicides, many of the justifiable homicides in this sample are likely to be falsely labeled as non-justifiable homicides.  On top of that, the error is greatest in urban areas, areas where more blacks are likely to have used guns defensively. 

The appendix for the paper doesn't appear to understand that many of the states don't collect data on justifiable homicides in every year and that even when they do report some data, not all the jurisdictions within the state collect the data.

5) I don't understand why a regression similar to that reported in Table 4 isn't run on the data in Table 3.  Singling out when there is a single victim and single shooter, they are both male, they are strangers, and a firearm is used, but the SHR data used by Roman goes beyond that in ages and region.