Media Matters gets it wrong yet again, new critique of piece that I had in the Wall Street Journal

Media Matters is back to its old ways attacking my work.  I have previously responded to Media Matters' past claims here, here, and here.  Now Media Matters is being critical of the op-ed that I had in the Wall Street Journal last Thursday.

Media Matters attacks (full text below) the surveys by the National Association of Chiefs of Police (NACOP), Police Magazine, and San Diego Police Officers Association that were referenced in my Wall Street Journal piece.  Instead, Media Matters prefers one small survey that over-samples urban areas.

Why does Media Matters prefer the single survey presented in a American Journal of Preventative Medicine (AJPM) article to the 23 NACOP and other surveys? Let us compare the AJPM article and the NACOP surveys since they are the ones that both deal with command officers. Both were conducted through the mail, but each year, for 23 years, the NACOP survey was sent to all of the roughly 22,000 chiefs of police and sheriffs in the United States.  By contrast, the AJPM article was only done one time during 2002/2003 and they sent out a letter to just a small fraction of these police chiefs, merely 574, or fewer than 3 percent of the total.

In addition -- something Media Matters conveniently neglects to mention -- the researchers for the AJPM article purposely did not examine a representative sample of police chiefs, limiting themselves to only a subsample of those cities with more than 25,000 people.

One difference between the NACOP and the AJPM article surveys is that the latter randomizes who they sent the survey to, something that Media Matters seems to think is an advantage of the AJPM article survey.   But that just shows how little Media Matters understands about statistics.  "Randomization" is a method researchers are typically forced to use because they only have limited resources to contact a smaller number of people.  Randomization is always imperfect to some degree in that it never perfectly represents the entire population; it is merely an an attempt to get a sample that is reasonably representative of the entire population.  The NACOP did not have send out a mailing to a subset of police chiefs and thus did not have to randomize because they sent the survey to the entire set of chiefs of police and sheriffs in the US. That is of course vastly better.

The response rate for the NACOP surveys was between 12% and 14%, thus ranging from about 2,700 to 3,150 per year.  By contrast, the AJPM article had responses from 405 chiefs.  Although the smaller survey had a better response rate, it represents a very small number of the approximately 22,000 police chiefs and sheriffs across the country.  The NACOP survey had more than 7 times the number of respondents in any single year that the AJPM article had.  And it has been repeated 23 years, including recent years.

As Media Matters correctly points out, the answers from the NACOP survey are not weighted by the population characteristics.  However, Media Matters neglects to mention that the survey in the AJPM article doesn't do that either.  Media Matters attacks the Police Magazine survey for not carefully weighting for "the demographics of its survey population to be representative of the general population," but they don't hold the survey in the AJPM article to the same standard.  Indeed, it is clear from the AJPM article that they heavily oversample urban areas (62.4%) in comparison to suburbs (36.3%) or rural areas (1.2%). 

This huge difference in the number of responses collected can easily explain the differences in the results.  The 2010 NACOP survey shows that 77% of chiefs of police and sheriffs supports very liberal concealed carry laws.  Twenty-three percent of respondents who either oppose or are neutral on concealed carry implies a number that is larger than the entire number surveyed for the AJPM article, let alone the 236 who say that civilians should be "prohibited from carrying a firearm in public places."  

There are also problems with the questions used by the AJPM article.  Two of the four the questions from the AJPM article that are referenced by Media Matters are extremely vague and are likely to elicit agreement simply because of their vagueness. For instance, agreeing that "Civilians [should be] prohibited from carrying a firearm in public places," cannot reasonably be interpreted as opposing concealed carry by law-abiding citizens who pass a criminal background check.  If an urban police chief could think of one place that they would oppose people openly carrying guns in public places, they would agree with this statement.

The "child access laws" question pointed to by Media Matters is equally vague.  The question could mean anything from a minimum age requirement for purchase of a gun to whether a child can only use a gun in the presence of an adult to prohibiting those under 18 access to guns under any circumstances. 

As to Media Matters attacks on the NACOP as an organization, they rehash accusations from two decades ago. But in addition to the the National Association of Chiefs of Police (NACOP), Police Magazine, and San Diego Police Officers Association surveys, there are many other surveys of police that come to similar results (see pages 14 and 15 in my book More Guns, Less Crime (University of Chicago Press, 2010, third edition)).  Another is available here: Stephen L. Christopoulos "Survey of Police Officers in Lehigh and Northampton Counties Pennsylvania."

Media Matters attacks the NACOP for not making information of their survey available, but Barry Shephard, the NACOP's Executive Director, wrote me: "We do not have record of any inquiry or request for information at any point in time from Media Matters."

Click on pictures below to make text larger.  Because Media Matters has rewritten their comments in the past, I am providing a screen shot here of what they wrote.

Other notes

Links to police surveys available here.

The 574 of the police chiefs contacted by the AJPM survey were out of the 1,290 cities with more than 25,000 people.