Why does Media Matters prefer the single survey presented in a American Journal of Preventative Medicine (AJPM) article to the 23 NACOP and other surveys? Let us compare the AJPM article and the NACOP surveys since they are the ones that both deal with command officers. Both were conducted through the mail, but each year, for 23 years, the NACOP survey was sent to all of the roughly 22,000 chiefs of police and sheriffs in the United States. By contrast, the AJPM article was only done one time during 2002/2003 and they sent out a letter to just a small fraction of these police chiefs, merely 574, or fewer than 3 percent of the total.
In addition -- something Media Matters conveniently neglects to mention -- the researchers for the AJPM article purposely did not examine a representative sample of police chiefs, limiting themselves to only a subsample of those cities with more than 25,000 people.
One difference between the NACOP and the AJPM article surveys is that the latter randomizes who they sent the survey to, something that Media Matters seems to think is an advantage of the AJPM article survey. But that just shows how little Media Matters understands about statistics. "Randomization" is a method researchers are typically forced to use because they only have limited resources to contact a smaller number of people. Randomization is always imperfect to some degree in that it never perfectly represents the entire population; it is merely an an attempt to get a sample that is reasonably representative of the entire population. The NACOP did not have send out a mailing to a subset of police chiefs and thus did not have to randomize because they sent the survey to the entire set of chiefs of police and sheriffs in the US. That is of course vastly better.
The response rate for the NACOP surveys was between 12% and 14%, thus ranging from about 2,700 to 3,150 per year. By contrast, the AJPM article had responses from 405 chiefs. Although the smaller survey had a better response rate, it represents a very small number of the approximately 22,000 police chiefs and sheriffs across the country. The NACOP survey had more than 7 times the number of respondents in any single year that the AJPM article had. And it has been repeated 23 years, including recent years.
As Media Matters correctly points out, the answers from the NACOP survey are not weighted by the population characteristics. However, Media Matters neglects to mention that the survey in the AJPM article doesn't do that either. Media Matters attacks the Police Magazine survey for not carefully weighting for "the demographics of its survey population to be representative of the general population," but they don't hold the survey in the AJPM article to the same standard. Indeed, it is clear from the AJPM article that they heavily oversample urban areas (62.4%) in comparison to suburbs (36.3%) or rural areas (1.2%).
This huge difference in the number of responses collected can easily explain the differences in the results. The 2010 NACOP survey shows that 77% of chiefs of police and sheriffs supports very liberal concealed carry laws. Twenty-three percent of respondents who either oppose or are neutral on concealed carry implies a number that is larger than the entire number surveyed for the AJPM article, let alone the 236 who say that civilians should be "prohibited from carrying a firearm in public places."
There are also problems with the questions used by the AJPM article. Two of the four the questions from the AJPM article that are referenced by Media Matters are extremely vague and are likely to elicit agreement simply because of their vagueness. For instance, agreeing that "Civilians [should be] prohibited from carrying a firearm in public places," cannot reasonably be interpreted as opposing concealed carry by law-abiding citizens who pass a criminal background check. If an urban police chief could think of one place that they would oppose people openly carrying guns in public places, they would agree with this statement.
The "child access laws" question pointed to by Media Matters is equally vague. The question could mean anything from a minimum age requirement for purchase of a gun to whether a child can only use a gun in the presence of an adult to prohibiting those under 18 access to guns under any circumstances.
Media Matters attacks the NACOP for not making information of their survey available, but Barry Shephard, the NACOP's Executive Director, wrote me: "We do not have record of any inquiry or request for information at any point in time from Media Matters."