Why does Media Matters prefer the single survey presented in a American Journal of Preventative Medicine (AJPM) article to the 23 NACOP and other surveys? Let us compare the AJPM article and the NACOP surveys since they are the ones that both deal with command officers. Both were conducted through the mail, but each year, for 23 years, the NACOP survey was sent to all of the roughly 22,000 chiefs of police and sheriffs in the United States. By contrast, the AJPM article was only done one time during 2002/2003 and they sent out a letter to just a small fraction of these police chiefs, merely 574, or fewer than 3 percent of the total.
In addition -- something Media Matters conveniently neglects to mention -- the researchers for the AJPM article purposely did not examine a representative sample of police chiefs, limiting themselves to only a subsample of those cities with more than 25,000 people.
One difference between the NACOP and the AJPM article surveys is that the latter randomizes who they sent the survey to, something that Media Matters seems to think is an advantage of the AJPM article survey. But that just shows how little Media Matters understands about statistics. "Randomization" is a method researchers are typically forced to use because they only have limited resources to contact a smaller number of people. Randomization is always imperfect to some degree in that it never perfectly represents the entire population; it is merely an an attempt to get a sample that is reasonably representative of the entire population. The NACOP did not have send out a mailing to a subset of police chiefs and thus did not have to randomize because they sent the survey to the entire set of chiefs of police and sheriffs in the US. That is of course vastly better.
The response rate for the NACOP surveys was between 12% and 14%, thus ranging from about 2,700 to 3,150 per year. By contrast, the AJPM article had responses from 405 chiefs. Although the smaller survey had a better response rate, it represents a very small number of the approximately 22,000 police chiefs and sheriffs across the country. The NACOP survey had more than 7 times the number of respondents in any single year that the AJPM article had. And it has been repeated 23 years, including recent years.