Fundamentally, there is a right way to gather statistical data, and there’s a wrong way. A few months ago, a survey that purported to question the consensus surrounding anthropogenic global heating illustrated, in egregious fashion, the wrong way. The “survey” asked poorly phrased and biased questions designed to be spinnable to DemandDebate.com’s propaganda goals. The method of distributing the survey bypassed all non-U.S. IPCC scientists and yet claimed to applicable to all IPCC scientists. And the survey population was self-selected, rendering the results statistically meaningless for the larger population. In addition, DemandDebate.com’s executive director, Steven J. Milloy failed to keep the data of his few respondents confidential as also required by the American Association of Public Opinion Researchers guide to professional behavior.
While hardly as bad as the propaganda produced by Steve Milloy and DemandDebate.com, a survey commissioned by Colorado’s Donnell-Kay Foundation is another example of a survey that was used more broadly than can be justified given its methodology. And in the process, the results were unnecessarily spun by the Donnell-Kay Foundation in order to support greater state spending on public education.
A few weeks ago, I came across Colorado Confidential’s interview of Matt Samelson of the Donnell-Kay Foundation (DKF), a private family foundation that operates in Colorado to support public education through direct grants, lobbying, and research. In 2004, the DKF launched a project intended to document the physical status of all of the public school facilities in the state of Colorado called Crumbling Classrooms. As part of the project, the DKF sent their Capital Requirements Survey to all of Colorado’s 178 school districts. The key results of the survey, alluded to by Colorado Confidential and as reported at Crumbling Classrooms’ “Problem” page are as follows:
- A quarter of school buildings are functionally inadequate.
- Nearly a third of elementary schools and one out of every five middle and high schools are too small..
- One-third of high schools have inadequate science facilities, and one-third are technologically inadequate.
According to the Crumbling Classrooms site, these results were applied to the entire state from a sample of less than half of all Colorado school districts. The problem is that the detailed methodology of the survey doesn’t appear to justify these broad conclusions.
According to the Capital Requirements Survey, Appendix A, “Detailed Methodology”, 72 districts of 178 districts statewide responded to one or both of the two surveys that were sent out. The responding districts represented 59% of all students in the state, and the results were weighted regionally in order to better apply the results across the entire state. For example, since 53% of all public school students are in the “Metro” region, the results of 19% of superintendent surveys from that region would be scaled up to 53%, the 3% of responses from the Pikes Peak region would be scaled up to match the 18% of all students who attend schools in that region, etc.
In other words, the survey’s designers, the National Research Center, tried to limit the errors and make the survey’s results are meaningful as possible.
The problem is that the survey’s results cannot be applied outside the actual respondents, even if the results are weighted correctly. As with the DemandDebate.com survey mentioned above, this survey suffers from self-selection bias that makes its results meaningless outside the districts that responded to the survey. Statistical surveys require random samples in order to accurately estimate an entire population from a small sample, and self-select by respondents destroys the randomness of the sample – certain school facilities managers and superintendents responded to the survey, and we cannot know whether their response made them more likely to claim problems with their facilities or less. Given the fundamentally non-random nature of the survey’s data, no amount of weighting of data will make these results applicable beyond the respondent districts themselves.
In this case, the only conclusions we can draw are that a quarter of the school facilities sampled were “functionally inadequate,” one third of the high schools sampled had inadequate science facilities and are “technological inadequate,” and nearly a third of sampled elementary schools and 20% of sampled middle and high schools were too small.
In many respects, this lessens the impact of the DKF’s survey, but not in all respects. Remember that the responding districts still represent 59% of all public school students in the state of Colorado, and that those school districts had 956 facilities out of an estimated 1600 school facilities statewide. 25% of 956 facilities is still 239 school facilities that are “functionally inadequate,” far too many for a state that supposedly values education very highly (and yet elects anti-intellectuals like Bruce Benson to be President of the University of Colorado).
Partly as a result of the Donnell-Kay Foundation’s ongoing efforts in this arena, the Colorado state legislature and the governor are now looking more closely at this issue, and with greater statistical rigor, than the issue had been looked at in at least the previous eight years. So as a catalyst for further study, the survey has served a valuable purpose even as it has been eclipsed by more recent data. It’s unfortunate that the Donnell-Kay Foundation, via the Crumbling Classrooms website, felt the need to mis-represent the survey as something that it isn’t – a statistically valid representation of all the public school districts in Colorado. The data was powerful enough to not need any spin to make it effective, spin that will only make the Donnell-Kay Foundation less effective in its public education lobbying efforts over the long run.
Thanks to Colorado Confidential for putting me onto this story.