Apropos of nearly nothing, Matt Yglesias, of whom I'm usually quite fond, commits my favorite fallacy in a recent post.
The gist of it is this: Texans are experiencing a backlash against their push for standardized tests as a way of improving education outcomes. They led the way in promoting standardized testing nation-wide, and Yglesias fears they may lead the backlash. Why is this worrisome? Well, because Texans have been doing a great job improving their education outcomes. Just look at their standardized test scores!
Without being too flip, this does get at something important. It should not be surprising that focusing on standardized testing will improve scores on standardized tests. Those who oppose an increased focus on standardized testing, I imagine, don't do so because they think it is self-defeating. They do so because they think other things in education are more important.
One thing that we might prefer is that students are taught to be creative, which is difficult to measure in a standardized way, but is clearly a valuable goal. Or teachers in general might just be better at educating children if they are given more freedom to teach to their own and their students' strengths, and this might be a better outcome than students doing marginally better on a single measure.
Following Socrates, I see education as making good citizens, not making good workers. But clearly it's going to be much easier to measure whether or not we're making better workers.
There are obviously lots of good reasons to want standardized measures of education quality. And perhaps these tests are the best way to improve education quality! But we certainly don't learn that by just looking at the results of these tests. We need to consider other possible measures (perhaps, unfortunately, longer-term measures, like students' future outcomes), and consider what costs we may be incurring by focusing on the standardized tests.
Unfortunately, it's a mundane epistemological fact that we can't know or measure everything. That's part of what makes these quetions so hard.