Friday, January 25, 2013

How Science Can Lie

Scientists want to discover interesting things. So they decide on something they're interested in, figure out a way to measure that thing, then run through dozens of trials with different subjects and variables and control groups. Then they publish their results and we learn new things, all the better for the world!

That's the hope, anyway. One trouble that arises is that lots of interesting things are difficult to measure. So we might pick a stand-in for that thing, and measure that, and hope the correlation between the stand-in and actually interesting thing is close enough. Any good scientist knows this, of course, and any good scientist will explain how their measurements are conducted and what possible problems might arise from this method of measurement.

But not every scientist is a good scientist. And we don't always hear about scientific studies from the scientists themselves, but a bizarre creature known as a "science reporter." And it's quite easy to take the simplest readings of the results of a study and draw superficial conclusions, and once these conclusions are taken as scientific fact by the media or public, that toothpaste is never going back in the tube.

Let's take an example. Consider happiness research. Happiness is really important, so naturally lots of people want to research it. But I'm not really sure what happiness is; and I'm sure that there aren't great ways to measure it.  There are several bad ways to measure it, one of which is probably most common, which is asking people how happy they are.

Now the problem becomes clear. What you're measuring is how people report their emotional state. And one thing that might correlate to is what their actual happiness level is like, but it also might correlate with what they want you to think about them. Or whether or not they ate breakfast that morning. Or their relative emotional state compared to those around them, or compared to people on TV, or compared to how happy they think they should be.

And if we're studying whether or not higher income makes you happy, it's important to be able to tell the difference between it actually making your happy, or making you more likely to want other people to think that you're happy. Those are not the same things at all.

But like I said, people want to discover interesting things. It's a mundane epistemological point that there are lots of things that are very difficult to know, and some of these things would be interesting to know. Sometimes, when people want to know things that are very difficult to know, they'll just take their best guess at how to measure it. And then they convince themselves that because there's science behind it, they must be right.

All of which is just to say that conclusions are different from results. We must be careful not to report conclusions of studies as if they are the results. The best way to reach solid conclusions in science (these problems may be more common in social sciences, but they apply to all science) is to try to find many different ways to measure what it is you're trying to study, and see if the different measurements react similarly to different variables. Which means you might have to run a lot of different studies, and be very cautious in publishing conclusions. You also have to have the self-critical awareness to ask, "What, besides what I'm trying to study, might I actually be measuring?"

Again, none of this is news to any decent scientist. But critical thought is important to me, and to the world, and I see a lot of bad reporting about science. It's worth the time to remind ourselves of these things, because it's too easy to accept claims at face-value.

No comments:

Post a Comment