crimes and misdestimation

Story on an NYT blog today about a woman in the Netherlands who may be serving a life sentence because of an error in statistical reasoning and an example of sampling on the dependent variable.

Although the physicist explaining the error makes an error of his own:

[S]uppose that police pick up a suspect and match his or her DNA to evidence collected at a crime scene. Suppose that the likelihood of a match, purely by chance, is only 1 in 10,000. Is this also the chance that they are innocent? It’s easy to make this leap, but you shouldn’t.

Here’s why. Suppose the city in which the person lives has 500,000 adult inhabitants. Given the 1 in 10,000 likelihood of a random DNA match, you’d expect that about 50 people in the city would have DNA that also matches the sample. So the suspect is only 1 of 50 people who could have been at the crime scene. Based on the DNA evidence only, the person is almost certainly innocent, not certainly guilty.

The error is that if the police had picked the person up as a suspect completely at random and found that their DNA had a 1 in 10,000 match to that found at the scene of the crime, then, yes, the person is most likely innocent. But, police tend to pick up suspects for nonrandom reasons, and the more the nonrandom reason is related to the actual probability that the person is the culprit, the less relevant the 1 in 50 calculation is and the more relevant the 1 in 10,000 probability is. Because there isn't a neat way of synthesizing this into a new probability estimate, people jump from one bad way of reasoning about the problem to another bad way of reasoning about the problem.

what predicts how much coverage a tragedy gets?

So, the New York Times is currently running stories on its front page about 32 students and faculty at Virginia Tech being murdered and 160 citizens of Iraq being murdered. Enormous tragedies, all, obviously. Yet, the VT shootings will presumably get at least 10x--and perhaps more like 100x--the total column inches of coverage in the NYT than these Iraq bombings will get (indeed, a telling part of the coverage of Iraq bombings is how commonly and easily they are lumped together).

I'm not making any judgment on the wrongness or rightness of this as a journalistic practice, but: I've always thought it would be interesting to do a study of the relationship between the number of deaths in a tragic event and the number of column inches a story gets, and then what are the other factors that lead events to get more or less coverage.

Even just for the deaths of individual soldiers, I think it would be an interesting graph to see how (I presume) coverage of individual soldier deaths has declined as the war has gone on, despite the rate of soldier deaths being, if anything, remarkable for their relative consistency over time.

I'm presuming there is research on this and it's more that I'm not aware of it. If anyone is and has pointers to finding it, let me know.

freaks and geeks

I uploaded a recent article in the New Republic on Freakonomics and economists fretting about the rise of cute little studies in their discipline. My stance, which I did not do an especially good job of articulating in this panel of Freaknomics at the Eastern Sociological Society meetings, is that sociology should, at least from a strategic standpoint, be on the side of the freakonomists in their battle against the structural-modellers in economics, because the former stakes out a position that is far more congenial to prevailing tendency of sociologists to be stronger with understanding data and process than they are at math.

Update: Link fixed.

nothing up my sleeve (p < .05)

My ability to figure out magic tricks increased greatly the day I realized that magic tricks are optimal in an important sense. A trick is basically as impressive as it can be without giving away its secret. When the magician saws the woman in half, the parts you can see are all you can see. If the trick would be even more impressive if the woman wiggled her toes but she doesn't wiggle her toes, there's a reason there's no toe-wiggling.

It's distressing the extent to which I have been able to export this epiphany to the evaluation of quantitative social science.

I was recently talking to a colleague who does not do quantitative research about a paper we had both, in very different contexts, read. The paper addressed its substantive question using Approach A. It could have used Approach B instead. Approach A is good enough for the standards of where the paper was published, but Approach B would be the approach preferred by more quantitatively-discerning types. The paper acknowledged the existence of Approach B but made substantive and statistical arguments for why Approach A was superior. In talking to my colleague, I explained that these arguments were really not very good arguments, and that, indeed, people who understand the technical issues are not going to very persuaded by results from Approach A because you really need results from Approach B to be able to assert the conclusions of this paper with any real confidence.

The thing, though, was that I went on to take for granted that analysis using Approach B wouldn't yield statistically significant (i.e., publishable) results. My colleague asked how I could be so certain of this, since no results from Approach B were reported in the paper.

I replied that, if Approach B would have yielded the same results as Approach A, the author would have announced this fact to assuage the concerns of people like me. Especially because the author clearly understands how to do Approach B and it would have only taken, say, five minutes to check. So when I saw that the paper contained these weak arguments for the superiority of Approach A over Approach B, and made no mention of what the results would have looked like using Approach B, I read this as basically equivalent to the paper containing a giant invisible footnote that said "We tried Approach B and it doesn't work."

I hate this.

institutional review boards have no jurisdiction over the dead

Three people in the past month or so, independent of one another and with various degrees of committment to the idea, have mentioned to me the possibility of doing an ethnographic study of funeral homes. If you, too, are working on this topic, I guess I would urge you not to dally.

Is this social science having a late-blooming Six Feet Under effect? Have funeral homes always been an attractive topic for ethnographers but something about the topic has prevented their from being (to my knowledge) The Great American Funeral Home Ethnography?

As a different matter regarding ethnography, I was having a conversation with an acquaintance recently about a prominent sociology ethnography in which the author, with the consent of the research participants (members of a minority group living in poverty), used their real names. The acquaintance was of the position that this was definitely wrong and asserted that their view was the consensus among people who do ethnographic research. I have to admit I don't really understand this as a general position. I do understand it in the obvious, but special, case in which naming an informant would allow one to determine the identities other people who don't want their names used. Otherwise, it seems like newspaper editors have the right idea in fretting about negative consequences of anonymous sourcing; namely, that there is basically no accountability for the writer to represent the source accurately rather than tweaking statements in ways that suit the author's argument. I recognize that people who do interview-based studies get very cross when someone says "I think you're just choosing quotes that fit your argument" or, worse, "How do we know you aren't just making this up?" But, irritation is not quite the same thing as counterargument. I can understand the idea that confidentiality is unfortunately what must be offered to get interviewees to provide honest participation, but the idea of swaddling it in ethicky goodness even for participants who express no reluctance about speaking on the record--this I don't buy.