At first, I wasn’t going to say anything about Jonah Lehrer‘s piece on Grantland about the problems with statistical analysis because I’m not sure anyone cares to hear anyone involved in statistical analysis argue in favor of it. It’s hopelessly meta navel-gazing. Eventually, though, a thought occurred to me that I hope might be somewhat interesting: Lehrer is totally right. But he’s also wrong.
Lehrer focuses his argument on an analogy between statistical analysis in sports and the rating of cars, noting that countable attributes (horsepower and fuel economy) tend to be overvalued when they ultimately have little to do with the satisfaction drivers get out of their vehicles. And he’s right. There is plenty of evidence that people put more emphasis on what is counted, even–and possibly especially–in sports. I’ve written as much in the past.
Here’s where I disagree with Lehrer: Counting stuff in sports is not new. Best I can tell, it’s been done in baseball about since the first time anyone picked up a bat. The analytics movement isn’t about counting stuff; it’s about counting the right stuff. In Lehrer’s analogy, APBRmetrics (and its cousins in other sports) isn’t the horsepower and fuel economy ratings. Those are RBIs and points per game and other stats that are easy to track but harder to correlate to success. The better analogy, in my view, is that advanced statistics are analogous to the measures of consumer happiness–which are, after all, statistics themselves.
Certainly, there are elements of the horsepower problem in the analytics community. There’s a tendency in many quarters to dismiss what cannot be measured as irrelevant, or at least not as important as it really is. Witness the changing perception of the importance of defense in the sabermetrics community within the last decade as the ability to quantify the value of fielding has improved, for one. That’s a great example of the fact that statistical analysis isn’t static–it’s constantly evolving to do a better job of measuring what matters.
Lehrer’s analogy also points to the importance of making sure that the statistics we use correlate to winning. That’s the ultimate goal, not being efficient or putting up the best numbers or anything else. When done correctly, analytics should do just that as one tool of many used by good decision-makers.