I sparked the most minor of Twitter controversies 12 days ago.
Imagine seeing these statistics about two bands for a year (numbers 100 percent made up):
Free Energy 60 (30/50)
U2 60 (30/50)
Someone who didn't know better could be forgiven for thinking, "Wow, I didn't realize that Free Energy were such a big deal" or "...that U2 had fallen off so much." The real issue here, though, has nothing to do with fame. The real issue is that the typical size of the venue is completely different for these two bands. Pretend with me for a minute that Free Energy had a few dates at Soldier Field, and that U2 snuck into a tiny bar a few times:
Free Energy 0 (0/2) 63 (30/48)
U2 55 (24/44) 100 (6/6)
Obviously, U2 is much more capable of selling out any venue around, but forgetting to account for this can skew the meaning of the numbers. In statistics, this is called Simpson's Paradox, the always-looming possibility that there's some unknown group distinction causing real differences in the numbers.
Like every reader of college basketball writing, I love Luke Winn's weekly power rankings. It was a table in last week's discussion of Kentucky that sparked this article. It showed that the Wildcats' points per possession allowed to that point in the season were essentially identical with or without Anthony Davis on the floor:
The interesting thing about this -- to me, at least -- was how differently four different writers approached this information, since everyone had identical data and held identical opinions. I'll post the Twitter conversation, and then run through why everybody was right and everybody was wrong in their reactions.
@DrewCannon1, 3:58 PM: I love @lukewinn's power rankings, but please don't believe UK's defense is better with Davis on the bench. (1/2)
@DrewCannon1, 4:00 PM: So many more possessions, percentagewise, with Davis on the bench against Radford/Penn State/Marist/Portland than UNC/KU/etc. (2/2)
@MattNorlander, 4:01 PM: @DrewCannon1 Exactly. That's where I side in w/ the pragmatism/eyes over the #s. No argument that Eloy > Alien. Data incomplete, misleading.
@lukewinn, 4:02 PM: My god, are ppl overreacting to that Davis stat. I'm not suggesting Vargas should start. Just pting out a very surprising, season-long fact
@lukewinn, 4:04 PM: "UK is better off in long term w/o Davis in its def. lineup" might be the easiest argument to attack, ever. I'm not making that argument.
@GoodmanCBS, 4:05 PM: @lukewinn And this is why numbers can be misleading.
@DrewCannon1, 4:05 PM: @MattNorlander I think that's absolutely a numbers argument. Just didn't take a confounder into account. Simpson's Paradox and all that.
@GoodmanCBS, 4:06 PM: @lukewinn You can play without Anthony Davis. I'll play with him -- and I'll beat you by 30.
@lukewinn, 4:07 PM: Dude. RT @GoodmanCBS: @lukewinn You can play without Anthony Davis. I'll play with him -- and I'll beat you by 30.
@GoodmanCBS, 4:08 PM: @lukewinn You are right. That was out of line. I'll beat you by 40, not 30.
@lukewinn, 4:13 PM: RT @nusscoug: Goodness, people. @lukewinn simply pointed out a statistical oddity and said it raised interesting questions. Not a referendum on "stats."
From most to least statistically-inclined:
1. There's an identifiable confounding factor, and I know what it is.
So we start off with me, terrified of the potentially incorrect implications of statistics since I was eight years old after reading Bill James's Whatever Happened to the Hall of Fame? I'm concerned by the disproportionate possessions Davis played against strong competition, so I arrogantly hit Twitter, claiming I know exactly why the numbers look the way they do. If you control for level of opponent, weighting each possession by the difference between opponents' adjusted offensive rating (as measured by kenpom.com a few days ago) and the points per possession they scored with Davis on/off court, you get this:
Which, yeah, makes it look a little saner. Instead of a 0.003 PPP advantage for keeping Davis on the bench, it spits out a 0.030 PPP advantage for having him in the game. Still, given Davis's extraordinarily impressive reputation (and, to my eye, performance) as a defender, you'd expect something more than this.
So maybe there's also a lineup configuration problem, a typical issue with "with or without you" analysis like this. Maybe it's a small sample size problem, and Marist and Penn State just weren't getting any rolls when Davis happened to be on the bench. I do think, largely because of the other chart in Winn's same power rankings, that we should be watching Eloy Vargas's defense a little more closely to see if he's demonstrating surprising effectiveness on that end.
2. Relax, it's just a number.
I don't think anyone, least of all Winn, would suggest that Kentucky is no worse defensively with Davis on the bench. Winn saw a number, thought it was interesting, and forwarded it on to his readers.
Here's the issue with this approach: If a number is meaningless, then why does anyone care about it? And, if a number has meaning, then you can't end an argument with it if you don't agree with its most obvious interpretation. A number like defensive points per possession certainly has meaning, in the same way that "wins while a starting pitcher was on the mound" certainly has meaning. But if you don't interpret the differing circumstances correctly, then you can end up making questionable claims -- or, more dangerously, seem to.
To be fair, what Winn said in his most recent power rankings essentially equates to what I said above: "Hey, Eloy Vargas might be pretty good, too, everybody."
3. This is why we need to balance numbers with the eye test.
In his tweet Matt Norlander did something I've done thousands of times in my life up to this point: evaded the backlash to an unpopular statistical argument by pointing out the need for the eye test.
People sometimes act like statistics and what happens on the court are separate entities, like you have to pay special attention to an ugly 23-point game to figure out it wasn't that impressive, and not just know the scorer was 4-of-18 from the field. For me, the eye test is there to fill in the blanks left by statistical analysis. I use the eye test to figure out why a team's turnover percentage is so high, and whether we should expect that to change. I use it to figure out whether a specific player is a good off-ball defender. I use it to project whether a mid-major kid will be able to perform at the same level against high-major defenses. But a lot of times when a numerical argument seems silly, it's just because there's a confounding factor somebody forgot to include.
When there's a statistic that disobeys the eye test, the answer isn't "Well, thank God for the eye test." It's "Why do the numbers turn out that way?" Only when that line of thought hits the abyss does the eye test become necessary.
4. This is why statistics can't be trusted.
It makes sense to me to mistrust statistics at some level; I certainly do. If you look at the national points per game leaders as a list of the best offensive players in the country, it's normally (though not so much this year) a ridiculous proposition. But once you take into account opponents, pace, shooting efficiency, turnovers, rebounding...it starts to approach the eye test pretty successfully. And since the statistical record isn't prone to heuristics and biases, it's usually worth checking out the standout guys who don't fit the media consensus and figuring out why those guys stand out.
At the end of the day I think most of the backlash against stats comes from a combination of misunderstanding, complexity, and incompleteness.
Have you ever tried to calculate a player's offensive rating? I have. It's outrageous. It takes forever. It makes perfect sense, but the overall equation is literally too long to fit into one Excel cell, so it takes a while to understand why each step is being taken. If I hadn't done so, I'd never feel comfortable quoting it -- and it takes enough time that you really have to believe it's worth it.
Hey, John Gasaway! Want to know why people continue to use rebound margin? It's not because reasonable minds think it's better. The beauty of rebounding percentages is that the improvements are easy to explain and hard to refute. But, if I'm not being given the percentages, I'll rarely bother to calculate them. Same goes for possession percentage and turnover percentage.
Finally, it's entirely true that there are important parts of basketball totally unnoted by statistics. If a kid's lockdown defense, without steals, holds a defense together, the typical stats don't teach us that. If someone is a cold-blooded killer in the clutch, we usually don't have enough data to say anything confidently one way or another. If a kid has off-the-charts work ethic, traditional stats won't help us predict his improvement accurately. But that type of knowledge can be understood in the context of the unchanging, unbiased information we already have.
As long as numbers decide who wins and loses, they will matter.
As long as numbers can be misinterpreted by accident or purposefully abused to back a personal argument, they will often be meaningless.
Drew Cannon is a college student and a regular contributor to Basketball Prospectus. Follow him on Twitter at @DrewCannon1.
This free article is an example of the content available to Basketball Prospectus Premium subscribers. See our Premium page for more details and to subscribe.
Drew Cannon is an author of Basketball Prospectus.
You can contact Drew by clicking here or click here to see Drew's other articles.