Last week, a Henry Abbott post on his True Hoop blog entitled "Stat Geeks and Coaches" offered a bold conclusion.
"What a miserable article to read if you're a coach," Abbott wrote. "All those late nights of film study. All that competition for your job. All those tricks learned at conferences. All those books by the masters you have internalized.
"And now there is evidence to support the notion you could be replaced by a deck chair."
Inevitably, even if Abbott was speaking with tongue planted firmly in cheek, the comment drew some attention. The article Abbott referenced, written for Slate.com by Ryan McCarthy, drew on a forthcoming study by economics professor Dave Berri. As part of the follow-up to Wages of Wins, the influential 2006 book Berri co-wrote with fellow economists Stacey Brook and Martin Schmidt, Berri will take a look at the influence of coaching in the NBA in Stumbling on Wins, to be released in 2009.
In the study, done with co-authors Michael Leeds, Eva Marikova Leeds and Michael Mondello, Berri did not find that coaches could be replaced by deck chairs. The results, to the extent that they have been shared from a study that is not yet finalized, did nonetheless apparently manage to call into question the impact of coaches. I'll let Berri explain from a clarifying post last week on the Wages of Wins blog.
"Certainly I suspect the coaches in the NBA know much more than the 'idiots' - or deck chairs - in the stands. Our study, though, did not explore these differences. What we did try and do is explore the differences in NBA coaches. And our study found that in many cases, there were not any substantial differences. In sum, although one has to acquire substantial knowledge to be an NBA coach, there isn't much one of these coaches is able to do to differentiate himself from his peers. Consequently, players perform in a similar fashion for most NBA coaches."
A few weeks ago in this space, I described evaluating coaches as "an extremely difficult task for statistical analysts in all sports." To the extent that we've struggled to find a way to apply numbers to coaches, Berri et al.'s study is ingenious. Most objective, systematic methods for evaluating coaching struggle to isolate the effect of coaching from player quality and the work of general managers in acquiring players. By focusing on how players have played under different coaches--surely controlling for age and other factors that might affect performance--Berri and company manage to avoid these pitfalls.
The results described in the Slate.com article are interesting. At the very extreme--Phil Jackson rated best at getting more out of his players, while Tim Floyd had the most negative impact--the results match conventional wisdom. Floyd was in fact the only coach whose rating was worse than neutral by a statistically significant amount, while seven coaches were significantly better. That's a good sign, inasmuch as poor coaches should not be coaching long enough to reach statistical significance.
The result Berri has chosen to focus on is this: Of the 19 coaches included in the study (chosen because of their longevity and having coached for multiple teams), 11 did not come out as statistically distinguishable from each other or average. Here's the key line, pulled from the quote above: "...there isn't much one of these coaches is able to do to differentiate himself from his peers."
This may be true, but it is hardly proven by this study. In order to get the benefit of isolating coaching performance, the authors of the study have traded off comprehensiveness. Improving player performance via player development or offensive system is just one of many ways an NBA head coach can demonstrate his value, and it's not even clear that it is the most important. Considering the issue subjectively, I would say the single area where a coach can have the greatest impact is in creating and implementing a defensive scheme. There's also the matter of personnel evaluation, both in terms of assisting the front office in identifying talent and distributing playing time among those already on the roster.
Already this young season, we've seen new coaches turn things around in both of those ways. Let's take the latter first. In the Slate.com article, Berri took a shot at one of his favorite targets, Knicks center Eddy Curry, telling McCarthy, "Think about it. What is a coach going to say that will get Eddy Curry to rebound?" Fair enough. What D'Antoni can do is bench Curry, and that's just what he's done, giving his minutes to more productive youngsters Wilson Chandler and David Lee. (Curry has been out with a sore knee the last two weeks, but was not in the rotation before that.) Without any major infusion of talent--though he is playing well, Chris Duhon does not count--the Knicks have already won seven games, nearly a third of the way to last year's total of 23.
The other noteworthy coaching change came in Milwaukee, where Scott Skiles replaced Larry Krystkowiak on the sidelines. 26-56 a year ago, the Bucks are also near .500 in the early going at 7-9. In this case, there was an offseason makeover that saw new GM John Hammond bring in Luke Ridnour, Richard Jefferson and several reserves as well as adding surprising rookie Luc Richard Mbah a Moute in the second round of the draft. Yet those additions, with the exception of Mbah a Moute, are not known as defensive players. So it seems the coach has had more than a little to do with Milwaukee's defense improving from dead last in the NBA a year ago to ninth thus far in 2008-09.
That kind of defense is very much out of character for the Bucks. To find the last time Milwaukee defended better than league average, one has to go all the way back to 1990-91, the last full year of the Del Harris era. On the other hand, stout defense has become Skiles' trademark. When he took over the Bulls early in the 2003-04 season, he improved their defense from 21st in the league to 16th. The following year, with a full training camp under Skiles, Chicago rose to second.
Let's imagine that we put Skiles through a mini-study looking for the impact of his coaching by studying player performance. Six veterans have seen appreciable playing time for the Bucks this season. Here's how their per-minute Winning Percentage rating compares this year to last.
Player 0708 0809 Diff
Ridnour .414 .427 +.013
Sessions .569 .576 +.007
Bell .392 .335 -.057
Jefferson .489 .441 -.048
Villanueva .452 .520 +.068
Bogut .564 .576 +.013
AVERAGE .480 .479 -.001
Three players have seen their performance essentially continue unchanged. Charlie Villanueva has started the season strong, while Jefferson and Charlie Bell have struggled. The bottom line? Skiles has no apparent effect on the performance of his players. This makes sense, given that the biggest improvement the Bucks have made has come in terms of defending opponent shots. Milwaukee blocks few shots, the only measure of shot defense at the individual level. The only way what Skiles has done would show up in this kind of study would be with the application of a heavy team adjustment that considers team defense. To call that player performance would be something of a stretch.
(My ratings do include team defense, but only credit players for the percentage of the team's minutes played, capping the adjustment at 20 percent of the team's performance in the most extreme scenario. Berri's Wins Produced model does include a full team adjustment. In this study, he apparently chose instead to use his Win Score rating, which does not have the same adjustment.)
Statistical analysis has yet to shed much light on the value of coaching in the NBA. Clearly, this is a difficult task. Sabermetrics is much further along in terms of exploring many issues than is its basketball cousin, yet the value of managers has not been firmly established in baseball, either.
The study conducted by Berri and company is a worthy effort which adds to the literature. That coaches do not appear to consistently have impact on the individual performance of their players is an important and noteworthy conclusion. Still, it appears that, in discussing the study, Berri has overreached. His argument that coaches make little difference is premature at best, and ignores many of the other ways coaches can win or lose games that do not factor into the study.
So sleep soundly, NBA head coaches, at least after you've finished watching film or drawing X's and O's into the early hours. You might get too much credit when your team wins and too much blame when it loses, but there's no reason to believe your work is for naught.
Kevin Pelton is an author of Basketball Prospectus.
You can contact Kevin by clicking here or click here to see Kevin's other articles.