Right now, NBA analysts everywhere are struggling with the same problem news organizations faced last night--just how quickly to draw conclusions from small samples of data. And while we're unlikely to have Karl Rove question our math on live television, there is the risk of embarrassment if we write off a struggling team too quickly or leap to accept a hot-starting team.
The question I often get asked this time of year is how many games we need for results to become reliable. And while I totally understand that thinking, it paints the issue too black and white. The reality is that statistics are always subject to uncertainty, which is why even player performance over a full season tends to be about equally valuable to play the previous two years when it comes to projecting the future. So the better way to think about the issue is to consider uncertainty a sliding scale that decreases as the season goes on.
To try to study the issue at the team level, I went back to game-by-game results from the 2010-11 season--throwing out 2011-12 because of the weirdness of the post-lockout schedule. My method was to look at teams' performance game by game and how it compared to final numbers from the regular season. I considered both straight point differential and the version we use adjusted for opponent and location. As a sanity check, I also added the standard error at each game based on the observed standard deviation over the course of the season. All three figures are graphed on the following chart:
I would say this graph looks about like I expected. Results are particularly noisy after the first couple of games, but then make steady, slow progress until the end of the season. Adjusted results are especially useful during the first week of the year; after that point, they tend to conform pretty closely to point differential. Alas, schedule-adjusted differential isn't quite as useful after two games as that chart indicates, since it's adjusted based on full-year figures, not an equally noisy set of opponent performances. So schedule matters in the early going, but we may not be able to tell how difficult schedules actually are right now.
The standard error line is of similar shape to what we actually observe, though consistently above it because of the issue I mentioned earlier--even 82 games aren't enough to truly capture team (or player) performance. There is still uncertainty in the data that isn't captured by the other methods, which assume the final mark is the true one.
Taken together, all three measures suggest that performance starts to flatten out somewhere around 25 games or so. If I was forced to put one number on when results become reliable, that would be the point, when two years ago teams were on average within two points of their final differential, adjusted or otherwise. Of course, that still means about one team per season will be at least four points away from where it finishes, but for the most part teams have shown their general level of play by the holiday season.
Now let's consider what we know at this point, a week into the regular season. Teams with between three and five games played were on average about 4.5 points away from their final regular-season point differential in 2010-11. That's a huge figure. It suggests about a third of the league will end up at least that far from where their point differential is right now. It's easy to pick out some candidates, like Boston (-7.0), Dallas (+9.8), Detroit (-13.3), New Orleans (+5.3), New York (+19.3), Orlando (+9.3) and Philadelphia (-9.7). The other factor for some of these teams is that, of course, team ability is not a fixed figure; it trends up and down throughout the year as teams add and lose players. So a team like the 76ers, playing without Andrew Bynum, could have even more room for improvement than the numbers indicate.
At the same time, the example of 2010-11 does show that early results far away from expectations can have some meaning. Even after just three games, only three teams were more than 10 points away from their final differential. The most any team changed the rest of the way was 11.5 points. That's encouraging news for the Knicks, who have a far better differential through three games than any team managed in 2010-11, and the surprising Magic. As ESPN Insider's John Hollinger pointed out on Monday, truly bad teams--the kind we figured Orlando would be--just don't win consecutive games as comfortably as the Magic did to start the season.
Player performance tends to stabilize a bit more quickly. Historically, analysts have used three benchmarks as cutoffs--250, 500 and 1,000 minutes. A starter can get to that first round figure, the smallest threshold at which I would ever seriously consider player performance, within the first couple of weeks of the season.
The interesting thing about player statistics is that there are a variety of different denominators, which means they stabilize at different rates. The denominators on rebounds (available missed shots), assists, steals and blocks (team plays) are so large that they tend to be highly reliable over small samples. The same is true of player tendencies (usage, and the percentage of plays used on twos, threes, free throws and turnovers). We see far more volatility in shooting statistics, since their denominators (shots attempted) are much smaller--especially in the case of three-point percentage. So it's worth casting a warier eye toward hot shooting starts than players dominating in other ways.
From time to time, mistakes are inevitable. Actually, that's part of the fun of watching sports. But by better understanding the uncertainty in early-season performance, we can avoid calling either the election or the NBA too soon.
For a comprehensive preview of the 2012-13 season, check out Pro Basketball Prospectus 2012-13, now available in .PDF and paperback formats.
Kevin Pelton is an author of Basketball Prospectus.
You can contact Kevin by clicking here or click here to see Kevin's other articles.