Home Unfiltered Articles Players Baseball Prospectus
Basketball Prospectus home
Click here to log in Click here for forgotten password Click here to subscribe

Click here for Important Basketball Prospectus Premium Information!

<< Previous Article
Premium Article Non-Stars (02/22)
Next Article >>
Premium Article High Seeds Beware (02/23)

February 22, 2012
Made in the NCAA
The RPI's Birth, Triumph, and Encirclement

by John Gasaway

Printer-
friendly
Contact
Author


Jim Van Valkenburg

Last week I spent two days in a hotel meeting room in Indianapolis, as staffers from the NCAA walked approximately two dozen media members through a mock selection exercise for the Division I men's basketball tournament.

Over the course of what proved to be an incredibly illuminating yet fatiguing session, the NCAA was forthrightly eager to illustrate just how small a role the Ratings Percentage Index plays in actual deliberations when teams are being evaluated. Men's basketball committee chair Jeff Hathaway, an unfailingly affable and engaging sort who gives little indication that he holds arguably the most powerful position in college basketball, circulated around the room and struck up conversations that called attention to the RPI by noting that no one was paying attention to the RPI. "How we doin'?" he'd ask enthusiastically. "See how the RPI never comes up?"

When the session wrapped up, I shook hands with NCAA executive vice president Greg Shaheen and media coordinator David Worlock, climbed into my rental car, and drove to St. Louis. I wanted to hear about the RPI from someone who has no connection to the NCAA but who has a very vital connection to the statistic in question. I wanted to hear about the RPI from the man whose father invented it.

The RPI was formulated in the fall of 1980 by Jim Van Valkenburg, a staff member with what was then called the NCAA's Statistics Service. If the criterion at hand is pure influence, the equation Van Valkenburg created can only be termed a stunning success. For over three decades now the RPI has been the central team-sorting mechanism "in the room" as the NCAA men's basketball committee has selected and seeded each year's field. Over that time the tournament has grown to the point where it has an 11-digit TV deal and Selection Sunday is unselfconsciously capitalized.

Van Valkenburg died in 1995. As I shook hands with his son, Jim L. Van Valkenburg, in a Starbucks in South St. Louis, it was hard to believe there was any connection between this genial landscape designer and the widely discussed denunciation of the RPI that Scott Van Pelt had delivered a few days before on ESPN Radio. Van Valkenburg's son told me yes, he's a huge sports fan, but no, he hadn't heard the clip in question.

As I did my best Van Pelt ("Picture a guy that's walking around with a big Walkman on his hip the size of a toaster, who's flipping over his cassette tape, who wants to run home to program his VCR on his standard definition television"), the son of the inventor of what the ESPN host termed the "worst metric in sports" smiled widely and, utterly unoffended, laughed merrily at the invocation of the Walkman.

Outside the halls of the NCAA, Van Valkenburg is but little remembered today, yet you can make a case that few figures have had a larger or longer-lasting impact on college basketball. It is a peculiar artifact, surely, that earns equal measures of enmity from both Dick Vitale and Ken Pomeroy.

"He had such a broad mind for statistics"
In 1937 a 29-year-old sportswriter in Seattle named Homer Cooke began doing something that no one had ever done before in a systematic fashion. He began compiling college football statistics. Cooke pursued this endeavor on his own time and funded his own exertions until 1946, when a fledgling NCAA agreed to support Cooke in forming what was called the National Collegiate Athletic Bureau. The NCAB established its offices in New York City to be close to the wire services, and in its brief existence as a semi-autonomous entity in Manhattan it never employed more than six people.

The NCAB faced a monumental task: compiling statistics and, more specifically, maintaining an authoritative record book for a sport that was nearly 40 years older than the first incarnation of its putative governing body, the NCAA. At a time when all communications over any distance were conducted by mail, phone call, or telegram, these staffers were expected to master a mass of historical records that had been haphazardly kept and to maintain the standards for gathering such information going forward. This was the work of figures like longtime NCAB and NCAA staffer Steve Boda, whose knowledge of college and particularly Notre Dame football was and is renowned far and wide. When Van Valkenburg arrived at the New York City office in 1968, the vestigial NCAB had been operating under the auspices of the NCAA for nine years, and the centennial of college football was imminent.

A Korean War veteran, Van Valkenburg served as the Associated Press sports editor in Kansas City prior to joining the NCAA. Today you can type "Jim Van Valkenburg" into Google News, hit "Archive," and pull up facsimiles of eastern- and central-Kansas sports sections from the 1960s where Van Valkenburg filed stories on Jim Ryun, Bob Cousy, or Sandy Koufax.

People who work in sports are often told how lucky they are, and apparently that's exactly how Van Valkenburg felt. "My dad loved his job," Jim L. Van Valkenburg told me, "and he wrote with wit, humor, and passion about what he loved." In his years at the NCAA Van Valkenburg researched and wrote a narrative that accompanied the weekly statistical release from headquarters. "He would include human-interest tidbits about the national leaders," NCAA director of statistics Jim Wright told me in an email, "and also add context to the actual stats, pointing out the occasional anomaly in the numbers."

Van Valkenburg was clearly a prolific writer, but he was also eager to traffic in numbers when and where he thought they could be helpful. Long before he joined the NCAA he was working up cross-era comparisons that pitted Stan Musial's hitting prowess against that of Babe Ruth and others. "He had such a broad mind for statistics," Boda told me.

The four or five sports aficionados that Van Valkenburg joined in New York City functioned as a clearinghouse for and repository of the nation's college sports statistics. These same staff members also issued write-ups on games that had just taken place, and offered previews of the week ahead. All of the above is the work of what today we think of as a sports information director's office, and indeed many of the NCAA's early staffers arrived by way of an SID's office, a stint as a sportswriter, or both. For a good portion of the 20th century this is what the NCAA meant by a "statistics service."

Today we would intuit something quite different from such a label, but criticizing that department in retrospect for lacking statistical heft is not only ahistorical, it's unjust. Statistical heft had few if any gathering places for a good portion of the 20th century, as evidenced by a certain revolutionary baseball analyst and his gig as the security guard at the pork and beans canning plant. Indeed in its own time the Statistics Service was attacked from the opposite direction. Even as it was heavily used by beat writers racing to hit their deadlines, the NCAA's Statistics Service was occasionally assailed by columnists for being a bunch of narrow-minded pedants focused on minutiae.

"A two-week time lag for complete verification"
As part of a consolidation of its operations in 1975, the NCAA brought the Statistics Service to the main office in suburban Kansas City. The move brought Van Valkenburg into direct contact with two powerful forces that would shape the rest of his career: computers, and Walter Byers.

An August 1979 wire-service article introducing the NCAA's new passer efficiency rating (which Van Valkenburg also helped to develop) referred to the increased "computer capability" that the move to headquarters had put at the Statistics Service's disposal. That capability arrived with impeccable timing. The NCAA was revamping its basketball tournament's selection process. What had previously been a pro forma exercise that merely bracketed conference champions was now a much more complex undertaking. With the arrival of at-large bids in 1975, the NCAA needed an unbiased method that would allow the men's basketball committee to make meaningful distinctions between teams.

Walter Byers would see to it that the committee received the tools they required. The NCAA's longtime executive director is the stuff of legend, famous and/or infamous for both his rectitude and his reclusiveness. The Rev. Timothy Healy, then president of Georgetown, voiced perhaps the most widely quoted sum-up of the Byers mystique in the 1980s: "Does he really exist?"

Byers wanted Van Valkenburg and his colleagues to create a computer rating system for basketball teams. According to former NCAA executive vice president Tom Jernstedt, Byers sat Van Valkenburg and fellow staffer Dave Cawood down and told them in effect: This is what we need to help the committee do its work. Working with several NCAA analysts (as many as 13 by one count), Van Valkenburg sat atop something of a statistical Manhattan Project.

Van Valkenburg and his staff looked at using margin of victory in their new rating system, but rejected the idea for two reasons: it could encourage coaches to run up the score, and final scores "oftentimes do not reflect the closeness of games." Instead Van Valkenburg and his colleagues formulated a rating system that adjusted a team's Division I winning percentage based on strength of schedule and game location. The name "Ratings Percentage Index" apparently came about because it did not include the taboo word "Ranking" and because it was thought "RPI" formed a serviceable acronym.

What Van Valkenburg came up with was presented as the best alternative out of some 14 different rating systems, which he called "methods," that he, Cawood, and their colleagues had tested over a period of six months or more. In its original form when it was first used during the 1980-81 season, the Ratings Percentage Index was comprised of a team's winning percentage against D-I opponents (40 percent), plus three more factors: opponents' winning percentage, opponents' opponents' winning percentage, and a team's road winning percentage (20 percent each). The RPI's composition and weights have been adjusted in the years since (the current formula assigns differing values for home, road, and neutral-site results), but in its essentials it's still powered by winning percentages: your own, your opponents', and your opponents' opponents'.

The amount of manual labor required to calculate the RPI for all of Division I in 1981 was staggering. Each result of each game had to be entered by hand by NCAA staffers. In their initial announcement of the RPI's creation the NCAA gave some sense of how large a task harnessing all those final scores really was: "Team RPI rankings will not be made public during the current 1980-81 season, because the figures will not be up to date until the committee meets. Computer runs prior to that will involve a two-week time lag for complete verification."

"Have you ever tried talking to a computer?"
If in recent years you've seen stat-wielding writers calling upon an allegedly hidebound and traditionalist NCAA to jettison precedent and revamp their tournament selection process, it's a bit jarring to note how neatly and indeed perfectly these roles were reversed three decades ago. The roll-out of the RPI featured a quantitatively-oriented NCAA touting its newly minted data to a suspicious cadre of sportswriters who pledged their allegiance instead to tradition, simplicity, narrative, and conventional wisdom. In March 1981 Alan Goldstein greeted the RPI in the Baltimore Sun with this:

Have you ever tried talking to a computer? No, we're not talking about highbrows like R2D2 and C3PO of "Star Wars" fame. We're referring to the lowlife computer types that can take a $50 department store bill and turn it into the national debt....

That's what the NCAA basketball tournament fathers had in mind this year when they decided to go modern in their selection process.

The period between roughly the late 1970s and the mid-1980s was remarkably eventful in college basketball terms, and not simply because Magic Johnson and Larry Bird roamed the hardwood back then. In effect that period established the sport as we know it today, complete with a tournament field comprised of 64 to 68 seeded teams, a bracket populated by more at-large entrants than automatic bids, a contract with CBS, a Final Four held in a football venue, a three-point line, a shot clock, and the RPI. None of those things existed when John Wooden retired in 1975. All of them have been constants for the past quarter-century or more.

Guiding the tournament at this time of change was a core group of NCAA staffers with remarkably similar backgrounds: Byers, Jernstedt, Van Valkenburg, and Cawood. Another key figure was then Big Ten commissioner Wayne Duke, who at the time served as the chair of the men's basketball committee. (It was Duke who decided to have the 1981 national championship game proceed on schedule in the hours after an assassination attempt on President Reagan.) Byers, Duke, and Van Valkenburg all worked as sportswriters, sports information directors, or both. The RPI, the favorite target of abuse for generations of sportswriters, was created by ex-sportswriters.

Vic Bubas and the creation of "record vs. top-50"
The RPI was used not simply to identify the teams at the top of Division I but also, and perhaps more importantly, to sort out which conferences comprised the bottom of D-I. At the time of the RPI's inception the tournament field consisted of 48 teams, most of which still arrived in the brackets as automatic selections. For instance in 1981 the tournament field was comprised of 26 automatic bids, and 22 at-large selections. But the NCAA was in the process of shifting that balance in favor of the at-larges, and the RPI was enlisted to help rope off the weakest members of the tournament field. Expansion to 64 teams was on the table at that time, but it was by no means clear that it would happen. If the field stayed at 48, the NCAA was gearing up to draw the line on which conferences would be awarded automatic bids.

In 1983 the tournament field expanded from 48 to 52 teams, and that year champions from eight conferences that had been flagged by the RPI were sent to a "preliminary round," otherwise known as a play-in game. In 1983 those eight conferences were the Atlantic Sun, Northeast, MEAC, Ivy, SWAC, Horizon, East Coast, and America East. That year Ivy League champ Princeton won a "preliminary round" game against North Carolina A&T and then upset No. 5 seed Oklahoma State in the round of 48.

Nevertheless, the RPI was also used as more than a screening device for conferences seeking automatic bids. In the RPI's early years, the committee made some daring at-large choices, and having a fancy computer-generated bauble surely helped. For example in 1983 Louisiana-Lafayette, Utah State, and VCU all received at-large bids, which constituted something of a RPI-fueled small-school coup at a time when the entire tournament field included just 24 at-large entrants.

Today the NCAA calculates a Ratings Percentage Index for a total of 11 D-I sports, including women's basketball, men's and women's volleyball, men's and women's soccer, men's and women's lacrosse, field hockey, baseball, and softball. In addition a number of high school associations nationally use the RPI to rank their vast populations of teams.

It's likely that the enduring appeal of the RPI lies primarily in its ease of use. A layperson can't work up a quick Elo rating, but the RPI runs on nothing more complex than "win/lose" and "home/road/neutral." The RPI is a multisport contraption of ingeniously and perhaps even elegantly simple construction, and it does one thing: it evaluates how good a team in a given sport is at winning games. The larger the population of teams, the higher the number of games, and the fewer the alternatives in terms of competing metrics, the more likely it is that the RPI can shed at least some light.

One of the annual criticisms of the selection committee, at least from former players and coaches, is that it's lacking in so-called "basketball people," and thus has to rely on an artificial construct like the RPI. This is more than a little ironic, because in truth it was quintessential basketball types like former Duke coach Vic Bubas who first looked at how well teams performed against top-50 RPI opponents. Bubas had two interns compile that information, and arrived in the selection committee room with printouts that soon became the standard template for everyone on the panel. Printouts came to have almost emblematic status within the committee room, until by at least one telling Greg Shaheen arrived and brought the wonders of the modern laptop to bear on the selection process.

Van Valkenburg's triumph -- and Byers'
The paradox of Van Valkenburg's handiwork is that by formulating a metric that eschews any knowledge whatsoever of points scored, he in fact created the perfect conditions under which scoring margin, adjusted for strength of schedule, could and has become a surpassingly powerful evaluative tool. But those of us who use that tool should acknowledge the analytic free ride we've been given, courtesy of Van Valkenburg. The stats we're able to deploy in 2012 are great at evaluating teams -- but the only teams we've ever been able to evaluate have been ones striving to look good to Jim Van Valkenburg, and not to us.

The good news is the art of evaluating basketball teams has advanced considerably over the past 30 years. If one still wishes to keep scoring entirely out of the evaluative mix, systems like Jeff Sagarin's "politically correct" ratings or Kenneth Massey's ratings do just that. Alternately, if we want to harness the evaluative benefits of acknowledging points scored but steer clear of encouraging teams to run up the score, Dean Oliver recently introduced a rating system tailored specifically to those criteria. These are just a few of the systems available -- LRMC is still another -- that would give the committee a more accurate picture of the teams they're evaluating. And using a composite index comprised of several different rating systems would surely be a welcome step, one that would filter out the biases of each while relieving any one system of the pitiless scrutiny under which the lone, solitary RPI has had to labor.

I disagree with Seth Davis when he says that the RPI is "the best metric that anyone has come up with so far," but the superlative that I will offer wholeheartedly is that Jim Van Valkenburg advanced the committee's knowledge more than any other person ever has. The moment when the first results from the new RPI arrived in the committee room must have been incredible. In 1981 national powers like DePaul, Louisville, and a few others could be seen on TV on the weekend, but only a very small number of programs received that level of exposure. Pre-RPI, the committee had little to go on but final scores, winning percentages, and a few stray random glimpses of teams on TV. And then suddenly in their midst there arrived a metric that at least measured all of Division I with a common ruler. The RPI is not the best metric that anyone has come up with so far, but it did represent a great leap forward in 1981. And given what Van Valkenburg had at hand to work with and the amount of labor that any trial and error then required, his achievement can justly be termed a triumph.

His was a shared triumph. To duplicate what Walter Byers did, Mark Emmert would have to go to his staff tomorrow and charge them with inventing, from the ground up, the best and most advanced basketball team evaluation statistic available anywhere. Byers had his critics, but the man can't be accused of lacking audacity.

Today when Van Valkenburg's invention is faulted on the grounds that it doesn't perform as well as other rating systems that benefit from more recent advances, an oddly prevalent response is that the NCAA "isn't trying to predict anything." We've reached a strange impasse, surely, when we say that because the NCAA doesn't wish to project the performance of actual teams in an actual bracket they're selecting and seeding, we can't take a step back after the season is over and ask how well our performance measures -- the RPI or anything else -- are doing what they were designed to do. It's an impasse that takes too little notice of Van Valkenburg's example. He worked for months and went through no fewer than 14 iterations of different rating systems trying to find the best possible method. Van Valkenburg very clearly cared passionately about getting the best information possible into the hands of the committee.

"So why does the NCAA still use the RPI?"
In my experience I'm yet to encounter an NCAA staffer who isn't both very smart and an exemplary hoops citizen. These are extremely bright people who care deeply about basketball. And they clearly, manifestly, genuinely want the NCAA to continue using the RPI.

Nate Silver was one of my fellow attendees at the mock selection, and he too saw that the NCAA can and does bring together "bright," "good citizen," and "an ongoing commitment to using the RPI." By lunch of the second day when this had become unmistakable, Nate greeted me with a fatigued smile and a request for some basketball knowledge: "So why does the NCAA still use the RPI?"

I asked them, of course. They get asked that a lot, and they always say the same thing. The RPI is just one factor in the mix, and the NCAA is always looking to improve all of their methods. Both statements have a good deal of truth in them, but at the end of the day we're still confronted with the same paradox. People everywhere use the RPI to predict which teams the NCAA will select for the tournament, but it goes without saying that no one but the NCAA uses the RPI to actually evaluate teams.

The RPI began its life as the the best quantitative rating available for college basketball teams, but for years now it has drawn its power exclusively from its sponsor. If the RPI were dropped by the NCAA tomorrow, the metric would surely fall into benign desuetude in a heartbeat. Yet here we are. Why?

My sense is the reason the NCAA continues to use the RPI in 2012 is purely institutional. It's possible that discontinuing the use of something that has become so deeply entwined with their rhythms, org chart, and identity would would cause some degree of pain and trauma in their physical workspace on the day of the discontinuance and for days afterward. All organizations resist doing things that cause pain and trauma in their physical workspace. Usually they're compelled to do those things only by outside pressures, and the NCAA faces way fewer of those than a normal organization. Normal organizations that employ fewer people than one good-sized Wal-Mart don't have $10.8 billion in guaranteed revenues over 14 years. The NCAA gets to call the tune on this one.

"Dad was a realist"
Van Valkenburg retired on November 9, 1992, and he lived his remaining years in Fairway, Kansas. A diabetic, he passed away at the Johnson County Dialysis Center in nearby Lenexa on September 11, 1995.

On a clear and warm Saturday in February in South St. Louis, I've caught Jim L. Van Valkenburg after he's worked a full day but before he's gone home, and I've promised to be brief. Jim L. is a native Midwesterner who lived for seven years as a child in suburban New York City, and he's frankly appalled to learn that I, a native Midwesterner, have recently moved to suburban New York City. ("I think it took years off my dad's life, to be perfectly honest.")

We nail down some biographical details on his father, and I want to compare notes on things I've learned through research but he knows from personal experience about NCAA figures like Byers ("My dad had a tremendous amount of respect for Walter Byers") and Boda ("I think the NCAA hired my dad thinking he might be the one person in the world who could keep up with Steve Boda on sports"). But before we part ways, I want to put one last question to the son of the man who invented the Ratings Percentage Index.

"If your dad were sitting here right now," I say, "and I told him his RPI is still being used in 2012 even though a goodly number of outside observers think there are better metrics, what would he say?"

Van Valkenburg's son doesn't hesitate. "Dad was a realist. I think he'd say, 'If there's something better, use it.'"

Follow John on Twitter: @JohnGasaway. This free article is an example of the content available to Basketball Prospectus Premium subscribers. See our Premium page for more details and to subscribe..

John Gasaway is an author of Basketball Prospectus. You can contact John by clicking here or click here to see John's other articles.

1 comment has been left for this article.

<< Previous Article
Premium Article Non-Stars (02/22)
Next Article >>
Premium Article High Seeds Beware (02/23)

RECENTLY AT BASKETBALL PROSPECTUS
State of Basketball Prospectus: A Brief Anno...
Tuesday Truths: March-at-Last Edition
Easy Bubble Solver: The Triumphant Return
Premium Article Bubbles of their Own Making: Villanova, Temp...
Tuesday Truths: Crunch Time Edition

MORE FROM FEBRUARY 22, 2012
Premium Article Non-Stars: Best Active Players
Premium Article Good and Lucky: Inside Notre Dame's Turnarou...

MORE BY JOHN GASAWAY
2012-03-01 - Premium Article No Rebound, No Cry: Bad Rebounding Not a Pro...
2012-02-28 - Tuesday Truths: Penultimate Edition
2012-02-23 - Premium Article High Seeds Beware: Five Sleepers
2012-02-22 - Made in the NCAA: The RPI's Birth, Triumph, ...
2012-02-22 - Premium Article Good and Lucky: Inside Notre Dame's Turnarou...
2012-02-21 - Tuesday Truths: Crunch-Time Edition
2012-02-16 - Premium Article Where Did the D Go?: West Virginia's Collaps...
More...


Basketball Prospectus Home  |  Terms of Service  |  Privacy Policy  |  Contact Us
Copyright © 1996-2014 Prospectus Entertainment Ventures, LLC.