US News Rankings Methodology: What’s Really Behind the Numbers

The US News Rankings Methodology: What’s Really Behind the Numbers

People crave rankings. But what do these rankings mean? How are they actually formulated? Every fall, a new edition of the US News and World Report college rankings comes out, and while people love consuming them, they many times fail to ask what exactly is behind this US News rankings methodology.

Asking this question is important because these rankings are not perfect – no rankings are. And this reliance on a sometimes flawed methodology can be an issue when students prioritize rankings over personal fit during the college application process.

So, since every new release sets off a round of buzz about which schools rose and fell, this article is a look into what is behind the US News rankings methodology.

How the US News Rankings Methodology Is Problematic

In a survey conducted by the National Association of Colleges and Employers (NACE), students were asked what the most important factor was in selecting a college. The most popular considerations are listed in order below:

• Ability to Obtain a Good Job After Graduation
• The Cost of Attendance
• Visit to the Campus
• Size of the School and Student Body
• Admissions Potential for Top Graduate/Professional Schools

You will note that few of these factors are directly accounted for in the US News rankings methodology, though recent editions added a graduate earnings measure, while campus visits, school size, and graduate school admissions potential still play no role.

To take just that first point, the US News rankings methodology gives job prospects after college only a small slice of its weight, even though they are one of the primary concerns of applicants. A single earnings figure also says little about whether a school prepares graduates for the specific field you want to enter. If career outcomes matter most to you, read each college’s own career outcomes report alongside the rankings.

Inside the US News Rankings Methodology, Measure by Measure

Now, let’s take a look at exactly what a few of the rankings criteria are in the US News rankings methodology and why they are sometimes problematic. U.S. News publishes an explanation of its current criteria with each edition, and the formula has shifted noticeably over time. Recent editions put more weight on outcomes, such as graduation for Pell Grant recipients and first-generation students and graduate earnings, and dropped older inputs like alumni giving and class size. The measures below are the ones that have long shaped the list, and several remain part of it today.

1. Peer Assessment (Academic Reputation)

This “prestige” ranking is based on a survey sent to the country’s university and college presidents, provosts, and admissions deans as well as a few high-school officials asking them to grade all of the schools on a 1-5 scale. So, for instance, the president of one national university is asked to rate hundreds of other national universities. There are several flaws with this approach as it is hardly scientific. 

This is not to say the reputational rankings can’t be helpful. They certainly can be, but only if they are ranking institutions along one dimension and relying on people with specialized knowledge. A more accurate example than the US News rankings methodology is the Wall Street Journal’s method of ranking colleges exclusively based on the opinions of corporate recruiters whose opinions are directly relevant to the rankings at hand.

2. Graduation and Retention Rates

This metric combines six-year graduation rates with first-year retention rates. These rates are seemingly solid points to include in the methodology – particularly the first-year retention rate.

Yet certain top schools, such as Stanford, are notorious for college dropouts who go on to pursue startup ventures. The most famous example, as we all know, is Mark Zuckerberg dropping out of Harvard to found Facebook. Thus the issue with this particular metric is that schools like those in Silicon Valley, which pride themselves on their entrepreneurial spirit and support of student ventures, are penalized in the US News rankings methodology for dropouts who are going on to act as major contributors to the business and tech worlds.

3. Faculty Resources

The justification for this category is purportedly that the more satisfied students are about their contact with professors, the more they will learn. The underlying point – that student engagement is critical – is accurate; the problem is how to measure that. And the US News rankings methodology struggles to do so.

In older editions, this category was broken down into the following: the proportion of classes with fewer than 20 students, the proportion of classes with 50 or more students, faculty compensation, the proportion of professors with the highest degrees in their fields, the student-to-faculty ratio, and the proportion of faculty who are full time. US News has since dropped the class-size measures.

What do these statistics have to do with measuring students’ happiness about their contact with professors?

Almost nothing.

Statistics can be manipulated. For example, Clemson managed to move up from 38 to 22 at some point in USNWR’s rankings of public research universities. Clemson allegedly did things like artificially manipulating class sizes with no real benefit to the students. They (apparently) would try to bump sections with 20 students down to 19, but let a class with 60 students rise to 65 in order to try to improve their ranking in the “Percentage of Classes with Fewer than 20 Students” category. [1] These changes would have no real, tangible benefit on student interactions with professors, rather they were more of a means to play into the US News rankings methodology.

In 1996, President Gerhard Casper of Stanford University penned a letter to the new editor of USNWR sharply criticizing the artificial manipulation of school ranks from year-to-year as a transparent ploy to sell more magazines. [2] 

4. Student Selectivity

The rankings race encourages schools to increase their selectivity. How do they do this? It’s simple. They make it easier to apply to the school. Stanford – the most selective school in the US in recent years – holds this title at least in part because it has lowered application fees and streamlined the technology needed to apply to allow thousands more students to easily submit applications. Although, Stanford’s Dean of Admissions has spoken out against the obsession with selectivity as simply another way to manipulate rankings. But if applicants are responding to the US News ranking methodology, then schools must as well. Always remember: colleges are businesses too. Even public universities. 

5. Financial Resources

More money does not always correlate with educational outcomes if it is not spent in ways that directly impact students. It is many times, rather, spent on something such as artificially inflating salaries in order to manipulate the US News rankings methodology. Schools with large endowments do not necessarily allocate all of their resources to directly benefit students’ needs on a year-to-year basis.

6. Graduation Rate Performance

This might be the most educationally counterproductive statistic of them all. The US News ranking methodology tries to measure the effectiveness of a school by using a “predicted graduation rate” and comparing it with an “actual” graduation rate. Former Stanford President Gerhard Casper captures why this is a problematic metric with a poignant example in his 1996 letter:

“The California Institute of Technology offers a rigorous and demanding curriculum that undeniably adds great value to its students. Yet, Caltech is crucified for having a “predicted” graduation rate of 99% and an actual graduation rate of 85%. Did it ever occur to the people who created this “measure” that many students do not graduate from Caltech precisely because they find Caltech too rigorous and demanding – that is, adding too much value – for them? Caltech could easily meet the “predicted” graduation rate of 99% by offering a cream-puff curriculum and automatic A’s. Would that be adding value? How can the people who came up with this formula defend graduation rate as a measure of value added? And even if they could, precisely how do they manage to combine test scores and “education expenditures” – itself a suspect statistic – to predict a graduation rate?”

Essentially, this point of the US News rankings methodology punishes good schools for challenging their students to become better.

7. Alumni Giving Rate (Since Dropped)

US News has since removed alumni giving from its formula, and the old measure shows why. [3] Keeping a nearly perfect score in this category was not particularly difficult. Just ask every one of your alumni for one cent. We see plenty of schools doing things like this by sending dozens of e-mails per year asking students for a mere one dollar contribution. Is this how you want to make any part of your decision on where to attend college? Whether enough people have been bothered to donate one dollar?

How to Use Rankings in Your College Search

None of this is meant to attack US News and World Report. It is meant to provide insight into the issues with rankings generally, and our look at what college rankings actually tell us goes further.

The US News rankings methodology is flawed because there is simply no direct, objective way to measure the quality of an institution. As Malcolm Gladwell writes, there is no way to measure “how well a college manages to inform, inspire, and challenge its students. So, the U.S. News algorithm relies instead on proxies for quality – the proxies for educational quality turn out to be flimsy at best.”

And many times, it is difficult to tell if schools are making the best decisions based off of rankings. For example, if UChicago has increased their selectivity in recent years (the schools has one of the highest standardized test score averages), then they will climb in rankings. This selectivity does not impact students. But if a school’s climb in the rankings allows it to attract more distinguished faculty, or gain more resources for students on campus, then perhaps it is a good thing that they played into the US News rankings methodology.

Applicants buy into rankings, which increases schools’ responses to rankings, which increases the importance of rankings generally. It’s all essentially a positive feedback loop, and because US News revises its formula over time, a school can rise or fall without changing much at all. [3]

These rankings can be useful for those who are just getting familiar with various colleges in the United States and their primary abilities. But that is as far as your reliance on these rankings should go because the quality of a particular college cannot be measured by a set of data points in a one-size-fits-all fashion. Additionally, amongst the top schools, the difference in their rankings is practically statistically insignificant because they score so similarly. This means that if you’re deciding between the #1 ranked school and the #5 ranked school, there is really no difference in quality – only minute points hidden within the US News rankings methodology.

Looking at the different rankings is a good place to start building your college list. But parsing through the US News rankings methodology shows that there is no perfect way to rank a school. And more than that, no rankings can inform which school will be the best for you, individually. The best school isn’t necessarily the school with the top ranking or the school with the highest SAT score averages or the school with the most prestigious professors. It’s whatever school is the best fit for your needs, major, potential career, personality, and interests, which is why it pays to pick a college that’s right for you and build a balanced college list around that fit. As with anything related to education and your future, you need to ask the tough questions, and do the research. There are no shortcuts to hard work in education. Your goals won’t work unless you do!

Ready to Take the Next Step?

Get a clear, honest read on which path actually fits your goals, and the specific next steps to build a competitive application. Click here to book your complimentary consultation.