One of the criticisms of the U.S. News rankings is the way it reports its results. U.S. News uses a process where they gather a lot of data, run it through the Willy Wonka gobstopper machine, and out pops some magical raw number. U.S. News then gives each school an overall score that is the percentage of that school’s number compared to the top number (Yale, which is reported as 100). But then, U.S. News racks and stacks those scores and simply reports a ranking. The actual score of the variable that U.S. News is trying to measure – quality – falls out. (For this discussion, I’m assuming, probably incorrectly, that the U.S. News instrument measures what U.S. news claims it measures – quality).
There could be a huge drop in quality between a school ranked 40 and a school ranked 41, or hardly any drop in quality between schools ranked 50 and 100, and the reader would not know. U.S. News goes to all this work and then reports a number that is pretty much meaningless. And for some people, those numbers may matter. A potential student who is deciding between schools may factor those numbers into her decision, along with cost, location (near family, near a major legal market), bar passage rates (already factored a little bit into the quality score), job prospects, etc.
Probably the best way to report the scores would be in standardized units (z-scores) where U.S. News would calculate the mean of the raw scores and then report each school’s standard deviation above or below that mean. If the z-scores of two schools are pretty close together, then it might not make any sense to spend more to go to a school with a slightly higher z-score. If the z-scores are far apart, it might.
We could figure out the z-scores (for the overall scores, at least) except that U.S. News doesn’t report the overall scores for the fourth quartile of schools. We could also take the data that U.S. News provides and calculate a raw score for each school, trying to replicate the way that U.S. News comes up with its raw score, and then report out z-scores based on that. One problem is that many of the schools in the fourth quartile are missing important data, and another problem is that that would take some real effort.
I think the best we can do without much effort is to graph the distributions of the overall scores using Brian Huddleston’s U.S. News data. I did that, and a couple of things jump out.
One is potential categories. “T-14” has become a category based on lack of movement across the magical 14-15 boundary. Looking at the following chart, if there is a significant drop somewhere (a step), then there is a large quality difference between the abutting units. On the y-axis is the U.S. News overall score. The x-axis is just a discrete unit with that score (I have removed ties). We see steps in quality at the two arrows.
Looking back at the original data, that first step is between schools ranked 7 and 8. For school 8 to catch school 7, it has to increase its overall score by four. (One tie aside, the steps between the top six schools are also pretty big – three or four points). The second step is between schools ranked 17 and 18. For school 18 to catch school 17, it has to increase its overall score by 5. From 8 to 17 and then from 18 and beyond, the differences between units is only one or occasionally two overall points.
If we are to do what humans love to do – draw circles around things and create categories – then we would have this:
We do seem to have a category at T-7 and one at T-17. We should expect limited movement within or out of T-7, and limited movement in or out of the next category, T-17. After school 18, shuffling up or down a few spots could occur anywhere on that segment. And there doesn’t seem to be anything magical about that 14-15 point.
The next chart may be more useful for potential students. The flatter spots show areas that are pretty sensitive – small changes in overall scores can cause some pretty wild swings. For example, a school ranked 94 can jump to 87 with a one-point increase in overall score; to 82 with two points; and to 78 with three. (Recently, Oregon has had some pretty big swings in this zone.)
The overall quality difference between 78 and 94 isn’t that much and probably would not translate into a different law school experience for that potential student. And it might not justify a difference in cost. Looking just at the rankings, though, the potential student wouldn’t know.
(I thought the following scatterplot was pretty cool. Space to the right of a dot means that several schools are tied at that point. You can see the T7, T17 gaps.)
Posted by Eric Carpenter on November 19, 2015 at 05:31 PM
Comments
Sorry, *not* thought to be worthwhile.
Posted by: Paul Horwitz | Nov 21, 2015 10:54:57 AM
Mea culpa. (I could have just said without sarcasm, about a comment earlier up, that even *if* the project was thought to be worthwhile, people spend their free time in all kinds of ways, including commenting on blog posts.)
Posted by: Paul Horwitz | Nov 21, 2015 10:54:12 AM
Orin,
I suspect he isn’t charging them at all. He’s taking money from his students and using it to provide free food to children. While feeding the children is a noble goal, surely we shouldn’t be calling on law students to fund such efforts — and funding it with debt, no less.
Posted by: Derek Tokaz | Nov 21, 2015 8:44:35 AM
Derek, Paul didn’t say how much he was charging them for dinner.
Posted by: Orin Kerr | Nov 21, 2015 1:10:12 AM
What incredibly useless information.
Posted by: Anon | Nov 20, 2015 1:56:34 PM
Look at Paul bragging about how he can afford to feed multiple people. Such hubris.
Posted by: Derek Tokaz | Nov 20, 2015 12:59:12 PM
This misunderstands what students use rankings for. It is a proxy for the industry’s snobbiness. Where the school you attend is ranked determines what jobs you will be allowed to compete for. So yes there is a T7 as we a T17 but also a T3 and a T1. However, the difference in scores (z or otherwise) between 78 and 86 are irrelevant. At that point geographic factors predominant. If you want to practice in Mississippi and go to the top ranked school in Mississippi you will be allowed to compete for almost any job in the state. Not tenure track law professor jobs, don’t be silly those are reserved for nobility, but most any other job including even district court clerkships and AUSA. On the other hand, if you go to a similarly ranked school in NYC, LA, or Chicago, I hope you took a lot of courses in small firm marketing.
Posted by: Ben Dov | Nov 20, 2015 10:37:36 AM
It’s true. Tonight I not only made dinner for my kids and hung out with them, and folded laundry while watching a movie; I even found time for purely trivial activities like reading and commenting on a blog post.
Posted by: Paul Horwitz | Nov 20, 2015 5:00:25 AM
My sarcasm detector just exploded.
Posted by: anon&on | Nov 19, 2015 11:53:30 PM
It’s wonderful that law professors have enough free time to engage in worthwhile exercises like this.
Posted by: Lois Turner | Nov 19, 2015 6:29:58 PM