A typically nice piece by Louis Menand in this week’s New Yorker discusses a new book by Philip Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? The book suggests, in brief, that experts are regularly overoptimistic in guessing both probabilities and outcomes of matters within their sphere of expertise. That’s not a surprise, as Menand notes, to those who are interested in the lessons of psychology for individual and group decision-making, as many increasingly are in the law. One nice quote from Tetlock, drawing on Berlin, sums up some of his thinking about why some people make better prognosticators than others:
“Low scorers look like hedgehogs: thinkers who ‘know one big thing,’ aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who ‘do not get it,’ and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible ‘ad hocery’ that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.”
Now, the first thing I must say about this quote is that it’s so nice to be reassured that there is indeed a small saving grace to the fact that I’m a dilettante. The other thing it does is make me reflect on what seems to me to be the often poor predictive power of law professors and other “experts” in forecasting outcomes of appellate court cases — especially in the Supreme Court, where the conditions of uncertainty are greatest, due to the difficulty of the questions presented and the weakened force of precedent. (For some evidence as to this poor predictive power, see here.)
A number of related factors seem to increase the possibility that “expert” law professors will make poor forecasters. First, leading law professors are, indeed, often hedgehogs: they have attained their fame by advancing one big idea, in an academy that loves unified field theories of some area or other. This tendency is exacerbated by the fact that, if that big idea comes early in one’s scholarly career, the luminary may well devote some portion of his or her remaining productive career to defending, extending, and rehashing the idea that made his or her reputation. Second, this is not an accident. As I’ve suggested before, hiring committees and law review editors often select for “brilliance” — for the prospect that a professor will forcefully and eloquently advance new and novel and provocative and counter-intuitive ideas. The poor plodders who are merely “right” may remain in the shadows. (Not always, to be sure. FYI, this last point is not an effort to justify my own plodder status, because that’s not my status; in fact, I am often brilliantly wrong in my scholarship. [TENURE COMMITTEE: PLEASE REDACT LAST SENTENCE.])
Third, there is the siren song of legal normativity. I mean “normativity” here both in a policy/ideological sense — some outcome must be achieved; therefore, predictively speaking, it will be achieved — and in a methodological sense, ie. a stringent originalist may, in predicting outcomes, overrely on that methodology in making a forecast. (Did Randy Barnett exhibit both tendencies in predicting the outcome in Raich, as well as displaying an advocate’s natural optimisim? Perhaps so. [UPDATE: Randy Barnett has suggested to me, cheerfully enough, that he did not in fact predict an outcome in Raich. These are the wages of quick posting, and I’m happy to concede the point. I may have been thinking about what I read as his fervor in pre-Raich postings, but that’s not the same thing as a prediction I’ve left the sentences as they are as a self-reproof.]) To this last point must be added the possibility that the policy/ideologicial/methodological commitments of some legal “experts” may lead them not just to unintentionally err in their predictions, but to deliberately and publicly exaggerate the likelihood that the Court will reach some desired outcome. Notwithstanding their scholarly and professional commitments, many high-profile legal experts are also exquisitely political animals and are more than willing to engage in a little public spin.
In light of all this, I wonder, finally, whether it is not a category mistake to describe at least some leading legal academics as “experts” for purposes of forecasting. I emphasize that last point — for purposes of forecasting. There are all kinds of reasons to favor the “brilliance” and hedgehog-ism I have described above. It sparks new ideas and carves out new vistas for thinking about the law — and, as Menand notes, when their one big idea is right it can be “really and spectacularly right.” But that skill consists more often of changing the landscape, as MacKinnon did, for instance, than of predicting the future. So, for prediction purposes at least, the question stands: If the academy sometimes overselects for brilliance and underselects for that congeries of foxlike tendencies, leavened by actual experience in the real world of litigation (a trait that is increasingly limited for high-tier law professors), that make up what we might call “common sense,” “judgment,” or “practical wisdom,” then why does it make sense to turn to these people in the first place when we are attempting to predict the outcome of a given case? (Hence the square quotes throughout this piece when referring to lawprofs as “experts.”) Perhaps, rather than turn to the leading professor in some field to predict the Court’s ruling in some case, it would make more sense to survey any ten randomly selected practicing appellate lawyers.
Posted by Paul Horwitz on December 3, 2005 at 12:25 PM