Tag Archives: law school rankings

“GAMING THE REPORTING”?

Posted on by
Reply

In a recent interview with Lee Pacchia of Bloomberg News, U.S. News & World Report’s director of data research Robert Morse explained this year’s only revision to his law school rankings methodology. Morse gave different weights to various employment outcomes for class of 2011 graduates. But he didn’t disclose precisely what those different weights were.

Morse said that such transparency worried him. Full-time, long-term jobs requiring a legal degree got 100 percent credit. But he didn’t reveal the weight he gave other employment categories (part-time, short-term, non-J.D.-required) because he didn’t want deans “gaming the reporting of their results.” It was an interesting choice of words.

Teapot tempests

In some ways, all of the attention to the changes in this year’s rankings methodology is remarkable. Certainly, a school’s employment success for graduates is important. But the nine-month data point for which the ABA now requires more detailed information accounts for only 14 percent of a school’s total U.S. News ranking score. To put that in context, consider some of the more consequential rankings criteria.

Fifteen percent of every school’s U.S. News score is based on a non-scientific survey of practicing lawyers and judges. The survey response rate this year was only nine percent.

Likewise, the “peer assessment” survey that goes to four faculty members at every accredited law school — dean, dean of academic affairs, chair of faculty appointments, and most recently tenured faculty member — accounts for 25 percent of a school’s ranking score. It asks those four individuals to rate all ABA-accredited law schools from 1-to-5, without requiring that any respondent know anything about the schools he or she assesses.

Taken together, the two so-called “quality assessment” surveys comprising 40 percent of every school’s ranking are a self-reinforcing contest for brand recognition. As measures of substantive educational value, well, you decide.

Game of moans

But if, as Morse suggests, his concern is “gaming the reporting,” he must be worried that some deans would either: 1) self-report inaccurate data; or 2) otherwise change their behavior in an effort to raise their school’s ranking. He’s a bit late to both parties.

Scandals engulfed prominent law schools that submitted false LSAT and GPA statistics for their entering classes. But how many others haven’t been caught cheating? No one knows. As for permissible behavior that accomplishes similar objectives, examples abound.

For years, deans seeking to enhance the 12.5 percent of the rankings component relating to median LSAT scores for J.D. entrants have been “buying” higher LSATs through “merit” scholarships. Need-based financial aid has suffered. Ironically, those merit scholarships often disappear after the first year of law school.

Likewise, the faculty resources component is 15 percent of every school’s ranking. But it encourages expenditures — and skyrocketing tuition — without regard to whether they benefit a student’s educational experience.

Whom to blame

Morse establishes the criteria and methodology that incentivize behavior producing these and many other perverse outcomes. But he doesn’t think that any of the current problems confronting the profession are his fault.

“U.S. News isn’t the ABA,” he told Pacchia. “U.S. News doesn’t regulate the reporting requirements…[W]e’re not responsible for the cost of law school, the state of legal employment, the impact that recession has had on hiring, or the fact that 10 or 20 new law schools have opened over the last couple decades. We’re not responsible for the imbalance of jobs to graduates. No, I think we’re not responsible. I think we’ve helped prospective students understand what they are getting into than they were previously.”

Of course, the problem isn’t just the flawed rankings methodology itself. Also culpable are the decision-makers who regard a single overall ranking as meaningful — students, deans, university administrators, and trustees. Without their blind deference to a superficially appealing metric, the U.S. News rankings would disappear — just as the U.S. News & World Report print news magazine did years ago.

Cultural obsession

Pervasive throughout society, rankings may be a permanent feature of the legal profession. But it’s worth remembering that they’re relatively new. Before the first U.S. News list of only the top 20 law schools in 1987, prospective students and law schools somehow found each other.

Today, rankings facilitate laziness. The illusory comfort of an unambiguous numerical solution is easier than engaging in critical thought and exercising independent judgment. Forgotten along the way is the computer science maxim “garbage in, garbage out.”

TOO LITTLE; TOO LATE

Posted on by
3

The ABA is thinking about punishing law schools that lie. What courage!

At the front end of the experience, intentionally inflated undergraduate GPAs and LSATs for Villanova’s admitted students led to an ABA censure in August. The school must now employ an independent compliance monitor for two years. Next up in the hot seat: the University of Illinois College of Law. Now, at the back end, the ABA is considering imposing penalties on law schools that misrepresent graduate job placement data.

This one-school-at-a-time approach misses the larger targets. Along with many law schools’ dubious sales tactics, the ABA itself has contributed to the chronic oversupply of lawyers.

Don’t let a recent Wall Street Journal article about the declining number of law school applicants fool you. Excess supply persists. Although total applicants are down ten percent from last year, the number of students starting law school has actually been rising. Meanwhile, the projected growth in new attorney jobs remains far below what’s required to achieve full employment for lawyers hoping to work as lawyers.

In the fall of 2002, first-year enrollment was 48,400. By 2009 — the last year for which the LSAC has published information — it had climbed to 51,600. In other words, demand still exceeds supply. This year’s ten percent applicant drop — to 78,900 — won’t prompt schools to reduce capacity. Rather, it will encourage growth.

And the ABA isn’t stopping them. Between 1970 and 2010, the number of law schools increased from 144 to 200. During the same period, the total number of law students soared from 64,000 to 145,000.

Meanwhile, the Bureau of Labor Statistics estimates that there will be only 98,000 net additional legal jobs for the entire decade ending in 2018. At current enrollments, law schools will produce five times that many graduates; baby boomer retirements won’t bridge that gap.

Last year’s drop in applicants may mean that some recent graduates are giving more thought to whether law school is the right path. That would be great news for them and the profession. Unfortunately, the accreditation of new schools and the growth of existing ones is bad news for many would-be lawyers.

Having facilitated a situation that continues to inflict tragic consequences on many unsuspecting victims, the ABA has avoided leading serious remedial efforts. In light of its recent punt on the requirement that law schools report meaningful information about their graduates’ employment status, its now-contemplated scrutiny of individual schools’ placement statistics rings hollow. To wit: the Wal-Mart greeter with a law degree still counts as employed.

The ABA’s piecemeal approach won’t solve the problem. Most law schools are prisoners of short-term profit-maxizing business models and metrics. That’s why too many resort to half-truths or outright deception to enhance U.S. News rankings, pump up demand, and put tuition-paying butts in classrooms.

Until students understand the deep methodological flaws in the U.S. News rankings, too many deans will continue manipulating them. Independent audit of the data that schools submit would help. But it should be part of a larger strategy: providing better information to prospective law students long before they sit for the LSAT.

The law can be a noble calling, but it’s not for everyone. When those enrolling in law school understand what’s ahead — including the possibility that their dream jobs won’t be there — they make better decisions and the entire profession wins. Here’s the harsh truth that will surprise many recruits: Some deans don’t act with much nobility when it comes to pursuing tuition dollars.

In an 1891 letter to his fiance, Louis Brandeis wrote: “If the broad light of day could be let in upon men’s actions, it would purify them as the sun disinfects.” Twenty years later, he was less optimistic about improving human behavior when he focused instead on practical remedies for misconduct: “Sunlight is said to be the best of disinfectants; electric light the most efficient policeman.”

The ABA isn’t going to start stripping schools of their accreditations, but it can put them under brighter lights. Adding surveillance cameras and a few more cops on the beat wouldn’t hurt, either.

THE US NEWS RANKINGS ARE OUT!

Posted on by
Reply

[UPDATE: This post first appeared on April 16, 2010. On January 1, 2011, Northwestern’s former dean, David Van Zandt, became president of The New School in New York.]

Earlier this week, I spoke with one of my former Northwestern undergraduate students. Headed for a top law school this fall, he surprised me with this remark:

“A lot of my classmates are waiting to send in their law school deposits until the latest US News rankings come out this week.”

Seriously?

Virtually every law school dean has condemned US News’ annual effort to do for law schools what the Am Law 100 has been doing for big firms. Those of you reading my “PUZZLE PIECES” installments know that annual profits-per-partner rankings haven’t brought out the best in us. It’s all part of a larger contemporary phenomenon: the MBA mentality of misguided metrics.

Unfortunately, students aren’t listening to the unanimous chorus of skeptical law school deans. It’s easier to follow the simplistic approach of a lonely outlier, Northwestern’s David Van Zandt: however wrongheaded, metrics matter.

For a decade, he has refused to join colleagues criticizing US News’ fatally flawed methodology. (See, e.g., Brian Leiter’s analysis) A self-styled maverick, Van Zandt insists that ratings are relevant consumer information.

His position proves too much. Not all misinformation should be allowed to pollute decision-makers’ minds. That’s why fraud and misrepresentation causes of action exist. There’s another problem: pandering to the US News criteria distorts law school administrators’ decisions. Once misguided metrics become governing principles, thoughtful reflection disappears. Teaching to the test is easier than creating imaginative lesson plans.

Lately, metrics seem to be foresaking the maverick. In 2009, Northwestern dropped from 9 to 10 in the US News overall standings; this year, it fell to 11.

Rationalizing the decline, Van Zandt says that his innovative programs haven’t gained traction because of “resistance within a conservative profession.” He argues from aneccdotal evidence that the future will vindicate him. Apart from his inconsistency in crediting a positive rating that suits his purposes but discounting it when things breaks badly, some might accuse him of magical thinking.

Is it time for Van Zandt to back away from his isolated defense of the US News listings? Sure, but it won’t happen. In an April 13 Above the Law post, he urges even more rankings, however dubious their value.

In the end, he’s a misguided metrics kind of guy — at least until Northwestern drops again next year. [UPDATE: It did — to 12th, but by the time the news hit, Van Zandt had already left to become president of The New School in New York.]