A recent survey, “What Courses Should Law Students Take? Harvard’s Largest Employers Weigh In?” by Harvard Law School Professors John Coates, Jesse Fried, and Kathryn Spier, has assumed a life that its sponsors never intended. For example, a recent Wall Street Journal headline implies that the survey provides a roadmap to success: “Want to Excel in Big Law? Master the Balance Sheet.” Likewise, some cite the survey in taking unwarranted shots at proposals to make law school more experiential.

Such misinterpretations of the Harvard survey might spring from a condition that psychologist Stephan Lewandowsky would call motivated reasoning: “the discounting of information or evidence that challenges one’s prior beliefs accompanied by uncritical acceptance of anything that is attitude-consonant.” In other words, people often see what they want to see, even when it isn’t there.

The HLS survey

Harvard sought curriculum input from an important constituency, namely, some big law firms. The questionnaire went to 124 practicing attorneys at the 11 largest employers of Harvard graduates in recent years, including my former firm Kirkland & Ellis. Among the respondents were 52 litigators, 50 corporate/transactional attorneys, and 22 regulatory practitioners. The tiny non-random sample is not even a representative slice of a typical big law firm practice.

Harvard didn’t ask attorneys to identify law school courses that might improve a student’s chances of getting a job. It couldn’t. The 11 firms represented in the survey hire virtually all new associates from their own second-year summer programs. They base those hiring decisions on first-year grades because, at the time they extend offers, there are no other law school grades to consider. Whatever courses students might take in the second or third years make no difference to their big law firm employment prospects.

Harvard also didn’t ask lawyers to identify courses that might help graduates become equity partners. That would be silly because there are no such courses. Even among Harvard graduates, fewer than 15 percent of those who begin their careers as new associates in big firms will become equity partners many years later.

So what did the survey investigate? The questionnaire could have read: You work in one of 11 big firms that serve corporate America. Your firm already hires many Harvard graduates. What courses can we offer that will make those newbies most useful to you when they start work?

Answering only the questions asked

Even within its narrow scope, the HLS questionnaire limited the range of permissible responses. For example, three questions focused exclusively on courses in “business methods” and “business organizations” (“BM” and “BO” — no laughing). Here’s Question #1:

“HLS has a variety of business methods courses that are geared towards students who have little or no exposure to these areas. For each of the following existing HLS classes, please indicate how useful the course would be for an associate to have taken (1 = Not at all Useful; 3 = Somewhat Useful; 5 = Extremely Useful).”

Respondents had to choose from among seven options: accounting and financial reporting, corporate finance, negotiation workshop, business strategy for lawyers, analytical methods for lawyers, leadership in law firms, and statistical analysis/quantitative analysis. Accounting and financial reporting placed first among all responses; corporate finance was second. Big deal.

Likewise, when asked to look beyond the seven business methods choices in identifying useful courses, respondents predictably chose corporations, mergers & acquisitions, and securities regulation as the top three. For decades, those classes have comprised the heart of most second-year students’ schedules. Again, no news here — and no magic formula that produces success in big law.

The options not offered

As for the misguided suggestion that the survey trashes experiential learning, only one survey question asked attorneys to identify the most useful courses outside the business area. Evidence, federal courts, and administrative law topped the list. But respondents didn’t have the option of choosing trial practice or any other experiential course because they didn’t appear on the questionnaire’s multiple-choice list of permissible answers.

So let’s return to some of the headlines about the HLS study.

Does the survey suggest that students taking business-oriented courses will be more likely to get jobs? No.

Does the survey suggest that students will be more likely to succeed — even in big law — if they take more business-oriented courses? No.

Does the HLS survey deal a blow to proponents of experiential learning? No. (In fact, an experiential option — negotiation workshop — did pretty well, placing third out of seven possible responses to Question #1.)

Desperately seeking something

In the end, any effort to overplay the survey collides with the authors’ concise summary: “The most salient result from the survey is that students should learn accounting and financial statement analysis, as well as corporate finance.” For that conclusion, no one needed 124 big law attorneys to complete an online questionnaire.

As the legal profession makes its wrenching transition to whatever is next, perhaps the unwarranted attention to the Harvard survey reflects a measure of desperation among those searching for answers. Motivated reasoning isn’t making that search any easier.