Wednesday, April 11, 2007

The Word You're Looking For Is "Cherry-Picked"

Thomas Toch and Kevin Carey complain that "it's an open secret that many of our colleges and universities aren't challenging their students academically or doing a good job of teaching them." I think the Education Sector folks overplay this particular canard in general, but the specific evidence offered in this editorial I find especially unimpressive:
In the latest findings from the National Survey of Student Engagement, about 30 percent of college students reported being assigned to read four or fewer books in their entire senior year, while nearly half (48 percent) of seniors were assigned to write no papers of 20 pages or more.
That report is here. I'm not sure what the big deal is about half of seniors not being required to write 20 page reports. Obviously, it's harder to sustain one's topical focus over 20 pages than over 2 pages, but it's not as though the great majority of college graduates go on to write lots of 20 page reports in the work force either. Besides, if you scroll down just one more page in that same report, you find that 91% of seniors were required to write at least one report of 5-19 pages in length.

For that matter, I don't quite get the big deal about the number of assigned books, either. I think if Toch and Carey had broken the numbers down by academic major, I think it would be somewhat clearer what was going on. So, for example, even though 27% of seniors were assigned 4 or fewer books, engineering majors were twice as likely to fit that description (38%) as social sciences majors (18%). It's not obvious to me that these numbers are evidence of some ominous scandal.

If the objection is just supposed to be that kids in college aren't being adequately challenged, that might very well be the case, but it's certainly not obvious to me that the NSSE report as a whole backs up that claim. Some results that go conspicuously unmentioned by Toch & Carey:
  • When asked to judge the extent to which their exams had "challenged you to do your best work" on a scale of 1-7, with 7 being the most challenging, 81% of seniors answered with a 5 or higher, and very nearly half (49%) responded with a 6 or a 7.

  • 66% of respondents said that over the course of their senior year they "often" or "very often" "learned something that changed the way you understand an issue or concept". Only 3% of seniors said that never happened.

  • One-third had had a "culminating senior experience" like writing a thesis, and roughly another third were planning to do so.

All in all, there's just not a lot here to suggest that seniors weren't busy and challenged. So no, I don't think that Kevin Carey and Tom Toch punk American higher ed this time around. I think they do kind of a lame job, really.

Update: And way to feed the right-wing nonsense machine, guys.


Rebecca C. Brown said...

I think the underlying problem with 50% of bad journalism is that doom-and-gloom sells better than the truth. How could they fill their pages with reports that college students are working hard, learning critical thinking skills, and being challenged? What self-respecting mother of a high school senior would get up in arms about that?

I also like your break-down of why those measurements of the underlying construct of "being challenged" (e.g., papers over 20 pages, etc.) are stupid and misleading.

Paul said...

I think that the biggest factor that lead to this particular piece's badness is that Toch and Carey are high-ups in a think tank the purpose of which is education reform. I happen to like a lot of what they do. At the same time, though, the more problems they can "find" with education, the more important the Education Sector is going to appear.

Rebecca C. Brown said...

Though I'm probably guilty of this very transgression, I don't think it's appropriate to exaggerate and cherry-pick in order to incite change. Or, rather, it's okay to selectively cite the most shock-worthy facts when supporting an overall truth ("education is in trouble"), but not to the ends that you fabricate a false conclusion ("college students aren't being challenged").