The Endarkenment at Home:  Benchmarking Academics (guest post by Elijah Millgram)

The Endarkenment at Home: Benchmarking Academics (guest post by Elijah Millgram)


The Great Endarkenment: Philosophy for an Age of Hyperspecialization is a new book by Elijah Millgram (Utah). In the book, Professor Millgram looks at the implications of our becoming, more and more, “a society of specialists” in which “communication across the barriers between the professions and disciplines is our own very pressing problem,” a problem that “threatens not just our more ambitious enterprises but the successful management of our day-to-day lives.”

Since the book sounded fascinating, and its topic clearly relevant to the philosophy profession, I invited Professor Millgram to contribute a series of brief posts* on various ideas from it (the book can be ordered here). His first post is below. Subsequent posts will appear over the next month or so on a weekly basis. Professor Millgram will be keeping an eye on the posts so please do comment or raise questions for discussion.


 The Endarkenment at Home:  Benchmarking Academics
by Elijah Millgram

If you’re an American academic, it’s quite likely that your administration is using a newish application to track your department’s ‘productivity’; at my home institution, data from Academic Analytics’ subscription product is folded into department budgets.

At first glance, the product works more or less as you’d expect. It helps itself to publicly available data on publications, grants and even prizes; it massages the counts into various formats (for instance, rankings of departments in the same field, nationwide); it presents the scores in an easy-on-the-eyes graphic display.

But the more closely you look, the less sense that display makes. I assume most readers of this blog are philosophers, so I’ll use philosophy as my example. And I’ll describe the version of the product I worked with a little while back; the company is adding features even as we speak, some of them in response to the concerns I’m about to describe.

When you’re assessing a philosopher’s publications, journal articles matter, but chapters contributed to edited volumes matter roughly as much; the Academic Analytics database contains the articles, but not the book chapters. In philosophy, which journal an article is placed in matters a good deal; but journal rankings don’t figure into the productivity scoring. Books are counted, but edited volumes aren’t distinguished from single-author monographs, and press placement again doesn’t figure into the scoring.

The display looks impressive and authoritative, but the information it provides is worse than useless: it’s the sort of incomplete that doesn’t even work as a partial guide, and the outputs it produces strike an informed insider as bizarre. During my first encounter with this product, I remember seeing my own institution’s philosophy department ranked nationally right in between Pittsburgh and Purdue.

This is a very practical problem for many of us. As rankings like these come to drive budgets, individual faculty will find themselves faced with professionally perverse incentives: stop publishing book chapters; stop caring about which journals you publish in; take on editing projects and forget about actually writing books.

It’s easy to be cynical, and decide that your Dilbert-like administrators don’t care whether the numbers make sense, as long as they have something to show higher-ups and outsiders to ‘justify’ their decisions. No doubt that’s sometimes true, but in fact there are many academic decision makers who really do want to do the right thing by their colleges and universities.

It’s also easy to be cynical about the provider. But I made contact with the firm, and I’m convinced they’re trying hard to do it right:  they mean well. The philosophically interesting questions turn up when you ask how administrators were put in the position of needing a supply of numbers, and not being able to afford to care whether the numbers mean anything at all. And the philosophically interesting question on the other side of the coin is how it came about that a well-intentioned business, one that genuinely wants to supply a much-needed service to academic administrators, found itself marketing a product that looks like this.

Universities are a microcosm of a society that is ever more highly specialized. The specialists produce work to standards that only that sort of specialist can understand and apply. But specialists in different fields have to be administered by someone who isn’t that kind of specialist.

In our case, he’s a different sort of specialist:  an academic administrator. He can’t himself tell whether the work is any good. And although the different disciplinary standards register, as moral philosophers sometimes say it, incommensurable values, he has to allocate budgets across disciplines, and so he needs numbers he can compare.  So he desperately needs someone to provide assessments.

But that someone—in this case, the firm providing the subscription product—is in the very same position as your dean: its employees aren’t specialized academics, but rather software developers, and they don’t understand, and can’t apply, the standards used to assess the processes and outputs of work in different academic disciplines. And even if they were themselves academics, the company itself would then have the problem your dean has: it wouldn’t be able to tell if its own employees were doing their job right. The craziness in your university’s administration is a symptom of a developing problem that is much broader than academia, and one that no one nowadays knows how to solve.  I’m calling it the Great Endarkenment.

I think that managing disciplinary specialization isn’t just a social problem, but a philosophical problem. However, I’ve gone on about it long enough for now; I’ll post another entry, in a week or so, aimed at explaining why I think so.

There are 9 comments

Your email address will not be published. Required fields are marked *

  
Please enter an e-mail address