The Endarkenment at Home:  Benchmarking Academics (guest post by Elijah Millgram)

The Endarkenment at Home: Benchmarking Academics (guest post by Elijah Millgram)


The Great Endarkenment: Philosophy for an Age of Hyperspecialization is a new book by Elijah Millgram (Utah). In the book, Professor Millgram looks at the implications of our becoming, more and more, “a society of specialists” in which “communication across the barriers between the professions and disciplines is our own very pressing problem,” a problem that “threatens not just our more ambitious enterprises but the successful management of our day-to-day lives.”

Since the book sounded fascinating, and its topic clearly relevant to the philosophy profession, I invited Professor Millgram to contribute a series of brief posts* on various ideas from it (the book can be ordered here). His first post is below. Subsequent posts will appear over the next month or so on a weekly basis. Professor Millgram will be keeping an eye on the posts so please do comment or raise questions for discussion.


 The Endarkenment at Home:  Benchmarking Academics
by Elijah Millgram

If you’re an American academic, it’s quite likely that your administration is using a newish application to track your department’s ‘productivity’; at my home institution, data from Academic Analytics’ subscription product is folded into department budgets.

At first glance, the product works more or less as you’d expect. It helps itself to publicly available data on publications, grants and even prizes; it massages the counts into various formats (for instance, rankings of departments in the same field, nationwide); it presents the scores in an easy-on-the-eyes graphic display.

But the more closely you look, the less sense that display makes. I assume most readers of this blog are philosophers, so I’ll use philosophy as my example. And I’ll describe the version of the product I worked with a little while back; the company is adding features even as we speak, some of them in response to the concerns I’m about to describe.

When you’re assessing a philosopher’s publications, journal articles matter, but chapters contributed to edited volumes matter roughly as much; the Academic Analytics database contains the articles, but not the book chapters. In philosophy, which journal an article is placed in matters a good deal; but journal rankings don’t figure into the productivity scoring. Books are counted, but edited volumes aren’t distinguished from single-author monographs, and press placement again doesn’t figure into the scoring.

The display looks impressive and authoritative, but the information it provides is worse than useless: it’s the sort of incomplete that doesn’t even work as a partial guide, and the outputs it produces strike an informed insider as bizarre. During my first encounter with this product, I remember seeing my own institution’s philosophy department ranked nationally right in between Pittsburgh and Purdue.

This is a very practical problem for many of us. As rankings like these come to drive budgets, individual faculty will find themselves faced with professionally perverse incentives: stop publishing book chapters; stop caring about which journals you publish in; take on editing projects and forget about actually writing books.

It’s easy to be cynical, and decide that your Dilbert-like administrators don’t care whether the numbers make sense, as long as they have something to show higher-ups and outsiders to ‘justify’ their decisions. No doubt that’s sometimes true, but in fact there are many academic decision makers who really do want to do the right thing by their colleges and universities.

It’s also easy to be cynical about the provider. But I made contact with the firm, and I’m convinced they’re trying hard to do it right:  they mean well. The philosophically interesting questions turn up when you ask how administrators were put in the position of needing a supply of numbers, and not being able to afford to care whether the numbers mean anything at all. And the philosophically interesting question on the other side of the coin is how it came about that a well-intentioned business, one that genuinely wants to supply a much-needed service to academic administrators, found itself marketing a product that looks like this.

Universities are a microcosm of a society that is ever more highly specialized. The specialists produce work to standards that only that sort of specialist can understand and apply. But specialists in different fields have to be administered by someone who isn’t that kind of specialist.

In our case, he’s a different sort of specialist:  an academic administrator. He can’t himself tell whether the work is any good. And although the different disciplinary standards register, as moral philosophers sometimes say it, incommensurable values, he has to allocate budgets across disciplines, and so he needs numbers he can compare.  So he desperately needs someone to provide assessments.

But that someone—in this case, the firm providing the subscription product—is in the very same position as your dean: its employees aren’t specialized academics, but rather software developers, and they don’t understand, and can’t apply, the standards used to assess the processes and outputs of work in different academic disciplines. And even if they were themselves academics, the company itself would then have the problem your dean has: it wouldn’t be able to tell if its own employees were doing their job right. The craziness in your university’s administration is a symptom of a developing problem that is much broader than academia, and one that no one nowadays knows how to solve.  I’m calling it the Great Endarkenment.

I think that managing disciplinary specialization isn’t just a social problem, but a philosophical problem. However, I’ve gone on about it long enough for now; I’ll post another entry, in a week or so, aimed at explaining why I think so.

guest
9 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Associate Professor
Associate Professor
6 years ago

True, outsiders and administrators sometimes have trouble assessing the quality of academic work. However, it is hard to tell whether this is just a technical problem or an “in principle” problem. I don’t see why it isn’t just the former (or though maybe I shouldn’t say “just”). Couldn’t Academic Analytics consult with a bunch of philosophers to get the information they need (some of which you describe) to supplement their algorithms?Report

Lisa H
Lisa H
6 years ago

The book sounds fascinating! Concerning „Academic Analytics“, however, shouldn’t we consider the possibility that such a company should not exist in the first place? Should we help it to get better at what it does, or should we try to convince our departments that a) the vast majority of academics are self-motivated and really try to do the best research they can, b) insofar as there are good reasons to measure and compare output, this requires judgment that cannot be replaced by algorithms, however finegrained? The hyperspecialization we live in pushes towards quantification whenever possible (for the reasons mentioned), but might there be places where we should resist it?Report

Ram Neta
6 years ago

If Academic Analytics could consult with philosophers in order to determine what information they need to bake into their algorithm to make it issue reliable verdicts, then could we outsource all tenure and promotion refereeing to Academic Analytics also? That would save tenure and promotion referees a bunch of time.Report

Griff
Griff
6 years ago

My department had the choice to be counted among the departments at our university who get assessed and ranked according to such Academic Analytics. Although this may have raised our profile (some), the faculty resisted – on the grounds of worries about pigeonholing and overspecialization. The worry was that even IF high-ranking journals are distinguished from articles in edited volumes and editing a volume is distinguished from authoring a monograph, that scholars would be encouraged to focus on one and only one topic, since it is easier to publish on a topic once you’ve published on it before. They were worried it would stifle academic creativity and lead to our scholars failing to branch out for fear of rejection (by journals, etc.). I applaud the choice.Report

PeterJ
6 years ago

It seems a good example of the way in which the method of measurement can alter the thing being measured. In business this system would be called ‘Management by Objectives’. If the system is set up badly the entire company can be destroyed. Perhaps the banking crash would be an example. Too much focus on quantity and efficiency and little on quality and effectiveness since these are more difficult to measure.Report

AnonProf
AnonProf
6 years ago

“The specialists produce work to standards that only that sort of specialist can understand and apply. But specialists in different fields have to be administered by someone who isn’t that kind of specialist.
In our case, he’s a different sort of specialist: an academic administrator.”

1) this “different sort of specialist” doesn’t seem to be very special. If someone specializes in academic administration and and doesn’t administrate academics properly, is that really a specialization?
2) They can’t understand and apply our standards. Does this work the other way. Do you really think you understand and can apply the standards of academic administrators?
3) Who administrates the administrators?Report

UofUundergraduate
6 years ago

I recently took a seminar on Wittgenstein––and, to be honest with everyone reading this, I did not understand him. The class was probably the most rewarding class I have ever taken, even if I understood very little. The professor I took it from really intimidated me (and told me to stop calling him “professor”), but I never doubted his good intentions (and continued to call him professor out of respect). He worked harder on my homework than I did and understood the class better than the class itself. I think he probably would be able to answer these questions better than me.

Thus, with that said, I cannot speak to the problem from the level of my professor, but maybe I can ask a few questions as an undergraduate would. Also, thank you for the book link––I just ordered it. I am going to lend it to a few of my former classmates after finishing it. I hope something good comes of it.

Here is my open question to anyone:

What do the readers think of this statement: “the craziness in your university’s administration is a symptom of a developing problem that is much broader than academia, and one that no one nowadays knows how to solve.” Is this the case? If so, how do we began our philosophical investigations? I am still struggling with that. Maybe if we could some more light on that, we’d be able to see the problem.Report

Elijah Millgram
6 years ago

Hi Ram; I can’t quite tell if that was tongue in cheek, though I sympathize about what’s happened to the promotion/tenure process; there’s some discussion of that in the book also, in the concluding chapter. For what it’s worth, and taking the suggestion at face value, my sense is that the tool is primarily intended to give administrators snapshots of academic units (departments, colleges, etc.), rather than to assess individuals; if what you care about is roughly how well a department is doing, it’s not such a big deal if a publication here or there slips through the cracks. Right now, that would be a problem if you were using it on individual faculty members.

I hadn’t thought of the incentives Griff mentions; that’s interesting. If there was a summary of the reasoning, or the back and forth, produced at the time, it would be great to have a look at it.

Responding to AnonProf, it does seem to me that whether academic administrators are specialists and whether they’re effective are different questions. (In the Middle Ages, physicians were specialists, working to disciplinary standards that they had internalized and that outsiders did not understand, even though, in retrospect, we don’t think they were very effective.) And speaking for myself, as an outsider, I don’t feel like I’m on top of the standards that many academic administrators seem to live by.Report

Elijah Millgram
6 years ago

And further following up on the comments from Lisa H and Associate Professor, I’m very sympathetic to the idea that the institutional management path we’re now going down is the wrong one. Here’s one reason why. In The Great Endarkenment, I argue that each successive attempt to solve the problem has just produced another instance of it.
To give you a sense of how that runs, here’s the iteration Associate Professor looks like he has in mind: Academic Analytics is run by well-intentioned people, and they’ve had a few sit-downs with yours truly. But now, (i) how do they know that what they’re hearing from me is a fair representation of the discipline’s internal standards and guidelines? You know, some people think I have pretty nonstandard views about some things. And (ii) there’s a limit to how much nuance a specialist can convey to a nonspecialist, as I’ve noticed in these meetings. E.g., I can suggest they start harvesting book chapters, but I’ve had a much harder time trying to articulate what’s conveyed by *which* edited volume such a chapter is placed in.
Over and above the argument that’s supposed to explain why the problem is hard-in-principle, that’s what’s done most to convince me that it is: we haven’t been able to make it go away. If the problem was easy or ‘merely technical,’ wouldn’t one of those previous attempts have solved it, or at least made more headway than we’ve seen?Report