Cognitive Biases and Limitations of Search Committees

Cognitive Biases and Limitations of Search Committees


A philosopher whose last name starts with a letter towards the end of the alphabet writes in:

I wonder how often members of search committees work through alphabetized stacks of dossiers?  I recently had a few conversations with people who have been on search committees, and both mentioned working through an alphabetized stack.

I work in phil cog. sci. and psychology, and some of my work deals with fatigue effects in evaluations—whereby evaluations can become harsher the longer someone has been at it, and overall we just become inefficient once something becomes tedious. It’s why I grade anonymously and try to do it with small batches.

There has certainly been a lot of discussion of how bias can enter into the job market and interviewing, and in many cases it can be very hard to fix.  This is an easy one to flag and fix.

There are other cognitive biases that can enter into the decisions of search committees. Here’s one brief piece issued by the government of Singapore that does a fairly good job of describing them in the context of hiring, including contrast bias, anchoring and adjustment, order effects, confirmation bias, representativeness bias, and others. It also contains some suggestions about standardizing hiring procedures so as to minimize the impact of such biases.

It would be helpful to hear what steps search committees are taking, if any, to take into account the effects of these kinds of biases. At least one department is asking that writing samples be prepared as if for anonymous review. What is your department doing?

Zillah

(image: from Edward Gorey’s The Gashlycrumb Tinies)
guest
14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
P.D.
6 years ago

Another common thing is to consider applications in the order in which they arrived. I am not sure that this is any more or less fair than a random order.Report

Benjamin Yelle
Benjamin Yelle
6 years ago

Dammit.Report

Shane Wilkins
Reply to  Benjamin Yelle
6 years ago

Right there with ya buddy.Report

ejrd
ejrd
6 years ago

An extremely helpful (and timely) reminder. I also love Edward Gorey. This may just be the perfect DN post.Report

metamorphic
metamorphic
6 years ago

I’ve had this issue on search committees and found it to be heightened when searching in other long lists, like searching for someone in a speciality among a list of one hundred or so. When looking for one person to invite for something, you scroll down till you find someone suitable, and start working through that way. It heightens the problem, since instead of searching for the single very best candidate, you search for someone who is quite good enough, of which there may be a number. But you get to the earlier-alphabetical candidates first.

My strategy is to alternate when I go looking: start from A and go down, then next time start from Z and go up, then start from M and go in reverse, then start from M and go forward. Doesn’t compensate entirely, but does markedly improve things. There are many excellent philosophers buried in the end of the alphabet at the bottom of the page!Report

Tom Hurka
Tom Hurka
6 years ago

If several people are reading the same stack, they should read them in different orders, e.g. one in alphabetical and the other in reverse alphabetical order. Then no application gets read late in the day by every reader. I remember being on a selection committee for an academic honour and finding a nominee I thought was terrific had been ignored or ranked low by all the other committee members. The nominee’s name began with “W” and the other members were all worn out and not paying much attention by the time they got that far.Report

Aaron A. Aaronson
Aaron A. Aaronson
6 years ago

I don’t see a problem with this…Report

Mr. Zykowski
Mr. Zykowski
Reply to  Aaron A. Aaronson
6 years ago

I do.Report

"I am the Alpha and the Omega -- the Beginning and End -- the First and the Last."
"I am the Alpha and the Omega -- the Beginning and End -- the First and the Last."
6 years ago

Now you understand where I’m coming from.Report

Matt Zwolinski
6 years ago

A-freakin’-men!Report

Fritz Allhoff
Fritz Allhoff
6 years ago

Our HR site sorts them by order of submission, not last name. It’s actually such a bad system that it’s not even sortABLE by last name, whether A->Z or Z->A. So the cognitive biases would be temporal, not alphabetical.Report

Christopher Hitchcock
6 years ago

My colleague in economics, Leeat Yariv (notice the ‘Y’) has found an alphabetical prestige effect due to the ordering of names on co-authored papers when they are listed alphabetically.
On searches, the applications will be read over a space of weeks by several people. If there is any simple fatigue effect, it is unlikely to systematically affect (say) W’s as opposed to C’s. It might be that on a given day I look at files from ‘Sh’ to ‘Te’. In that case, Dr. Teller is the unlucky one. But it is unlikely to work out that way at all the places Dr. Teller is applying.
Committee members will start looking at applications as soon as there are an appreciable number. So I suspect that if there is any effect of this kind it has more to do with how early the application comes in. E.g., if you are the best candidate from among the first 25 to apply, at least you occupy a salient position, and it seems plausible that you will be more easily remembered than if you are the best among the last 25 to apply (in which case there may already be 10 or 20 who are comparable).Report

Nancy J. Matchett
6 years ago

We’ve asked for supporting materials prepared for anonymous review, with the exception of letters and CVs (largely following the excellent advice provided by Olberding et. al. here: https://c.ymcdn.com/sites/apaonline.site-ym.com/resource/collection/D03EBDAB-82D7-4B28-B897-C050FDC1ACB4/FeminismV13n2.pdf). Since this means that every file is split into two parts, there is an additional, built in randomizing effect regarding the order in which any particular candidate’s materials are read — the set that includes names is filed alphabetically while the set with anonymized materials is filed by the week in which the application arrived (and no particular order within weeks — though this was not intentional, it turns out that files in the anonymized stack have been shuffled quite a bit as they are checked in and out by committee members). Alas, in a TT search, it is very difficult to control for fatigue …Report

harry b
6 years ago

In a large enough discipline (not Philosophy — but, say, Education, or History) couldn’t we check whether this is a serious problem by looking at the last names of all people applying in a particular year and the last names of all people hired in that year?Report