Cognitive Biases and Limitations of Search Committees
A philosopher whose last name starts with a letter towards the end of the alphabet writes in:
I work in phil cog. sci. and psychology, and some of my work deals with fatigue effects in evaluations—whereby evaluations can become harsher the longer someone has been at it, and overall we just become inefficient once something becomes tedious. It’s why I grade anonymously and try to do it with small batches.
There has certainly been a lot of discussion of how bias can enter into the job market and interviewing, and in many cases it can be very hard to fix. This is an easy one to flag and fix.
There are other cognitive biases that can enter into the decisions of search committees. Here’s one brief piece issued by the government of Singapore that does a fairly good job of describing them in the context of hiring, including contrast bias, anchoring and adjustment, order effects, confirmation bias, representativeness bias, and others. It also contains some suggestions about standardizing hiring procedures so as to minimize the impact of such biases.
It would be helpful to hear what steps search committees are taking, if any, to take into account the effects of these kinds of biases. At least one department is asking that writing samples be prepared as if for anonymous review. What is your department doing?
Another common thing is to consider applications in the order in which they arrived. I am not sure that this is any more or less fair than a random order.
Dammit.
Right there with ya buddy.
An extremely helpful (and timely) reminder. I also love Edward Gorey. This may just be the perfect DN post.
I’ve had this issue on search committees and found it to be heightened when searching in other long lists, like searching for someone in a speciality among a list of one hundred or so. When looking for one person to invite for something, you scroll down till you find someone suitable, and start working through that way. It heightens the problem, since instead of searching for the single very best candidate, you search for someone who is quite good enough, of which there may be a number. But you get to the earlier-alphabetical candidates first.
My strategy is to alternate when I go looking: start from A and go down, then next time start from Z and go up, then start from M and go in reverse, then start from M and go forward. Doesn’t compensate entirely, but does markedly improve things. There are many excellent philosophers buried in the end of the alphabet at the bottom of the page!
If several people are reading the same stack, they should read them in different orders, e.g. one in alphabetical and the other in reverse alphabetical order. Then no application gets read late in the day by every reader. I remember being on a selection committee for an academic honour and finding a nominee I thought was terrific had been ignored or ranked low by all the other committee members. The nominee’s name began with “W” and the other members were all worn out and not paying much attention by the time they got that far.
I don’t see a problem with this…
I do.
Now you understand where I’m coming from.
A-freakin’-men!
Our HR site sorts them by order of submission, not last name. It’s actually such a bad system that it’s not even sortABLE by last name, whether A->Z or Z->A. So the cognitive biases would be temporal, not alphabetical.
My colleague in economics, Leeat Yariv (notice the ‘Y’) has found an alphabetical prestige effect due to the ordering of names on co-authored papers when they are listed alphabetically.
On searches, the applications will be read over a space of weeks by several people. If there is any simple fatigue effect, it is unlikely to systematically affect (say) W’s as opposed to C’s. It might be that on a given day I look at files from ‘Sh’ to ‘Te’. In that case, Dr. Teller is the unlucky one. But it is unlikely to work out that way at all the places Dr. Teller is applying.
Committee members will start looking at applications as soon as there are an appreciable number. So I suspect that if there is any effect of this kind it has more to do with how early the application comes in. E.g., if you are the best candidate from among the first 25 to apply, at least you occupy a salient position, and it seems plausible that you will be more easily remembered than if you are the best among the last 25 to apply (in which case there may already be 10 or 20 who are comparable).
We’ve asked for supporting materials prepared for anonymous review, with the exception of letters and CVs (largely following the excellent advice provided by Olberding et. al. here: https://c.ymcdn.com/sites/apaonline.site-ym.com/resource/collection/D03EBDAB-82D7-4B28-B897-C050FDC1ACB4/FeminismV13n2.pdf). Since this means that every file is split into two parts, there is an additional, built in randomizing effect regarding the order in which any particular candidate’s materials are read — the set that includes names is filed alphabetically while the set with anonymized materials is filed by the week in which the application arrived (and no particular order within weeks — though this was not intentional, it turns out that files in the anonymized stack have been shuffled quite a bit as they are checked in and out by committee members). Alas, in a TT search, it is very difficult to control for fatigue …
In a large enough discipline (not Philosophy — but, say, Education, or History) couldn’t we check whether this is a serious problem by looking at the last names of all people applying in a particular year and the last names of all people hired in that year?