I very much doubt that I would be able to provide anything like reliable judgments of philosophical quality based on the names of individuals in faculties, without spending an enormous amount of time reading people’s work. Although I’ve been in professional philosophy for nearly ten years, and have gained at least some familiarity with a large number of philosophers, I think I know next to nothing about a large majority of the philosophers whose departments you’d be asking me to rate. At a minimum, I might be able to find and skim the CV of every member of a department in half an hour or so, but to employ any of my own philosophical skill to give anything like an expert opinion, I’d have to read and engage with people’s work. Since we’re talking about roughly a hundred departments, this represents a daunting task to say the least. (Even just skimming CVs, at 30 minutes per department, would take some 50 hours.) Or I could skip most departments, limiting my attention to those containing my friends and those colleagues I’ve interacted with enough to have an opinion about already; but I’d worry about selection bias, and even those departments are made up mostly of people I don’t know. I’m just not comfortable contributing a meaningful opinion about the quality of someone’s work without spending orders of magnitude more time than I could offer if I wanted to.
So writes Jonathan Jenkins Ichikawa (UBC) in his response to an invitation from Berit Brogaard (Miami) to be an evaluator for the Philosophical Gourmet Report (PGR), posted at NewApps. Invitations to evaluate departments for the PGR went out earlier today, according to a post by Brian Leiter at Leiter Reports. A few hundred philosophers are supposed to have received them. Perhaps some of the invitees would care to comment on the epistemic demands of the task.
UPDATE: (10/30/14): Eric Schwitzgebel explains “Why I Will Be Contributing Rankings to the Gourmet Report.”