Quality Control, Methodological Bias, and Persistent Disagreement in Philosophy


Recently, mainstream philosophy journals have tended to implement more and more stringent forms of peer review (e.g., from double-anonymous to triple-anonymous), probably in an attempt to prevent editorial decisions that are based on factors other than quality. Against this trend, we propose that journals should relax their standards of acceptance, as well as be less restrictive about whom is to decide what is admitted into the debate.

We start by arguing, partly on the basis of the history of peer review in the journal Mind, that past and current peer review practices attest to partisanship with respect to philosophical approach (at least). Then, we explain that such partisanship conflicts with the standard aims of peer review, and that it is both epistemically and morally problematic. This assessment suggests that, if feasible, journals should treat all available and proposed standards of acceptance in philosophy as epistemically equal, and that philosophical work should be evaluated in terms of the novelty and significance of its contribution to developing thought in ways that are of value. Finally, we show, in a programmatic way, that improving the current situation is feasible, and can be done fairly easily.

That is the abstract of “Pluralism and Peer Review in Philosophy,” by Joel Katzav and Krist Vaesen, forthcoming in Philosophers’ Imprint. It is worth a read.

In arguing that the profession, through its journal editing, adopt a pluralistic approach to philosophical methodology, the authors recognize that their central opponents will be philosophers who think that the mainstream “analytic” Anglo-American philosophical method is superior to alternatives and that their proposals would mean a lowering of philosophical standards. Katzav and Vaesen argue that the distinctiveness of the analytic philosophical method should not be mistaken for its superiority.

They begin by describing how a factor contributing to the dominance of the analytic style in mainstream philosophy was the preferences of journal editors: Moore’s and Ryle’s editorships of Mind are the central examples.

They then note that this mainstream (“m”) style is characterized by what the authors call “m-rigour” and “m-clarity”:

M-rigour predominantly characterises argumentation that aims to uncover the correct answers to substantive philosophical questions [and] argumentation that is adversarial…

The standards of m-clarity can plausibly be partially characterised in terms of the requirement that an author’s work be relatively easily understood by other mainstream philosophers, especially by those having the work as their area of specialisation. The standards of m-clarity also plausibly involve the requirement that concepts, propositions, inferences and larger-than inference level argumentative structures that make up written philosophical work be presented using one of a limited repertoire of standardised language forms; at the most general level, these forms will borrow from symbolic logic and/or regimented, informal reasoning, but they will also borrow from more domain specific forms such as the terminology of recent analytic metaphysics or recent meta-ethics.

Both m-rigour and m-clarity could be elaborated upon but, the authors say:

What matters is that the mainstream philosophy journals we are concerned with are, in employing m-rigour and m-clarity as criteria for assessing submissions, partisan with regard to which approach to philosophy they serve and thus which papers they are willing to publish.

Readers will no doubt differ over whether this sounds like unwarranted partisanship or just the application of reasonable standards. The authors think that a look at the proper functions of peer review will show it is unwarranted partisanship. Peer review has epistemic and non-epistemic functions:

The epistemic role of peer review is assessing the quality of research. From the hundreds of thousands, or even millions, of manuscripts submitted on a yearly basis, it weeds out those which are not sound, significant or novel… The authoritative nature of peer review is widely supposed partly to rest on the fact that work is to be judged on shared standards of excellence and expediency

As regards non-epistemic certification, peer review serves as an evaluative standard in hiring, conferring tenure, promotion and grant committee decisions. Peer-reviewed articles are taken to demonstrate a scholar’s ability to (continue to) contribute to a field, in a way recognized by that field and, at least in comparison to invited contributions, independently from the scholar’s personal and professional networks.

So when Moore, around 1924-1925, changed editorial policies at Mind “so as to favour some of the approaches to philosophy that were still thriving in Britain over others that were doing so,” the authors argue that he was undermining the epistemic role of peer review “because the epistemic standards of evaluation employed during these editorships were not shared among the scholars served by the editorships.”

The problem is not merely historical, argue Katzav and Vaesen:

It is natural to think that the community currently served by, say, Mind, just comprises contemporary, mainstream philosophers. This community does, it might be suggested, share standards for evaluating philosophical work. Thus it might be thought that, although the journal has recently been partisan about how philosophy should be carried out, its peer review practices have been in accord with the standard requirements of peer review.

Such a line of thought is, however, too quick. Mind and other prominent journals collectively play an important role in determining how resources such as jobs, funding and research time are distributed among academics working, and seeking to work, in academic philosophy in the United States of America, the United Kingdom and other countries. Accordingly, the community these journals serve extends well beyond those who accept the standards of mainstream philosophy; it includes, for example, many working in African philosophy, feminist philosophers critical of mainstream philosophy, many continental philosophers and many of those who see themselves as trying to bridge the analytic-continental divide. The community in question also includes those who are not quite a good fit for any particular, widely adopted approach to philosophy. The journals being considered are thus, deliberately or not, partisan about whom they support from among the community of actual and potential academic philosophers in countries they influence. [emphasis added]

What is especially interesting about this paper, in my view, is how Katzav and Vaesen then raise skepticism about the philosophical enterprise (based on the argument from disagreement) to argue against the bias in peer review in favor of mainstream analytic philosophy. This part of the paper resists an easy summary, but here is one excerpt:

Thus far, our discussion recommends responding to the argument from disagreement by accepting that no philosophical approach has been established to be a reliable means of answering substantive philosophical questions and trying nevertheless to legitimate some standards of acceptance in philosophy. We have also suggested that, once the reliability of mainstream philosophy is held to be an open issue, there appears to be no good reason to prefer its standards… 

We do not deny that there is widespread agreement among mainstream philosophers about how to do philosophy. Our point is that, nevertheless, mainstream philosophers who have examined the issue of the reliability of philosophy do not provide agreed grounds for supposing that any available approach to philosophy is reliable…

[W]e argue that, in the absence of intersubjectively agreed grounds for taking available approaches to philosophy to be reliable, there are good epistemic and moral reasons not to prefer any particular available approach to philosophy in peer review.

The authors discuss a variety of objections to their arguments, and claim that review processes in philosophy need to be changed.

Two desiderata for adequate review processes in philosophy journals are suggested by the previous section. The first desideratum is that these processes should treat all available and proposed standards of acceptance in philosophy as epistemically equal (irrespective of who puts them forward). This desideratum is suggested by the thought that pluralism about philosophical approaches makes sense given the lack of an established to be reliable philosophical approach. The second desideratum is that review of philosophical work should include evaluating such work in light of the novelty and significance of its contribution to addressing philosophical questions that we need to address as humans and indeed, more broadly, in light of its contribution to developing thought in ways that are of value. This desideratum is suggested by the observation that, plausibly, addressing questions that we need to address and developing thought comprise an important part of what is of societal value in philosophy apart from its potential ability to answer substantive philosophical questions in a reliable way.

Both of our desiderata clearly recommend that approaches to philosophy according to which it has something other than the correct answers to its questions as a goal—e.g., literary value, suspension of belief, a purely heuristic role in discovery—should be included among the repertoire of approaches to philosophy. The second desideratum also recommends pluralism in approaches to philosophy as the historical record shows that a wide variety of philosophical approaches, including the proposal of novel approaches, have been of value. 

What changes do they recommend? Here are some quotes from the paper illustrating some possibilities (see the paper for details on them):

  1. With regard to fostering pluralism of approaches in philosophy, the easiest option is for journals to adopt editorial policies that are pluralist about approaches to philosophy and, correspondingly, for them to diversify their editorial boards and review committees so that each paper is assessed by standards that match its approach. Mind has, very recently, partly gone down this route.
  2. A further change that retains the basic set-up of the current peer review system would be to lower the bar for acceptance. Journals could set rejection rates at, say, around 60% rather than at above 90%.
  3. Additionally–also easy to implement–journals might disclose, at the time of article publication, relevant review reports (with optional blinding of reviewer names), author responses and editorial decisions.
  4. More inclusive policies regarding who participates in peer review, or even the elimination of peer review. A feasible way of broadening the participation in peer review involves following [a] public reviewing practice [in which] any paper which passes a basic quality check is put online and made available for public review and discussion.
  5. [Editorial bias] could be avoided by allowing authors to curate their own papers. Authors, rather than editors, would thus decide whether and how to revise in light of comments received, when to consider their paper to be final or to retract.

Read the whole thing.

Julie Mehretu, “Sehkmet”

guest
34 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Matt McAdam
Matt McAdam
4 years ago

I was happy to find a more easily accessible copy of the paper at Vaesen’s site here: http://home.ieis.tue.nl/kvaesen/peerreview_ONLINE.pdfReport

Paul Whitfield
Paul Whitfield
4 years ago

I appreciate their thoughts on this issue, and the recommended changes, especially 3 and 4, sound amazing.Report

Arthur Greeves
Arthur Greeves
4 years ago

I don’t oppose all of their practical suggestions, but the notion that we should “treat all available and proposed standards of acceptance in philosophy as epistemically equal” is complete nonsense. It certainly doesn’t belong to the discipline of philosophy.Report

Arthur Greeves
Arthur Greeves
4 years ago

What if my proposed standard is “papers that include references to John Stamos should be published”?Report

Mason Kesha Georgie
Mason Kesha Georgie
4 years ago

Mainstream standards are heavily biased against belief in extra-sensory perception. I don’t know how many philosophers currently disagree with this standard, but I hope to increase their number until they’re as numerous as those who think Tuvel’s paper shouldn’t have been published because of the identities of the authors cited in it.Report

Critilo
Critilo
Reply to  Mason Kesha Georgie
4 years ago

Proportioning assent to testable claims with the evidence currently amassed for those claims–in the case of ESP, none–is not bias. No purported evidence for any specific claims on ESP’s behalf, including that published in peer-reviewed journals, has yet passed muster under scrutiny, and the overwhelming majority of it has been systematically and repeatedly falsified.

http://www.skepdic.com/esp.html

What strong evidence do you feel there is, and how (in specific terms please) has it been ignored, downplayed, or otherwise been the object of bias?Report

Arthur Greeves
Arthur Greeves
Reply to  Critilo
4 years ago

“Proportioning assent to testable claims with the evidence currently amassed for those claims” is ruled out by the proposal in the original post, which explains that all proposed standards should be considered “epistemically equal”.Report

Daniel
Daniel
4 years ago

I’m pretty sure you just failed to recognize trolling, critilo.Report

Critilo
Critilo
Reply to  Daniel
4 years ago

Given Poe’s law, I confess my jaded indifference to the personal motivation behind most Internet comments. If a poster seems genuine, sifting minutiae and weighing sincerity are generally less important to me than the simple fact that, for every concern troll or jester who posts arguments with disingenuous intent, there is a non-trivial number of people who actually believe what the troll or jester pretends to believe. I’d rather address the substance of a troll’s post and end up rolling my eyes at a trite “Gotcha!” than look for ulterior motives under the bed every time I read a comment that isn’t overtly sarcastic or comical.Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Critilo
4 years ago

I don’t think you were being trolled at all. MSG is being sarcastic, but is making a genuine point.Report

Hey Nonny Mouse
Hey Nonny Mouse
4 years ago

The issue is a very difficult one. Just to make it more difficult, on what grounds if any do we give preference to approaches that are accepted by large groups of philosophers instead of tiny cliques, or just one? On what grounds do we give more weight to the opinions of full professors over what is acceptable over the opinions of assistant professors? On what grounds do we give more weight to supposed “experts” in a field, given that they are no more able to produce definitive results than anyone else? Perhaps most problematically, on what grounds to we give more weight to approaches used inside the academy than to approaches used by those who are not professional philosophers. Standards used for, say, moral and political arguments on Fox.com are very different from those we generally use?Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Hey Nonny Mouse
4 years ago

Bad grammar and typo’s throughout. Sorry. Need more coffee before public speculation.Report

Cautious Until Tenured
Cautious Until Tenured
4 years ago

I think this is an important discussion to be having. That said, I’m an unrepentant defender of the mainstream approach.

The disagreement objection is interesting, but I’m not sure that it is, ultimately, convincing, since it assumes that the goal of philosophical inquiry is to reach some consensus on philosophical questions. But while we may HOPE for a convergence of this kind, I think we lost any right to EXPECT convergence around the time of Hume and Kant.

Russell was correct both in thinking that the value of philosophy lies in its ability to expand our conception of what is possible, and that we can gain a kind of genuine knowledge through the process of conceptual analysis. We use thought experiments and propose principles to test relations between our concepts, and find which combinations are impossible, and which are not.

If THIS is the goal of philosophy, then I consider contemporary philosophy to be a rousing success. There is a vast explosion of theories that have done a wonderful job of, collectively, expanding our conceptions of what is possible, and testing the limits of those possibilities as well. This has been enabled by contemporary methodology, with its use of rigorous and careful argument. This style of philosophy allows us to draw distinctions, and it is these distinctions that allow us to do the fine-grained work of conceptual investigation. Without distinctions, thought is muddled and conceptual investigations remain hopeless.Report

Professor Schneider
Professor Schneider
4 years ago

While the motivations of the authors’ approach are laudable, I think many of the kinds of worries noted could be assuaged by going in entirely the opposite direction. Rather than to ease up the acceptance rate in journals, we should only have ONE journal. (Hear me out). The One Journal will have an acceptance rate of, say, .001%, and the referees will be (in each case) over 100 of the leading scholars in the field. Everything that is not accepted in the One Journal can of course be posted to authors’ individual websites. Papers that are accepted in the One Journal will have been peer-reviewed by just about everybody who is relevant in the field! (Perhaps this could be up to 500 people). There could be a cut-off system in play, of course, so that not every paper gets read by 100+ people. However, all *accepted* papers must pass a threshold where some majority of 100+ leading scholars recommend publication. This would fix several problems. For one thing, there’s less scope to complain that a paper’s acceptance was due to a few biased referees. After all, effectively everyone will have had a say. Secondly, there will be less crap published, and it will be easier to stay on top of the literature (now, it’s impossible to read everything). Who will be the editors of the One Journal? Simple: all the editors of present journals vote democratically to make editorial decisions at the One Journal.Report

Greg Gauthier
Reply to  Professor Schneider
4 years ago

One Journal to rule them all, One Journal to find them,
One Journal to bring them all and in the darkness bind them.Report

Untenured Ethics Professor
Untenured Ethics Professor
4 years ago

A. Did Moore exclude British idealism from Mind because he was unfairly prejudiced against the view or because defenses of British idealism were opaque or poorly reasoned? Pointing out that there were many British idealists in academia at the time Moore was the editor of mind does not show that Moore’s decisions not to publish their work were misguided or “partisan.” Instead, he may have seen malarkey for what it is.

B. Though philosophers may not have decisively solved any of the Big Questions, there are approaches to some Big Questions that have been decisively ruled out. As Gutting points out, there are conceptual distinctions we know to be valid or useful.

C. At a time when philosophy is institutionally under attack, advancing the false claim that philosophy has no accepted standards is counter-productive.Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Untenured Ethics Professor
4 years ago

What standards would you say are universal among philosophers?Report

Untenured Ethics Professor
Untenured Ethics Professor
Reply to  Hey Nonny Mouse
4 years ago

When this rubric for undergraduate papers was posted a few weeks ago, there was a lot of discussion about whether rubrics for undergraduates are pedagogically useful and whether the criteria in this rubric are weighted correctly. Some readers pointed out that the rubric sets the bar for originality too high for an undergraduate paper. But aside from the point about originality, there was no debate about the whether the listed criteria for an excellent paper are indeed criteria that an excellent undergraduate paper would meet.

The criteria for good undergraduate philosophy papers are also criteria for good professional work in philosophy. For professionals, of course, it is appropriate to demand originality. It is true that some widely-admired older philosophical works do not meet the standards of clarity we now demand of our students and of each other. These philosophical works are admired despite their opacity, not because of it. The demand for greater clarity in contemporary philosophy is an improvement in the field.

http://dailynous.com/2017/05/24/impressively-detailed-philosophy-paper-grading-rubric/Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Untenured Ethics Professor
4 years ago

I dig those standards very much, but they are definitely not universal. For instance, not everyone thinks that an undergraduate paper must consider counter-arguments to the thesis.Report

Untenured Ethics Professor
Untenured Ethics Professor
Reply to  Hey Nonny Mouse
4 years ago

Would you agree that a professional philosophy paper should consider obvious counter-arguments to its thesis? Is there serious controversy about this?Report

serious controversy
serious controversy
Reply to  Untenured Ethics Professor
4 years ago

Bernard Williams: “In a way that will be familiar to any reader of analytic philosophy, and is only too familiar to all of us who perpetrate it, this style tries to remove in advance every conceivable misunderstanding or misinterpretation or objection, including those that would occur only to the malicious or the clinically literal-minded. This activity itself is often rather mournfully equated with the boasted clarity and rigour of analytic philosophy. Now, it is perfectly reasonable that the author should consider the objections and possible misunderstandings, or at least quite a lot of them; the odd thing is that he or she should
put them into the text. One might hope that the objections and possible misunderstandings could be considered and no doubt influence the text, and then, except for the most significant, they could be removed, like the scaffolding that shapes a building but does not require you after the
building is finished to climb through it in order to gain access.”Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Untenured Ethics Professor
4 years ago

Yes, I agree. Yes, this is controversial. Indeed, I tried and failed to have a requirement introduced for our senior assignment paper that students must consider objections to their thesis.Report

Untenured Ethics Professor
Untenured Ethics Professor
Reply to  Untenured Ethics Professor
4 years ago

Let me rephrase the requirement: if there are obvious objections to an argument (i.e. objections that a reasonably charitable reader is likely to have and to which good replies are not obvious), the objections should be discussed in the text.

The Williams quotation is consistent with upholding this requirement.Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Untenured Ethics Professor
4 years ago

I do not know what those who see no need for objections would say about “obvious objections”. However, “obvious” is so subjective that I doubt it can serve as a standard. Objections that seem obvious to me are not obvious at all to some other philosophers. Certainly, some philosophers either don’t think that obvious objections need to be considered in student papers, or routinely do not see obvious objections to the arguments presented by students for positions that they (the philosopher) themselves support.Report

Untenured Ethics Professor
Untenured Ethics Professor
Reply to  Untenured Ethics Professor
4 years ago

Some standards are vague. Some standards are somewhat subjective in their application. They are, nonetheless, standards. We can agree what the standards are without always agreeing about how they should apply.

One could argue that when there is significant disagreement about how to apply our standards, editors and reviewers should always apply those standards in the most lax way possible. But one would need to make an argument for this position. It’s not obviously correct.Report

Aaron V Garrett
Aaron V Garrett
Reply to  Untenured Ethics Professor
4 years ago

Regarding A, if Moore excluded defenses of British Idealism without inspecting the particular arguments, then yes he was prejudiced and perhaps destructively so. A parallel case can be seen in his attack on Westermarck. Few philosophers today would think that the kind of naturalist emotivism that Westermarck argued for was a non-starter for the reasons that Moore gave. Moore’s arguments seem far more questionable. Westermarck did not commit the “howlers” that Moore ascribed to him. Today the sort of position that Westermarck argues for flourishes, although very few philosophers know who he is and though there are many weaknesses in his arguments. But the position was essentially mocked into exile by Moore. I think that’s terrible for philosophy — although this is a different point to the one made above.Report

Greg Gauthier
4 years ago

Being on the outside looking in, I never quite understood why the review processes or methodological approaches or ideological orientations of any particular journal would matter so much to anyone. If you don’t think Mind has the right approach, submit your paper to a journal that does take an approach you approve of. If there is none, then *start* one. Publishing is easier than ever, these days.Report

Arthur Greeves
Arthur Greeves
Reply to  Greg Gauthier
4 years ago

Thus revealing that the goal here is not to “provide a venue” for different approaches, but rather to co-opt the prestige of already existing journals in the service of ideals these journals have not previously espoused.Report

Greg Gauthier
Reply to  Arthur Greeves
4 years ago

So, you view this more as a sort of political conquest, than an attempt to plea for access to a limited resource?

Or, maybe its both. political conquests have often been driven by access to limited resources. But if the barrier to entry to publishing is so low, and getting published is so cheap and plentiful these days, what would be the resource being conquered? Is it really “prestige”? And what would that mean? Isn’t it really, some kind of institutional power that the invaders think they’ll be gaining? Are they right to think that this is what they’ll gain? I mean, something like “prestige” or respect has to be earned or laboured for, in some way. It’s not like coal or forested lands. That prestige, presumably, is the fount of any institutional power these journals presumably have. But if that’s the case, then political conquest would literally evaporate whatever gains these people might think they’re achieving. In other words, who would take Mind seriously any more, after such a coup? Would not the journal eventually just become an echo chamber full of conquistadors, high-fiving each other for their “wins”?

If they realize this, then perhaps the goal is not actually to co-opt the prestige, but to destroy it. If they don’t realize this, then I’m sort of back where I started. Either way, I’m not at all sure what the intentions are, but it doesn’t seem like a good idea to me.Report

Hey Nonny Mouse
Hey Nonny Mouse
Reply to  Arthur Greeves
4 years ago

I don’t buy the idea that we should “treat all available and proposed standards of acceptance in philosophy as epistemically equal” any more than you do. But I don’t think that ulterior political motives are required for holding that position. A lot of people read Mind (and rightly so), which makes it a good venue for philosophical discussion. Compare this to the Tuval case. Many people criticized Tuval’s critics for saying that, according to their standards, her paper should not have been published in Hypatia. By why does it matter that the paper be allowed to be published in Hypatia when Tuval could just start her own journal. Well, Hypatia is read by a lot of philosophers, which makes it a good venue for philosophical discussion.Report

Arthur Greeves
Arthur Greeves
Reply to  Hey Nonny Mouse
4 years ago

“But I don’t think that ulterior political motives are required for holding that position.”

I’m using the principle of charity here. I cannot think of any NON-political reasons to hold a position like that except reasons that make the holder of the position seem positively anti-rational. I prefer to think that the holder of such a position is rational, but is using her rationality in service to a subjectively inspiring political idea.Report

Greg Gauthier
Reply to  Hey Nonny Mouse
4 years ago

“…Well, Hypatia is read by a lot of philosophers, which makes it a good venue for philosophical discussion…”

This is a rather static view of the market for intellectual commerce. If she started her own journal (perhaps in cooperation with others), perhaps it would become more widely read than Hypatia. Hypatia itself didn’t exist until 1982, when a group of women saw a need for a journal in which female philosophers were “relatively safe from a critical point of view” (https://vimeo.com/13360820).

Perhaps the authors of this paper critical of Mind, need to take the same entrepreneurial approach.Report

Eric
Eric
4 years ago

It is a bit shocking to hear that some philosophy instructors do not believe that consideration of and response to objections or counter-arguments is an essential component (and thus should be a requirement) of undergraduate philosophy papers, or argumentative essays in general (allowing exceptions if the “paper” assignment is merely expository). The reasons for such a requirement, and the fact that they’re dispositive, should be obvious to any philosopher, or indeed any intellectually-minded person.Report

Nichole
Nichole
4 years ago

I think Doron Zeilberger is onto something with his idea to more or less get rid of journals. (His opinion 156 expresses this.)

With online publication and retrieval being so common anyway, the journals serve primarily as gatekeeper anyway. Why not have people find their own reviewers who would then put their name on it as reviewer when they think it’s good enough? E.g. Argle writes a paper and revises and revises and asks Bargle to take a look. When Bargle thinks it’s good enough, Argle puts it online including the information that Bargle approves of it. Argle may also ask Carlyle and Donna to look. Carlyle might say he won’t put his name on it without a revision, and Donna might refuse to be associated with it at all.Report