Competent Referees for Controversial Ideas
“I think Jeff McMahan is in for the surprise of his life when the submissions to his new journal start coming in.”
I recently thought of that message, which a friend (like me, an admirer of McMahan’s) sent me several years ago when the Journal of Controversial Ideas (JCI) was first announced.
I feel for McMahan and his co-editors of the journal, Peter Singer and Francesca Minerva. They’re providing what they take to be an important service to the world—as they put it in a recent editorial, “to help to protect and preserve academic freedom, as well as freedom of thought and expression in general”—but they have to work with the materials others send them, and it’s a mixed bag: some interesting philosophy on important topics that could have been published in a range of very good “mainstream” journals, and then things like some white guy arguing that it should be okay for him to say the n-word.
The journal was brought back to my attention by a reader who noted a recent exchange in the journal on the topic of the intelligence of immigrants.
In “Intelligence of Refugees in Germany: Levels, Differences and Possible Determinants,” authors Heiner Rindermann, Bruno Klauk, and James Thompson argue that the average IQ of refugees to Germany is lower than the German average, and that this “hurts” Germany because it “lower[s] the average ability level.”
The article was subject to a blistering reply, “Controversy Requires Competence: Comment on Rindermann et al.” by Eric Turkheimer and K. Paige Harden, which draws attention to the methodological problems with the original study. For example:
One predictor of immigrants’ test performance is a rating of their “evolution,” which was derived from two purported characteristics of their home country—“skin lightness” and “brain size.” What is called “skin lightness” was not quantified by measuring the light refracted off people’s skin, as has been done, for instance, in carefully conducted genomic studies documenting that Southern African populations with the oldest genetic lineages have the lightest skin pigmentation within Africa. Instead, a “student”, not otherwise described, consulted a map in a mid-century book, Le razze e i popoli della terra [Races and peoples of the earth, 1953/1967], by the Italian geographer, Renato Biasutti, who sketched global variation in skin tone based on existing ethnographic reports and his imagination. The reader is left to infer that darker skinned people are less “evolved.” Country-level estimates of “brain size” were obtained in similar fashion. Rather than, for example, measuring participants’ global brain volume using fMRI (functional magnetic resonance imaging), as is standard in neuroscience, a student simply consulted a map from Beals, Smith, and Dodd (1984), which we reproduce here… an outdated method, to say the least.
The “skin lightness” and “brain size” variables were then combined using “factor analysis.” We put “factor analysis” in scare quotes because the researchers did not perform the statistical analysis of variance-covariance matrices that commonly bears that name. One of the most basic rules of factor analysis is that any factor solution requires at least three indicators. Typing “FACTOR” in one’s SPSS code is not the same as conducting a meaningful statistical analysis. Their “factor”—essentially the median of two country-level ratings derived from the shadings of fifty-year-old maps—is what they variously call “evolutionary ancestry,” “evolution: G factor”, or simply “evolution.”
We are surprised, to put it mildly, that peer reviewers approved this meaningless analysis and the resulting factor label for publication in a professional scientific journal.
They go on, as “the statistical blundering continues”, to say that the author’s conclusions “are not empirically supported arguments. They are speculations accompanied by error-ridden statistical analyses of dubious data.”
As with many journals, articles at JCI may appear online before being assigned an issue number, which may explain why the original article (accepted in March, 2024) and the reply piece (accepted in August, 2024) appear in the same issue, alongside an editorial (October, 2024) in which the editors comment on both.
They describe the extensive refereeing the original article was subject to:
This paper was originally submitted to the journal in September 2021, so it took us an unusually long time to complete the review. In part, the delay was due to the fact that the paper involves empirical research and analysis of raw data in German, which required us to find a trusted German-speaking data analyst to check the raw data and ensure the results could be reproduced. We had also sent the paper to an English-speaking data analyst who had conducted a preliminary analysis and provided the authors with some initial feedback about the more technical parts of the paper, which the authors incorporated, but we felt that it was important to find someone who could rerun the analysis in the original language.
As we understand it, not all journals go to these lengths when reviewing a paper. But we believe that having an external data analyst rerun the analysis of the raw data is good practice, and we plan to continue to do this in the future to guard against faulty or distorted analysis of data…
The paper went through three rounds of reviews, and at each round of review, the paper was amended and sent back to the reviewers. After three rounds, the four reviewers were satisfied with how the authors had addressed their comments. We editors also read the paper twice and sent our comments to the authors at various stages of the review process.
But they also acknowledge a challenge that seems inherent to the project of a journal focused on “controversial ideas”: the scarcity of willing and able referees:
The topic itself adds a second layer of complexity. Because the topic of group differences in IQ is so taboo, there are not many qualified academics working in this area, so it took us some time to find qualified reviewers.
Compounding the problem is that in areas on which there are very few researchers, it may be harder for others to use the usual indicators and heuristics for assessing competence. (Recall the refereeing aspect of this debacle.)
In response to Turkheimer and Harden’s critique, the editors write:
It’s possible that all these reviewers were mistaken and failed to detect the problems raised by Harden and Turkheimer, but as already explained, the paper passed a strict process of peer review (three of the four reviewers are… respected psychology professors), and we had no reason to reject the paper on the basis of the peer-review process.
No real-world procedures are error-free. And the fact that the editors of JCI are doing something prone to facing steeper obstacles (when it comes to finding competent referees) is not itself objectionable. But perhaps there are better ways to try to overcome this obstacle?
Discussion welcome. But, please, not on the IQs of different groups of people. Thank you.
Related:
Philosophy Journal Hosts Debate on “Jewish Influence”
“Journal of Controversial Ideas” with Pseudonymous Authors to Launch Next Year
Solidarity Instead of Pseudonymity: an Alternative Strategy for “Controversial Ideas”
So, what is the problem? A paper gets through peer-review, someone finds a mistake and publishes a reply correcting the results. Isn’t this how science is supposed to work?
Justin pulls some quotes from the reply piece that aren’t exactly favorable to JCI. But I think he actually understates how bad all of this looks. I for one am embarrassed that my fellow philosophers run a journal that publishes “research” that can be correctly described in the reply piece in the following ways:
“Human intelligence and human evolution are controversial areas of scientific inquiry that require the highest levels of scientific rigor and editorial discretion, which are absent here.”
“We are surprised, to put it mildly, that peer reviewers approved this meaningless analysis and the resulting factor label for publication in a professional scientific journal.”
Examination of computer output provided to us by the authors gives a clue regarding why the authors did not report the standard errors in the main text: The model is so misspecified that they couldn’t compute them correctly. The output on which Figure 3 is based includes a half-page of capitalized error reports, including “THE STANDARD ERRORS OF THE MODEL PARAMETER ESTIMATES MAY NOT BE TRUSTWORTHY.” [original emphasis]
“For a moment, let’s set aside, if we can, the substantive point that these pseudo-analyses are meant to support. We ask the reader: In whatever uncontroversial area of science you happen to work, if you were to submit a paper based on a factor analysis of two variables obtained by consulting the map shown in Figure 1, using statistical output that was preceded by extensive error reports, producing results tabled without standard errors, would you expect the paper to be accepted at any legitimate scientific journal?
“The authors do not have a scintilla of data supporting their contention that the test scores of immigrants have anything whatsoever to do with their “evolution.” They have only centuries-old prejudices about strangers with dark skin.
“In the end, Rindermann et al.’s paper did make us contemplate a controversial idea:
Is the academic journal dead? After all, publishing today has nothing to do with printing
presses and nothing to do with free speech. Anyone can disseminate anything in five
minutes. By calling a collection of papers posted online an “academic journal,” the editorial board is implicitly making a claim about the benefits of expertise for discerning whether an idea is worthy of discussion, and moreover, whether the idea has been developed in a way that merits the attention of the scholarly community. If the paper is representative of the discernment of some of the world’s most eminent philosophers and scientists, we fear for the future of academic publishing.”
Brutal.
The criticism that appear in the discussion article and that Justin quoted, discuss the supplementary file of the manuscript. I failed to understand the criticism because none of the references the critics mention are even cited in the manuscript (but they are on the reference list). Perhaps the reviewers were simply focusing on the manuscript and missed the supplementary file, where – if my reading is correct – the mistakes appear in. It can be difficult to spot the errors, if the errors are not in the manuscript you are asked to review.
Well, I admit that I’m not qualified in the specific field but one would have thought that being a specialist wasn’t really required to spot the questionable nature of rating someone’s degree of evolution, whatever that’s supposed to mean.
Here is a simple, and obviously enthymemetic argument for why a journal that implicitly invites defenses of racism and eugenics has an obligation to *retract* certain articles that turn out to be intellectually unsound. The reason is not epistemic, in the sense that there is an epistemic obligation to retract articles that are shown to advance a false thesis (although if one is a contextualist of a certain sort, then perhaps there are epistemic reasons to retract the article).
So here is the argument: Racism and eugenics directly contribute to the unjustified mass oppression of people, and often of people who already suffer grave injustice. So, any defense of such views risks producing and/or deepening a terrible injustice. But, we have strong moral reasons not to produce and/or deepen terrible injustices. We have even stronger reasons not to produce and/or deepen terrible injustices through publicizing falsehoods. So there are strong moral reasons not to publish anything false that might contribute to such oppression. But, if one has already published falsehoods that risk producing and/or deepen terrible injustices, then a wrong has been committed. So, one has both a primary duty to prevent further injustice and the means is to retract the article, and a duty of rectification in the face of injustice to which one has already contributed to retract the false article. So, there are at least two strong reasons to retract the article.
But, someone might say, how do we know the article contains falsehoods if it already went through peer review? Well, if experts in the field show the article contains significant errors, then one has good reason to doubt the article’s veracity. This should significantly increase confidence that one is producing and/or deepening terrible injustices through publishing falsehoods. At this point, one should err on the side of avoiding risky behavior. (Eg if one initially believes they are sober enough to drive but then an expert points out lots of evidence that one is too drunk to drive, then out of caution, one should not drive, even if one still has some doubts about whether they are in fact too drunk to drive.)
Here is a simple and not obviously enthymemetic argument why JCI should absolutely not retract this article on the grounds that it leads to injustice.
Journals should follow their explicitly stated policies and not make ad hoc exceptions to them, even in the face of external pressure.The explicitly stated policy of JCI is to retract papers only when they satisfy the retraction criteria identified by the Committee on Publication Ethics.The retraction criteria identified by the Committee on Publication Ethics do not include that a paper leads to injustice.(I also think JCI’s policy is sensible, indeed obviously so, as is COPE’s intentionally narrow grounds for retraction, but that’s a different matter.)
Whether the paper ought to be retracted on grounds of containing serious statistical errors is another matter; certainly that’s not outright implausible on the basis of the quotes Justin gives.
I was arguing for a retraction policy and implicitly that the JCI’s retraction policy is a bad one. I was not arguing for an ad hoc exception being made to (what I implicitly argued was) a bad retraction policy.
Having now read the paper (and its supplementary material, and the full note by Turkheimer and Harden), I would upgrade “not outright implausible” to “almost certainly correct”.
Presumably, one of the reasons for the journal’s commitment to free inquiry is that it’s a good way to root out false views. Granted that there are two effects here. On the one hand, people are more likely to believe things the more they hear them. But on the other hand, people are also more likely to believe things that seem true and not to believe things that seem false; and one way to make things seem false is to thoroughly and publicly shred the arguments for them apart. (Whereas, if views are suppressed rather than refuted, people who are already disposed to believe them only tend to become more convinced, somewhat like children who have been told never to talk about sex.) Do you have reason to think it’s more beneficial—i.e., a better way to make sure dangerous ideas aren’t kept alive—to retract papers like this rather than to let them be publicly embarrassed?
For the most part I don’t see an issue here. Editing always involves trading false positives against false negatives, and any process that lets interesting work through will sometimes let work through that shouldn’t have been let through. (I’ve read plenty of papers so bad as to make me think “who on earth thought that should be published?”) JCI is no different – of course the nature of the journal means that it’s more likely that a bad paper causes controversy, but the whole point of JCI is to avoid being deterred by factors like that.
If there is a problem identified here, it’s more mundane. All three of JCI’s editors are philosophers, and most (not all) of the editorial board at the journal are humanists. And while officially the journal is open to publications on any controversial idea in any academic discipline, virtually everything published so far at JCI has been philosophy, broadly construed. But this paper is a technical, scientific paper.
The editorial team at JCI clearly has competence to assess papers in philosophy, even broadly construed, but it is much more challenging to set up a review process that really covers every discipline: editing requires domain-specific academic judgement both in selecting referees and in interpreting their reports. I don’t think it would be unreasonable for a supporter of JCI (like me!) to question whether the journal ought to be cautious about publishing work of a primarily scientific / technical nature, however controversial or otherwise it might be.
(All this without prejudice to the specific paper under discussion, which I have not read.)
It is true that most of the editorial board are “humanists” but there are about 15 who are psychologists, economists, and social scientists (with one physicist and one statistician) and maybe a couple from medical sciences. You’d think that might be enough to find reviewers for what appears to be a psych/anthro paper- but maybe “controversial ideas” is just to broad, when it is meant to include controversies about (any?) discipline? I dunno. I’m less confident than you that the editorial team has competence to assess papers in philosophy, “even broadly construed” but maybe they’re only thinking of “controversial ideas” in the value theory part of philosophy? Would JCI be a good place to send a paper defending Berkeleyan idealism, external world skepticism, “the Given”, logical positivism, verificationism, Popper’s criterion of falsifiability, rejection of the necessary a posteriori, or even perhaps the Many Worlds Interpretation of QM? (OK, maybe the latter isn’t so controversial thanks to you)? I guess “controversial” is limited to “socially controversial”? Even that might be too broad for one journal, if it includes ANY discipline. If so, I suspect they’ll run into this problem again.
According to the editorial, the EB isn’t involved in editorial decisions.
On editorial competence: I think it admits of degree. The editors certainly aren’t competent to act *as reviewers* in every area of philosophy, but they know the institution of philosophy well enough to know how to identify and reach out to appropriate editors, and understand the subject matter well enough to interpret referee reports. Of course that’s imperfect (other things being equal I prefer journals to specialize more, for exactly that reason) but JCI by its nature cross-classifies in ways that make it difficult to specialize too far… it’s a balance.
I see – so having 10 psychologists on the board might help in finding referees for this paper, but not for evaluating whether to publish them, since they’re not involved in editorial decisions. That does seem to put the current EB in a difficult spot: why should they feel competent to judge and interpret the referee reports from scientific disciplines.
OK: now I have read it. I stand by the basic shape of this post but I think I underestimated how severe the “mundane” problem I mentioned is. See longer post below.
You should really reconsider your entire reasoning process here.
They literally published a response piece that highlighted a number of issues which merit retraction under COPE standards. Since those are the standards they use for considering retraction, and they accepted that response piece in advance of publication of either piece, it’s absurd for them not to have gotten a second (or fifth or whatever) opinion prior to publication. If you have the most restrictive stance on retraction a journal can have, and you intentionally court controversial topics (“are immigrants of other races genetically inferior?”) you should probably make sure you aren’t publishing articles that have fumbled methodology and data six ways from Sunday, and, you know, advertise that this was brought to your attention in advance of publication by publishing a reply piece effectively taking you to task for publishing garbage pseudoscience?
Editors are where the buck stops. You can’t say everything went right here when so much went wrong.
Did they appear simultaneously? I had gathered from Justin’s OP JCI has continuous publication and that the Rindermann et al paper was published before the Turkheimer and Harden response, even though they’re in the same issue.
If that’s wrong and they had the response paper before publishing Rindermann et al, then yes: I’m inclined to agree with you. (I don’t think “getting a fifth referee” is quite right: there are different processes to consider retraction.)
The data on their site (which maybe would be worth glancing at before weighing in? Worth considering!) for the worthless racist article says:
Received: 8 Sep 2021 / Accepted: 15 Mar 2024 / Published: 30 Oct 2024
The info for the first of *three* reply pieces is:
Received: 18 Apr 2024 / Accepted: 6 Aug 2024 / Published: 30 Oct 2024
So this volume effectively contains a symposium on the garbage racist piece with wastebin-worthy methodology.
In my view, this is the sort of thing that fully undermines the editorial judgment behind the journal (along with many of the other pieces they’ve published but those have more to do with general merit than COPE methods and data standards). It’s all a bit “what are we even doing here?”.
At any rate, while it may feel aggressive for me to suggest that you do minimal investigation before weighing in here, I do think the degree to which you had to walk back your tentative support of the journal on this particular case lends some weight to the recommendation that you not speak out in their defense on publishing racist gibberish or on the manner of their doing so without even you reading over the table of contents for the issue in which they did so. Perhaps a greater familiarity with the journal would consistently dampen your support?
Okay, I’m just not interested in engaging with people at this level of hostility and (as you say) aggression; life is too short. I’ll be over here hanging out with the small child in Justin’s comment policy.
I am reminded of some remarks made by Noam Chomsky about 50 years ago:
“If there is any purpose to investigation of the relation between race and some capacity, it must derive from the scientific significance of the question. It is difficult to be precise about questions of scientific merit. Roughly, an inquiry has scientific merit if its results might bear on some general principles of science. One doesn’t conduct inquiries into the density of blades of grass on various lawns or innumerable other trivial and pointless questions. But inquiry into such questions as race and IQ, appears to be of virtually no scientific interest…Since the inquiry has no scientific significance and no social significance, apart from the racist assumption that individuals must be regarded not as what they are but rather as standing at the mean of their race category, it follows that it has no merit at all. The question then arises. Why is it pursued with such zeal? Why is it taken seriously?”
This question, according to those who take it seriously at least, has great significance for development economics, immigration policy, and anti-discrimination law, among others. I am not saying they are right, but I do not think it is true to say that is has *no* scientific interest.
As an example – Acemoglu et al. just won a Nobel prize for a theory whose main weakness is *precisely* that it cannot show that IQ and economic development are not positively corelated.
How is that the main weakness of AJR? There are no serious social science arguments linking IQ to economic development, so even showing that there is a correlation between the two is meaningless as a challenge to their argument. I’m a social scientist who works in development economics, and I haven’t seen anybody raising that critique. So even if it is a weakness because you happen to believe that there might be a relationship there, that’s an idiosyncratic critique that only you happen to have formed.
There’s obviously a scientific significance insofar as it bears on the important question of the causes of various disparities in social outcomes. Moreover, we may begin to uncover the genetic factors (if there are any) of various cognitive traits in general without meaning to tie it to race at all, and then simply notice afterwards that those factors happen to differ between groups in some important way.
It may well turn out to be the case that there are no biological links between ethnicity and intelligence whatsoever, but you can’t just judge the inquiry valueless from the outset.
Further confirmation that most “controversial” ideas are controversial because they rest on a shoddy epistemic basis.
In the spirit of “sunlight is the best disinfectant” one might think this is a good episode, for if *that* is the best justification one could come up with, the hypothesis can be safely discarded.
On the other hand, one would have thought that we were in this epistemic position (knowing that the hypothesis can be safely discarded) to begin with, so nothing has been achieved here except to re-litigate something that did not need to be re-litigated.
Be that as it may. One might be justifiably annoyed that some push to have “academic freedom” redefined to mean that we must create conditions that allow everyone with a bias and a post-hoc justification to pursue such shoddy re-litigation of their biases. One might be particularly disappointed that it is our fellow philosophers who have taken up that regrettable task.
As for the matter of whether the article should be retracted: does it matter? Bad work in a journal dedicated to bad work is of no consequence. If JOC has aspirations to be taken seriously one day, they might consider retracting, I guess.
A possible solution would be to change the journal to the Journal of Controversial Philosophical Ideas and to only publish stuff they can find competent referees for.
(Another possibility is the Journal of Controversial Ideas with No Implication that the Ideas are Correct – To Be Clear, our Desiderata Include ‘Controversial’ at the Top and Other Things like ‘Clear Writing Style,’ but We’re Not Going to Try to Make Sure We Only Publish Correct Things. I’m mostly joking, but given the best one can say about publishing a lot of stuff in the journal is that it gives people a chance to refute it, maybe it would be worth explicitly jettisoning any kind of commitment to truth.)
Having now read the relevant material: I think there is a fairly serious issue here, but it doesn’t have anything to do with the controversial nature of the topic.
Turkheimer and Harden (TH) claim to identify a number of egregious methodological and statistical errors in the paper:
(1) it determines population-level brain size and skin color through estimates by eye from maps in outdated data sources
(2) it combines these into a country-level ‘evolutionary’ measure using a meaningless statistical pseudoprocess
(3) it fails to provide significance data for its reported results, ‘justifying’ those on the basis of some general citations about the need to be cautious in the interpretation of significance data.
(4) it substantially misdescribes its path analyses in several respects, and makes errors in calculating them.
(5) it was based on computer output (provided to Turkheimer and Harden by the authors) that was used despite its producing many error and warning measures.
So far as I can tell they are correct in all these observations. (The fourth is a little above my pay grade, the fifth is based on data I don’t have access to.) But the real issue is that these are five issues with the paper each of which in isolation would invalidate or render unreliable a major part of the paper’s conclusion.
Four observations:
1) TH’s objections provide ample prima facie grounds for retraction. The COPE guidelines for retraction (which JCI follows) say in relevant part that editors should consider retraction when “[t]hey have clear evidence that the findings are unreliable, either as a result of major error (eg, miscalculation or experimental error), or as a result of fabrication (eg, of data) or falsification (eg, image manipulation)”. If TH are correct, that bar is comfortably crossed.
2) The fact that the paper has passed extensive peer review is not relevant to the retraction case here. Peer reviewers will not always catch all errors (indeed sometimes it will be impossible for them to do so because they do not have full access to relevant data). By definition a paper being considered for retraction has passed peer review; if passing peer review was sufficient grounds to disregard strong evidence of data or statistical error, the COPE retraction guideline would be meaningless.
3) The editors’ note on this paper misdiagnoses the issues. They describe a careful and extended review process, and say that in those circumstances they ‘could have rejected the paper only because of fear of criticism for publishing it’. But the issue isn’t whether they should have published it but whether they should leave it published given the issues TH raise.
I sympathize with the editors’ caution here. Philosophy has seen a number of truly egregious cases where people on social media called for published papers to be retracted due to controversial content, nominally on the grounds of errors of scholarship missed by the reviewers; indeed, cases like these, which threaten academic freedom, are part of why JCI exists. But the cases are not alike. In previous DN comments on this, I’ve suggested that papers should be withdrawn on content grounds if (a) the data they report is not correct (e.g. it was fabricated or mistranscribed from other sources) or (b) the paper has serious errors of a mathematical or statistical nature, but not just because the editors reconsider their academic judgement. TH’s criticisms (at least, those I describe above) are very much (a) and (b). Retraction on these kinds of grounds is standard in science and does not impinge on academic freedom.
4) I think JCI’s editors need to think seriously whether the journal as currently set up is really in a place to accept first-order scientific research. Its editors are all philosophers; its editorial board contains a few scientists but the editors tell us that the board is not involved in editorial decisions. Assessing work in a given area requires editors familiar with the scholarly community in that area (in order to source good reviewers); it also requires that the editors are familiar enough with the subject matter to appropriately assess and interpret reviews (since the final academic decision is and must be the editors’).
I don’t intend to cast doubt on this or any other review process carried out by JCI. But I am concerned that the journal is in general not well set up to publish this sort of work and risks undermining its reputation by doing so. My advice to the editors is either to narrow the scope of the journal somewhat to exclude papers carrying out first-order scientific research (which does not preclude the discussion of extant research results), or, if they think the journal’s goals require it to continue to consider technical scientific papers, to recruit associate editors who are respected subject-matter experts in the appropriate areas – as of course the editors are in philosophy.
I agree with your final point here –
What really puzzles me is why they sweated so much to publish this paper of all things? Like, some editors reject articles because they cannot be bothered to go through even one round of revision, and they went through three (plus all the back and forth with finding a german-speaking statistician). And it is not as it this paper was some ground-breaking masterpiece, it would have at best been a minor contribution to a niche (and very questionable) debate.
I really love the JoCI, they really published a lot of good stuff, but I struggle to see the rationale here.
I assume it’s because they sent it out, and then the reviewers said to accept it. I can see how their hands would be tied at that stage.
Not really, no. Editors have ultimate decision on whether to go with the publication or not.
I’m aware that they have the ultimate decision.
I suspect the editors regret sending it out for review and, given the consensus view about race and IQ, did so expecting the scientists who reviewed it to reject it. But if it received only ‘accept’ verdicts from four respected referees, given that the editors themselves are not scientists, it would be very hard for them to justify not publishing it, in a way that didn’t look seem they were doing so because it was such a controversial idea.
I once had a paper rejected (by a different journal – I’ve never submitted to this one) after two rounds of R&R. After the second round, both reviewers said to accept it. And my paper wasn’t even racist!
I referred a paper for them. Unfortunately, it was garbage based on conservative internet talking points.
I’m not sure what this is supposed to tell us – I’ve reviewed bad papers for many really good journals. Did they publish this garbage paper? If not, then what concern is supposed to arise from this?
Yes, obviously that’s true. It’s a deliberately incomplete anecdote, not data. And to their credit, it wasn’t published. It’s just that I strongly suspect the journal primarily attracts a certain type of garbage submission. And if I hadn’t been R2, it might well have gone through. As this one did.
Do you have any evidence to support these suppositions? I certainly have never thought I had reasons to make such assumptions about any of the many journals that have sent me bad papers – and then rejected those papers upon receiving my negative review…
One reason stems from the nature of the problems on the submission I reviewed, where “controversy” was ginned up by misrepresenting the literature, and not actually engaging with it. Another is in the OP. Another comes from some of the work I’ve read there.
None of these are dispositive, let alone proof. But they’re enough to raise my eyebrows, and they’re enough that I’m not too surprised to see that at least some wholly indefensible work slips through. There are only so many referees, after all, and they only have so much time and energy. If you throw enough garbage at them, some is bound to slip through.
If nothing else, that they need a more robust desk-rejection policy. Or, indeed, a desk-rejection policy at all.
Tom, they publish a lot of very bad papers, I hope this helps you form a judgment about the quality of this journal!
And I routinely referee articles that are garbage based on leftist talking points.
Not like this. There was very little scholarship, just the talking points parroted by non-philosophers. And what scholarship there was systematically misconstrued the literature so as to position the paper as controversial, when in fact the people coming under fire _agree_ with the author. (And had no trouble publishing their controversial views in mainstream generalist and subfield journals.)
Yes, there are bad papers everywhere. This was easily one of the poorest I’ve seen, but that happens. But, as I said above, I strongly suspect that such submissions are rather more common at JCI, which means that referees have to be more vigilant.
Is that really true?
Nearly a quarter (23.88%) of the articles published in JCI are trans panic nonsense. The proportion of articles dedicated specifically to Alex Byrne’s views against trans people is shocking. The endeavor was not very promising at the outset but the results have been astonishingly embarassing.
I fully agree with this. There have also been at least two papers that I recall defending blacking up. I think it tarnishes the reputation of the editors.
What are my “views against trans people,” exactly?
While I won’t go back and look for comments, I seem to remember another retraction-worthy paper being accepted to Hypatia a few years back, and the tone from many commenters being that editors can’t be expected to catch everything, that some things rely on trust that authors are sincere and competent, and that such mistakes shouldn’t warrant doubt of the merits of the journal’s integrity. People taking this as a smoking gun should make sure they apply their standards consistently.
Without wishing to systematically reopen the Hypatia discussion: there was never even a prima facie case for retraction there, because (as I and others argued at length at the time) the paper did not contain clear errors of a logical/mathematical or empirical kind that could justify retraction under COPE guidelines. In the present case there is a pretty good prima facie case for retraction, but it would need to be on the grounds of errors of that kind (which Turkheimer and Harden very plausibly argue for) and not general concern with the paper’s qualitative thesis.
(And I at least have no concerns with the integrity of JCI or its quality as a humanistic journal, though I do have some concerns as to whether it’s appropriately constituted to review first-order science.)
I’m not saying the cases are analogous vis. the merits of the paper’s retraction, I agree with your assessment that there are grounds for this. My issue was with those trying to take the original acceptance as evidence of the editors’ lacking incompetence or integrity, on which it’s harder to see a bright line difference in the cases.
The journal also published an article that engages with Rindermann et al, “On what matters for obligations to refugees,” that was received on Oct 21, accepted Oct 22, and published Oct 30. That is a remarkable turnaround time.
The demand for ‘retraction’ is misguided. The faults of the paper are: weak methodology, weak conclusions, and a lack of philosophical substance. Its topic, however, ‘group differences in IQ’ is not something that should be discounted/suppressed in a philosophy journal. After all, the journal is focused on ‘controversial ideas’.
Retraction would be appropriate in cases of plagiarism, fraud, deception, or unethical experiments (think of the Nazis) on the research subjects. Rindermann et al just wrote a poor paper, and the replies in the same issue take the authors to task. So there is no danger of readers – who are unfamiliar with scientific methods – being mislead. J.S. Mill (On Liberty, 1859) would approve.
However, the paper could have some philosophical relevance in the area of Global Justice with regard to the question: when should immigrants and/or asylum seekers be admitted to a country? It suggests that the ability to contribute to the economic productivity of the host country should be a criterion.
Rindermann et al don’t consider that immigrants who don’t have the right qualifications (or qualifications not being recognised – very common in Germany) and/or don’t speak the language of the host country yet, can still be economically productive. See all the corner shops in the UK and the US – who runs them?
We have seen during the pandemic how important delivery drivers, cleaners, bus drivers, supermarket workers, care workers etc. are. So, in the Rindermann et al paper the notion of ‘economic productivity’ is reduced to being an entrepreneur.
Their study finds the average IQ of migrants to be 85 and for Germans it is 100. The IQ of migrants ‘is certainly too low to form the basis for a second German economic miracle’. The authors ignore that low-skilled or unskilled jobs are important, and modern economies are usually short of these types of labour.
It is important to understand the political context in Germany. In 2015 Chancellor Merkel didn’t wait for the EU to formulate a common policy on migrants, but instead accepted close to 1 Million asylum seekers, mostly from Syria (fleeing civil war), to Germany. This has led to resentment in the population and has increased the popularity of the far right party AFD.
I read the paper by Rindermann et al as an attempt to put a scientific gloss on this wide-spread resentment within the population: We will let only those in who have a high IQ – and who will benefit us. But there was actually no need to link this to IQ. The authors seem to take a utilitarian position – that, by itself, could do the trick.
Rindermann et al should have taken their ideas to its logical conclusion: Germans with a lower than average IQ should be urged to emigrate, because they are unlikely to be ‘economically productive’, preferably to countries where their lower IQ is the norm.
Disclosure: I published a paper in philosophy of mind/trans theory in the same issue.
You seem confused about the purposes of retraction. The widely-accepted COPE guidelines for retraction give a list of possible reasons for retraction, and the very first is when:
“They [the editors] have clear evidence that the findings are unreliable, either as a result of major error (eg, miscalculation or experimental error), or as a result of fabrication (eg, of data)”
Turkheimer and Harden provide detailed reasons to believe that the findings are unreliable as a result of major error. If T&H’s criticism is correct, then the paper very clearly falls within the scope of retraction as commonly understand in scholarly publishing.
I’m aware of the COPE guidelines. I take a Millian line in this case, as I have indicated in my post – the criticism of Rindermann et al is there in the issue for everyone to see. Furthermore, the editors of JOCI have announced that there will be a reply from Rindermann et al in the next issue. I look forward to it.
I will upgrade my assessment then: you seem deeply confused about the purpose of scholarly publishing.
To stay with Mill, rather than resorting to personal attack, epistemic humility is always a good position to take. There is always the possibility that we are wrong. Secondly, the conclusions by Rindermann et al about immigration deserve to be discussed openly. They can be read independently of the IQ ‘research’ and could be relevant in the wider debate about Global Justice.
Do you think all papers with flawed methodologies should be published in this journal (so long as they meet other criteria – they’re racist or whatever, etc.)? Or is the rule “if you sneak your flaws past the reviewers, your paper stays published, but if reviewers catch your mistakes prior to publication, you don’t get published”?
I don’t like this obsession with retractions. I have no ‘appetite for destruction’ – unlike many social justice activists. I have explained that there is stuff in there that needs to be discussed. Long live J.S. Mill!
I wasn’t asking about your stance on retractions. I understand you do not want papers like this to be retracted. I am asking about your stance on publishing them in the first place. Do you think the journal should:
1) Have a rule that it will publish papers even if they make methodological mistakes such that they do not give us good reason to accept the claims they are advancing?
or
2) Have a rule that it will try not to publish papers like this, but of course sometimes mistakes will be made and papers like this will end up published anyways (and not get retracted, because, again, I understand that you do not want the journal to retract anything)?
I think “flawed methodology” isn’t quite the right way to describe the issue here. (A study shouldn’t be withdrawn because it was badly designed). I think the main reason for retracting papers on empirical or statistical grounds is that those errors are not readily detectable just by reading the paper. If you have to trawl through supplementary material, or consult the raw data or code, to identify errors, that’s not realistically doable for most readers, and so by publishing the journal is saying that it doesn’t believe there are errors like that. If it gets evidence that there are, it makes sense to retract or correct.
My (fairly procedural) point is that
(a) the scholarly norm for dealing with papers that have errors of a logical/mathematical kind or inaccurately report data is editorial retraction / correction rather than the publication of a response, that (b) the T&H reply is therefore a prima facie case for retraction, and (c) if JCI wants to publish empirical and data-analysis work but to deviate from that scholarly norm, they should say so explicitly and publicly as part of their editorial policy statement.
At the moment they just say that they will only retract for reasons covered by COPE guidelines, which I assume is there as a (very welcome) signal that they are not going to entertain social-media pressure to retract controversial arguments, but which certainly implicates that they *do* think retraction is warranted for the usual science reasons. I am inclined to think that this just isn’t something the editors have given much thought to hitherto, since I don’t think they are especially familiar with scientific publication norms, and have mostly been publishing humanities-style work.
(If there is a suffiicently substantive part of the Rindermann et al paper that is not reliant on their data analysis, there are ways to handle that: major corrections or partial retractions. And in any case none of this would remove the paper from the public record: retracted papers are still available.)
I agree, there could be an erratum added – if indeed errors occurred. And I believe ‘there is a sufficently substantive part of the Rindermann et al paper that is not reliant on their data analysis’. More here: https://miroslavimbrisevic.wordpress.com/2025/02/17/appetite-for-destruction/