In Development: Philosophy Archive & Journal with Crowd-Sourced Peer Review
Imagine a website philosophers can join to post their papers for reading, reviewing (on a wiki), and upvoting/downvoting by other members, and which will periodically publish a journal comprised of a selection of these papers (ones that make it through a review process they qualify for by getting enough upvotes). That’s what Populus will be once it is up and running. “Think the non-horrific parts of Reddit meets PhilPapers,” says Populus founder David Faraci (Georgetown), in a post about it at PEA Soup.
Why create this? Faraci says:
Populus seeks to reimagine the publication process to take fuller advantage of the resources afforded by the Internet to provide a better experience in publishing, reviewing, and archiving.
For example:
Authors [on Populus] have access to real-time feedback on their work. They are able to see at all times whether their work is being read and how it is being received, both in terms of votes and feedback on the wiki. Is your work getting less upvotes than you’d like? Are there comments in the wiki you believe you can address? Simply address them and upload a new version…
The hope is that philosophers will see reviewing works on Populus as a duty of citizenship within the philosophical community, much as many do now when asked to referee for other publication venues. However, because the review process relies most heavily on votes, reviewing for Populus is less arduous than reviewing for other journals. If there is a paper you are interested in reading, there is little cost to reading it through Populus and voting. Populus will also incorporate game-like incentives, such as status boosts (e.g., ‘top reviewer’). It is TBD whether these will be purely symbolic or will come with tangible incentives, such as additional voting power…
As Populus membership grows, having work on Populus will make it more visible to the philosophical community. But unlike with other archives, authors also gain insight into their peers’ assessment of their work. As time goes on… information may become available on what other works appeal to the same population as yours. Citation indexing is also planned, and thus eventually Populus plans to offer real-time information on impact factor.
Faraci is currently looking for feedback on and assistance with the project. In his post at PEA Soup he writes:
- I’d like to put together an editorial board whose association with the project will boost its credibility. I anticipate this’ requiring little actual work. If you are a famous person who likes my idea and would like to get on board, that would be great.
- This project will likely require some funding. I’m looking for suggestions for sources.
- I need people with web development or other relevant technical experience who would like to donate (or, if we get funding, be paid for) their time.
- I’m looking for people who want to help or get involved in any other way, especially ones with the general entrepreneurial skills I lack.
Thoughts on the project are welcome in the comments. Those interested in helping can also email Faraci directly at [email protected].
Hi,
I think this is a great idea; if it works, it could perhaps replace academia.edu (whose most valuable feature is the possibility to do sessions and which, as far as I can see, is becoming increasingly unattractive.) Good luck!
To prevent some of the obvious pitfalls, perhaps you should restrict voting rights to people who have a good record of helpful, unbiased commenting. This would be work intensive, but hopefully doable if you find enough people to serve as editors for different areas.
I like this idea very much–thanks to everyone involved in launching it.
I do have one question: Will reviewers read the papers knowing the author’s identity? I get that the author identity has to be paired with the paper for Populous to work like a journal. There has to be a record of who wrote what for our CVs. But maybe papers could spend the first week (or couple of weeks, depending on how fast reviews pour in) on the website without the author’s name attached?
There are probably better strategies for ensuring some modicum of blind review. But knowledge of who wrote a thing can (and often does) influence reviews of that thing, so some strategy for blunting that effect seems important.
This is really important. At the BJPS, we get vast numbers of submissions and even once the highly inappropriate papers are weeded out, there are still a large number of perfectly good papers left. My mind boggles at just how many at least half-way decent philosophy papers there are floating around at any one time. Even before we start to worry about the effects of various biases on the rating of papers, what heuristic are readers likely to use in choosing which papers to read? Because if the authors’ identities are listed, it will reintroduce all the biases that at least some journals have sought to eradicate with anonymising etc.
One suggestion would be that the papers should be anonymous for as long as the author deems necessary. While anonymous, the papers can gather upvotes and move onto the top of the list. Once the author rips off their mask, perhaps that should freeze the score.
This does leave open the problem where I post something anonymously and e-mail all my friends to tell them to upvote it. Once when I was kicking around a similar idea over beer at a conference, someone said, “Oh! So it’s Reddit for philosophy!” And Reddit is kind of notoriously prone to abuse and sometimes an absolutely horrible place for people who aren’t straight white male bros. It might be good to try to bring in some advice from professional community managers on social media, especially people with expertise in stopping harassment.
I think this is a fantastic idea and could be a great reform to a journal system that is under great strain, even aside from all the ways in which non-open-source journals make it more difficult to access research that anyone with a web connection could distribute freely.
I remember that Andy Cullison was working on something similar a number of years ago. David might be able to get some insights from Andy’s experience.
Anca: The plan is to have unrestricted voting initially and see how that goes. But I am certainly open to having that change as things progress. One possibility, as you suggest, is for the editorial staff to either restrict voting to certain members or give some votes more weight than others. One problem with this is, as you note, that it’s time and resource intensive. But the larger problem is that, at least initially, I’d hoped to have the system be triple blind, in that the editorial staff won’t know who is doing the voting. (Though as I note on the site, we might be able to anonymously give some members boosts in virtue of their voting record without knowing who they are.) Another possibility is that eventually vote-weighting will be done automatically, in virtue of (say) past voting record as compared to others’.
Luke and Beth: All journal submissions will be anonymous, so readers will not know whose paper they are reading (or, at least, won’t know it from us). Once the papers are published in the journal, they will be signed. Archive records of work published elsewhere will not be anonymous. I’m not sure yet about archive records of unpublished work. On the one hand, if someone just wants to have a paper up there under their name, as they might on Academia.edu, that seems unobjectionable. On the other hand, there’s the worry that someone can upload a signed archive record, get votes on it, and then turn it into a journal submission. I need to think about this more.
Jonathan: Someone mentioned this on the PEA Soup thread, too. And Andy just posted there saying he’s going to offer some comments here. I look forward to them!
All.
I was working on something like this years ago. It was called “Sympoze” and my only limit was time. I had a group of about 200 volunteer referees ready to go, and I had a system set up. But once I took on more professional responsibilities at the APA (and now in my position at The Prindle Institute for Ethics), I didn’t have time. I would love to see a project like this come to fruition, and I’d be happy to talk more with the Populus team about my experiences.
In regards to the first two bullet points:
1. I had an editorial/advisory board of about 20 philosophers. Their role was going to be to vet referee volunteers by subject area. It won’t be enough to simply have people who are famous listed as advisers. It needs to be clear that they are doing something to ensure quality. My idea was to have a pool of philosophers that were authorized to vote in a category based on having been screened by the advisory committee.
2. Your question about funding depends on what the legal status of Populus would be (in particular whether it will be a for-profit or non-profit entity) and how much revenue you’re shooting for. Sympoze was part of a for-profit, not because I wanted to make tons of money, but because I saw quicker routes to funding. And it worked. We secured funding, but we had to have a compelling story about profit through advertising and the market for advertising shifted drastically. The nature of marketing online has changed so much, that I’m not sure our business model would work now. If you were going to be for-profit, I’d want to know more about the business model.
If you were a non-profit, then there are a variety of grants you would be able to apply for, but the question would be about long-term sustainability would still be an issue. You would need to convince a grant maker that you have a plan for sustaining the project after their 2-3 year investment. There is good news here. If you thought you could maintain the site (after an initial kickstart) with something like $10,000 a year in advertisements from academic presses, then this might be the way to go.
Again, I’d love to see something like this launch and I’m happy to talk more about it.
Andrew,
Glad to hear you’re available for brain-picking. I expect I’ll take you up on that. The major difference between our projects, it seems, is that I’m aiming to start with as open a voting system as possible (within the confines of the field) and then narrow only as necessary, rather than starting with votes from experts. (Though I expect in practice there will be more similarity than this suggests, since at least at first I will probably have to send a lot of “hey, you’re in this area, would you read and vote on this paper?” emails.)
My intention is to remain non-profit, at least for now. I’ve found some grants that I’m considering applying for, but first I’m going to have a go at developing the site myself (along with some kindly volunteers). If I can get it up and running, then the overhead should be fairly low. The only obvious costs will be things like the domain, which is minimal, and server space, which I’m ultimately hoping I could get a university to provide. Are there associated costs you ran up against that aren’t occurring to me?
I’m unsure if we should decide what’s good philosophy by popular vote. In theory, referees are supposed to comment on whether the paper meets the standards, not on whether they LIKE the paper. But this Populus system looks as if it’s one where what’s popular, rather than good or right, rules supreme.
Is it just me, or is that a recipe for bad philosophy?
As stated in the prospectus, “members are expected to use the same criteria they would if acting as referees for any other journal: upvoting only if they believe the work warrants publication as-is or with a reasonable amount of editing, downvoting only if they believe that the paper does not warrant publication in its current or any similar form.” It is certainly possible that people will not do this, that they will simply vote for what they like. But, crucially, the same is true of referees under the traditional system. Now consider three possible systems:
1) The system isolates those members of the larger group that are acting in good faith and takes their views into account.
2) The system aggregates the views of as many members of the community as possible.
3) The system randomly selects members of the larger community and takes their views into account.
I think that arguably, (1) is the best of these systems, while (3) is the worst. (2) is a compromise. The idea is that since most people act in good faith (and have a reasonably good sense of their own expertise), but we can’t figure out which are which, we’re more likely in any particular case to get the right answer if we take everyone’s views into account, rather than risk putting all the power in the hands of members of the bad faith minority.
My suspicion–one I expect others share–is that the current system, perhaps even any feasible version of the traditional system, is closer to (3) than to (1). If that’s true, then that’s reason to think that (2) is the way to go. Populus is an experiment in implementing (2).
Of course, it could turn out that the current system is closer to (1). Or it could turn out that the subset of the community that participates in Populus is not the same as–and behave worse than –the subset that participates in the current system. Or it could turn out that the same population simply behaves less well in this context than when getting a Please Review This email from some official email address. In those cases (and others!) Populus might turn out to be a recipe for bad philosophy. But this isn’t something we can know from the armchair; we have to run the experiment.
Will they be given these instructions whenever they vote or just once?
I fear that people will default to voting what they like without reflective analysis.
How about trialling some kind of “post-publication peer review” system in philosophy that is analogous to the wildly successful “PubPeer” in the (largely I think biomedical) sciences? https://pubpeer.com/
Although PubPeer has proved its worth as a means of detecting problems with research data, and scientific fraud, I would see the potential of this in philosophy to be opening up conversations outside the usual peer-groups who communicate with one another, some of whom have become very closed. Also, if a published paper fails to cite good work that has already been done in the area…that could be noted too.
It strikes me that something like this might be layered onto the PhilPapers infrastructure without too much effort.