A simple solution to peer review problems

Famous computer scientist and Roomba co-founder Rodney Brooks writes about the problems of peer review in academia. He notes that peer review has some important virtues even as the way it’s currently practiced generates many problems and pathologies too. Brooks says, “I don’t have a solution, but I hope my observations here might be interesting to some.” I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified.

That way, the paper is available to those who find it useful. If a researcher really thinks the peer reviewers are wrong, the researcher can state why, and why they’re leaving it up, despite the critiques. Peer-review reports can be kept anonymous but can also be appended to the paper, so that readers can decide for themselves whether the peer reviewers’ comments are useful or accurate—in my limited, but real, experience in English lit, they’ve been neither, and that experience seems to have been echoed by many others. If a writer wishes to be anonymous, the writer can leave the work as “anonymous” until after it’s been submitted for peer review, which would allow for double-blind peer review, and that double-blindness would help remove some of the insider-ism biases around people knowing each other.

Server costs for things like simple websites are almost indistinguishable from zero today, and those costs can easily be borne by the universities themselves, which will find them far lower than subscription costs.

What stands in the way? Current practice and setup. Plus, Elsevier and one or two other multi-billion-dollar publishing conglomerates that control the top journals in most fields. These giants want to maintain library fees that amount to thousands of dollars per journal, even if the journal editors are paid minimally, as are peer reviewers and so on. Only the companies make money. Academics live and die based on prestige, so few will deviate from the existing model. Publishing in top journals is essential for hiring, tenure, and promotion (the tenure model also generates a bunch of pathologies in academia, but we’ll ignore those for now).

There are pushes to change the model—the entire University of California system, for example, announced in 2019 that it would “terminate subscriptions with world’s largest scientific publisher in push for open access to publicly funded research.” In my view, all public funding bodies should stipulate that no research funded with public money can be published in closed-access journals, and foundations should do the same. There is no reason for modern research to be hidden behind paywalls.

It would also help if individual schools and departments quit making hiring, tenure and promotion decisions almost entirely based on “peer-reviewed” work. Those on hiring, tenure, and promotion committees should be able to read the work and judge the merit for themselves, regardless of the venue in which it appears.

Coronavirus and the need for urgent research has also pushed biomed and medicine towards the “publish first” model. Peer review seems to be happening after the paper is published in medRxiv or bioRxiv. One hopes these are permanent changes. The problems with the journal model are well known but too little is being done. Or, rather, too little was being done: the urgency of the situation may lead to reform in most fields.

Open journals would be a boon for access and for intellectual diversity. When I was in grad school for English (don’t do that, I want to reiterate), the peer reviewer reports I got on most of my papers were so bad that they made me realize I was wasting my life trying to break into the field; there is a difference between “negative but fair” and “these people are not worth trying to impress,” and in English lit the latter predominated. In addition, journals took a year, and sometimes years, to publish the papers they accepted, raising the obvious question: if something is so unimportant that it’s acceptable to take years to publish it, why bother? “The Research Bust” explores the relevant implications. No one else in the field seemed to care about its torporous pace or what that implies. Many academics in the humanities have been wringing their hands about the state of the field for years, without engaging in real efforts to fix it, even as professor jobs disappear and undergrads choose other majors. In my view, intellectual honesty and diversity are both important, and yet the current academic system doesn’t properly incentivize or reward either, though it could.

In the humanities, at least being wrong and “peer reviewed” doesn’t carry some of the costs that being wrong and “peer reviewed” can in the sciences.

For another take on peer review’s problems, see Andrew Gelman.

How do you judiciously help someone whose work isn’t very good?

This question keeps reappearing in various guises: How do you help someone whose work isn’t very good? Simply saying “This sucks” isn’t helpful and is usually taken with offense. A sufficiently screwed up work may also be unrecoverable. But making minor changes and saying, “It’s great!” often isn’t helpful either, because the work isn’t great and false praise is a lie. Those seeking criticism should be tactful enough not to ask, “Is it good?”, but often they aren’t and it leaves critics and editors in an awkward position.

I’m a writer, so I tend to see stuff from bad writers, but the same principles apply to other people with other domains of expertise. I developed my method of commenting on bad writing years ago, when a former student and now friend asked me to read a few stories she’d written for a creative writing class. Given her age they weren’t terrible; I made some comments, fixed a couple of minor things, and suggested some books that might speak to her.*

She asked if I thought the stories were good, but fortunately she asked via email so I had a few minutes to think about my response. I replied that I’d reframe the question: if she keeps writing, reading about writing, and developing her own sense of good writing, in four or five years she’ll reread her stories and be able to decide for herself whether her work was any good. I mentioned that when I was 26 or so, I no longer thought the stuff I’d written from 18 – 22 was any good. She got the point, I think, and seemed to appreciate what I was saying without saying.

And what I told her was and is true: I don’t think much of that early work now. But I also wouldn’t be where I am today without having written what I did then. In addition to being true, that sort of advice has the advantage of being tactful. I think John Irving said that every writer who seeks feedback really wants to be told, “It’s perfect. Don’t change a thing.” But of course nothing is perfect and editors exist for a reason (so do therapists; the reasons may be more closely related than we’d like to commonly assume).

* Anyone interested in writing ought to look at this list, which I still think good. I periodically re-read every book on it. In some sense no good writer ever fully stops being a beginner.

Publishing is always changing

Guess what time period this quote describes:

Publishers had to improve the way they did business. So they tried several things: [. . . and] in general, they became less worried about literary merit and more about salability as the ultimate criterion in accepting a manuscript.

It could have be from last week’s New York Times, except with “publishers” perhaps replaced with “Amazon,” but the overall gestalt is there, complete with the carping about the lowering of standards when entrenched powers are losing their powers. But D. G. Myers wrote it in The Elephants Teach, and the quoted passage applies to the early 20th Century.

Someone out there is always lamenting the deplorable state of literary merit these days, but someone has been always been lamenting it, just like some old person is always lamenting kids these days. Don’t listen. The next big thing is probably not going to come from the old guard.

Careers—and careerism—in academia and criticism

Careers in criticism examines what D.G. Myers thinks can be done about the possible problem of lousy literary criticism. It’s worth reading, but I suspect that the other problem, which goes undiscussed in this post, is the difficulty of deciding what is good criticism: many people complain that lots of academic and other criticism is bad (I probably count myself in their ranks much of the time), but they tend to disagree with what would be good in its stead. Deciding is particularly hard in a field where wildly divergent ideas of what constitutes quality exists. Therefore you get… gridlock, high school politics, and so forth.

How to solve this? Myers says:

[Elberry] thinks that I am suggesting that “critics should write about less well-known books,” but I suggest this only as a method, a practical expedient, for undertaking their real responsibility: namely, to contribute to literary knowledge. The demand upon critics (in the university and out) must be, not to “write something new and different,” but to add something new and different to the store of human understanding.

I bet that most people who are writing just to “write something new and different” would argue they are adding to the store of human knowledge. I definitely agree with Myers’ formulation on a high level but am not sure how to implement this on a lower level. The best ideas I can come up resolve issues in academic publishing: right now, it can take years to publish an essay in a peer-reviewed journal, which then locks it behind pay walls on the Internet. The length raises the obvious and uncomfortable question: if it takes three years to publish a paper, is the paper really that important? That this process takes forever is hardly new; Lucky Jim mocked it in the 1950s.

My solution: have peer-reviewed journals “publish” online, and have publication be a link to the author’s paper on the author’s website. The journal’s editor could also copy that paper to their own site after anonymous peer review. That way, the information is freely available, especially to people in countries where most universities can’t afford journal subscriptions under the present model; the theoretical “size” of a journal could be limitless, although the practicalities of reading would probably still limit that size; there would still be a recognized body of work that makes up, say “Modern Fiction Studies;” and the journal could still issue a print edition every n months or years for those who prefer it. This would cause the journal to lose the revenue stream that currently comes from publishers, but that stream seems to be so small that universities could replace it in return for the prestige of housing the journal. Alternately, the exceedingly low cost of web publishing—one could buy server hosting with 200GB+ per month transfer limits and so forth for $100/month—could obviate the (relatively) high cost structures that journals already have while reducing barriers to entry.

Current top-notch journals have no incentive to adopt this model, as it would challenge their hegemony, but if lesser journals began adopting it and scholars preferred it, the quality in my wiki-like journal would rise, and competition might force top-notch journals to adopt the same strategies if they’re going to retain their position. Since publishing in English lit seems mostly a prestige and influence game, this strategy has few drawbacks I can perceive. If anyone knows of a reputable journal (which is to say: one backed by a university with at least a few years of regular publication) that’s already doing this, I’d love to hear about it.

The other change is one I read about in Freakonomics, the blog: require peer reviewers to say publish/no publish on each paper, and give comments, rather than giving comments with the implication that, if they’re not taken, one will automatically be rejected. Rather than having a three- to four-draft round-robin time-waster of questionable benefit, a peer reviewer would have to say “yes/no,” on the first iteration in its current condition, and the reviewer’s comments would be an option rather than requirement. This structural change seems less important than the one above.

Anyway, given that I’m in grad school for English lit, expect more on this topic in the future, since I’m now tasting the peer review that many others have called bitter and find that they’re mostly right.

EDIT: Myers has a follow-up post, with a response to some of my comments, here.

%d bloggers like this: