Tag: academia

Posted on

What I do when I review

I currently wear a number of hats in terms of academic publishing. I’m an author (and co-author). I am often asked to be a peer reviewer. I am the editor-in-chief of a small, but historical, regional journal. And I am an academic editor – some journals would call a similar position an associate or subject area editor – for PeerJ.

(As an aside, it should be noted that I do not get paid for any of these roles in any way. I, and other scientists, do this type of work because we feel that it is our responsibility to contribute to the process of scientific communication. Without proper and reputable avenues of communication, science would rapidly grind to a lurching halt as we each worked in isolation in our own little realm. The scientific endeavor has always relied on communication, and communication is only accomplished if it is also facilitated.)

Over the next while, in my sporadic blogging fashion, I plan to write down some of my thoughts about each of the roles that I mentioned above. Specially, I would like to write briefly about the mechanics, the philosophy, and perhaps some of the side issues that arise in each role. Today I’m beginning with the role of peer reviewer.

Peer review is the cornerstone of scientific communication. Not all of our communication as scientists is peer reviewed (e.g. most conference presentations or posters are not peer reviewed). But prior to results and analyses being entered into the permanent scientific record in the form of a journal paper, the work is reviewed by two or more independent referees.

The process of peer review has been fairly standard over the years. The reviewers receive the paper (or just the title and abstract) from a journal editor or associate editor. The potential reviewer has a preliminary look at it to see if they are able to review it. If they can and are willing to do the work, they read the manuscript carefully and provide a report back to the editor. Often the report comes with recommendations to accept as is (rare), accept with minor or major revisions, reject with an invitation to resubmit, or to outright reject. The journal editor compares the various reviews from the referees and arbitrates a final decision.

So, what do I do when I receive a request to review a paper? And how do I go about reviewing? Here are some of my thoughts in the form of do’s and don’t’s:

  • Do consider reviewing a paper when a request is sent to you. We are all busy, and sometimes we are so busy that we need to set reviewing tasks aside for awhile. But that cannot and should not be a permanent reality for a practicing scientist. As long as I am submitting papers to journals and am relying on the good graces of volunteer referees, I should be willing to do the work on the other end of the equation as well.
  • Do not review a paper if you are not qualified to do so. If the paper is out of your realm of expertise, don’t pretend that it is. Your review will not be helpful. On the other hand, there might be a reason that the editor asked you to do the review. Perhaps one portion of the paper is highly specialized and in your area. While you might not be qualified to comment on all aspects of the paper, you should be able to comment on that part. The editor should have outlined this to you in their request. But feel free to ask them if you find yourself puzzled by a request.
  • Do not review a paper if you perceive a conflict of interest. Not every editor knows every possible collaborative or collegial arrangement. If you know or suspect that there is a conflict of interest, let the editor know the details and then let them decide on your eligibility to review.
  • Do help the editor, even if you are not going to do the review. If you cannot review a paper for the reasons listed in #2 or #3, provide the editor with some alternate names of colleagues who are qualified to do the work.
  • If you accept the assignment, do be cognizant of the deadline and do your level best to meet it. Nothing is worse from an author’s point-of-view than waiting for months to get the reviews back on their paper. Nothing, that is, except for being an editor having to poke recalcitrant reviewers while also fending off increasingly irate emails from authors.
  • Do follow the journal’s specific reviewing guidelines. Not all journals are the same. All reputable journals expect scientific rigor and appropriate analysis. But there is variation beyond that (e.g., some journals look for “impact” while some do not).
  • Do review as you would wish to be reviewed. In other words, follow the golden rule of reviewing. No author wants to hear the bad news that there are flaws in their analysis or reasoning. But neither does any author want to publish a flawed manuscript. So tell it like it is. Use your expertise. But – and this is an important “but” – be respectful. Whether you are recommending accepting or rejecting the paper, give constructive and useful feedback. Note the positive aspects of the work. Explain what you think could be improved in clear terms. Be as extensive in your comments as you need to be; but never be blunt and brief to the point of being insulting. If you have taken the job on, then do a good job. A good job always entails more than six lines of halfhearted text. Be helpful, be kind, and be honest. In short, be professional.
  • Do be willing to re-review the paper if necessary. If you do end up rejecting the paper or recommending major revisions, let the editor know that you would be willing to have another look at the paper if or when the authors resubmit it. Since you now have some of the best knowledge of the state of the paper, you are among the best placed to assess the recommended changes.
  • Do record your work in your CV. You are not being paid for the work, but it is part of your contribution to the scientific community. Immediately after finishing a reviewing task, record the task (not revealing the authors’ names or other identifying information, of course) in your CV so that you don’t forget about it.
  • Do maintain confidentiality as expected. Most journals still use an anonymous (and sometimes double-blind) review system. That means that you are obliged not to reveal details of your review unless the authors and you both agree at some point. There are a few caveats to this. First, if you wish to sign your review, you can reveal yourself to the authors, but even then, you cannot discuss the details of the review with others. Some journals, such as PeerJ or F1000Research, encourage open peer review. If that’s the case and you choose to abide by that system, then the entire review process will be made public. But, even then, you must maintain confidentiality until the paper and the accompanying reviews are published. Again, be professional.
  • Do look forward to reading the paper in the literature. If you have done a good job of reviewing it, you should take some degree of pride in the outcome because you have had a (hopefully) positive influence on the direction of science.

Peer reviewing takes time and effort, but it is also a rewarding experience. Besides allowing scientists another avenue for participation in the scientific process, it also exposes us to new ideas and cutting-edge thinking. And, above all, it ensures rigor in the scientific record. So enjoy the work, learn from it, and take pride in doing a good job.

Posted on

Whither peer review?

If you’ve been working in science long enough to have published at least one or two papers, you are already well-acquainted with certain aspects of the process:

  • Our current system of anonymous peer review has been a resounding success in terms of furthering the scientific endeavor.
  • Anonymous peer review has been around for a long time now and has carved itself a firm niche within academic culture.
  • A good reviewer is worth their weight in gold (or ink?). Their suggestions, even when graciously rejecting your article, can be used to strengthen the work for eventual publication.
  • Thankfully, most reviewers are good reviewers. Most take the time to carefully and thoughtfully train their lens of critical expertise on the submissions that they receive. In most cases, the eventual published products benefit from the (usually mainly unrewarded) referee’s effort.
  • A poor reviewer, on the other hand, is one of the most aggravating people that you will ever encounter. Poor reviewers take many forms. There are the ones that seem to have not read your paper in the first place and ask questions about things that are explicitly mentioned in your submission. There are those who seem to have an agenda, either scientific or otherwise, and who wear that agenda on their lab coat sleeve. And there are those who obviously don’t have the time or inclination to give a proper review and so either cursorily reject (usually) or accept your paper but who offer no helpful advice in their five-sentence paragraph to the editor. There is no real recourse for response; no real opportunity for dialogue. The review is the review is the review. Good, bad, ugly, or very ugly.
  • The system can be slow, not necessarily because of careful consideration by reviewers, but simply because a manuscript can sit for weeks or months on someone’s desk before they get reminded the seventeenth and final time by the journal editor to complete the review.
  • No one has ever received tenure or promotion on the basis of their careful and fair reviews of others’ articles. Conducting reviews is vital to the ongoing work of science,  but is a generally thankless job.

There are any number of peer review horror stories out there. Some of them are real. Some of them stem from the fact that nobody likes to get their work rejected. So it’s tempting to ascribe villainous motives to the anonymous reviewer(s) who stopped your article in its tracks. It is often hard to differentiate a legitimate beef from sour grapes.

Sir Winston Churchill is reputed to have said, “(i)t has been said that democracy is the worst form of government except all the others that have been tried.” And the same might be said for anonymous peer review. The fact of the matter is that peer review has served science well and continues to do so to this day. But that doesn’t mean that the current system is the pinnacle accomplishment of the scientific publishing process. Life evolves. Culture evolves. Technology evolves.

To stretch the evolutionary analogy, are we witnessing something akin to directional selective pressure on the anonymous peer review process? If so, where is the process being pushed? Are there better forms of reviewing that we have not yet tried because, until recently, our technology would not permit them? As technology changes, will peer review also change and become better – both for the scientists involved and for the furthering of our scientific knowledge in general?

Along with the recent discussion about more open science  and more “crowd” involvement in the process, we are also hearing some interesting ideas about changes to the review process. One such idea was recently presented by James Rosindell and Will Pearse at the PLoS Biologue blog:

Peer review is an essential part of science, but there are problems with the current system. Despite considerable effort on the part of reviewers and editors it remains difficult to obtain high quality, thoughtful and unbiased reviews, and reviewers are not sufficiently rewarded for their efforts. The process also takes a lot of time for editors, reviewers and authors.

And their solution:

We propose a new system for peer review. Submitted manuscripts are made immediately available online. Commissioned and/or voluntary reviews would appear online shortly afterwards. The agreement or disagreement of other interested scientists and reviewers are automatically tallied, so editors have a survey of general opinion, as well as full reviews, to inform their decisions.

…

In our proposed system, users would log into the system and get the opportunity to vote once for each article (or reviewers comment), thereby moving it up or down the rankings. Access could be restricted to those within the academic world or even within an appropriate discipline, so only appropriately qualified individuals could influence the rankings. The publication models of established journals would be preserved, as full publication of an article can still take place once the journal is satisfied with the scientific community’s reception of the work.

There are certainly attractive elements to this idea. First, of course, is the idea of online publication of what amounts to being a preprint. This gives the authors official priority and it gets the results out to the community as soon as possible. It also allows some semblance of “democratization” as the review process would no longer be a one-way street. And, of course, it forces reviewers to be responsible for their comments and decisions; the lack of such accountability being one of the biggest issues with the system of anonymous peer review.Referees would also receive explicit credit for their good, and not-so-good, reviews. A great reviewing track record may be the sort of thing that could actually be rewarded within the academy. There would be a real incentive to conduct good reviews.

However, I have concerns as well. Just as with “liking” on Facebook, this has the potential to become a popularity contest. And science is not about popularity. It is about truth. And truth can come from unpopular sources. There is also the likelihood that some researchers in highly competitive fields will only sign on to such a system with extreme reluctance due to the fear of being scooped.

Beyond that, would already overworked researchers really take quality time to thoughtfully comment on preprints? And, would there be ways to game the system, analogous to people trying to increase their search engine rankings? Finally, what about small and boutique journals? The authors of the new peer review proposal envision a marketplace where editors bid for articles within the ranking system. As the editor of a small, regional journal, I am worried about what would happen to journals like the one that I oversee. Would we be able to win bids for quality papers? Or would we get lost in the shuffle after over 100 years of service to the scientific community?

As with the shifts that are occurring with the move toward open access and away from impact factors, I am positive that peer review will also have to change. And it’s good to see that people are thinking about how those changes will come about. Hopefully some of the various concerns with the intended and unintended consequences of changing the system will also be thoughtfully considered. There’s nothing wrong with moving quickly as long as you apply the brakes appropriately around the corners.

A quick post script: It should be noted that the peer review process is not a monolithic edifice of utter similarity across the board. Some journals (e.g., BMJ) have been practicing open peer review for quite some time now. And some new journals (e.g. PeerJ) are also pushing into new territory on this front.