Blog – Do we need to peer review the process of peer review?

Welcome back to my corner of the Dementia Researcher blogs! Does that title seem weird? I hope not, it’s something I’ve been pondering on in recent weeks. Firstly, let me start by saying thanks to everyone who reached out to me on the back of my last blog about spending some time in hospital. I am happy to report that I have been out for a month and a half now and continuing to do well! This has resulted in me going back to work and resuming my duties. One of those duties… finishing off writing a paper.

This got me thinking about the whole process of how we publish new scientific data. There is a general pipeline that all researchers follow:

• Step 1 – Come up with a hypothesis to put to the test. It helps if this is based on previous findings/ideas. Don’t just go with something mental and baseless. Nobody will fund that!
• Step 2 – Design experiments to test that hypothesis, conduct them and analyse the data
• Step 3 – If you generate interesting and “novel” (I hate that word) data select a suitable target journal and draft it into a draft scientific publication
• Step 4 – Send your shiny manuscript off to the peer review gods and pray they don’t immediately rip it to shreds, thus destroying your hypothesis and your dreams.

Every step in the process is a complex one but, for me, step 4 is the biggest minefield. I have control over steps 1-3. If experiments don’t work as planned I can react and change tact accordingly. My hypothesis can evolve. It’s fluid. So many research projects go down completely unexpected branches off the back of “bad results”. Peer review, however, is completely out of your hands. For those of you who aren’t sure… this is what happens.

Once you send your draft to a journal you are essentially saying “I think this study is of significant interest to your readers for you to publish it”. The editor of the journal has first refusal on whether they think it could be of interest. They will usually give it a quick skim and either say “no thanks try somewhere else” or they will then send it to peer review. If it goes to peer review, the editor then finds 2 or 3 researchers that work in a similar field to the one your study is based in. For example, if you work on cell based models of Alzheimer’s, the editor will invite people who also work in that field to review your paper.

If a reviewer accepts, their job is to go through you draft with a fine tooth comb. They scrutinise your data to see if it is tight and coherent. They analyse the draft as a “story” i.e. does it tell us anything new and interesting? They also check things like the quality of writing and data presentation. Ultimately, they have to answer one question. Have you presented enough interesting, exciting and useful data to justify the publication of your work? They then have 4 options:

• Option 1 – Accept the paper with no corrections required (the perfect and rarest of scenarios)
• Option 2 – Accept subject to minor tweaks (this is great you just need to tweak some of the wording or adjust some of the data figures)
• Option 3 – Accept subject to major revisions (there’s something cool here but you need more data. Back to the lab!)
• Option 4 – Reject (I think reviewers are overzealous with this one)

Sounds like a rigorous system right? In theory it is but after 9 years in research, I really see the flaws. It can be better. Let’s start with my opinion on paper rejections. On multiple occasions I have had reviewers reject a paper. That’s common for any researcher. What is annoying though is how often the feedback you receive is either unhelpful or questions their decision. The reviewer has to give a reason for the rejection and I’ve had quite a few rejections where they have gone “you are missing this data or that experiment to back up your claims”. I have read those comments and thought “I DO have that data I just didn’t include it”. At this point there is nothing you can do. The paper has been rejected. Can we all agree to do one thing in peer review please? Start swapping rejections for major corrections? They might already have the data you thought was missing! On top of that can we stop normalising the idea that you have to find corrections to suggest? It’s okay to accept with no corrections if the paper is good. Corrections for the sake of it are totally pointless!

Secondly, reviewers don’t get paid any extra to be part of the process. Most reviewers do a great job but I have had some that have clearly not read the paper and just rejected it based on a skim read. Fortunately on those occasions the editor has stepped in but I can bet hundreds of interesting papers get rejected every month because the reviewer has no incentive to do the job properly. It takes quite a long time and I understand why anyone would prioritise tasks that pay the bills. This can make the process long, arduous and ultimately unrewarding when you wait months just to have your paper rejected with minimal feedback. Journals charge a fortune for subscriptions, publications fees etc. Why do they expect their reviewers to work for free? Share the load guys!

I could keep listing issues with peer review but I will finish on perhaps the most worrying. Competitive judgement I will call it. It is not too common but it does happen. It’s when a reviewer gets a paper, sees it as a “competitor” to their work and rejects it so they can get theirs out first. I have heard many researchers say “we sent out paper to journal X, it got rejected and a few months later a very similar paper was published”. This needs to be addressed. Science should be 100% transparent. It’s not a competition. We are here to benefit people from our ideas and discoveries.

Peer review may be the best system we have at the moment but that doesn’t mean we cannot improve it. It’s 2022, maybe it’s time we peer reviewed peer review.

Dr Sam Moxon

Author

Dr Sam Moxon is a biomaterials scientist at the University of Manchester. His expertise falls on the interface between biology and engineering. His PhD focussed on regenerative medicine and he now works on trying to develop 3D bioprinting techniques with human stem cells, so that we better understand and treat degenerative diseases. Outside of the lab he hikes through the Lake District and is an expert on all things Disney.

Follow @DrSamMoxon