Disallow anonymous reviews?

[tkuhn]

I had a discussion the other day with somebody arguing for enforcing and not only encouraging non-anonymous reviews. The main argument was that reviewers should stand by their reviews and not be given the opportunity to hide behind their anonymity or even provide reviews despite covert conflicts of interest.

I am still inclined to allow anonymity but strongly encourage attributed reviews, but I think it's worth discussing. I don't think anything really bad would happen if all reviews had to be attributed, but in a small number of cases there can be valid reasons for reviewers to stay anonymous.

As a middle ground, we could force reviewers to state their reasons when they opt for anonymity. This might give us interesting insights.

I am interested to hear what others think.

[phitzler]

On SWJ, reviewers can opt for anonymity by simply checking a box. We get about 20% anonymous reviews. I do not have any hard data on this, but I believe that not allowing anonymous reviews, or strongly pushing for anonymous reviews, will introduce a bias. On the one hand, I am rather certain that it will deter some reviewers. On the other hand, it will probably significantly help papers by prominent authors. It will also make the task of correcting for positive bias even harder for the editors and EiCs - even with the SWJ system we have to carefully weigh each review, and sometimes read between the lines, while almost ignoring the "score" given by the reviewer (The scores tend to be too positive very often, compared to the review text).

[micheldumontier]

In addition to being formally recognized on the final paper, perhaps we need another incentive - that the weight for anonymous reviewers will be half of that than non-anonymous reviewers. that said, our system will requires reviewers to login with orcid ids, so we can track their performance relative to their peers.

[phitzler]

I would not formalize the weight. My experience is that you simply need to look at all the reviews, i.e. at the content, and weigh all the arguments. That a reviewer is anonymous is sometimes important, although the effect may be mainly in the other direction: Sometimes a reviewer is rather direct in assessment and wants to protect her or himself by remaining anonymous. If it's good arguments, they should not be discounted. On the other hand, a non-anonymous review which doesn't bring good arguments shouldn't be taken into consideration quite as much. It's hard to believe, but we did have very weak four-line reviews which were non-anonymous (I guess the reviewer missed that the name would be public).

[tkuhn]

Thanks for sharing your experiences. 80% of non-anonymous reviews is a very impressive number. So, reviewers mostly don't really mind not being anonymous. This is really good news.

Yes, forcing reviewers to be non-anonymous would probably introduce a bias, but then again almost all decisions about the reviewing process do. The biggest bias is probably introduced when editors decide which reviewers to invite...

I was also thinking that we shouldn't have a formalized weight but make sure the editors mainly look at the arguments and focus on the ones that they think are valid and important.

What about having a simple checkbox to tick if reviewers want to stay anonymous, plus an optional text field where they can state their reasons for this decision? This alone might make reviewers think about their reasons to stay anonymous a bit more, and might nudge some of them to have their reviews attributed.

[phitzler]

I think the optional field is a great idea. In fact I'll look into getting this for SWJ as well.

[essepuntato]

Sorry to enter in the discussion only now. I do not have direct experience in dealing with these stuff within an academic venue, but while I really love the approach introduced by @phitzler in SWJ, I would also like to have a more drastic approach for this new journal.

Thus, while I understand the pros and cons of having anonymous reviews vs. non-anonymous reviews vs. leave-the-reviewer-decide-to-be-anonymous-or-not, I would fully support the idea of having only non-anonymous reviews. Then, I know, its up to the editors (that have a quite important role within this framework) to balance appropriately the reviews according to possible bias. But I firmly believe that obliging reviewers to be more responsible for what they write (by signing explicitly their reviews) is the (mandatory) right way to follow.

My two cents.

[micheldumontier]

There are valid reasons for why a reviewer, who is providing valid feedback, may desire to remain anonymous. These are issues related to power or discrimination, perceived or otherwise previously manifested, which could also result in a reviewer not providing a review, thereby equally introducing a bias. I think it's ok for us to ask for a reason, and that this reason will be kept confidential to the editors alone. perhaps after a year or two, we can learn the real reasons why people want to remain anonymous.

[essepuntato]

Hi @micheldumontier,

I see your point. But, if you are looking for answers for anonymity, why to use an optional text field for feedback? Wouldn't it be better to ask for mandatory feedback in such cases?

[micheldumontier]

yes, i'm ok with it being mandatory in this case.