Not signed in (Sign In)

Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

  • Sign in using OpenID

Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

Welcome to Math2.0
If you want to take part in these discussions either sign in now (if you have an account), apply for one now (if you don't).
    • CommentRowNumber1.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 12th 2012
    • (edited Oct 12th 2012)

    It would be helpful for the future of peer review to have a standard checklist to guide referees as to how to construct their report. I think it would encourage higher quality reports and reduce (at least the appearance of) opinionated decision-making. Also, if we are to move to post-publication review, which seems likely, it will be useful to have a fairly standard (but customizable) template.

    I have received something like this for some journals, and for some CS conferences, but it is not the norm in my experience. I have read of the need of them in experimental fields (“is the data freely available?”, etc). I am thinking of a more math-specific list. If it is a good one, we could ask organizations such as IMU or COPE to endorse it.

    So, what items would you like to see in such a list? I suggest we separate correctness, novelty and estimated importance and have a 3-section list. I will start with the empty list, so as not to unduly influence anything, but will participate later.

    • CommentRowNumber2.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 14th 2012

    To clarify, I have just read every question on MathOverflow found by searching for “referee”, and some are relevant to this. I also found via Google Search some suggested checklists from various non-mathematical journals and writers. But I have not found anything useful enough to make my suggestion unnecessary. I can expand on the reasons why I think such a list (comprehensive!) is good but I would rather we just get on with it. Is this forum actually alive? I don’t see how this project is appropriate for Math Overflow, and welcome any advice as to which forum I should present it.

    • CommentRowNumber3.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 14th 2012

    By the way, I just realized that academia.stackexchange exists, and is interesting, but it also fails to answer the question.

    • CommentRowNumber4.
    • CommentAuthorHenry Cohn
    • CommentTimeOct 14th 2012

    I think it would be hard to make much impact with a standardized checklist for referees. When journals give me a checklist, it has almost no influence on my report: I write the report I would have written anyway, and then I spent a minute or two addressing the questions from the checklist (generally I have nothing interesting to say about that question if I didn’t already address it in my report). I don’t think I’ve ever run across a thought-provoking question on a list like this. It’s generally obvious stuff; for example, SIDMA’s list is as follows:

    1. Is the paper’s subject appropriate for SIDMA?

    2. Is the work correct and original or, if a review paper, of wide appeal?

    3. Does the paper meet the policy standard for being of the highest quality?

    4. Is its presentation clear and well organized?

    5. Is the notation well-conceived and consistent?

    6. How does the paper relate to current literature?

    7. Are the references complete, relevant, and accurate?

    8. Does the title accurately characterize the paper? (If not, please suggest an alternative in your report.)

    9. Does the abstract properly summarize the paper without being too vague?

    10. Does the introduction relate the paper to contemporary work and explain the purpose of the paper?

    11. Is the length of the paper reasonable, given its content?

    12. Is the quality of the English acceptable?

    These lists probably have some value for new referees (although I hope first-time referees are getting some mentoring from someone beyond just written instructions), and they may prompt lazy referees not to skip over issues, but I’d guess that they improve only a small fraction of the reviews.

    • CommentRowNumber5.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 14th 2012
    • (edited Oct 14th 2012)

    Thanks. The union of a few checklists like this might be a good start. If people have them from their recent referee assignments, or are editors and can supply, that would make the task easier. I have seen this SIDMA one but couldn’t find in my email some others that I recall.

    Refining the categories above seems useful. #2 is a big point. TRUE and NEW should have more detail. I would like to give some guidance in a checklist on how to check NEW - it is a lot harder than most people think, in my experience.

    A few points of a more optimistic nature:

    • Even a small fraction is better than none.
    • It is indeed the case that many people refereeing are inexperienced. My proposal to separate out the correctness and significance testing and farm out more of the latter to junior researchers would exacerbate that.
    • I certainly never got any mentoring. Steven Krantz’s book on mathematical writing would be the closest and it came out after I had already started refereeing I think.
    • It is never easy to change the behaviour of those set in their ways - see the Elsevier boycott trajectory. However we need to think about the next generation. Peer review is changing whether we like it or not, because the current system is going to collapse under its own weight. There are simply not enough good pre-publication refereeing jobs being done relative to the huge number of papers.
    • This checklist can be used for post-publication review. I can foresee more competition for recognition as the number of papers grows. Most have no readers and attracting a decent “score” may become very important. A standard scoring system would help.
    • Best practice guidelines are a good thing even if experienced people want to ignore them.

    Comments on my motivation for this list:

    • I am going to be managing a journal which I intend to try to improve substantially in terms of process. I have received many ridiculous referee reports. My least favourite type are full of value judgments that are not substantiated, claims that the referee could prove a stronger result more easily, … These are just not useful. I don’t want to be doing this to authors submitting to my journal. A list would at least surely exert some kind of restraint on the ego of some of these referees.
    • Such a checklist should be publicly available to authors - it should improve quality of submissions if they believe that referees will be this rigorous, even if they aren’t quite as rigorous as the list implies.
    • In general, I really think that a substantial number of active members here and on Math Overflow seem to live in a different world from the vast majority of working mathematicians. We don’t live near and interact socially with the mathematical establishment and personally know a lot of Fields Medallists. We attended respectable institutions but are not hanging around a top 5 graduate school. Nor do we know many editors of journals personally. Some of us suspect that considerations other than quality (in particular, fashion) influence decisions such as grants, hiring and acceptance of journal papers. I have been shocked at the unscientific, poorly thought out and illogical opinions (really prejudice) displayed at times by quite well-known people, (the words “everyone knows that …” usually give it away). We are not certain that we know the “right” way to do everything, including refereeing. We don’t care about Inventiones and we are actually interested in our work being applied to something socially useful in our lifetime. I realize that “we” might actually mean “I, and no one else”.

    Therefore, I favour anything that improves the appearance of fairness and the making decisions based on merit.

    • CommentRowNumber6.
    • CommentAuthorAndrew Stacey
    • CommentTimeOct 17th 2012

    I like the idea of such a list, but I think I’d attack it from the other end. As a user of publications what do I want to be able to assume when I see the magic words “this paper has been refereed” (though it’s more common to see “this paper has been peer reviewed” which is almost always incorrect).

    To that end, my list is a bit like this:

    1. As a reader then I use “this paper has been refereed” as a first line in limiting my search. Although probably I use the almost synonym “this paper has been listed in MathSciNet” as the real filter. So my first criterion would be “Is this worth listing in MathSciNet?”

    2. As an evaluator then I use “this paper has been refereed” again as a first line in limiting my evaluation of a candidate’s CV. So my second criterion would be “Is this worth appearing on a person’s CV?”

    3. As an author then I use “this paper has been refereed” as the first line in limiting what I put in my references. So I take it to mean “If there’s a mistake in it, someone else is going to be embarrassed too”. So my third criterion would be “If someone spots a mistake in this, am I going to be embarrassed if it becomes public knowledge that I refereed it?”

    I realise that this isn’t all that helpful to you as none of it is relevant to the actual journal. But then the information as to which journal an article is published in is never anything that I take into account in any of the above scenarios.