## Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

## Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

• CommentRowNumber1.
• CommentAuthorMark C. Wilson
• CommentTimeMar 27th 2012

I chose an academic career at least partly out of idealism. Bitter experience has shown that academics may not be motivated by the same incentives as people in other professions (e.g. salary) but can be just as selfish. So, I want to know what I get out of refereeing. I hear that it is becoming harder and harder to find good referees. Right now I have been waiting 6 months for reports on 2 of my papers, neither of them long and difficult to follow, submitted to different journals. I can understand the plight of referees, as I don’t have time to write my own papers, let alone read many others (perhaps I should spend less time on online discussions). I usually finish my job on time, but sometimes at the expense of thoroughness (I like to think that I find a Pareto optimal solution to the multicriteria optimization problem).

One problem as I see it is that the secrecy of referee reports means that only the editor dealing with the referee has any idea of the quality of the referee report (and referees don’t get feedback on the quality of their refereeing from anyone other than the author, who may be biased). Journals often don’t give clear instructions as to how to referee, which makes it harder. A system where more credit is given for refereeing (some kind of reputation score?) might go a long way toward finding good referees. Perhaps separating out the correctness and significance checking would be good. For mathematics we primarily want referees for checking correctness. This is a task that can be well performed by relatively junior people who have more time and an incentive to impress their elders. If they can get credit more publicly for this work, so much the better. I haven’t yet thought of a good mechanism for assigning credit, however. But if refereeing is considered an integral part of the research enterprise, it needs to get more credit. As does writing surveys.

Apologies of some of this has been answered in other threads. They are getting a bit stale now and search is not as easy as I would like.

1. You write that in math we need referees mostly to check correctness, but it seems to me that this is the part that is not assumed to be thoroughly accomplished (many of my elders have told me right from the beginning thinks like “you have the charge for writing flawless papers, because you cannot expect the referee to find your errors”). I feel like the editors want above all to know whether the paper is suitable to their journal, and they stress it when asking me to referee.

Your idea of using youngsters to check correctness is quite good, and in fact I have been told that it is quite usual for Princeton grad students to check papers submitted to the Annals. I guess it is not the only journal doing this.

• CommentRowNumber3.
• CommentAuthorMark C. Wilson
• CommentTimeMar 28th 2012

I guess I have a crisis of confidence in refereeing (“pre-publication review”) and correctness checking in mathematics was really the only reason I can think of to keep it (I am not at all sure about other disciplines). Decisions on significance belong to a bygone era of scarcity of outlets. We now have plenty of places to publish and decisions on significance will increasingly be measured by citation indices, prizes, perhaps selected paper networks, etc. Journals and their suitability criteria don’t make much sense because as far as I know no one reads them in their entirety any more - the paper is the natural unit and most are found via arXiv announcements, Google search, conference talks, etc (no doubt better tools for finding the good and relevant stuff among the deluge of papers will be developed soon - for example I would like to follow certain authors easily far more than I would follow any journal).

The main point I wanted to make is that we need to incentivize refereeing, or it will essentially die out. The increasing pressure to publish and the credit one gets for doing so must be counterbalanced by incentives to read carefully and be critical of research findings. I don’t see that happening at the moment. Since many people here still seem to like traditional journals, provided they are cheaper, I would like them to say how they think we can deal with the problem.

• CommentRowNumber4.
• CommentAuthorzskoda
• CommentTimeMar 30th 2012

Decisions on significance belong to a bygone era of scarcity of outlets.

Well, I do not quite agree. If there is a typical importance deficiency like lack of examples and applications etc. it will be iterated statistically in most of refereeing processes in which a particular paper is submitted. So, the author will be compelled to care about it: colleagues will warn, look you will have a problem with publishing this, he will be aware if he had such a moment in previous publications of similar kind or in similar area, he will try to remedy the problem; sometimes the author will abandon an idea which was so unfruitful, and sometimes will care more to find applications. So the pressure of importance is in general a good pressure and scientific environments in which it is nurtured are usually better ones.

• CommentRowNumber5.
• CommentAuthorMark C. Wilson
• CommentTimeApr 2nd 2012

The importance filter can be applied after publication (“postpublication review”, citations, …). I see no real benefit in allowing the tastes and opinions of a few people, no matter how widely acknowledged as strong mathematicians, to prevent correct work from being published. If that work is clearly suboptimal in that the author could improve it with reasonable effort, then by all means reject. But rejection as “not interesting” is nonsense - it may be OK for CS conferences, but for mathematics, which aspires to be timeless, it is a bad idea.

• CommentRowNumber6.
• CommentAuthorzskoda
• CommentTimeApr 3rd 2012
• (edited Apr 3rd 2012)

to prevent correct work from being published

It can be published in low level publication niche. The space in journals and even more the time we have to read others work is limited. So we do not want to be messed up with nicely looking papers which offer little substance.

It is to prevent correct but unimportant place to replace the important work, to prevent taking space in our limited time. If I have rare time to open Annals of Math. I do not want to waste time to something what is not important. Math would be much better if people would not spend so much time to various polution, arbitrary work.

As far as “timeless” modifier, look at mass of work in your favorite historical period, say in 1930-s and compare an average article to something what is an important contribution and see how little will you will have to read former and how much to read the latter.

By the way, Mark, we must have met around 1993-1995 in Madison ?

• CommentRowNumber7.
• CommentAuthorMark C. Wilson
• CommentTimeApr 7th 2012

Hi Zskoda - I don’t remember any such meeting, but I was there 1990-95.

I agree - too much is published for us to read it, yet people do need to keep publishing. Many works should be published (i.e. disseminated) in (essentially) write-only venues, provided that they are properly abstracted and keywords are assigned, so that they can be found by search engines if someone is doing a background search. How to decide which papers should actually be read in detail is a hard one, and perhaps competitive conferences could work. The fact that so few papers are read and the process by which this is decided is suboptimal has been discussed often. Noam Nisan has written about readerless publications and I have an old blog post on “Journals we might need”. I think Math Reviews could work well if we could get multiple reviewers for each paper. There are a few other ideas on this forum too. However it is still very important that correctness checking be done.

I haven’t had a compelling answer from anyone to the original question. Why referee? Not why it should be done, but why should an individual do it - what are the incentives?

• CommentRowNumber8.
• CommentAuthorHenry Cohn
• CommentTimeApr 8th 2012
• (edited Apr 8th 2012)

Regarding individual incentives for refereeing, I think it’s mainly social pressure rather than a cost-benefit analysis. There are certainly benefits to refereeing - reading papers in your area that you might not have chosen to study otherwise (so you get a broader perspective, and perhaps identify problems worth suggesting to students even if they don’t change your own research agenda), helping authors, feeling good about contributing to the journal and being recognized as an expert, impressing editors with your insight or reliability, etc. However, for anyone senior enough to say this with a straight face, I think it’s clear that the optimal selfish strategy is to accept refereeing requests only if the paper looks interesting and to turn down other requests with an apologetic note indicating the large number of other papers you are (supposedly) refereeing. Fortunately, I don’t think many people do that dishonestly, if any.

It’s kind of amazing that the system works as well as it does, and that could be good (it’s very robust) or bad (the long tradition and universal agreement are key, and if we damage this system, we may never be able to replace it with anything that works as well). I think ambiguity plays an important role in enforcement. Nobody really knows what happens if you don’t do your duty. Will editors suspect you of laziness or dishonesty, and be annoyed? Will this information spread and give you a bad reputation among influential people in your subfield? Could it lead to worse letters of recommendation? Will editors be less diligent about ensuring your papers get refereed and published quickly if you have been an unreliable referee? Will a university promotion and tenure committee (or post-tenure review committee) someday ask which journals you have refereed for, and is there any chance they might detect it if you lie?

Probably there won’t be serious career consequences to refusing to referee (with repeated excuses, not an explicit refusal) or doing a bad job, but it’s hard to say for sure, and that ambiguity is crucial. You can’t even tell whether you are getting away with it yourself, or whether you are suffering undetected harm. Even someone with no conscience or sense of duty knows that a research career in mathematics is a delicate thing, and won’t want to take unnecessary risks.

By contrast, if there were a clear-cut and minor penalty for bad refereeing, like losing a modest number of MathPoints, then refereeing would be much more susceptible to a cost-benefit analysis, and more people might decide they had plenty of MathPoints already. (In the other direction, a clear-cut and major penalty could be really unfair, and this is a difficult thing to balance.)

One of my hopes for future publishing systems is to leave things vague enough that it discourages this sort of cost-benefit analysis. I think we’re all better off if people referee out of a desire to help or a sense of duty, rather than because they’ve decided it’s a net benefit to them personally.

2. Henry Cohn's insightful comment reminds me of a passage in "Freakonomics" by S. Levitt and S. Dubner (http://en.wikipedia.org/wiki/Freakonomics), where some kindergarten introduced small financial penalties for parents who were late to pick their children up from kindergarten. The net effect was that late pick-ups became much more common. Before the penalties were introduced, parents felt that late pick-ups are unfair to people working at kindegarten, since they had to stay after hours etc. After the penalties, it was no longer matter of human decency, but simple cost analysis ("late pick-ups is something I can buy, so why not"). It might be important to keep such lessons in mind when trying to revamp review system (any institutional system, really).
• CommentRowNumber10.
• CommentAuthorzskoda
• CommentTimeApr 8th 2012
• (edited Apr 8th 2012)

In Soviet Union referees where getting significant pay for their work I was told, and hence were motivated to do the refereeing properly. I do not know why such system is completely out of discussion, at least for some niche of scientific publications.

• CommentRowNumber11.
• CommentAuthorHenry Cohn
• CommentTimeApr 9th 2012

Regarding paying referees, one issue is that I really like the idea that we’re working together to referee papers for free, so I’d be sad to give it up (although it could make sense if it really leads to better reports). Another is that it seems difficult to judge how much of a difference it might make. For example, there were presumably a number of big cultural differences in the Soviet Union, and it’s difficult to disentangle them from the payment issue.

But the thing I’d worry the most about is irreversibility: people are very unhappy if you ask them to do the same thing for less money. My guess is that in an experiment with paying referees, the payment would lead to a modest increase in willing to referee and in amount of time and effort spent. However, if the money ran out or if it didn’t seem like an efficient use of funds, then it might leave the system in a worse state than before the experiment.

For example, the way I’d imagine paying referees is as follows. There would be a two-stage refereeing process: first, senior mathematicians weigh in on how interesting and important the paper is (without receiving any payment), and then junior mathematicians are paid to check the mathematics and writing very carefully. [This division would help avoid questions like “What if the leading expert doesn’t want to do more work?” or “How could you pay Terry Tao enough to make this a sensible use of his time?”] My fear is that once you move to a system like this, you’ve established that a careful reading is worth money, and it will be harder to get anyone to do it for free in the future.

• CommentRowNumber12.
• CommentAuthorTom Leinster
• CommentTimeApr 9th 2012

I wouldn’t referee for money, unless it was an unrealistically huge amount. Like many academics, I’d rather have more free time than more money.

However, I’d continue to referee because it’s the right thing to do. I agree with Marcin Kotowski: the comparison with the Freakonomics kindergarten example is an excellent one. See Tim Harford’s article Does the altruism theory help anyone at all? for more along the same lines. The main point, as I understand it: once you muddy the waters by introducing money as an incentive, people are apt to forget about other incentives. (Here I use “incentive” in the economists’ sense: it can be anything that drives you, including idealism.)

I also agree with Henry that retreating from a paid system would be very hard.

• CommentRowNumber13.
• CommentAuthorzskoda
• CommentTimeApr 12th 2012

This money is just complimentary, though it was substantial in that system: it was not the only reason why people refereed and it was not always the same. In west, refereeing books (out of science) is still paid in a similar way. In physics, some journals give some small money to the authors. I know that Reviews in modern physics have articles only by invitation and the authors get something like 500 dollars for the article. This is negligible for most of those top people (to publish there one needs to be rather close to Nobel Price level in physics) but still it gives them the reason to feel honoured and also it is pleasant. Feynman used to collect 1$for patent given to the patent office and buy candy for it. Others would not take that one$ but some friends of him got into the habit finding that it is nice to have those candy to cheer up :) As far as additional income, most of mathematicians are still in not so developed countries and most of them have multiple jobs to survive. My last job at IRB had the same salary per month as I get for two semesters of teaching a graduate course at the university regarding that the university pays the external contractors about 5 times less than the regular teachers. Still it makes it a more serious work once you know you are paid for that extra work. There were situations at the same university where people would get no pay at all for teaching a graduate course, if they were employed elsewhere. When you talk to the ordinary people from outside of academia and tell them that some part of your work is not paid, they often consider that you are an idiot and that you have no abilities or something. Complimentary pay, even very moderate, makes things more serious and more acceptable in a number of situations.