Not signed in (Sign In)

Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

  • Sign in using OpenID

Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

Welcome to Math2.0
If you want to take part in these discussions either sign in now (if you have an account), apply for one now (if you don't).

Peer Review: improving citation indices

Bottom of Page

1 to 7 of 7

  1.  
    • CommentRowNumber1.
    • CommentAuthorMark C. Wilson
    • CommentTimeMar 21st 2012

    Daniel Lemire and colleagues are aiming to find a better algorithm to measure importance of research articles by incorporating the context in which the citation is made (for example, distinguishing between “courtesy citations” inserted to placate referees and real pointers to important work). They need some data and it looks like a low burden for each researcher to provide it. Check out http://lemire.me/blog/archives/2012/03/20/from-counting-citations-to-measuring-usage-help-needed/ for more.

    I think we have passed the point of no return with bibliometrics in evaluating researchers and articles. They will be used, so it is to our benefit to ensure that less bad ones are used.

    • CommentRowNumber2.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 28th 2012

    A related topic: can we and should we have some recommendations for how to cite properly? If citation indices are going to be more highly used for evaluation of research(ers), as I believe they will in the near future, surely it makes sense to try to adopt reasonably standard practices. I am sure this has been discussed on this forum but can’t find it, so am adding it here. MathOverflow has some discussion on related topics, but not this one as far as I can tell.

    Ideas:

    • limit the number of citations allowed in any one paper (so as not to debase the currency any further)
    • explain to people that courtesy citations (where there is no intellectual debt to acknowledge, but just to show to the author is aware of other work) should be avoided as much as possible except in survey articles
    • explain to beginners that citations should be to the first, the best and the most recent work on the topic (it is not necessary to list absolutely all work, since the point is to assign credit for good work and to help newcomers to navigate the literature)
    • CommentRowNumber3.
    • CommentAuthorHenry Cohn
    • CommentTimeOct 29th 2012

    I wouldn’t endorse the idea that we should change our citation behavior for the sake of bibliometrics. Citations serve a really important purpose, and it’s not worth compromising that to help with their side application in evaluation.

    limit the number of citations allowed in any one paper

    Of these ideas, this one seems the most problematic. In extreme cases, I can imagine saying someone was citing too many papers. (Imagine a brief note with a gratuitous 1000-item bibliography of all work ever done in this area.) However, within reason I believe authors should be allowed to cite as many papers as they consider appropriate.

    explain to people that courtesy citations… should be avoided as much as possible except in survey articles

    explain to beginners that citations should be to the first, the best and the most recent work on the topic

    I’d argue that many papers should be a little more like survey articles, rather than less. Not to the point where it becomes a distraction from the main purpose of the paper, but a bit more background for beginners is usually helpful, even sometimes for experts. I don’t recall ever reading a paper where I wished the authors had cited fewer sources, but I have read papers I thought should have cited more.

    • CommentRowNumber4.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 29th 2012

    I worry that citation measures will be increasingly used, and we can’t stop it. So I would like them to have a more standardized meaning. They do serve an important purpose, but I am not sure that there is agreement on what that purpose is.

    Here is another semi-random thought. Maybe we should have two types of citations, and they should be distinguished. The first type is everything that is essential for the paper, in the sense that without the cited work, the present work is incomplete. The second type is for all other citations (mostly courtesy). Papers could have a separate section for all the second type citations, and the first type ones are cited when needed in the body of the work. These would have to be enforced by a journal, for example. I can see that there would be problems implementing this, because it might be tempting for authors to include all arguments so as to shorten the first list, or include almost none by citing far too many other works (although the latter presumably will make the present paper seem rather weak so I doubt authors would do it). It might also be tempting for some not to give credit to earlier work. I suppose that incentive is already there in some sense (at least, it is easy to be lazy in looking for that earlier work) and it is an issue that should be addressed by refereeing, but I’m not sure about exactly how.

    If citation measures are not used for evaluating quality/impact of research, then what? I am very suspicious of judgments on the quality of mathematical works made by small in-groups of “experts”. Widespread (“crowd-sourced”) ratings might be good, but the problem many people foresee is that almost all works will receive no ratings.

    Of course if we are only about the quality of research produced overall, most of these considerations are less of a concern (for example, there are big incentives now to collaborate on papers, since the credit does not seem to be divided between the authors - this can be a problem if we evaluate individuals, but isn’t a problem for the overall research enterprise). But I very much doubt we will achieve this kind of world very soon. In fact it seems to me that trends are going the other way.

    • CommentRowNumber5.
    • CommentAuthorAlexander Woo
    • CommentTimeOct 29th 2012
    • (edited Oct 29th 2012)
    First of all, it's not at all clear what 'important' means in the context of a paper. For example, you might cite some paper that asked the question you are answering, but you use no techniques from that paper. Is that 'important'? Some citations from my papers are undoubtedly important to the paper and some are undoubtedly not, but it is not clear to me whether to consider many (perhaps even a majority) of the citations important or not.

    On a different point, since mathematics is part humanities and part science, it perhaps makes some sense to get a sense of how the humanists are trying to handle this problem. I don't know anything about this.

    (Going even further, in the arts it is rather difficult to see how this could be solved at all. It seems hard even to show that Mozart is more influential (rather than more popular) than Salieri. It seems even harder to show that Elliot Carter is more influential than Vincent Persechetti (i suppose someone with certain tastes could argue this) or Norman Dello Joio (I think that's less debatable).)

    Finally, I think the real problem is that we are trying to distinguish between epsilon and epsilon+delta^2. Most of us have practically no impact on human affairs at large; our biggest impact is being part of the critical mass needed to pass on the culture of mathematics (or the culture of our area of mathematics). Essentially there is a scarce resource (jobs in this case) to allocate, and the cost of collecting the information to allocate this resource on a rational basis greatly exceeds the benefits of allocating this resource rationally. So it is actually rational to allocate this resource on a somewhat arbitrary basis. Would you be happier if we made this explicit by taking all the qualified applicants for a job and hiring one of them at random?
    • CommentRowNumber6.
    • CommentAuthorMark C. Wilson
    • CommentTimeOct 29th 2012

    I agree with most of Alexander’s comments - we should only really be making quite crude distinctions, perhaps at most 4 equivalence classes of researcher ability. I do advocate randomness in decision-making in some situations where it still seems to be socially unacceptable. In this case math researchers usually also teach and do administration, so evaluation is not just based on research. And I am past worrying about getting a job (unless something very bad happens where I am now). I am more concerned about concentration of resources such as funds for PhD students and postdocs, which, if crude quantity-based measures of research output are used (and unfortunately they are rather more widely than we would like) just lead to even more production by those with the extra resources, but not necessarily an advance in science or a good use of human capital. (That is close to my record length for an English sentence).

    • CommentRowNumber7.
    • CommentAuthorPeter Krautzberger
    • CommentTimeOct 30th 2012
    • (edited Oct 30th 2012)

    Unsurprisingly, other people have encountered the same issues regarding standards for citations. Here’s a short introduction by Martin Fenner to CiTOs which some reference managers already support.

1 to 7 of 7

  1.