Not signed in (Sign In)

Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

  • Sign in using OpenID

Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

Welcome to Math2.0
If you want to take part in these discussions either sign in now (if you have an account), apply for one now (if you don't).
    • CommentRowNumber1.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 29th 2012

    Yes Minister is, of course, required viewing. (Not particularly for this forum, just required in general.) For those that haven’t had the pleasure yet, it’s a British comedy centred around a hapless government minister and his top civil servent (who runs rings around him, most of the time). The reason I bring it up is that in one episode, the minister (Jim Hacker) gets worried about how he’s doing in office and goes around asking everyone how he’s doing. Just about everyone says something like, “I think you’re doing alright.” to which he invariably replies, “But am I doing alright, or just ’alright’?”. Namely, is it a positive “alright” or a negative one? The problem is that he doesn’t know how to evaluate his performance so he goes round asking others to do it for him, but because no-one really knows then he’s never content with the answers.

    That’s me, really. I don’t know how to evaluate my own progress as a mathematician, so I go around asking others (via journals) to do it for me, but at the back of my mind is a nagging worry that the answers I’m getting don’t really make sense. This was worse, for me, when I was a new post-doc with no published papers: I didn’t know if I was “doing alright”. And in the subsequent 10 years (gosh! I’ve been a mathematician for a decade), I can identify particular signs that I must be on the right track, but they’re few and far between.

    All of which is just a long-winded way of saying: an extremely important thing to discuss here is that of evaluation. Proposals for enhancing the arXiv, or bringing mathematicians together, or implementing “Web2.0 for mathematicians” (if anyone can explain that to me) are great - or perhaps great - but aren’t the revolution that we need. To truly get out from under the thumb of the commercial publishers, and to make mathematics a professional occupation, we need to figure out a robust, reliable, fair, and public method of evaluating ourselves.

    I don’t believe that journals were ever meant to be used for evaluation purposes. But there wasn’t anything else around, so they became a convenient method. The lesson to learn from that is that if we don’t figure this out for ourselves, the people who have to make these evaluations will find some way of doing so for us - and it probably won’t be a method that we like. So if we throw out journals but only replace them with an enhanced arXiv, with ratings and comments and so forth, then it will be used for evaluating mathematicians even though no mathematician in their right mind would say that that was a good method. Scott’s linked to a post by Izabella Laba, and there’s been some follow-up discussion here (which Izabella has taken part in, I’m pleased to see), highlighting why some might be unhappy with this. Alexander Woo has commented that this must work not just for the top of the profession, but for everyone.

    I’ll register right now that I would be extremely unhappy with any system that evaluated my work (and thus implicitly me) by any sort of activity surrounding it. As a very concrete example, consider my paper on piecewise smooth loops. I would actually quite like it if someone used this to evaluate me as I think it shows my mathematical talents quite well. But I will be amazed if it gains many citations, or is even read by many people, as it’s really saying “Don’t do this.” and I suspect that people will be quite happy to be told that without knowing the gory details (so won’t read it) and won’t cite it as they won’t do what I warn against.

    So what criteria should we use to evaluate ourselves, and how should we measure ourselves against these criteria?

    What I don’t disagree with the journal system is that we should be evaluated by our mathematical output. However, I would like it to be recognised that there are many different types of output. This is something that I feel is missing (or not emphasised enough) in Tim Gowers’ proposal. Of course, I could just be reacting to the fact that in that system, I don’t think I’d measure up very well! The thing is, I’m a frog. It took me a while to come to terms with that, but now I am and I’d prefer to be evaluated as such and not as a failed bird.

    And that’s really the heart of what I’d like to see: an evaluation process whereby the person being evaluated has some direct input into the evaluation process. As others have said, a system whereby we try to evaluate every paper that is produced would creak under the strain. But how about a system whereby when someone needed an evaluation then they submitted a small number of their recent articles with some criteria that they would like them assessed against (probably drawn from a standard set). So I could choose one article that showed my technical ability, another which showed that I was well-connected to the mathematical community, and so on. I’d send this to some professional mathematical organisation to be evaluated, and then I could use that evaluation when applying for jobs, or funding, or tenure, or whatever. As I could choose the papers and the criteria, I could tune the evaluation to the specific situation in which I wanted to use it. This is slightly different to just sending in my best 4 papers to, say, the job search committee because the evaluations could be used at the early stage, whereas I suspect that the papers are merely weighed at that stage and only actually looked at if I get to the short-list.

    Of course, that’s a long way from a proposal! But hopefully it’s enough to get people thinking about it.

    • CommentRowNumber2.
    • CommentAuthorzskoda
    • CommentTimeFeb 29th 2012
    • (edited Feb 29th 2012)

    I have been writing for few years a little LaTeX document in Croatian about evaluation (so far 40 or so pages) in theoretical science. The basis point was not to have any agenda, but rather to reflect on the distinctions between various kinds, aspects and purposes, to have at least a basis to think about the problem. For example, you say above

    What I don’t disagree with the journal system is that we should be evaluated by our mathematical output.

    as if it were something obvious, but you are clearly wrong even there. Somebody who wants to evaluate may evaluate different things, e.g. how useful member of a mathematical community you are, or how good you are for some department moneywise (how many grants and patents you bring), or how good you are explaining math to others (teacher, mentor, group leader) or how good chances you have to hit something really big (this is again not measured by steady output but by the depth of the strongest ideas of yours). There are people who dedicate to solve lots of small problems and those who spend years for working on a big conjecture. Andrew Wiles for a while had no output, but insiders had strong reasons to believe that he was up to something big, even then. Not only there are different types of output, not all aspects of being good are about the rate of output, some are about the prospects, ability to hit eventually.

    One should also masure effectiveness. Some people have double output, but spent 10 times more public money, being in a position to do so. Some committees ask you to list grants and results. More grants and more results you had, the better you get in the evaluation. On the other hand, by common sense a guy who had the same level of results with less grant money is, of course a better mathematician: he performs better in tighter conditions. But it may show that he might be a worse fighter for grants, what some institutions punish for as if it were a measure of academic quality. Is obtaining a grant or a prize a form of an output ? Clearly not but most committees take it for that.

    So, if you want to find the measure of evaluating some aspect of yours being a mathematician, then make sure which aspect of being a mathematician you actually evaluate.

    • CommentRowNumber3.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 29th 2012

    Somebody who wants to evaluate may evaluate different things, e.g. how useful member of a mathematical community you are, or how good you are for some department moneywise (how many grants and patents you bring), or how good you are explaining math to others (teacher, mentor, group leader) or how good chances you have to hit something really big (this is again not measured by steady output but by the depth of the strongest ideas of yours).

    I would actually class all of these as “output”. The only part that is difficult is the example of Andrew Wiles. We can see in hindsight that his work was worth pursuing, but it could have been hard to justify during it (I don’t know - I’ve only read the pop-lit versions!).

    I think we really do need to think about what sort of output we value. A good exposition can be worth its weight in gold, for example. I’m in complete agreement with:

    So, if you want to find the measure of evaluating some aspect of yours being a mathematician, then make sure which aspect of being a mathematician you actually evaluate.

    and that’s part of what I was trying to get people to think about.

    • CommentRowNumber4.
    • CommentAuthorzskoda
    • CommentTimeFeb 29th 2012
    • (edited Feb 29th 2012)

    I made some changes to my text above, in the meantime (added things about grants) when you were answering. I know lots of people who do things quietly like Wiles and have much bigger things there than some constant output-ers publishing things which lead to nowhere, so I can not understand what does it mean “exception” there, apart from the fact that I have chosen a famous example.

    A good exposition can be worth its weight in gold, for example.

    Again, it is not a measure of a strength of a mathematician, but of usefulness to the community. So as long as you mix this with output of results you are mixing apples and pears. Efficiency, level, knowledge, creativity, focus, cooperative skills, and morality are so different aspects.

    • CommentRowNumber5.
    • CommentAuthorzskoda
    • CommentTimeFeb 29th 2012

    Do you really want to answer how useful I am to mathematics community (in that case you have to subtract all your operating costs, including grants, and you can add community service, cleaning your office, saving on computer maintance and usefulness of mathematical textbooks and exposition) or how good I am in discovering new mathematics ?

    • CommentRowNumber6.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 29th 2012

    So as long as you mix this with output of results you are mixing apples and pears.

    Only if I insist that they be measured on the same scale. If, however, I say that both apples and pears are valuable types of fruit and evaluate an apple by comparing it with other apples, and a pear by comparing it with other pears, then there’s no problem with this.

    Also,

    Again, it is not a measure of a strength of a mathematician, but of usefulness to the community.

    This, again, should be considered as part (only “part”) of a mathematician’s output.

    (Frankly, I would quite like it if all my work on the nLab and similar could count somehow. But that’s probably a long way off.)

    • CommentRowNumber7.
    • CommentAuthorzskoda
    • CommentTimeFeb 29th 2012

    If I am hiring you as a project member, why would I care about your output in refereeing papers or community work ? Just because you call it “mathematicians output” ? Do you want to evaluate output or a particular quality which is needed. In a project one needs a researcher, period. In teaching a class one needs a teacher, period. In a department one needs attraction of grants, good research and good teaching. A community funding a researcher may need applicability of research and exposition of the results to the public. Now you stubbornly say, this is all output: but the same output for different evaluation purposes is of different value. It is not about wheather the pears or apples are more worthy, but what criterion to weigh the pears against apples. In some criteria one of them may be even negligible or even of negative value.

    • CommentRowNumber8.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 29th 2012

    but the same output for different evaluation purposes is of different value.

    Absolutely!

    So the job advertisement or funding proposal or whatever says, “We want a mathematician with X,Y,Z qualities.”. I then look through all that I’ve got and select some representative “pieces of output” (of some variety) which I think best show that I have qualities X,Y,Z. I get them certified, or maybe I already have them certified, and send them to the committee.

    I’m not talking about a linear scale. At no point do I want to say “3 apples = 2 pears”. But a hiring committee should say, “We think that for this job, we put more value on apples than pears” and then as a candidate, I say, “These papers, lectures, blog comments, show that I’m good with apples.” and that - I hope - gets me through the first round.

    • CommentRowNumber9.
    • CommentAuthorzskoda
    • CommentTimeFeb 29th 2012

    Yes, I know, but the BELIEF that the committee knows which thing is measured by which kind of fruit is unfounded. Once a measure is on the market, by inertia it is applied to things which it should not apply for. So it is important not to mix and scale, but to reason sharply. E.g. as I said, having the same results with more grant money means lower efficiency/productivity. The standard productivity measure is output/input where the input is in general a combination of human and material resources. Do you really think that grant proposal reviewers think about that ? They think weather some CV looks bright. And getting a grant looks bright. Never mind that it is on other side of the equation, PROVIDED the results are the same and provided further grant attraction is not a purpose of the grant.

    • CommentRowNumber10.
    • CommentAuthorAlexander Woo
    • CommentTimeMar 1st 2012
    As a cautionary example, one might look at our general failure to evaluate teaching.

    As I understand it, in many countries, teaching is basically not evaluated at all, and mathematicians (as well as other academics) are hired, promoted, and fired based entirely on their research, even though as much as 50-90% of their job is teaching. Basically, the system relies on the goodwill of individuals to do the teaching part of their job appropriately, though obviously terrible cases are noted.

    At most institutions in the United States, the situation is not quite as dire. Academic staff are evaluated on their teaching. However, many institutions evaluate teaching entirely or almost entirely based upon student evaluation surveys collected near the end of each term. Everyone acknowledges that, unless your definition of 'good teaching' is 'happy students', this is a woefully incomplete way of evaluating teaching. Almost everyone then turns around and says that, unfortunately, we simply don't have the time and resources to invest into better ways to evaluate teaching. The only exceptions are the few colleges and universities (or, in some instances, individual departments or groups of departments) for which good teaching is really important.

    The truth is that, for most purposes, easy and cheap is more important than good. (This is a general fact about everything, not just evaluation. Many people, myself included, became academics in part to run away from this fact, but one can only run so far. In the rest of society outside academic settings, this is so obvious no one needs to make the point.)

    Each year in the United States, somewhere around 5000 different people apply to around 1000 academic jobs, just in Mathematics. Some jobs get nearly 1000 applicants, though perhaps 400 is more typical. Most people seriously on the job market apply to nearly 100 jobs or even more. I'm afraid, Andrew, that suggestions along your lines are about an order of magnitude too much work for everyone involved where hiring is concerned.
    • CommentRowNumber11.
    • CommentAuthorDavidRoberts
    • CommentTimeMar 1st 2012

    Each year in the United States, somewhere around 5000 different people apply to around 1000 academic jobs, just in Mathematics. Some jobs get nearly 1000 applicants, though perhaps 400 is more typical. Most people seriously on the job market apply to nearly 100 jobs or even more.

    this is really depressing…

    • CommentRowNumber12.
    • CommentAuthorAndrew Stacey
    • CommentTimeMar 1st 2012

    Everyone acknowledges that, unless your definition of ’good teaching’ is ’happy students’, this is a woefully incomplete way of evaluating teaching. Almost everyone then turns around and says that, unfortunately, we simply don’t have the time and resources to invest into better ways to evaluate teaching.

    Let me edit that slightly:

    Everyone acknowledges that, unless your definition of ’good research’ is ’high impact factor’, this is a woefully incomplete way of evaluating research. Almost everyone then turns around and says that, unfortunately, we simply don’t have the time and resources to invest into better ways to evaluate mathematical research.

    The first sentence still rings true. The second is slightly different because the numbers are smaller and it affects only those directly involved. I believe that the mountain to climb for evaluating mathematical research is not so high as for evaluating teaching. It is still a mountain, and so - recognising that - my goal here is not to climb it but to make enough noise that some body like the AMS decides that it’s worth figuring out the best way to do it.

    I’m afraid, Andrew, that suggestions along your lines are about an order of magnitude too much work for everyone involved where hiring is concerned.

    Then I haven’t explained it properly. Actually, I was trying to avoid explaining it too carefully at all as I wanted to merely hint at it and see what others said without making my ideas too explicit as then the discussion would have been about specifically those ideas not the general context. However, maybe I’ve made things too obscure.

    The idea that I have would not involve any more work than we are currently doing. It just reorganises it a little. The core is that there is a(t least one) central body which is responsible for evaluating a mathematician’s “output” (which can be more than research articles). If I think I’m going to need it, I can get some of my work evaluated. Maybe I should even pay for it, I’m not sure. Then when I want to, say, apply for a position, I can use that evaluation as part of my submission. So the hiring committee gets the evaluations, which can be a fairly simple scale, to do the first round of filtering. No more work for them than currently.

    Here’s what I see are the advantages of this:

    1. It doesn’t assume that everyone is looking for the same thing. The hiring committee can specify a minimum profile and it is up to me, as the candidate, to tailor my application to fit it.
    2. It gives the candidate some measure of control. I think that this is quite important. If I’m presenting myself as a candidate, I would like to be given the opportunity to show myself in the best light. It’s only natural! This builds in the ability to do that, and as it is built in then it is transparent and - hopefully - less likely to be abused.
    3. It doesn’t mean more work. The evaluation is pretty much what a referee currently does and is arranged by the prospective candidate well in advance. Moreover, as the evaluation is against a public set of criteria, the referee’s job should be easier. The results can be standardised so that the hiring committee’s filtering is simple to do.
    4. It is transparent. As the evaluation is done by a few central bodies, the standards can be public.

    This is my impression of what proper professional organisations (architects, lawyers) currently do.

    • CommentRowNumber13.
    • CommentAuthorzskoda
    • CommentTimeMar 1st 2012

    Andrew, the situation with lawyers is not one should strive for. It became a profession which is more parasitic than a service to a community; it is important how to twist things into formal framework more than a substance. One has a proof that somebody killed but if a defendand can find a trick how to dismiss the right to present the proof, he will be free, despite that all know he is guilty.

    It is a good idea that some new sources of bookkeeping the output record and its virtues be created. I find it less good idea to compress those into scales: each department has not only its own criteria but also informal biases. Like I know of a top level mathematician applying every year for numerous positions just to raise his own salary at his institution; most departments dismiss his excellent application, as they know he is not much likely to accept. Some people like to hire a person who does not travel much, as this is their preference. Real liking of somebody appears at the INTERVIEW where people see if they are really likely to interact with a person; this chemistry is so much. Now if you have an official, central body, what could happen ? A pressure from dean to hire a person with higher number on the official scale, even when the people in the department do not feel so.

    Finally, I do think that making a formal file with formal criteria is much more work than an informal application. I have sent few days ago a job application where only the titles of the documents (like research proposal) without formal structure are requested. It was easy to adapt what I already have, my informal LaTeX files which I use for other presentation purposes.

    I would like to be given the opportunity to show myself in the best light. It’s only natural!

    It is a learned behaviour. I have missed an advisor whom I wanted in the beginning of the graduate school as I was too shy to ask. I thought, he saw I was a good student and he will suggest this to me. I am raised that way: in school I was always spotted to be good and never claimed myself to be such. When I finished graduate school I wanted to go home and felt disgussed with the application procedures in which one has to self-praise. It is not the way I was raised, to self-praise. It is a disgusting activity. So I did not apply for a job. But then I was told of one place where they seeked for a position and told me to contact them by email and so on. So it went informally. I did not ask people for recommendations, it was against my spirit at the time, but gave to the future employer the phone numbers of people who know me well. And they done it for me.

    Needless to say, after many temporary employments and grant writing I got used to write about myself, and self-praising in aspects where I can find some arguments in my favour. I am now doing it, but by no means I can say I am still comfortable with it, I am perplexed every time I ask my colleagues for recommendations (and many things I skipped just for that), and by no means I can consider this self-praising activity natural. Natural would be to be spotted by people who follow the work in the field, and who will therefore want to hire you.

    On the other hand, I do agree with an aspect of your statement. I mean, I agree that, if we are to present ourselves, then we should be requested to do this more by description of results than by list of publications. List of publications may have coauthors, the numbers mean less than content. One should instead describe main results, regardless where they are published, if so, and the evaluator should ask the peer reviewers if your assesment that those results are indeed original and indeed that important holds. I think that Ngo’s “I have proved the fundamental lemma” means much more than “I have published in Annals”.

    • CommentRowNumber14.
    • CommentAuthorAlexander Woo
    • CommentTimeMar 1st 2012
    Andrew - what do you imagine an evaluation would look like? Could you provide an example?

    I simply am not able here to imagine one that could be usefully read in 15 seconds. (Whereas it is possible to count total papers and count total top journal papers (by whatever your criterion of 'top' is) and even compare the counts to time since PhD in 15 seconds, unless the number is so big to be clearly big enough.) (Of course, out of the 1000 jobs, these criteria are only important for perhaps 200.)

    I am pessimistic on evaluation in general. I think the costs of doing it well are high enough that most people will rather live with bad evaluation than pay for good evaluation. Sure the AMS can put together a task force and talk about it and come out with recommendations, but then a few wealthy schools will follow the recommendations, and everyone else will apologize for ignoring them. After all, given the market, if you evaluate badly, you hire someone who is 1% worse than the best person you could have gotten. What difference does that make? Is it worth spending an extra person-week of time to erase the 1%?
    • CommentRowNumber15.
    • CommentAuthorBen Webster
    • CommentTimeMar 2nd 2012

    It doesn’t mean more work. The evaluation is pretty much what a referee currently does and is arranged by the prospective candidate well in advance. Moreover, as the evaluation is against a public set of criteria, the referee’s job should be easier. The results can be standardised so that the hiring committee’s filtering is simple to do.

    I think this is tricky; mathematicians spend a shocking amount of time on hiring committees, so in theory there could be an enormous time savings from centralizing some of that evaluation. On the other hand, while time spent on service is relatively poorly rewarded in the current system, I can’t help but guess that similar work for some distant, unconnected organization would counted even more minimally (at the very least, it would increase the danger of being assigned other department service). Of course, universities don’t have to do things that way, but it’s a real danger.

    • CommentRowNumber16.
    • CommentAuthorHenry Cohn
    • CommentTimeMar 2nd 2012
    There are also other arguments against centralization. Hiring always involve some unfairness, randomness, and bias, but the fact that it is done independently in many places helps mitigate. I'm not convinced it would be good for the field if we all relied on a central authority for a first screening. Maybe it could do a better job than just about any single department, but any imperfections would be magnified by its central role. (And this is not even getting into the question of whether it's possible to establish public standards that inspire confidence and do not allow obvious ways to game the system.)
    • CommentRowNumber17.
    • CommentAuthorAndrew Stacey
    • CommentTimeMar 2nd 2012

    Henry, This type of evaluation is meant to replace paper-counting and impact factor. Note that these are already centralised: my paper count is the same in Norway as it is in the US! Moreover, although it would by its nature be crude and not good at fine filtering, it couldn’t be worse than paper-counting and impact factor.

    But my purpose in proposing it isn’t to say that this is the way things should be done but to try to get people to think about how it ought to happen. Are you really saying that you’d rather have impact factor and paper-counting than any other system?

    Ben, You are already doing this work. Whenever you referee a paper, you are in effect being asked to decide: “Should this count for one more on this person’s paper count?” and “Is this paper at an appropriate level for this journal’s impact factor?”. You may not be explicitly asked to do this, but that is what you are doing as far as the author is concerned.

    I’ve seen a lot of remarks along the lines of “I would stop sending my articles to commercial publishers but I need to do so for hiring/tenure/grants” so what I’m proposing is sort of like saying that we should use journals only for this purpose. Except that once we accept that, we don’t actually need the journal itself any more: just some organisation to do the admin in sending articles to referees.

    More generally, there are two main parts to all of these discussions and it’s important to keep that clear. There is “How can we improve the journal system for the good of mathematicians doing mathematics?” and “How can we improve the journal system for the good of mathematicians begin evaluated?” Here, I’m focussing on the latter.

    • CommentRowNumber18.
    • CommentAuthorAndrew Stacey
    • CommentTimeMar 2nd 2012

    Oh, I forgot this bit:

    we all relied on a central authority for a first screening

    which would you rather: ISI or AMS? That’s the choice!

    • CommentRowNumber19.
    • CommentAuthorHenry Cohn
    • CommentTimeMar 2nd 2012
    That's a good point about paper counting and impact factors. How often are they actually used for screening, though? I don't have a good feeling for how hiring processes vary between institutions, countries, etc. My impression is that in US research universities, impact factor counts for little or nothing to mathematics hiring committees; paper counting is more subtle, since presumably the length of someone's list of papers has some impact on anyone who looks at the application, but I'm not aware of numerical cut-offs. However, my only direct experience with hiring is at Microsoft Research (which takes an academic approach to hiring but is not a university) and MIT (which is not representative of academia overall).

    I don't know as much about this issue as I should. I don't suppose anyone has any useful reading to suggest?
    • CommentRowNumber20.
    • CommentAuthorAlexander Woo
    • CommentTimeMar 2nd 2012
    Andrew - you cannot divorce the question of "How do we improve the system for the good of mathematicians being evaluated?" from the question of "How do we improve the system for the good of mathematicians doing evaluating?"
    • CommentRowNumber21.
    • CommentAuthorAndrew Stacey
    • CommentTimeMar 2nd 2012

    Alexander: I’m not separating those two questions. I presume that you’re referring to my last paragraph:

    More generally, there are two main parts to all of these discussions and it’s important to keep that clear. There is “How can we improve the journal system for the good of mathematicians doing mathematics?” and “How can we improve the journal system for the good of mathematicians begin evaluated?” Here, I’m focussing on the latter.

    In the two questions, the mathematicians are the same. What is different is what activity they are engaged in. When I’m doing mathematics then I couldn’t care less what journal an article was published in, and even less what the impact factor of that journal was. I don’t think I’ve ever read an article simply because of where it was published (hmm, maybe expository articles now that I think of it). One side of journal reform is making the system actually help me do my research by making it easier for me to find the article that I need and help me make the best use of it. The other activity is evaluating mathematicians. This is an external activity in that I’m most likely to be having to justify my evaluation to some non-mathematicians. So for that, I need evidence which I can use to say, “See, this person can do Hard Maths”.

    Although these two sides can complement each other, and can actually happen simultaneously, there is a very big difference in that the first is internal to mathematics and the second involves interacting with the Big Wide World. At the moment, journals are used for both and do an amazingly bad job in both respects. With all the talk of comment sites and so forth, I wanted to bring the other side into play - which is, to my mind, the harder part.

    • CommentRowNumber22.
    • CommentAuthorAlexander Woo
    • CommentTimeMar 3rd 2012
    Andrew: I am trying to emphasize that, when one is evaluating mathematicians, one has concerns other than the accuracy of the evaluation, and in most cases, those concerns are quite significant, especially when differences in the accuracy of the evaluation are relatively small.

    It is said that 'Second-rate mathematicians hire third-rate mathematicians,' and fixing that requires a lot more than fiddling with metrics.