Not signed in (Sign In)

Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

  • Sign in using OpenID

Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

Welcome to Math2.0
If you want to take part in these discussions either sign in now (if you have an account), apply for one now (if you don't).
    • CommentRowNumber1.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 10th 2012
    • (edited Feb 10th 2012)

    I’m glad to see this forum!

    Here’s my recent thinking:

    People seem to get frozen in indecision when contemplating big serious changes to the current journal system. The consequences are too huge. So, I’m thinking we could hasten the evolution of good ideas by doing something quick, easy and fun. Something that’s not too painful to set up. And something that we use because we enjoy it, not because it’s good for us.

    I’m imagining something like this:

    A system linked to the arXiv, which looks a lot like the arXiv: one page per paper. If you click to download the paper you get sent to the arXiv. But you can also comment on the paper, and/or give a simple ’rating’, on a scale from 1 to 5. When you first join the system, you could say whether you want your ratings to be visible to everyone, or only contribute to an average rating.

    Next to every paper you could see how many people have clicked to download it, its average rating, and a link to the discussion thread.

    In a further refinement, this could all be combined with a reputation system, so that people could gradually develop fancier and fancier algorithms to decide which papers are ’good’.

    BUT: the short-term goal is not to provide a system for ranking job candidates or carrying out official peer review - merely to provide a fun place to talk about papers and help decide what we might like to read.

    I think this “lowered ambition” will make it easier for people, especially youngsters, to explore new ways of doing things.

    Unfortunately I don’t have the skill or desire to do the programming. But I’m imagining that perhaps it’s not too hard to start a system vaguely like the one I describe.

    Ideally what would happen is that people would enjoy the system but dislike certain aspects of it - and then people would improve it, or create competitors. After a certain point, people would start taking one of these systems seriously enough to base hiring or tenure decisions on it, in the same way that they use journals now. But not right away.

    • CommentRowNumber2.
    • CommentAuthorxbad
    • CommentTimeFeb 10th 2012
    There once was such a system, scirate.com. It appears to be down at present. See http://dabacon.org/pontiff/?p=3782 for more information.
    • CommentRowNumber3.
    • CommentAuthorCschaffner
    • CommentTimeFeb 10th 2012
    we are currently trying to revive it: https://github.com/cschaffner/scirate
    Any help is appreciated (please email me know if you're interested in helping). The basic scirate.com allowed users to "vote" for arxiv-papers which were then displayed in descending order of votes. One could also make comments, but I don't think that functionality was heavily used. I loved and appreciated Dave Bacon's idea of crowd-sourcing the daily routine of looking through the long list of new arxiv-papers, it saved me quite some time every day.

    I completely agree with John that what he describes is a good system to "have out there", and it does not seem too hard to actually implement. I hope we'll be there soon.
    • CommentRowNumber4.
    • CommentAuthorCharles Rezk
    • CommentTimeFeb 10th 2012
    I don't really care for or see the need of a numerical rating system. But having a default place to host a discussion for each and every individual paper could be incredibly useful. (And if having a rating system makes it "fun", and so encourages people to use it, I can live with it.)
    • CommentRowNumber5.
    • CommentAuthorCharles Rezk
    • CommentTimeFeb 10th 2012

    Incidentally, the quickest and easiest thing to do is to encourage every mathematician to post all their new papers to the arxiv. Even though there are about 100 new math papers a day being put on arXiv, that’s still probably less than half of all published papers.

    • CommentRowNumber6.
    • CommentAuthorYiftach Barnea
    • CommentTimeFeb 10th 2012
    If you build it, they will come. By they I mean people who need numbers to rank people. These numbers will be used in hiring, promoting, etc. But the system you suggest is so open to abuse that I think it would be a really bad idea to implement it.
    • CommentRowNumber7.
    • CommentAuthorPeter Krautzberger
    • CommentTimeFeb 10th 2012
    • (edited Feb 10th 2012)
    • For commenting and “voting” there is also papercritic.com. This is a young and active project coming out of programming competition from Mendely and PLoS. They might be inclined to add a paper-view. Also, Mendeley has data on how many papers have been read, annotated, shared etc. This data is available through an API.

    • Generally speaking, encourage people to make their reviews public! After all, we all do it when we read papers.

    • Tell people to sign up at wordpress.com, with a pseudonym if they like, and start writing.

    • Tell them to sign up at researchblogging.org and cross-post reviews of arXiv papers.

    I would like to politely disagree with the If you build it, they will come-attitude. Without a culture of making multiple reviews public, I don’t think a platform will work (e.g. even Nature failed at that).

    • CommentRowNumber8.
    • CommentAuthorgowers
    • CommentTimeFeb 10th 2012
    I'm not terribly keen on numerical ratings -- I worry that there would be a certain amount of non-fun mixed in with the fun. However, comments, reviews etc. (which might implicitly rate papers, but in a polite way that wouldn't make an author think, "Hey, my paper's at least a 4, so why did that jerk give it a 3?") seem OK. I also worry that people who are somehow "popular" would get their papers more highly rated -- a phenomenon that applies to questions/answers on Mathoverflow.

    So my view is that to keep it quick, easy and fun, numerical ratings should not be part of it.
  1. There are lots of problems with most rating systems. For example on Mathoverflow questions which are in areas where there are more mathematicians using MO get higher ratings, just because there is a larger base of viewers and not for any intrinsic reason. Giving a 1-5 rating is better, but there are all sorts of consistency problems that could emerge.

    So maybe John's easy and fun idea is not easy enough? Maybe we should just have a site which can host a discussion for individual papers posted on the arxiv? This should be even easier to set up, right? I want to second Rezk's statement that this alone could be incredibly useful, and as gower's suggests this might implicitly rate papers afterall (but in a more subtle fashion).
    • CommentRowNumber10.
    • CommentAuthorxbad
    • CommentTimeFeb 10th 2012
    I have a concern about pubic rating systems. In the current refereeing system, there is no communication among reviewers, which has the consequence that the referee reports are independent assessments. This feature is worth trying to preserve. My fear is that, in a public system, the the early reviews will have undue influence on later reviews. A related concern is that early critiques will lead to rapid-fire revisions of manuscripts. It can take a real investment of time to do a serious review. This work will become all the more difficult once a manuscript becomes a moving target.
    • CommentRowNumber11.
    • CommentAuthorYiftach Barnea
    • CommentTimeFeb 10th 2012
    The anonymity of the referee has advantages and disadvantages. But the main strength of the peer-review system is its independence. The author has very little influence on the choice of referee. Thus, we can trust the process to some extent. In the system that is suggested the authors could influence the ranking. It is obvious that once there is such a system some people will use it to rank other people (that is exactly what we do by looking where people published). If you believe that no abuse will take place you are incredibly naive.
    • CommentRowNumber12.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 11th 2012
    • (edited Feb 11th 2012)

    If most people don’t like the idea of numerical ratings, we should skip that, or modify it. All I care about is that lots of people start using an arXiv overlay website to discuss papers.

    The first version will inevitably be imperfect; people will complain about it, but if a lot of people use, it they’ll keep trying to improve it, or set up better systems.

    I mentioned the idea of ratings merely because most forums that become extremely popular have some feature like this:

    • Facebook lets you ’like’ something

    • Google+ lets you ’+1’ something

    • Amazon lets you rate items from 1 to 5

    • Netflix lets you give movies from 1 to 5 stars

    • MathOverflow lets you click ’up’ or ’down’ on questions and answers to those.

    and so on. In some of these systems you can see who liked or disliked a given item; in others that information is hidden. Perhaps, for obvious reasons, it’s hidden in all systems where people can say they disliked something (instead of merely not saying they liked it).

    People love to complain about these systems, and of course they’re open to abuse, but they’re still useful, and they seem to really engage people. That’s why I suggested introducing one.

    Of course, none of these systems are attempting to be a substitute for ’peer review’,… and that’s the exactly the point! I’m not proposing a system that’s supposed to substitute for peer review. It’s only supposed to help you find papers you might like, and see what other people have to say about those papers.

    This lowered goal will make it easier for people to start such a website without becoming paralyzed with worry. However, if people feel inclined to become paralyzed with worry over a particular issue, even despite this lowered goal, we should sidestep that issue.

    It’s been pointed out that an arXiv overlay website for discussing papers already existed for papers on quant-ph. It was developed by David Bacon. People in the quantum information community enjoyed it a lot: they would go there instead of the arXiv to see the new papers and see which ones people liked. (I believe this system merely allowed you to ’like’ a paper, a bit like Google+ or Facebook.) Unfortunately David Bacon got a job at Google and became too busy to maintain the system, so it no longer exists. However, as noted here, people are now trying to revive it - thanks to the Elsevier boycott! And I know there are people with ideas on how to expand it to the whole arXiv.

    • CommentRowNumber13.
    • CommentAuthorScott Morrison
    • CommentTimeFeb 11th 2012

    I know Yossi Farjoun put some thought into something very much like this a while back. I’ve pinged him, hoping he’ll drop in here.

    • CommentRowNumber14.
    • CommentAuthorScott Morrison
    • CommentTimeFeb 11th 2012

    I fear that “build it and they will come” is wildly naive! Instead, projects like this must be useful at the point of use.

    Happily, I think there is definitely a way to achieve this in the current context. Most mathematicians receive a daily email from the arXiv listing new papers in chosen subject areas. If you’re at all like me, you want to subscribe to more than one area, but many (most? nearly all?) of the individual papers aren’t of much interest.

    My life would be noticeably better if some pre-filtering or sorting could be done! I’d love to have that email try to cluster papers by tighter topic areas than just the arxiv subject areas, and then only show me my preferred clusters. I’d love to have my attention automatically drawn to papers my friends liked! Indeed, I’d be happy just to have a list of every paper Noah glances at, just to be reminded of what a terrible reader-of-papers I am. :-)

    It sounds like David Bacon’s site achieved some of these things. Can we do more?

    • CommentRowNumber15.
    • CommentAuthorYiftach Barnea
    • CommentTimeFeb 11th 2012
    I have been trying to think how to address the worries I have. Let me suggest some ideas. I would suggest that the authors of a paper would be able to decide that they do not want to receive any comments on their paper. More importantly any comment will first be available to the authors and only if they approve it will be available to everyone. This way the system cannot be abused to heart people, also its reliability as a tool to rank people will be so low that no one will use it this way. On the other hand, most people don’t want to know which papers not to read but which ones to read, so negative comments are less helpful to them. However, the authors will still be able to see the negative comments and react to them. If we would like to make things a bit clearer, we could make it visible how many comments were not approved by the authors.
    • CommentRowNumber16.
    • CommentAuthorgowers
    • CommentTimeFeb 11th 2012
    One additional thought is that while I'm still not keen on giving a rating to papers, there might be a place for something like Amazon's option to judge whether a comment/review is helpful/unhelpful. So if somebody gives a particularly insightful assessment of a paper and many people find it helpful, that assessment would be trusted more (in general, though one might have to be careful how to interpret the data).
    • CommentRowNumber17.
    • CommentAuthordarij grinberg
    • CommentTimeFeb 11th 2012
    I think that, rather than a 1-5 overall rating for the paper, there should be separate ratings about importance, readability, correctness and self-containment. I wouldn't be comfortable with downrating a reasonable paper just because it uses some nonstandard notations and doesn't care to introduce them; but I would give it a 3/5 in self-containment. But these all are technical details of a feature of little importance. The really important part, I believe, is the commenting, not the rating, **even if** evaluation is the purpose. Rating is just too easily manipulated, and even if it is not, people will think it is. You can create 20 user accounts which rate each of your papers a 5,5,5,5, but you can't have 20 sockpuppets give you positive reviews that don't look fake at the first sight.
    • CommentRowNumber18.
    • CommentAuthorzskoda
    • CommentTimeFeb 11th 2012
    • (edited Feb 11th 2012)

    The fact that popular sites Facebook, google+, amazon, netflix, MO have like buttons does not mean that their popularity mechanism makes them better in inducing the true development of personality etc. Popularity is not a measure of development and achievement itself. A Serbian kid growing up in California came to his parents place in Belgrade and was surprised that he was accepted to any children group to play with on the street, not preceded by marked belonging to a clique, clan or band, unlike what he learned in modern California, driven by TC shows, popularity culture, hi school prom craze. In fact I NEVER heard in my childhood in Croatia that the phrase “be popular” applies to kids. Word “popular” was about movie stars and so on, not about something relevant to our ordinary lives. This was few decades ago. Now it is probably changing, everybody needs to have a good picture to put on social site, and people kill when get defriended. Finally, the democracy in US is ridiculuous – the PR makes people choose whoever, all is lobbyied and advertised; voting is not discovery of the truth.

    I fully agree with more sensitive 11: especially this – for peer reviewing it is beneficial to be independent, that one referee does not know what the other one read. Also ASSIGNING a referee makes people discover things which are not easy to appreciate without through reading. It may be a boring looking paper, but the arguments while on the surface written by the same kind of letters and equation symbols are in fact a deep breakthrough. A casual audience will miss this, while true reader will be forced to understand. Once understood it is more likely to give a more balanced mark of the achievement.

    On the other hand, I do second that one should have pages of discussion for individual articles. Not collections of opinions, but more a wiki style. Look at wikipedia. If somebody writes something without arguments and references, he is likely to be overwritten. It is also possible to write remarks about the paper. If you observe some interesting relation to other areas, or that the result follows from something else it stays in the wiki. If you repeat the bullshitting which somebody else wrote before, your sentence will get into the history and erased from the new assesment, or labelled as irrelevant.

    The danger is also that the author himself will make his work more appreciated by answering to the craze. Instead, as Arnold used to say, one should work in old Russian style of sitting in the library and proving hard theorems. It is not that one should not value or finance education and dissemination of knowledge. But it should be DIFFERENT job and not interfering with appreciation/evaluation of primary results. If somebody is going to get a journalism prize let it not be called a theorem of the year, but a self presentation of the year or whatever. Mathematicians are sometimes subtle in distinguishing nuiances, and should do that even more emphatically here, where people starting mixing apples and pears.

    • CommentRowNumber19.
    • CommentAuthorKevin Walker
    • CommentTimeFeb 11th 2012
    I like this idea, and I agree with most of what John Baez and Scott Morrison wrote.

    Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I'll refrain for the moment.
    • CommentRowNumber20.
    • CommentAuthorzskoda
    • CommentTimeFeb 11th 2012
    • (edited Feb 11th 2012)

    and help decide what we might like to read

    One top mathematician whom I pretty much appreciate told me that he almost never goes to the library nowdays and reads only on the internet. He says, in my memory:

    I do not care what is published what is not, I google for the name of the author whom I trust or the title I suspect to be interesting, and find it on the arxiv or the homepage of the author. I do not need others (i.e. journal boards) to decide what I will read, I want myself to decide what I will read.

    This is not a view of a youngster but of an active mathematicians closer to the retirement than to the PhD. And of course he does appreciate peer advice on what he will read, but only from the colleagues he directly talks to and trusts, of those whose good lecture he might have heard on the topic and not from the “societies”, networks, pyramid schemes, points systems and alike.

    This is not to discourage building sites for discussions, wikis etc. – we need them, especially to diseminate the information from the rare centers of hi concentration of expertise in some areas of knowledge to the rest of the intellectual world – but rather to cautiously raise the awareness of the importance of the preservation of healthy basic old practices in refereeing, reading, teaching and research practices; those which the popularity chase and craze and related psychological and social rearrangements, new orders and disorders, eases and diseases do not properly replace. But some new technology will add them. There is so much more new in software and on the internet on the market, than the liking/popularity/points systems.

    • CommentRowNumber21.
    • CommentAuthortlawson
    • CommentTimeFeb 11th 2012
    • (edited Feb 11th 2012)

    I’m not in support of having any kind of “point” system papers on the arXiv. There are already a number of mechanisms in place that reward scholarship, and already more than enough incentives to produce widely popular work.

    I don’t think that it’s necessarily a bad idea to have some kind of “reward” system in place, but I think that people underestimate how carefully thought-out these point systems are. These don’t make the site inherently engaging or psychologically addictive on their own. They specifically provide some form of inexpensive reward for contributing to the site, and a reason to check back regularly, until using the site becomes a part of the user’s routine.

    So if you’re trying to organize large-scale adoption of a site and you want users to

    • collect public lists of literature that interests them,

    • submit short synopses or reviews of papers,

    • ask questions about papers,

    • respond to questions on their own papers,

    • make sure that even “lower-profile” papers get reviews, or

    • maintain professional community standards,

    then I feel like these are the things that you should be providing rewards for.

    • CommentRowNumber22.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 11th 2012

    (Kevin: remember that you can start a new discussion on any particular idea that strikes you. Just link between the two to keep them connected.)

    With all the ideas I read here, I try to examine them under the spotlight of “How would this make a difference to my daily work?”. A review site sounds fun, but I don’t know that I would use it, and I don’t think that I would be all that happy having my papers there. I’m worried that mine would fall into that category where the “number of people who read it is a number between 0 and 1”. The time that I want to talk about my work is when I’m doing it, not when I’ve written the paper. I like the quote from Dror-Bar Natan on MathOverflow (see also Fran&ccedila;ois Dorias’ website):

    Papers are written so that their author(s) can forget their content and move on to other things. Therefore when you write you should be very careful to put in enough of the big picture and enough of the details so you’d be able to reconstruct your thoughts 10 years later if you’ll need to, assuming you’ll forget everything but retain some familiarity with some basic principles of mathematics.

    So when I’ve written the paper and submitted it to a journal, I’m ready to move on to the next thing. I’m quite happy to talk about that paper, but I’m not so excited about it as I was. Indeed, often by the time I’ve written the paper I get a strange “flat” feeling about it as though I’ll be glad to see the back of it - at least for a short period.

    The time when I’d really like to talk about it is when I’m writing the paper. Call this the “alpha” stage. That’s when I’m sure of the basic ideas, but am aware that I might radically change how I think of them as time goes on. Then when I’ve written the paper, I don’t really want to be told of a whole new way of thinking about it, but I don’t mind being told that my notation is rubbish and I should have written it all in the passive voice. Call this the “beta” stage. Finally, I submit the article to a journal and hope that that’s dealt with.

    That’s as an author.

    As a reader, I have considerable sympathy with Zoran’s last post and with Scott’s further up. I don’t need to be told whether or not I should be interested in a paper. If it contains the result I want, then I’m interested. If it doesn’t, then I’m not. At least, that is, when I’m doing my Main Research - of course, sometimes I read articles just to find out what’s going on, but that’s a bit different. So I don’t want to be told that I should read a paper because it has 5 stars! But I would find it useful to know if there are any corrections or clarifications attached to the paper - this is subtly different to a review.

    • CommentRowNumber23.
    • CommentAuthoryfarjoun
    • CommentTimeFeb 12th 2012
    ARGH!!! I wrote a long comment and then it got erased since somehow I got logged out by the time I clicked submit. At this point I'm too annoyed to write it fully again, so here's the main points:

    I think that there are flaws in any point-based system like "+1" "like" etc. I suggest a similar but different approach based on Twitter.

    The idea is that each user can "read" any number of papers on the arxiv and also to "follow" any number of users (and thus see what papers they are "read"ing). from the resulting graph, many different scores can be formulated, and the users can each select to view next to each paper a subset of these scores. Good score formulas would render sockpuppeteering useless.

    To jump-start the system, one could use the download information from the arXiv (using hashed IP as the user) if they give it.
    • CommentRowNumber24.
    • CommentAuthorYiftach Barnea
    • CommentTimeFeb 12th 2012
    It seems that quite a lot of people would like to see some kind of ranking. If indeed such ranking will exist, that will be a negative incentive for me to put my papers in such system. I am sure there are many other people like me. The same is true for being exposed to negative comment. I am fine with receiving negative comments, but I would not like everyone to see them. So if you would like everyone (more or less) to take part in such a system, then you have to accommodate to such concerns. The fact that you are not worry about it is not relevant. The question is whether substantial amount of people are worried about it.
    • CommentRowNumber25.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 12th 2012
    • (edited Feb 12th 2012)

    Andrew wrote:

    I don’t need to be told whether or not I should be interested in a paper.

    I don’t ’need to be told’, either. That sounds pathetic. I want to hear what papers people are interested in - particularly, specific people I know and trust. And I’m not alone in this: lots of people want to know what their friends and respected colleagues are reading…. and what they think about those papers. They gossip about this endlessly at conferences.

    This is why I like the idea of a system where people are able to opt to let others see which papers they like. If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.

    On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody. Most people don’t have the energy to write a column like that, but maybe they’d have the energy click ’like’ on a paper now and then, and sometimes even write a paragraph saying something about it. The system I’m envisioning would make all that easy. Of course people could include links to blogs, too.

    • CommentRowNumber26.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 12th 2012

    Kevin wrote:

    Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I’ll refrain for the moment.

    No, do it! - unless you have some power to make those better people do it.

    • CommentRowNumber27.
    • CommentAuthorplm
    • CommentTimeFeb 12th 2012
    I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.

    Hopefully this would soon be officially integrated to arxiv, with the possibility to open one's article to comments or not, for privacy. I think some would be deterred from posting on the arxiv if it were like opening a thread in a forum, but it is not clear how well the (default) option to disable comments would remedy this; though I guess the decision to post on the arxiv already requires enough openness from the poster to accept possibly (rightly or wrongly) annoying comments.

    Someone who posted as Ralph on Tim's blog also mentioned this:
    http://gowers.wordpress.com/2012/01/29/whats-wrong-with-electronic-journals/#comment-15399
    • CommentRowNumber28.
    • CommentAuthorCharles Rezk
    • CommentTimeFeb 12th 2012

    I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.

    • CommentRowNumber29.
    • CommentAuthorzskoda
    • CommentTimeFeb 12th 2012
    • (edited Feb 12th 2012)

    28 I agreed above that the discussion sites are welcome, the disagreement was about a point driven system.

    21 tlawson says that if one is going to compile bibliography lists and so on, this should be driven by reward system. I have created hundred of nLab entries and put there many hundred references and do that on daily basis for a couple of years. If anybody would pressure me by having the points of my contribution publicly seen (daily followed) at semaphore I WOULD STOP CONTRIBUTING. (Edit: the fact that I am much less active in MathOverflow is partly because I dislike the atmosphere of the reward system there). Many people like me in mathematics find beauty and peace and do not believe that one creative involvement can be linearly compared to any other involvement. If one is creating, one is expecting to see something nice building up. I do not need point like rewards, I need to see that something is created and built, and that some things are solved my own way which I like in the process. I like that I can come back to my notes when I need them again. If I made something neat that this neatness helps me in future, and helps my students and so on. Of course, I like others contributing where I start. But why to mix social needs, cooperative works and so on, with the popularity and linearizing craze ? I know that US media dominate the world and that US culture is obsessed with point systems from kindergarten, but why would adult creative people be bogged with that ?

    P.S. I proposed accompanying arXiv with discussion features and hep-th-comment section for short remarks and replies instead of papers still in 2000 in a discussion with Vafa at a conference on orbifolds in Madison. But not in the faintest dream I thought of a point system and rewards, rather than posting arguments, comments, in other words stimulative CONTENT and not pure likes and dislikes.

    • CommentRowNumber30.
    • CommentAuthorYiftach Barnea
    • CommentTimeFeb 12th 2012
    29 +1 :-) (Sorry, I couldn’t resist.)
    I agree with Zoran, in some sense coming exactly from the other direction. I tend to be competitive. I don’t care about the point system in MathOverflow, because it has no real life affect, so it is harmless fun. But I really don’t want a point system on anything important like research. Would you have a point system at home? You changed the baby diaper, you get 10 points. Your partner took the kids to a birthday party, 20 points. Now, you are behind, what would you do?
    • CommentRowNumber31.
    • CommentAuthorKevin Walker
    • CommentTimeFeb 12th 2012

    As requested, here’s an attempt to summarize the discussion. Reading through the comments a second time they seem less diverse and unfocused than they did to me on the first reading, so perhaps this is unnecessary. “#n” means comment number n. I’ve probably overlooked some stuff; bring it to my attention and I’ll edit this post accordingly.

    Main idea (#1). A site where one can discuss Arxiv papers, possibly also with some sort of rating system or some way of finding papers which are popular in general or popular with particular people (such as the hugely influential Noah Snyder; see #14).

    Existing systems. Similar things have been tried before, so we should try to learn from these earlier attempts (scirate.com (#2, #3), papercritic.com (#7), David Bacon quant-ph thing (#12) [is this the same as scirate.com?]).

    Ratings. They could take various forms:

    • Actual direct ratings of papers, on a scale of 1 to 5, say.

    • Separate ratings for importance, readability, etc. (#17).

    • Simple “+1” or “favorite” type system.

    • Each person could maintain a list of papers that they like, but these lists would not necessarily be aggregated (#14, #23).

    • Perhaps (only?) comments should be rated (like on amazon.com) (#16, #21).

    Arguments in favor of ratings:

    • Many successful sites use a rating system, and perhaps the system is partly responsible for their success (#12).

    • It is important to be able to quickly and easily find your way to the most useful papers, and some sort of rating system would help with this (#14, #25, #28).

    Arguments against ratings:

    • Too many to list fully here. See #4, #6, #8, #9, #10, #18, #20, #24, #29, #30 (probably I missed a few).

    • Ratings are too arbitrary, which is maybe OK on mathoverflow when not much is at stake, but would be bad for a site that might influence hiring decisions, etc.

    • If the ratings are important, they will likely be gamed.

    • Ratings could make the site less fun.

    • CommentRowNumber32.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 13th 2012
    • (edited Feb 13th 2012)

    Rezk wrote:

    I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.

    Right. I should have made that more clear - and the website that does this should also make it very clear!

    pln wrote:

    I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.

    My friend Chris Lee, head of bioinformatics at UCLA, is interested in helping set up such a site. He’s good at programming and he’s written an interesting paper:

    I hope some people here can join with him in rapidly setting up some sort of ’quick, easy and fun’ website, while leaving open the ability to expand and improve it. I’ll keep talking to him and all of you to help coordinate things. I’ve tried to get him to post comments here himself, too.

    • CommentRowNumber33.
    • CommentAuthorJohn Baez
    • CommentTimeFeb 13th 2012

    By the way: one reason I hope this ’quick, easy and fun’ website gets started soon is that it will pressure people who dislike it to come up with something better. I’m not sure what the best solutions to our problems are. But I don’t think we can find out until someone tries something.

    • CommentRowNumber34.
    • CommentAuthorzskoda
    • CommentTimeFeb 13th 2012
    • (edited Feb 13th 2012)

    Tim Gowers in the article a-more-modest-proposal proposed a site which would be dedicated to mutual commenting of people (with point-pay for comments from author to the commenter) on their preprints, helping so to improve the papers which are not yet in the final stage. While credits are given, they do not feed the vanity in that case, but the direct need to have the points earned (for the job or pre-reviewer) in order to pay for your own papers being pre-reviewed and hence helped in their finalization and improvement. I like his proposal. The comments there are also very interesting.

  2. I’m wondering if it is ok to open a related thread (or better, a wiki page somewhere) to collect existing technologies, i.e., isolate that aspect of Kevin Walker’s summary #31 (thanks Kevin!).

    This could be both about technology (scirate, papercritic, altmetrics.org, +1 etc) but also anything to put these tools in perspective (for example this study on tweets predicting citations).

    While this thread shows that there’s need to discuss how we value these tools, I think it might be good to have a neutral thread/wiki page to simply collect facts and usage.

    Just like John’s #33 said, the sooner we start, the sooner we’ll know what works. (I would add: there won’t be one tool that works for everybody, but there will be equally useful tools so that everybody can find one that suits them.)

    • CommentRowNumber36.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 13th 2012

    Why not start a discussion here first (make it clear that it’s just for gathering information, not discussing it) and then when the amount of information gets unwieldy, we can look for a more permanent home for it. Wikis are easy to set up, and if needed then we could set up a wiki linked to this forum to gather information - if there isn’t already somewhere suitable. But I don’t want to scatter information too much too early.

    • CommentRowNumber37.
    • CommentAuthorAndrew Stacey
    • CommentTimeFeb 13th 2012

    John said in #25:

    If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.

    On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody.

    There is a huge difference between these two situations, and I like the second but dislike the first.

    I remember what led me to the idea of review panels. It was when I learnt about the Journal of Stuff I Like. Initially, I thought this was great. But as I thought more, I decided that it wasn’t such a great idea after all. The danger with it is that it leads to the Cult of the Personality. John is absolutely right - everyone would tune in to see what Witten or Atiyah or Deligne were reading (or Terry Tao or Tim Gowers or John Baez), but no-one would be all that interested in what Andrew Stacey was reading. So we’d be shifting the pressure from trying to be published in Annals to being mentioned by one of the Big Wigs. This feels wrong on so many levels.

    However, This Week’s Finds is a wholly different beast. I admit that I’ve only dipped in here and there, so I may have the wrong impression. That impression is that you (John) are more interested in telling us about a particular topic than about a particular paper. There’s always “Where to find out more” lists at the end, and it may be a paper that sparked the interest in the topic, but the central theme is of “Here’s something cool to know about” rather than “Here’s something cool to read”. Now that is far more useful. Whilst I feel that I don’t really need someone to tell me whether or not to read a particular article - once I have that article in front of me - I often feel that I need someone to say “If you want to learn about X then the papers to start with are A, B, and C”.

    So I see a big difference between setting up a system for adding reviews to the arXiv, and a system for making it easier to find papers by topic. The former doesn’t seem to add much usefulness, just a load of froth(!), but the latter makes it easier for me to do my job.

    • CommentRowNumber38.
    • CommentAuthorAlexander Woo
    • CommentTimeFeb 14th 2012
    I am a little apprehensive about this idea, and I'm more worried about it working too well than about it not working at all.

    The problem is that this service does nothing for those papers no one reads and those mathematicians whose papers no one reads.

    If this service works too well, and the research community abandons the journal system for a system where papers are tracked by how many mathematicians (or, worse yet, which well-known mathematicians) read and recommend them, then we lose the ability to give a minimal certifying stamp of 'this is valid and most likely correct and somewhat novel mathematics research' to uninteresting papers. The danger is that the 'research elite' (which I'm defining broadly enough to include I think everyone whom I've seen post on this forum so far) create a system that works for us and abandons everyone else.
    • CommentRowNumber39.
    • CommentAuthorScott Morrison
    • CommentTimeFeb 16th 2012

    @Alexander, I remember reading a great comment you made somewhere, where you talked about what the purpose of peer review is, for all the researchers outside of that ’research elite’ you mention. Now I can’t find it, even with the help of google. Could you point me to it if you remember the one I mean, or otherwise write somewhere (here, a blog, etc? :-) about this again? I agree people need to appreciate this point further, but I know I can’t articulate it quite right yet!

  3. John Baez wrote: "A system linked to the arXiv, which looks a lot like the arXiv: one page per paper .... But you can also comment on the paper ... "

    Imho, this is very good and necessary idea and actually it is almost-done - arXiv has system of trackbacks.
    So I think the pretty simple step to start right today - is the following:

    When one submits a paper to arXiv , he by himself creates a) say wordpress page (e.g. on his blog or some joint blog)
    which is devoted solely to this paper b) trackback to it.
    Everyone can leave comments/questions/ remarks on this wordpress page.

    That's it. We do not need new soft or whatever.

    The only thing I guess is may be worth to add is to create some mailing-list or better site, where those people who wants to get comments
    can show up themselves: "Hey guys comments to my paper are welcome". It is not necessary for those people who look all arXiv papers everyday
    because they will see the trackback with "comment on me" when paper appears,
    however it would be helpful for those who are looking arXiv irregullary... (E.g. I am looking QA, almost everyday, but alg-geom not so often).
    • CommentRowNumber41.
    • CommentAuthorzskoda
    • CommentTimeFeb 16th 2012
    • (edited Feb 16th 2012)

    Well, the email addresses are included in the papers and often there is a mark that the comments are welcome. This does not make many people commenting by email anyway. Web makes it sometimes easier, unfortunately many commenting sites will have their own resistrations which make a turn off to many. So it is good that it is not just trackbakc link but indeed one unique system for the whole arxiv. Gowers’ idea that we could have a system of credits for “reviewing” preprints, granted by the authors, who have some credits to give up, is a good incentive for certain kind of activity. Having possibly new arXiv categories of new micro-“comment” articles which could be stably cited is another. We need to discuss more mechanisms and forms.

  4. @Zoran, I agree that Gowers proposal is much bigger, just that it is first step can be done today within existing technologies and build up community doing this.

    Concerning private comments to authors by e-mails. I see the big differences, do not You ?
    • CommentRowNumber43.
    • CommentAuthorChris Lee
    • CommentTimeFeb 16th 2012
    Hi, I'm new here. I hear some people voicing various assumptions that open selection systems can never work as well as closed selection systems (i.e. journals). Before you assume that, we would love it if you would look at the specifics of what we're actually proposing. Sorry to be annoying, but it seems like everyone just ignored the link John posted above to our detailed proposal and analysis of the challenges of open selection networks (written by me, but based on ideas from many, many people; see the Acknowledgments, linked from John's post #32).
    • CommentRowNumber44.
    • CommentAuthorPeter Krautzberger
    • CommentTimeFeb 16th 2012
    • (edited Feb 16th 2012)

    @Andrew #37 I understand and share the worry. But I think it’s equally likely that we could end up with Mathematical Critics (much like, say, literature critics), i.e., people who are renowned for there knowledge and experience at identifying good research. Add to that the study I linked to in #35 – tweets may be able to predict citations. That’s not “Tim Gowers’s tweets predict citations”, it’s the “wisdom of crowds”. For a different experiment check Crowdometer.

    @Alexander Chernov #40 second that! (In fact, see #7) – wordpress can solve a lot of these problems out of the box.

    • CommentRowNumber45.
    • CommentAuthorzskoda
    • CommentTimeFeb 16th 2012
    • (edited Feb 16th 2012)

    44 With many proposals involving wordpress – what is the gurantee that the wordpress will stay free ? This is uninformed but serious question: I know there is a commercial owner of the wordpress web site, there are free user accounts, and there is some software there and some related software (and add-ons) in circulation. I do not know which portion of the software is transferrable to other sites retaining full functionality, how much of it is open source and what are ultimately the licences involved. I am sure some people here know – please let us know.

    • CommentRowNumber46.
    • CommentAuthorYemon Choi
    • CommentTimeFeb 16th 2012

    Peter at 44: isn’t the danger that “good research” becomes increasingly defined as “what those Big Name good critics like?” There are also people doing mathematics in systems or societies where the kind of openness taken for granted by this kind of model is not so easy, and the loudest voices may not be the wisest.

    I also share Alexander’s concerns at 38.

    • CommentRowNumber47.
    • CommentAuthorPeter Krautzberger
    • CommentTimeFeb 16th 2012
    • (edited Feb 16th 2012)

    @Yemon yes, there’s a danger. But how is that different from today? Editorial boards are precisely “Big Name critics”. Yet journals are so narrow in their design, that they will never say “hey, I just read this fantastic expository paper about this obscure proof that nobody understood; there’s nothing new, but it’s the best paper I read all month”.

    As I tried to argue with the study about tweets predicting citations, the “wisdom of crowds” approach is likely the most fruitful (and most democratic) and specifically eliminates this problem of “Prof BigWig said so”.

    @szkoda wordpress is open source. So yes, it will always be free. The distinction is between wordpress.com, a free hosting service, and wordpress.org the underlying software. But wordpress.com also lets you export anything from your site there and import into your own installation. I know a little bit about this since we’ve started boolesrings.org last summer, an experiment towards a wordpress hosting service specifically for mathematicians. We are working on a new site that would be open to everybody, but above all we focus on a decentralized approach, i.e., identify the open source tools that mathematicians might need for the purpose of their academic homepages (from citations management to course management). If you want to look at some of its potential, you might want to check out Joel David Hamkin’s site

    • CommentRowNumber48.
    • CommentAuthorzskoda
    • CommentTimeFeb 16th 2012
    • (edited Feb 16th 2012)

    To 46, let me give just a small unimportant comment: even Grothendieck was at times unhappy with such a great mathematician as Weil surpressing the unconsent voices within such an open and small circle like Bourbaki was (according to his memoirs). In current refereeing process editors control that even the strong opinions of experts will be taken into account mostly if they are concrete and well pronounced. It is not always good that the same expert makes an argument (recommendation) and the decision: the two step process with different subjects involved makes the personal opinions having less weight.

    47: thanks

  5. @zskoda Oops, sorry, I expanded my post a little while you were writing yours…

    • CommentRowNumber50.
    • CommentAuthorYemon Choi
    • CommentTimeFeb 16th 2012

    Peter: which crowds? I really think there are contingent factors to do with location and politics that are not always accounted for, at least in the corners of maths where I send my working time. You want to know how big a crowd care about the cohomology of Banach algebras and know what they are talking about? I could probably have them over for dinner in one go…

    And actually I trust journal boards or handling editors more than the small world of online big name bloggers. I don’t want to have to compete for the attention of e.g. Gowers, Tao, Kalai, Baez, Cameron, Ghoussoub, Hamkins, Laba, Krautzberger… ;) I just could do with the grudging approval of e.g. Blecher, Davidson, Elliott, Haagerup, Helemskii, Lau, Meyer, Rochberg, Smith, Toms, Voiculescu, or whoever, that my current effort is OK even if not of great “noncommutative spacetime!” interest.