Want to take part in these discussions? Sign in if you have an account, or apply for one below
Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.
I’m glad to see this forum!
Here’s my recent thinking:
People seem to get frozen in indecision when contemplating big serious changes to the current journal system. The consequences are too huge. So, I’m thinking we could hasten the evolution of good ideas by doing something quick, easy and fun. Something that’s not too painful to set up. And something that we use because we enjoy it, not because it’s good for us.
I’m imagining something like this:
A system linked to the arXiv, which looks a lot like the arXiv: one page per paper. If you click to download the paper you get sent to the arXiv. But you can also comment on the paper, and/or give a simple ’rating’, on a scale from 1 to 5. When you first join the system, you could say whether you want your ratings to be visible to everyone, or only contribute to an average rating.
Next to every paper you could see how many people have clicked to download it, its average rating, and a link to the discussion thread.
In a further refinement, this could all be combined with a reputation system, so that people could gradually develop fancier and fancier algorithms to decide which papers are ’good’.
BUT: the short-term goal is not to provide a system for ranking job candidates or carrying out official peer review - merely to provide a fun place to talk about papers and help decide what we might like to read.
I think this “lowered ambition” will make it easier for people, especially youngsters, to explore new ways of doing things.
Unfortunately I don’t have the skill or desire to do the programming. But I’m imagining that perhaps it’s not too hard to start a system vaguely like the one I describe.
Ideally what would happen is that people would enjoy the system but dislike certain aspects of it - and then people would improve it, or create competitors. After a certain point, people would start taking one of these systems seriously enough to base hiring or tenure decisions on it, in the same way that they use journals now. But not right away.
Incidentally, the quickest and easiest thing to do is to encourage every mathematician to post all their new papers to the arxiv. Even though there are about 100 new math papers a day being put on arXiv, that’s still probably less than half of all published papers.
For commenting and “voting” there is also papercritic.com. This is a young and active project coming out of programming competition from Mendely and PLoS. They might be inclined to add a paper-view. Also, Mendeley has data on how many papers have been read, annotated, shared etc. This data is available through an API.
Generally speaking, encourage people to make their reviews public! After all, we all do it when we read papers.
Tell people to sign up at wordpress.com, with a pseudonym if they like, and start writing.
Tell them to sign up at researchblogging.org and cross-post reviews of arXiv papers.
I would like to politely disagree with the If you build it, they will come-attitude. Without a culture of making multiple reviews public, I don’t think a platform will work (e.g. even Nature failed at that).
If most people don’t like the idea of numerical ratings, we should skip that, or modify it. All I care about is that lots of people start using an arXiv overlay website to discuss papers.
The first version will inevitably be imperfect; people will complain about it, but if a lot of people use, it they’ll keep trying to improve it, or set up better systems.
I mentioned the idea of ratings merely because most forums that become extremely popular have some feature like this:
Facebook lets you ’like’ something
Google+ lets you ’+1’ something
Amazon lets you rate items from 1 to 5
Netflix lets you give movies from 1 to 5 stars
MathOverflow lets you click ’up’ or ’down’ on questions and answers to those.
and so on. In some of these systems you can see who liked or disliked a given item; in others that information is hidden. Perhaps, for obvious reasons, it’s hidden in all systems where people can say they disliked something (instead of merely not saying they liked it).
People love to complain about these systems, and of course they’re open to abuse, but they’re still useful, and they seem to really engage people. That’s why I suggested introducing one.
Of course, none of these systems are attempting to be a substitute for ’peer review’,… and that’s the exactly the point! I’m not proposing a system that’s supposed to substitute for peer review. It’s only supposed to help you find papers you might like, and see what other people have to say about those papers.
This lowered goal will make it easier for people to start such a website without becoming paralyzed with worry. However, if people feel inclined to become paralyzed with worry over a particular issue, even despite this lowered goal, we should sidestep that issue.
It’s been pointed out that an arXiv overlay website for discussing papers already existed for papers on quant-ph. It was developed by David Bacon. People in the quantum information community enjoyed it a lot: they would go there instead of the arXiv to see the new papers and see which ones people liked. (I believe this system merely allowed you to ’like’ a paper, a bit like Google+ or Facebook.) Unfortunately David Bacon got a job at Google and became too busy to maintain the system, so it no longer exists. However, as noted here, people are now trying to revive it - thanks to the Elsevier boycott! And I know there are people with ideas on how to expand it to the whole arXiv.
I know Yossi Farjoun put some thought into something very much like this a while back. I’ve pinged him, hoping he’ll drop in here.
I fear that “build it and they will come” is wildly naive! Instead, projects like this must be useful at the point of use.
Happily, I think there is definitely a way to achieve this in the current context. Most mathematicians receive a daily email from the arXiv listing new papers in chosen subject areas. If you’re at all like me, you want to subscribe to more than one area, but many (most? nearly all?) of the individual papers aren’t of much interest.
My life would be noticeably better if some pre-filtering or sorting could be done! I’d love to have that email try to cluster papers by tighter topic areas than just the arxiv subject areas, and then only show me my preferred clusters. I’d love to have my attention automatically drawn to papers my friends liked! Indeed, I’d be happy just to have a list of every paper Noah glances at, just to be reminded of what a terrible reader-of-papers I am. :-)
It sounds like David Bacon’s site achieved some of these things. Can we do more?
The fact that popular sites Facebook, google+, amazon, netflix, MO have like buttons does not mean that their popularity mechanism makes them better in inducing the true development of personality etc. Popularity is not a measure of development and achievement itself. A Serbian kid growing up in California came to his parents place in Belgrade and was surprised that he was accepted to any children group to play with on the street, not preceded by marked belonging to a clique, clan or band, unlike what he learned in modern California, driven by TC shows, popularity culture, hi school prom craze. In fact I NEVER heard in my childhood in Croatia that the phrase “be popular” applies to kids. Word “popular” was about movie stars and so on, not about something relevant to our ordinary lives. This was few decades ago. Now it is probably changing, everybody needs to have a good picture to put on social site, and people kill when get defriended. Finally, the democracy in US is ridiculuous – the PR makes people choose whoever, all is lobbyied and advertised; voting is not discovery of the truth.
I fully agree with more sensitive 11: especially this – for peer reviewing it is beneficial to be independent, that one referee does not know what the other one read. Also ASSIGNING a referee makes people discover things which are not easy to appreciate without through reading. It may be a boring looking paper, but the arguments while on the surface written by the same kind of letters and equation symbols are in fact a deep breakthrough. A casual audience will miss this, while true reader will be forced to understand. Once understood it is more likely to give a more balanced mark of the achievement.
On the other hand, I do second that one should have pages of discussion for individual articles. Not collections of opinions, but more a wiki style. Look at wikipedia. If somebody writes something without arguments and references, he is likely to be overwritten. It is also possible to write remarks about the paper. If you observe some interesting relation to other areas, or that the result follows from something else it stays in the wiki. If you repeat the bullshitting which somebody else wrote before, your sentence will get into the history and erased from the new assesment, or labelled as irrelevant.
The danger is also that the author himself will make his work more appreciated by answering to the craze. Instead, as Arnold used to say, one should work in old Russian style of sitting in the library and proving hard theorems. It is not that one should not value or finance education and dissemination of knowledge. But it should be DIFFERENT job and not interfering with appreciation/evaluation of primary results. If somebody is going to get a journalism prize let it not be called a theorem of the year, but a self presentation of the year or whatever. Mathematicians are sometimes subtle in distinguishing nuiances, and should do that even more emphatically here, where people starting mixing apples and pears.
and help decide what we might like to read
One top mathematician whom I pretty much appreciate told me that he almost never goes to the library nowdays and reads only on the internet. He says, in my memory:
I do not care what is published what is not, I google for the name of the author whom I trust or the title I suspect to be interesting, and find it on the arxiv or the homepage of the author. I do not need others (i.e. journal boards) to decide what I will read, I want myself to decide what I will read.
This is not a view of a youngster but of an active mathematicians closer to the retirement than to the PhD. And of course he does appreciate peer advice on what he will read, but only from the colleagues he directly talks to and trusts, of those whose good lecture he might have heard on the topic and not from the “societies”, networks, pyramid schemes, points systems and alike.
This is not to discourage building sites for discussions, wikis etc. – we need them, especially to diseminate the information from the rare centers of hi concentration of expertise in some areas of knowledge to the rest of the intellectual world – but rather to cautiously raise the awareness of the importance of the preservation of healthy basic old practices in refereeing, reading, teaching and research practices; those which the popularity chase and craze and related psychological and social rearrangements, new orders and disorders, eases and diseases do not properly replace. But some new technology will add them. There is so much more new in software and on the internet on the market, than the liking/popularity/points systems.
I’m not in support of having any kind of “point” system papers on the arXiv. There are already a number of mechanisms in place that reward scholarship, and already more than enough incentives to produce widely popular work.
I don’t think that it’s necessarily a bad idea to have some kind of “reward” system in place, but I think that people underestimate how carefully thought-out these point systems are. These don’t make the site inherently engaging or psychologically addictive on their own. They specifically provide some form of inexpensive reward for contributing to the site, and a reason to check back regularly, until using the site becomes a part of the user’s routine.
So if you’re trying to organize large-scale adoption of a site and you want users to
collect public lists of literature that interests them,
submit short synopses or reviews of papers,
ask questions about papers,
respond to questions on their own papers,
make sure that even “lower-profile” papers get reviews, or
maintain professional community standards,
then I feel like these are the things that you should be providing rewards for.
(Kevin: remember that you can start a new discussion on any particular idea that strikes you. Just link between the two to keep them connected.)
With all the ideas I read here, I try to examine them under the spotlight of “How would this make a difference to my daily work?”. A review site sounds fun, but I don’t know that I would use it, and I don’t think that I would be all that happy having my papers there. I’m worried that mine would fall into that category where the “number of people who read it is a number between 0 and 1”. The time that I want to talk about my work is when I’m doing it, not when I’ve written the paper. I like the quote from Dror-Bar Natan on MathOverflow (see also França;ois Dorias’ website):
Papers are written so that their author(s) can forget their content and move on to other things. Therefore when you write you should be very careful to put in enough of the big picture and enough of the details so you’d be able to reconstruct your thoughts 10 years later if you’ll need to, assuming you’ll forget everything but retain some familiarity with some basic principles of mathematics.
So when I’ve written the paper and submitted it to a journal, I’m ready to move on to the next thing. I’m quite happy to talk about that paper, but I’m not so excited about it as I was. Indeed, often by the time I’ve written the paper I get a strange “flat” feeling about it as though I’ll be glad to see the back of it - at least for a short period.
The time when I’d really like to talk about it is when I’m writing the paper. Call this the “alpha” stage. That’s when I’m sure of the basic ideas, but am aware that I might radically change how I think of them as time goes on. Then when I’ve written the paper, I don’t really want to be told of a whole new way of thinking about it, but I don’t mind being told that my notation is rubbish and I should have written it all in the passive voice. Call this the “beta” stage. Finally, I submit the article to a journal and hope that that’s dealt with.
That’s as an author.
As a reader, I have considerable sympathy with Zoran’s last post and with Scott’s further up. I don’t need to be told whether or not I should be interested in a paper. If it contains the result I want, then I’m interested. If it doesn’t, then I’m not. At least, that is, when I’m doing my Main Research - of course, sometimes I read articles just to find out what’s going on, but that’s a bit different. So I don’t want to be told that I should read a paper because it has 5 stars! But I would find it useful to know if there are any corrections or clarifications attached to the paper - this is subtly different to a review.
Andrew wrote:
I don’t need to be told whether or not I should be interested in a paper.
I don’t ’need to be told’, either. That sounds pathetic. I want to hear what papers people are interested in - particularly, specific people I know and trust. And I’m not alone in this: lots of people want to know what their friends and respected colleagues are reading…. and what they think about those papers. They gossip about this endlessly at conferences.
This is why I like the idea of a system where people are able to opt to let others see which papers they like. If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.
On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody. Most people don’t have the energy to write a column like that, but maybe they’d have the energy click ’like’ on a paper now and then, and sometimes even write a paragraph saying something about it. The system I’m envisioning would make all that easy. Of course people could include links to blogs, too.
Kevin wrote:
Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I’ll refrain for the moment.
No, do it! - unless you have some power to make those better people do it.
I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.
28 I agreed above that the discussion sites are welcome, the disagreement was about a point driven system.
21 tlawson says that if one is going to compile bibliography lists and so on, this should be driven by reward system. I have created hundred of nLab entries and put there many hundred references and do that on daily basis for a couple of years. If anybody would pressure me by having the points of my contribution publicly seen (daily followed) at semaphore I WOULD STOP CONTRIBUTING. (Edit: the fact that I am much less active in MathOverflow is partly because I dislike the atmosphere of the reward system there). Many people like me in mathematics find beauty and peace and do not believe that one creative involvement can be linearly compared to any other involvement. If one is creating, one is expecting to see something nice building up. I do not need point like rewards, I need to see that something is created and built, and that some things are solved my own way which I like in the process. I like that I can come back to my notes when I need them again. If I made something neat that this neatness helps me in future, and helps my students and so on. Of course, I like others contributing where I start. But why to mix social needs, cooperative works and so on, with the popularity and linearizing craze ? I know that US media dominate the world and that US culture is obsessed with point systems from kindergarten, but why would adult creative people be bogged with that ?
P.S. I proposed accompanying arXiv with discussion features and hep-th-comment section for short remarks and replies instead of papers still in 2000 in a discussion with Vafa at a conference on orbifolds in Madison. But not in the faintest dream I thought of a point system and rewards, rather than posting arguments, comments, in other words stimulative CONTENT and not pure likes and dislikes.
As requested, here’s an attempt to summarize the discussion. Reading through the comments a second time they seem less diverse and unfocused than they did to me on the first reading, so perhaps this is unnecessary. “#n” means comment number n. I’ve probably overlooked some stuff; bring it to my attention and I’ll edit this post accordingly.
Main idea (#1). A site where one can discuss Arxiv papers, possibly also with some sort of rating system or some way of finding papers which are popular in general or popular with particular people (such as the hugely influential Noah Snyder; see #14).
Existing systems. Similar things have been tried before, so we should try to learn from these earlier attempts (scirate.com (#2, #3), papercritic.com (#7), David Bacon quant-ph thing (#12) [is this the same as scirate.com?]).
Ratings. They could take various forms:
Actual direct ratings of papers, on a scale of 1 to 5, say.
Separate ratings for importance, readability, etc. (#17).
Simple “+1” or “favorite” type system.
Each person could maintain a list of papers that they like, but these lists would not necessarily be aggregated (#14, #23).
Perhaps (only?) comments should be rated (like on amazon.com) (#16, #21).
Arguments in favor of ratings:
Many successful sites use a rating system, and perhaps the system is partly responsible for their success (#12).
It is important to be able to quickly and easily find your way to the most useful papers, and some sort of rating system would help with this (#14, #25, #28).
Arguments against ratings:
Too many to list fully here. See #4, #6, #8, #9, #10, #18, #20, #24, #29, #30 (probably I missed a few).
Ratings are too arbitrary, which is maybe OK on mathoverflow when not much is at stake, but would be bad for a site that might influence hiring decisions, etc.
If the ratings are important, they will likely be gamed.
Ratings could make the site less fun.
Rezk wrote:
I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.
Right. I should have made that more clear - and the website that does this should also make it very clear!
pln wrote:
I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.
My friend Chris Lee, head of bioinformatics at UCLA, is interested in helping set up such a site. He’s good at programming and he’s written an interesting paper:
I hope some people here can join with him in rapidly setting up some sort of ’quick, easy and fun’ website, while leaving open the ability to expand and improve it. I’ll keep talking to him and all of you to help coordinate things. I’ve tried to get him to post comments here himself, too.
By the way: one reason I hope this ’quick, easy and fun’ website gets started soon is that it will pressure people who dislike it to come up with something better. I’m not sure what the best solutions to our problems are. But I don’t think we can find out until someone tries something.
Tim Gowers in the article a-more-modest-proposal proposed a site which would be dedicated to mutual commenting of people (with point-pay for comments from author to the commenter) on their preprints, helping so to improve the papers which are not yet in the final stage. While credits are given, they do not feed the vanity in that case, but the direct need to have the points earned (for the job or pre-reviewer) in order to pay for your own papers being pre-reviewed and hence helped in their finalization and improvement. I like his proposal. The comments there are also very interesting.
I’m wondering if it is ok to open a related thread (or better, a wiki page somewhere) to collect existing technologies, i.e., isolate that aspect of Kevin Walker’s summary #31 (thanks Kevin!).
This could be both about technology (scirate, papercritic, altmetrics.org, +1 etc) but also anything to put these tools in perspective (for example this study on tweets predicting citations).
While this thread shows that there’s need to discuss how we value these tools, I think it might be good to have a neutral thread/wiki page to simply collect facts and usage.
Just like John’s #33 said, the sooner we start, the sooner we’ll know what works. (I would add: there won’t be one tool that works for everybody, but there will be equally useful tools so that everybody can find one that suits them.)
Why not start a discussion here first (make it clear that it’s just for gathering information, not discussing it) and then when the amount of information gets unwieldy, we can look for a more permanent home for it. Wikis are easy to set up, and if needed then we could set up a wiki linked to this forum to gather information - if there isn’t already somewhere suitable. But I don’t want to scatter information too much too early.
John said in #25:
If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.
On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody.
There is a huge difference between these two situations, and I like the second but dislike the first.
I remember what led me to the idea of review panels. It was when I learnt about the Journal of Stuff I Like. Initially, I thought this was great. But as I thought more, I decided that it wasn’t such a great idea after all. The danger with it is that it leads to the Cult of the Personality. John is absolutely right - everyone would tune in to see what Witten or Atiyah or Deligne were reading (or Terry Tao or Tim Gowers or John Baez), but no-one would be all that interested in what Andrew Stacey was reading. So we’d be shifting the pressure from trying to be published in Annals to being mentioned by one of the Big Wigs. This feels wrong on so many levels.
However, This Week’s Finds is a wholly different beast. I admit that I’ve only dipped in here and there, so I may have the wrong impression. That impression is that you (John) are more interested in telling us about a particular topic than about a particular paper. There’s always “Where to find out more” lists at the end, and it may be a paper that sparked the interest in the topic, but the central theme is of “Here’s something cool to know about” rather than “Here’s something cool to read”. Now that is far more useful. Whilst I feel that I don’t really need someone to tell me whether or not to read a particular article - once I have that article in front of me - I often feel that I need someone to say “If you want to learn about X then the papers to start with are A, B, and C”.
So I see a big difference between setting up a system for adding reviews to the arXiv, and a system for making it easier to find papers by topic. The former doesn’t seem to add much usefulness, just a load of froth(!), but the latter makes it easier for me to do my job.
@Alexander, I remember reading a great comment you made somewhere, where you talked about what the purpose of peer review is, for all the researchers outside of that ’research elite’ you mention. Now I can’t find it, even with the help of google. Could you point me to it if you remember the one I mean, or otherwise write somewhere (here, a blog, etc? :-) about this again? I agree people need to appreciate this point further, but I know I can’t articulate it quite right yet!
Well, the email addresses are included in the papers and often there is a mark that the comments are welcome. This does not make many people commenting by email anyway. Web makes it sometimes easier, unfortunately many commenting sites will have their own resistrations which make a turn off to many. So it is good that it is not just trackbakc link but indeed one unique system for the whole arxiv. Gowers’ idea that we could have a system of credits for “reviewing” preprints, granted by the authors, who have some credits to give up, is a good incentive for certain kind of activity. Having possibly new arXiv categories of new micro-“comment” articles which could be stably cited is another. We need to discuss more mechanisms and forms.
@Andrew #37 I understand and share the worry. But I think it’s equally likely that we could end up with Mathematical Critics (much like, say, literature critics), i.e., people who are renowned for there knowledge and experience at identifying good research. Add to that the study I linked to in #35 – tweets may be able to predict citations. That’s not “Tim Gowers’s tweets predict citations”, it’s the “wisdom of crowds”. For a different experiment check Crowdometer.
@Alexander Chernov #40 second that! (In fact, see #7) – wordpress can solve a lot of these problems out of the box.
44 With many proposals involving wordpress – what is the gurantee that the wordpress will stay free ? This is uninformed but serious question: I know there is a commercial owner of the wordpress web site, there are free user accounts, and there is some software there and some related software (and add-ons) in circulation. I do not know which portion of the software is transferrable to other sites retaining full functionality, how much of it is open source and what are ultimately the licences involved. I am sure some people here know – please let us know.
Peter at 44: isn’t the danger that “good research” becomes increasingly defined as “what those Big Name good critics like?” There are also people doing mathematics in systems or societies where the kind of openness taken for granted by this kind of model is not so easy, and the loudest voices may not be the wisest.
I also share Alexander’s concerns at 38.
@Yemon yes, there’s a danger. But how is that different from today? Editorial boards are precisely “Big Name critics”. Yet journals are so narrow in their design, that they will never say “hey, I just read this fantastic expository paper about this obscure proof that nobody understood; there’s nothing new, but it’s the best paper I read all month”.
As I tried to argue with the study about tweets predicting citations, the “wisdom of crowds” approach is likely the most fruitful (and most democratic) and specifically eliminates this problem of “Prof BigWig said so”.
@szkoda wordpress is open source. So yes, it will always be free. The distinction is between wordpress.com, a free hosting service, and wordpress.org the underlying software. But wordpress.com also lets you export anything from your site there and import into your own installation. I know a little bit about this since we’ve started boolesrings.org last summer, an experiment towards a wordpress hosting service specifically for mathematicians. We are working on a new site that would be open to everybody, but above all we focus on a decentralized approach, i.e., identify the open source tools that mathematicians might need for the purpose of their academic homepages (from citations management to course management). If you want to look at some of its potential, you might want to check out Joel David Hamkin’s site
To 46, let me give just a small unimportant comment: even Grothendieck was at times unhappy with such a great mathematician as Weil surpressing the unconsent voices within such an open and small circle like Bourbaki was (according to his memoirs). In current refereeing process editors control that even the strong opinions of experts will be taken into account mostly if they are concrete and well pronounced. It is not always good that the same expert makes an argument (recommendation) and the decision: the two step process with different subjects involved makes the personal opinions having less weight.
47: thanks
@zskoda Oops, sorry, I expanded my post a little while you were writing yours…
Peter: which crowds? I really think there are contingent factors to do with location and politics that are not always accounted for, at least in the corners of maths where I send my working time. You want to know how big a crowd care about the cohomology of Banach algebras and know what they are talking about? I could probably have them over for dinner in one go…
And actually I trust journal boards or handling editors more than the small world of online big name bloggers. I don’t want to have to compete for the attention of e.g. Gowers, Tao, Kalai, Baez, Cameron, Ghoussoub, Hamkins, Laba, Krautzberger… ;) I just could do with the grudging approval of e.g. Blecher, Davidson, Elliott, Haagerup, Helemskii, Lau, Meyer, Rochberg, Smith, Toms, Voiculescu, or whoever, that my current effort is OK even if not of great “noncommutative spacetime!” interest.