## Not signed in

Want to take part in these discussions? Sign in if you have an account, or apply for one below

## Discussion Tag Cloud

Vanilla 1.1.10 is a product of Lussumo. More Information: Documentation, Community Support.

1. Suppose most mathematical research papers were freely accessible online.

Suppose a well-organized platform existed where responsible users could write comments on any paper (linking to its doi, Arxiv number, or other electronic identifier from which it could be retrieved freely), or even mark it up'' (pointing to similar arguments elsewhere, catch and correct mistakes, e.g.), and where you could see others' comments and mark-ups.

Would this be, or evolve into, a useful tool for mathematical research? What features would be necessary, useful, or to-be-avoided-at-all-costs?

This is not a rhetorical question: a committee of the National Research Council is looking into what could be built on top of a World Digital Math Library, to make it even more useful to the mathematical community than having all the materials available. This study is being funded by the Sloan Foundation.

Input from the mathematical community would be very useful.
• CommentRowNumber2.
• CommentAuthorHenry Cohn
• CommentTimeFeb 18th 2013

I’m skeptical about commenting on specific papers, and I don’t see evidence of much demand for it: there’s considerable speculation about what a commenting system might lead to, but that’s different from an actual desire to write comments about papers. It could be that once a system crosses some threshold, in terms of number of users or usability, it will take off, but I’d guess that commenting on math papers will never become widespread. A few papers might get a lot of comments (some useful and appropriate, others less so), some would get a handful, and most would get none. This isn’t an argument against building a commenting system, and it could be worth experimenting with, but I don’t see it as something that should be a priority for the community or for funding agencies. (I think variants of this idea could be much more fruitful, and I’ll discuss them below.)

Marking up papers is much more delicate. In principle it could be useful: I’m sure tons of mathematicians would love to see Terry Tao’s mark up on whatever he read, and if he wanted to mark up any of my papers I’d certainly be interested to see what he had to say. However, I would not be happy to have a system that encouraged mathematicians to view papers with the sort of mark up I expect they would actually receive. Like many mathematicians, I really care about mathematical writing and presentation, and I do not believe most mark up would be an improvement. (I’d be unhappy if someone added clumsy explanations, even of points I admittedly should have elaborated on, or references and comments I thought were only loosely relevant.) Of course I can’t stop people from using whatever mark up system they choose, but it’s not something I think the community should encourage or mathematical organizations should endorse. The way mathematics is presented is really personal, and tacking on a crude form of crowd-sourcing does not feel like an improvement.

I see two fundamental issues with commenting systems that the community would have to sort out:

(1) Attaching comments to articles (like comments on blog posts) is fundamentally a broken idea, and implementing the system this way would lead to trouble. Many comments deal with several articles, conversations shift topic over time, and there’s great interest in having online mathematical discussions that are not necessarily about specific articles at all. If the system gets any substantial use, then either it will need a lot of enforcement of boundaries and rules about what is on topic, or you’ll basically end up with a crazy filing system in which everything is formally associated with some article but not in any logical way (the way today you find discussions in the comments of blog posts that have little or nothing to do with that post).

Aside from these practical issues, there’s also the question of how to think of comments as part of the scholarly literature. Are comments independent contributions in their own right, or ancillary materials bundled with the article they are commenting on and part of its permanent record in the literature? I would very strongly prefer the former: I greatly dislike the idea of people permanently attaching comments to my articles, while of course anyone is welcome to write independent comments that refer back to them. This should be reflected in the architecture of the system. Specifically, comments should be treated similarly to articles that cite a given article, and presented as additional, independent material that refers to it and that would likely be of interest to readers. I.e., the opposite of how blog comments are generally presented.

This would also address the issue of approval. Independent comments on someone’s paper clearly don’t need that person’s approval, while it’s not nearly as clear for attached comments.

So I’d envision a system in which a comment is a small but independent scholarly contribution that may refer to or comment on any number of papers (including none - would you really reject a comment that responded in some valuable way to a previous comment but was no longer really commenting on any specific paper?).

This, it seems to me, is what people really want, as a complement to formal research papers. They want a system for disseminating relatively short, lightweight but academically interesting ideas and commentary, stuff that doesn’t rise to the level of a paper yet still has a real audience. Some of these contributions would be comments on research papers, but most would not.

You can see this on the internet: there’s vibrant mathematical discussion, not much of which could be cast into the form of comments on papers. It occurs on blogs, social networking sites, discussion fora, and Q&A sites. This discussion is scattered around the internet, difficult to search or keep up with, and poorly archived. A lot of it could never really fit as part of the formal mathematical literature, but some of it could. It would be great to encourage the more scholarly parts of this discussion, disseminate them better, and archive them. I don’t know how best to do this, but I think it could really amount to something. By contrast, blog-style comments on research papers is a much narrower and less interesting possibility, and I don’t think it’s what the community really wants or needs.

(2) If a commenting or discussion system has serious academic value, then it must be treated as a serious part of the literature. For example, it should be impossible to delete or modify the historical record: if you update or remove a comment, then the original should always remain available for those who care. Furthermore, these comments should be carefully archived for future generations.

Of course the problem is that there is a serious tension between these constraints and the sort of speed and flexibility that make internet discussions so engaging. If you let people quickly post things online, with no delays or refereeing, you won’t always get archive-worthy material. Then you have a choice between either archiving it anyway (and perhaps embarrassing the author for years to come) or allowing some flexibility regarding what gets archived (which is a dangerous precedent).

I don’t know how best to balance these, and it would really depend on having the right community norms develop. What I’d hope for is something like the arXiv: light filtering to get rid of crackpots, a strong feeling that posting is serious and should not be done thoughtlessly, and a quick but far from instantaneous posting schedule. (Quick enough to enable an actual conversation, although a slow one taking place over days, but without the sort of instant gratification that leads people to post without thinking.)

Under these circumstances, I think one could get away without formal refereeing. After all, if you have refereeing, then how is it different from just encouraging very short papers? What I imagine is that the literature would have several parts: research papers (generally long, serious, carefully refereed) and comments (generally short, possibly less serious, and unrefereed).

As a final comment, one thing I believe the community does not need to sort out now is search, ranking, recommendation, reputation, etc. These are highly contentious issues, and if you push them too far you’ll offend people. (I would not participate in any publication-related system that assigned scores or encouraged voting on submissions, and I would be very unhappy if the World Digital Math Library made what I considered to be biased or inappropriate recommendations.) However, there’s an even more fundamental issue. This is a separate layer from the commenting system itself, and it should be treated as such. People may want to opt in or out of several different, competing recommendation systems, separate from the underlying library but built on top of it, and our ideas of what is fruitful or important will presumably evolve considerably over the next few decades. It would be crazy to hard-code today’s ideas into the library or the literature itself.

In this respect, I see the literature as being like the web. The web deals with dissemination, but makes no attempt to solve the problems of search or recommendation. Instead, some people find things through Google, some through Facebook, some through Twitter, some through reddit, etc. In the mathematical literature, we should have common standards for dissemination and archiving, but it would be foolish to try to standardize search and recommendation.

• CommentRowNumber3.
• CommentAuthorAndrew Stacey
• CommentTimeFeb 18th 2013

I agree with everything that Henry has said. Another post that says a lot of what I would say is this one by Izabella Laba on commenting both on her blog and on articles.

I would like discussion to take place, and I would like to be able to find that discussion, but I strongly feel that comments attached to articles is the wrong implementation of this. Rather than just repeat what Henry and Izabella have said, let me point out something that I think does work: the nLab and the nForum.

In case you don’t know what they are, the nLab is a wiki for people to record their notes on things. To quote from the main page:

The purpose of the nLab is to provide a public place where people can make notes about stuff. The purpose is not to make polished expositions of material; that is a happy by-product.

We all make notes as we read papers, read books and doodle on pads of paper. The nLab is somewhere to put all those notes, and, incidentally, to make them available to others. Others might read them and add or polish them. But even if they don’t, it is still easier to link from them to other notes that you’ve made.

Quite a lot of the content on the nLab is sparked by someone reading a particular paper, or papers on a particular topic. But the wiki format encourages contributions to be about the mathematics not the article. The fact that one can link pages means that you can write a summary of an article and put in links to all the supplementary stuff to be filled in later (if it isn’t already there).

But one thing we discovered early on was that the wiki format was not good for discussion. There were pages where people could list their recent edits, and people left “query boxes” for queries, but neither of these really worked. So the nForum was set up as an aide to the nLab to facilitate discussion. A lot of the posts on the nForum are of the form “I made some changes on page XYZ”, but every now and again there’s a discussion about some mathematics itself. If that coalesces into some greater understanding, it gets put on the nLab. Thus the wiki is also the permanent repository of the discussions.

Interestingly, there is a category on the nForum that is specifically for discussing particular papers and preprints. It has 18 discussions in it. There are nearing 5000 “discussions” on the nForum in total, with nearing 40000 comments. The nLab itself has over 7500 pages.

• CommentRowNumber4.
• CommentAuthorTobyBartels
• CommentTimeFeb 18th 2013

Interestingly, there is a category on the nForum that is specifically for discussing particular papers and preprints. It has 18 discussions in it. There are nearing 5000 “discussions” on the nForum in total, with nearing 40000 comments. The nLab itself has over 7500 pages.

To be fair, people tend not to use the categories correctly on the nForum. There are discussions about papers that get into the Latest Changes category by mistake. And there are discussions that get into that category correctly, but are still discussions about papers, because they refer to a page whose purpose is to describe the paper or its results. But if your main point is that people want to discuss the mathematics rather than the papers as such, then I agree.

• CommentRowNumber5.
• CommentAuthorRodMcGuire
• CommentTimeFeb 18th 2013

You are proposing some vague commenting system that has

1. An infrastructure that doesn’t yet exist.
2. An undefined community. One distinction is whether the system is geared toward experts, students, or those who are expert in one area but students in another.
3. No particular purpose - more specifically no tasks that community involvement is supposed to achieve.

An example of well purposed commenting systems are those used in software development bug tracking and resolution. Such systems can be somewhat open to the world at large. For example I have used Mozilla’s Bugzilla’s tracking of Firefox bugs to check whether some problem I think I’m having is a known bug and in a few cases I’ve reported a bug and had it fixed.

One particular task for whatever purposes you have is to build up the infrastructure. A first step would be to build a unified database that is supposed to hold entries for every math paper or book ever written (with pointers to electronic version if they exist) where each entry contains pointers to all the works it references. This task is well suited toward a somewhat open community contribution system and a more open bug tracking system for things like fixing typos and resolving who exactly an author listed as “J. Smith” is supposed to refer to.

What tasks could such a database be used for?

One simple task would be as a centralized recorder of typos and other more serious errata. Recent books sometimes have on-line errata files if you can find them though they are not always actively maintained. I once bought a recent book and found an on-line errata sheet for it. In reading it I found what looked like a lot more errors including a few sections that looked like they were only partially revised from an earlier system for presenting the material. When I contacted site for the book and sent a list of what I thought was more errors, the junior author eventually replied that the senior author had moved on and didn’t want to update the errors or have anything else to do with the book.

Another task use of a works database is somewhat evaluative. Such a database would put each work on equal footing including a large number of works which few people have read and seem to be of little import. As a crude metric one can use reference counts to determine how “important” a paper is though one gets no indication whether the references are positive, negative, gratuitous, indicate some slight similarity, or use a structure or result or generalize, specialize, re-formalize, or simplify it, along with other things.

The above issues are somewhat involved with the notions of validity and intellectual history. One could enlarge the database to include comments along the lines of the following (though they should include more details):

• J. Smith’s definition of octoplex is almost the same as an 8-squdiod previously defined by G. Smith(1974) which is not in the references.

• A. Smith(1980) pointed out an error in this which was fixed by B. Smith(1985).

• Theorems 3,4, and 5 were proved 10 years earlier by C. Smith(1965) which is not listed in the references.

• D. Smith(1995) shows that axiom 4 is redundant.

• E. Smith(1998) shows that these results are simpler to prove in the more general setting of XXX.

This (validity and history) is a fairly specific task domain which means that the criteria of whether a comment is “on topic” is simple. I would expect the vast majority of works to receive no comments but comments on some works will be contentious and there is need for a maybe moderated comment policy and a policy of signing what is promoted into the database - one would want to prevent the revisions wars that crop up on Wikipedia but still allow someone to provisionally claim priority.

I have no idea how many mathematicians would want to participate in such a task. Its usefulness depends upon its size and it may need a critical mass to take off. On the other hand, if you have a database of titles of all math works that allow comments you may very well need to partition off wrangling about priority so that it doesn’t pollute a less restrictive general comment mechanism.

I’ve mentioned two specific tasks. The real questions should be what tasks can such a commenting system achieve and would people really find them valuable.

• CommentRowNumber6.
• CommentAuthordarij grinberg
• CommentTimeFeb 18th 2013
• (edited Feb 18th 2013)

Three remarks I’d like to make on this:

1) There have been many precedents (both discussions and implementation) for discussion forums to arXiv papers, such as http://arxaliv.org/ . I don’t recall any of them being widely used, and while I don’t know what kept others from using them, I can tell what kept me from using them (and I absolutely admire the idea): They were tailored to short comments, similar to those under blog posts, with no complicated LaTeX (sometimes with no LaTeX at all). I remember trying to post some corrections to someone’s long formulas on arXaliv and failing because the formulas were too complex. A good forum for mathematical discussion should allow LaTeX and offer a selection of packages at least as wide as a typical schoolkids’ forum offers a selection of smilies. This way even those who don’t want to use the LaTeX will notice this and conclude “aah, this is a place for maths and not for one-liners and jokes”. Trolls will still post their longcats but at least they’ll have to draw them in xypic ;)

2) Rod McGuire: Unless I misunderstand what you mean by “evaluative”, I don’t think the advocates of annotation platforms (in mathematics, at least) have any evaluative applications in mind. Sure, annotation platforms will occasionally help expose underappreciated material and even correct hypes (though mathematicians tend to overcorrect, rather than correct, hypes), but it would be very unhealthy to place such expectations on the platform and its users. I certainly don’t want to be confronted with “Why are you commenting so much on epigonal papers A and B, and ignoring paper C which really laid the foundations of the whole subject?”. Mathematical comments should be viewed, first and foremost, as mathematical comments.

3) Andrew Stacey: No offense, but the reason I am not using the nLab is that I’m not a category theorist, and I suspect this holds for many other people. My impression of the nLab, possibly not up-to-date, is that it is a place where category theorists discuss the rest of mathematics from the viewpoint of the category theory; a very good project, but none I’d participate in before learning some category theory beyound Yoneda’s lemma, just as I wouldn’t write a Wikipedia page on something I have no idea about. Forums, by their very forum, look much more inviting to newbies. Not surprised about the ratio of discussions therefore.

• CommentRowNumber7.
• CommentAuthorIzabella Laba
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)
I agree with pretty much everything Henry Cohn and Andrew Stacey said about attaching comments to articles. The only kind of annotations that I'd see as appropriate in formal publishing venues would be those concerning validity and history (as in Rod McGuire's examples), and even that might be better accomplished via links to errata, notes, or follow-up papers.

As for why, in addition to what has been said already: first, informal discussions of my papers should not be part of my formal professional record, for many reasons. Second, there should be no expectation that I will respond to all such discussions, moderate, or even read them. I don't have the time, and on the other hand, the participants in such discussions might feel more comfortable without me looking over their shoulder. Third (and related), informal discussions are so valuable precisely because they are informal. We give ourselves permission to speculate, be vague and imprecise, make errors, maybe get off the track altogether - all of which are unacceptable or at least discouraged in formal publications, and for good reasons. Moving such discussions to formal platforms and imposing the formal requirements of correctness and precision would defeat their purpose, although if this coalesces into some better understanding of the subject, a note describing it might be posted somewhere (as in Andrew Stacey's nLab scenario).

@Darij Grinberg: "Unless I misunderstand what you mean by “evaluative”, I don’t think the advocates of annotation platforms (in mathematics, at least) have any evaluative applications in mind."

Actually, they do. Here are some suggested uses of comments from Gowers's post on epijournals:

"For example, the editor who accepts an article might wish to write a paragraph or two about why the article is interesting, a reader who spots a minor error might write explaining the error and how it can be fixed (if it can), and an expert in the area might write a review that could be very useful to hiring committees."

http://gowers.wordpress.com/2013/01/16/why-ive-also-joined-the-good-guys/

I'm finding that last example especially problematic. In the comments on the same post, Mike Taylor makes a comment to the effect that absence of discussion on an article would mean (to him, anyway) that the article is not very interesting. So, I think that the "evaluative" aspect will be there, even if it's an off-label use.
• CommentRowNumber8.
• CommentAuthordarij grinberg
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)

Izabella: where exactly did Mike Taylor make such a comment? I think the one comment he did is that if someone actively disables discussion of his paper, then he/she might be having some worries about the paper not surviving such a discussion very well. I believe (Mike doesn’t…) that even this is a “circumstantial evidence” kind of assertion and the people who should be making hiring decisions shouldn’t be relying on things like this, if you ask most mathematicians. (I don’t know about palaeobiologists; it might be that the amount of controversy in their field, fueled in part by creationist bullshit, has made people much more suspicious.)

In Tim Gowers’ blog, he only mentions the possibility of a review having a positive (in his opinion) influence on hiring, not claiming that any comment is a review. I don’t think I agree with even this narrow claim, though…

Ultimately, when talking about hiring decisions, I think we should ask not whether we’re going to mess up the system, but whether we’re going to mess up the system more than it already is. If people want to be unjust, they will be; if not, having more information at their hands will not make them more so.

About the “attachment to paper” vs. “independent contributions” issue: Yes, I believe (for the reasons mentioned by Henry and Andrew) that comments should be standalone contributions rather than attached to papers; but I don’t think this is really a big difference, because eventually someone will write a script which, when you look at a paper, collects all the backlinks to it from comments. It’s, of course, of symbolical significance that this is not done by the platform itself.

• CommentRowNumber9.
• CommentAuthorIzabella Laba
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)

http://gowers.wordpress.com/2013/01/16/why-ive-also-joined-the-good-guys/#comment-32656

" “It is tempting to think that every paper would have a lively, engaging and productive comment page. In reality, I expect that this would only happen for a few articles. The majority of papers might get one or two lazy comments.”

The solution to this is probably for us to write more interesting papers."

(The inner quote is from my comment, followed by Mike Taylor's response. For clarity, I should add that Taylor did not say anything about hiring. That was a quote from Gowers's original post)

The main difference between stand-alone contributions and comments attached to papers is who endorses the content, controls its quality and takes responsibility for it. You can of course use a search button to find everything that's ever been posted about a given paper, but the search results will likely depend on page rank (a reputable journal should be ranked higher than some crackpot "math critique" website), reflect distinctions between trustworthy sources and everything else, one's own engagement (or not) with websites, and so on. What gets posted where does make a difference, especially where it concerns the prime real estate.

Edited to add: unfortunately, additional data points are very easy to misuse, especially by administrators who don't really understand them. Several years ago the IMU produced a long document on using the citation index, with focus on how *not* to use it. I imagine they'll have to write a much longer one about internet comments. It's easy to say that if people want to be unfair, they will be, so nothing can make a difference anyway. In my experience, making it more difficult to be unfair (intentionally or not) can actually make a huge difference. But this is going off topic already.
• CommentRowNumber10.
• CommentAuthordarij grinberg
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)

Ah, that comment – sorry; I didn’t parse it carefully enough to understand its implications. I don’t agree with Mike here. I don’t think many mathematicians would.

Do you think pagerank is very informative when it comes to judging mathematical reliability? I’m not exactly sure of that, given how little google helps on technical questions nowadays…

• CommentRowNumber11.
• CommentAuthorIzabella Laba
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)
Just edited my comment above. As for pagerank, it obviously wouldn't be able to actually evaluate the correctness of a math proof, but it would likely rank a reputable journal higher than a vanity outfit, or Terry Tao's blog above some crackpot trying to disprove the Cauchy integral formula, so you're likely to get the more reliable results first. I could imagine that a journal website could run some Google Scholar-type search engine.

I'd like to add something from my experience. I've been on prize selection committees where nominations were, in large part, based on MathSciNet reviews of relevant articles. I pointed out, of course, that this was never the intended purpose of MathSciNet reviews - some referees write long praises of anything that cites their own work, others limit themselves to short summaries of content, others still just quote from the abstract, unaware that the review might be used to this sort of ends. The response was that we didn't have much else to go by. So, it's not a question of whether comments made in a different context *would* be used for evaluation purposes. The fact is that this happens already - fortunately, not very often.
• CommentRowNumber12.
• CommentAuthorRodMcGuire
• CommentTimeFeb 19th 2013

@7 Izabella Laba

The only kind of annotations that I’d see as appropriate in formal publishing venues would be those concerning validity and history (as in Rod McGuire’s examples), and even that might be better accomplished via links to errata, notes, or follow-up papers.

The problem with author controlled external links is that they often suffer from “bit rot”. For example an author at a university creates an errata sheet and then thinks it is done and can be forgotten about, but then someone else comes in and reorganizes the web site or does something else and breaks the link to that page.

For example your Harmonic Research Group page has bit rot. It contains the sentence “We invite you to learn more about our group and our research but the two links in that sentence are currently broken.

Similarly @1 Ingrid Daubechies recommends starting from the World Digital Math Library. However when you go to that page you find that it hasn’t been updated since 2006 and both the links in the first sentence “recommendations on best practices for retrodigitization and a draft document outlining CEIC’s vision of a distributed library of digitized past literature.” are broken.

Rather than monitoring thousands of sites to prevent their bit-rot of things like errata and updates in idiosyncratic formats it would be much easier to centrally administer this information. This is also needed for cases where the author is dead or otherwise doesn’t want to acknowledge errors.

• CommentRowNumber13.
• CommentAuthorJohn Baez
• CommentTimeFeb 19th 2013

I would like some way for me to be able to easily read lots of comments on people’s papers. Right now to find these comments I either use Google or trackbacks on the arXiv. But I think there could be something better.

To be honest, I mostly want to read my own comments on people’s papers, because I wrote a lot of them in This Week’s Finds, and nobody else writes nearly enough. I don’t have much trouble finding my own comments: I use Google, and use keywords that single out This Week’s Finds. But it’s harder finding comments when I don’t know who wrote them or where they are.

• CommentRowNumber14.
• CommentAuthorIzabella Laba
• CommentTimeFeb 19th 2013
@Rod McGuire

I agree that errata should be archived together with the paper, and that this should happen even if the author cannot do it or doesn't like the idea (an editor might have to step in if there's a dispute). I would include your examples 1 and 3 (missing references) in that category.

As for the "forward history" in examples 4 and 5... there's a lot of rot potential there, not because of broken links, but because of the fracturing of data. As you point out correctly, it's hard enough for us to keep our own webpages up to date (yes, I really should do something about that one). I don't believe that anyone will have enough time to post updates every time a theorem is proved, not just once, but on every page associated with every one of the related papers. Most likely, you'll end up with a lot of very incomplete histories posted across many different pages. I'd rather have a few decent expository papers or notes consolidating the information (archived someplace stable, of course).
2. Ignoring the discussion about commenting, I wanted to reply to the original posting.

Would this be, or evolve into, a useful tool for mathematical research?

I think the answer is "yes". And add: it already is. Despite the negative aspects of commenting, there are many people who experience fruitful discussions about their papers, talks, slides and other representations of their research both on blogs and elsewhere.

Annotations are untapped so far. The Open Annotation group is writing a W3C spec and projects like hypthes.is are working hard on realizing them. In addition, annotations are standard in the ebook world and as such used on a daily basis by both researchers and especially students (in eTextbooks).

Last week I had the opportunity to talk to Robert Sanderson, who is on the open annotation group and an information scientist at Los Alamos Labs. He described how Los Alamos is already using web-based annotation of (especially old) papers – both annotating math and annotating using math. At least on a small scale, it’s safe to say that annotating papers is already a fruitful tool for our community.

Input from the mathematical community would be very useful.

I’m not sure that’s really important at this point; coordination with other research fields dealing with these topics seems more relevant to me.

Of course, eventually the community has to embrace new tools and that will take serious effort if we don’t want to fall behind. Thankfully the mathematical community’s willingness (or lack thereof) will not matter – everyone will get these tools sooner rather than later.

Either way, I choose to be optimistic. Who thought MathOverflow could happen (say, five years ago)? Maybe it’s the mathematically inclined research librarians that will make the difference. Maybe it will be just another bunch of grad students.

• CommentRowNumber16.
• CommentAuthorAndrew Stacey
• CommentTimeFeb 19th 2013

Darij, you missed my points on two counts:

1. I quoted the figures on discussions on the nForum to show that whilst there may be a clamour for discussions on articles, then in practice I don’t see it and I have quite a lot of data to base that on. Although Toby’s point is true in that the figure is underestimated, on a forum that is specifically designed for discussion of mathematics, discussions of actual articles forms a fraction of a percent of all the discussions that take place. To get a better picture, I tried search for discussions that have the word “article” or “paper” in the title and the figures were similar. That’s not to say that articles don’t get discussed, just that these take place in a larger discussion.

2. When I gave the example of the nForum/nLab my intention was to say, “Here’s an example of something that is actually working.” meaning not that everyone should join it (though you’d be more than welcome to do so) but that it might be worth figuring out what exactly about it makes it work and learn from that rather than start completely from scratch.

(With regard to the “not being a category theorist”, may I recommend reading the discussion on the nForum Outside, looking in.)

Now let me try to add something to the discussion.

At Oxford (where I’m on sabbatical), we’ve been having a reading group on Functor Calculus. Now, naturally there are a few “backbone” articles that we focus on considerably. But “a few” means more than one, and there are also a lot of other articles that have proved interesting to follow up. Our discussions and notes have focussed primarily on the topic of functor calculus. Sometimes we might have something to say about a particular proof in a particular paper, and the odd comment could very easily be attached to a particular paper, but these aren’t substantive comments. The substantive stuff is where we take an argument and work through it highlighting the important stuff, linking in to ideas elsewhere. These comments might be useful to others, but they don’t link to a particular paper. Rather they link to the topic “Functor Calculus”. There are also papers that link to “Functor Calculus” and these sit together with the comments, but not in a particular hierarchy.

So if someone wants to design a useful place to discuss mathematics, they shouldn’t make articles the roots of the discussion tree. That should be topics.

The other thing I’d like to say is that I think that any new system has to fulfil two criteria for it to take off:

1. Be useful at the point of use
2. Don’t do anything new

I’ve expounded on the first before on this forum. Let me explain the second. I’m quite busy already. If you present me with a new system that does something that I’ve not yet any particular experience with then I’ll say, “Sounds interesting, I’ll take a look when I have a minute” and then forget about it. But if you say, “Here’s a tool that can help you collate your references, just feed in all those .bib files and it’ll index them and make them easier for you to find. You can easily add references direct from MathSciNet and the arXiv - no need to fill out horrible BibTeX forms any more. And when you decide that you actually want to use biblatex instead of bibtex it’ll be easy to switch. Oh, and every morning it’ll present you with the “What’s New” page from the arxiv with nice little checkboxes for you to import them directly into your database.”1 then I start getting interested and I probably don’t give a second thought to the bit that says, “Oh, and by the way. We’ll keep a record of which papers you add. Purely for statistical reasons and with no identifiable information contained. Just to help others know what you find interesting.”. The new system does something that already fits with my workflow, and makes it easier for me. So I adopt it, and then discover that it actually does even more that is also useful and gradually my methods change to fit. But at no point do I have to think “Do I contribute to this shiny new thing, or is it going to be another huge waste of time?”.

It’s spelled out in the purpose of the nLab: “You already do this, just try doing it online.”. Even that jump proves too much for some, but fortunately it proved doable for enough people and those that use it find that it genuinely is “useful at the point of use”. When I contribute to the nLab, I’m not “doing the world a favour”, I’m doing me a favour. As yet, I’ve not seen how a commenting system on articles does me a favour at the point of use.

1. Apart from the biblatex bit - because the export module for that isn’t written yet - this is an accurate description of my reference database management system.

3. I am pretty appealed by the possibility of commenting papers and mathematics in general, but rather agree with Henry Cohn. I would see something like a MathOverflow site, independant from arXiv and any repository and journal, where questions are replaced by comments and answer by follow-up comments. While I do not like the score part of MO very much, it is very important to let the site self-regulate. Maybe one could design it first without all this badge stuff, second with hidden scores that only serve to give moderator rights and take not-easy-to-use-for-evaluation actions (like prompting highly upvoted comments more often).

• CommentRowNumber18.
• CommentAuthorHenry Cohn
• CommentTimeFeb 19th 2013

I would see something like a MathOverflow site, independant from arXiv and any repository and journal, where questions are replaced by comments and answer by follow-up comments.

The stackexchange mechanics work pretty well for soliciting and ranking answers to unambiguous, answerable questions, but the further you move from that the worse they work, so I think a broader system would have to rethink everything more or less from scratch.

What I’m most interested in promoting is comments with real scholarly value, so instead of appearing as blog posts and then gradually sinking into the archives of the blog, they are more visible, better archived, and easy to cross reference. For this, I would not want to build in any game-style mechanics or voting, but rather deal with things more in the style of the arXiv. (Additional filtering or recommendation could be layered on top of this.)

However, ultimately the community needs to have outlets for all sorts of discussion, including things that are not about mathematics per se (career advice, Fields medal speculation, etc.). It’s important to have a broad range of outlets, so it’s clear where the latter sort of discussion would be appropriate and valued. The level and style of moderation could vary substantially between these sites.

From my perspective, for things we view as real parts of the scholarly literature, it’s valuable to standardize everything and have a central plan for archiving, so that scholarly contributions do not fall through the cracks. For things we do not view as part of the literature, it’s important not to try to standardize things. The world would benefit from having various mathematical discussion sites, with different styles and philosophies. If some interesting discussions fall through the cracks and don’t get broadly noticed or preserved, then that’s too bad, but it’s a price I’m willing to pay.

While I do not like the score part of MO very much, it is very important to let the site self-regulate.

One thing worth noting is that MO’s self-regulation has led to a lot of complaints and bitterness at times, with ongoing debates regarding when questions should be closed and whether there’s a small group of high-reputation users imposing their will on the site. On a site like MO, this is tolerable: people only care so much, and in any case it’s reasonable to say “this is the way the site works; if you don’t like it, go create a different sort of site”. However, the more central a commenting system became to mathematics (to the extent that participating was expected by the community and valuable for careers), the less compelling this sort of argument would be. Furthermore, the MO system itself is unpleasant for some fraction of mathematicians, which puts fundamental limits on its growth.

A useful comparison is the arXiv. A lot of crackpots and amateurs are absolutely furious at having their papers rejected, because they see this as being excluded from an important part of the research community. However, this doesn’t bother most academics, for two reasons: the moderation is almost invisible to mainstream members of the community, and it is very light (they err much more often on the side of letting papers in).

Anything intended as widespread infrastructure of mathematical communication needs to work like the arXiv. I.e., it should be acceptable and appealing to almost everyone, with any controversy confined to fringes of the community.

• CommentRowNumber19.
• CommentAuthorMark C. Wilson
• CommentTimeFeb 19th 2013

Some very insightful comments here. I agree with everything Henry has said, in every post, and also Rod McGuire. Apart from a complete database of papers (probably too much to ask, but let’s start there) with forward and backward links corresponding to citations, comments should be restricted to “scholarly” contributions such as bug fixes, weakened hypotheses, etc. These would not normally be published as a full paper, but deserve proper credit for the writer. In many cases authors have no incentive to do this even when they are informed of the corrections (as I know from experience), and centralizing them is much better than not doing so.

A potential problem is: what to do with such “scholarly” short comments that are actually wrong? Moderation could be quite expensive. Formal refereeing of comments seems extreme. I think that a fairly substantial barrier should be placed. Perhaps a reputation system will have to develop among commenters, or perhaps only established professionals will be allowed to comment freely, with others needing a sponsor.

I suggest that Math Reviews should be involved in this project somehow. Reviews ought to be better than they are. Currently, the incentive for contributing a good one (which can be very useful) is too little. Also, arXiV seems vey relevant; as well as learning from their experience, there should be a way to link it more directly to this proposed system.

• CommentRowNumber20.
• CommentAuthorHenry Cohn
• CommentTimeFeb 19th 2013

I suggest that Math Reviews should be involved in this project somehow.

Math Reviews definitely has valuable experience in this area and would be important to consult with, but there are two caveats. One is that too much dependence on a subscription service could be problematic, and the other is that MathSciNet revenues help support other AMS programs, so the AMS might have a conflict of interest regarding other systems that could decrease the demand for MathSciNet.

comments should be restricted to “scholarly” contributions such as bug fixes, weakened hypotheses, etc.

I’d be inclined to broaden it quite a bit beyond tweaks to existing papers. New examples, commentary, exposition, etc. could all make interesting comments. Basically anything in the range between an intellectually serious blog post and a minor paper, the sorts of things people usually don’t publish formally because it doesn’t seem worth the effort (and where would you submit it anyway?), but that still contain something valuable.

A potential problem is: what to do with such “scholarly” short comments that are actually wrong?

I’m not too worried about that, provided that the culture doesn’t encourage thoughtless posting. Everyone would understand comments aren’t refereed, and responsible authors would retract incorrect comments. Some incorrect comments would persist unretracted but be ignored because nobody cares, while others would attract follow-up comments by others explaining why they are wrong. (I hope informing the author first would be the default, but this would be a reasonable next step if necessary.)

• CommentRowNumber21.
• CommentAuthorIzabella Laba
• CommentTimeFeb 19th 2013
@Henry Cohn

"I’d be inclined to broaden it quite a bit beyond tweaks to existing papers. New examples, commentary, exposition, etc. could all make interesting comments. Basically anything in the range between an intellectually serious blog post and a minor paper, the sorts of things people usually don’t publish formally because it doesn’t seem worth the effort (and where would you submit it anyway?), but that still contain something valuable."

I'm not sure that we're still talking about "comments" as they are usually understood. The wider systemic problem is the dearth of appropriate publishing venues for expository papers and short contributions, and if such venues were established, I'm sure that they would be used widely and immediately. Refereeing could be replaced by some minimal screening (arXiv style), and the papers/notes could count as "unrefereed contributions" (on par with unrefereed conference proceedings, for example, except that they will be easier to find). Although frankly, I also would love to see more refereed, reputable journals for expository papers, both as outlets for dissemination of such work and as instruments of giving it the recognition it deserves.
• CommentRowNumber22.
• CommentAuthorHenry Cohn
• CommentTimeFeb 19th 2013
• (edited Feb 19th 2013)

Yeah, I agree that this is much broader than comments on specific papers, and your term “notes” would be more appropriate. Basically, I think there’s a lot of pent-up demand for some form of notes, and this system could easily encompass paper-specific comments in a way that made them easy to find (but avoided many of the disadvantages of blog-style comments). By contrast, I think it would be much harder to make a paper-specific commenting system work well.

Although frankly, I also would love to see more refereed, reputable journals for expository papers, both as outlets for dissemination of such work and as instruments of giving it the recognition it deserves.

I totally agree, although even if they existed I think a lot of valuable writing would not make it to that stage.

• CommentRowNumber23.
• CommentAuthorMark C. Wilson
• CommentTimeFeb 20th 2013

In some fields, letters to the editor of the journal can serve the purpose of short notes. Some journals have “Research Notes”. Perhaps all we need is for our journals to have sections not only for refereed articles, but also for “notes”, with the explicit statement that they are not refereed. I guess those sections of the journal would not be indexed in Math Reviews, for example.

I am keen to start something like this section in the journal I am managing.

• CommentRowNumber24.
• CommentAuthorHenry Cohn
• CommentTimeFeb 20th 2013

I think notes might work better if they aren’t associated with the name of a refereed journal. One reason is to avoid confusion about refereeing status, in either direction (people assuming notes are refereed or wondering whether refereed papers are notes - the citation format might make the status clear in principle, but in practice probably only to people used to this idea). Another is to avoid stigma: designating notes as the unrefereed section of a journal presents them as being much like refereed papers but not as good. Instead, I’d prefer to present them as a different sort of thing entirely, the same way blog posts aren’t just degenerate papers. Of course it’s hard to avoid some comparison with refereed papers, but I’d worry about conveying a message of “the author sent a submission to Journal X but asked them not to referee it”.

• CommentRowNumber25.
• CommentAuthorMark C. Wilson
• CommentTimeFeb 20th 2013

I guess this is a legitimate concern. Another possibility is newsletters. ACM for example has special interest groups, at least some of which have newsletters. Perhaps AMS and SIAM could start some forums for notes, as could other societies.

My idea was that journals really do have to justify their existence now, and adding expository material, even announcements of conferences, etc, could give them more of a reason to exist. Having notes seemed a good idea along these lines. Electronic Journal of Combinatorics has Dynamic Surveys, which seem useful. I am not sure what their refereeing status is, and how they are cited. And I have not seen this done in other journals, although it seems an excellent idea.

4. Somehow, it seems to me that the conversation is leading us to a kind of math blogging platform, where people would need to identify (probably with a soft filter like arXiv endorsement) so that posts are signed, giving some responsibility, and that takes care of technical aspects, archiving, classification etc. It could even be a single huge blog where every mathematician could be a writter. There could be only a posteriori moderation, triggered when a user flags a post. It could be interesting to limit comments to very short contributions, everything longer going to a post with links so that discussions can be followed but can easily ramify. Last, it would probably need a consolidation feature, so that when an intricate discussion lead to a rather short conclusion, the contributors can make the discussion shadowed in the background, with the state of the conclusion more prominent.

5. [Though I feel this now has nothing to do with the OP about annotating papers so maybe should be a separate thread.]

At the Joint Math Meetings, the AMS hinted at a social-network like site in the near future. This might be a platform people could use as Benoit described. Personally, I prefer a decentralized system where the researcher is in control. But democratic oversight from the AMS can make this a trustworthy enterprise.

• CommentRowNumber28.
• CommentAuthorHenry Cohn
• CommentTimeFeb 21st 2013

it seems to me that the conversation is leading us to a kind of math blogging platform

Hmm, I feel like it should be different somehow, but it may take more thought to articulate why and how. Partly it’s a matter of style rather than substance: blogs are generally focused on the writer and chronologically arranged, neither of which would fit how I think of notes, but this is a social rather than a technological issue. You could turn a math blogging platform into a math notes platform, but it would take some interface adjustments and some expectation setting to keep it from turning into a group blog.

One key psychological difference is that maintaining activity is considered important on a blog: your blog is viewed as a failure if you post only on rare occasions. By contrast, with notes I wouldn’t want people to post too often. (If many people were writing a note every week, then this would suggest that notes had ended up being used for much more trivial things than I had hoped.)

It could be interesting to limit comments to very short contributions, everything longer going to a post with links so that discussions can be followed but can easily ramify.

If it were set up this way, I think I’d choose not to have comments at all. Minor corrections and the like don’t need to be done in public (but can just be sent to the author), and serious comments can easily be made as notes in their own right. In general I would downplay the role of rapid discussion: I imagine notes as a way to disseminate short but relatively carefully thought out contributions, rather than a way to hold something like a real-time conversation in public. For that, discussion fora and blogs are probably better.

Last, it would probably need a consolidation feature, so that when an intricate discussion lead to a rather short conclusion, the contributors can make the discussion shadowed in the background, with the state of the conclusion more prominent.

This is another thing I think wouldn’t need an explicit feature. If several notes lead to a new, simpler understanding, then you can just write a new note explaining the simple conclusion (and citing the previous notes for those interested in following the chain back). The primary user interface shouldn’t be a chronological blog-style presentation, so the history shouldn’t be too distracting.

At the Joint Math Meetings, the AMS hinted at a social-network like site in the near future.

Ack, what was the context? UniPHY has failed, and I’m skeptical about the prospects of any disciplinary social network: it would take an impressive new idea to make the benefits high enough to attract a critical mass of users.

Systems like the arXiv are not very social, but they do much better by Andrew Stacey’s criterion of being useful at the point of use.

Personally, I prefer a decentralized system where the researcher is in control.

What counts as being in control? For example, having a centralized arXiv seems to me to be enormously better than just putting preprints on web pages.

• CommentRowNumber29.
• CommentAuthorAndrew Stacey
• CommentTimeFeb 21st 2013

Somehow, it seems to me that the conversation is leading us to a kind of math blogging platform,

Really? I hope not as I don’t feel that the concerns that some of us have raised have as yet been addressed.

Let me try to summarise my feelings on this. I have no problem with the ability to comment on articles in and of itself. It isn’t where I want my mathematical conversations to take place, but if others want to then I certainly can’t and won’t stop them. However, where it becomes a concern is in how those comments are viewed by others and in particular how my non-responsiveness to those comments becomes viewed.

In particular, I would not be happy if comments on articles were mistaken for either of internal or external evaluation (aka peer review; and by “internal” I mean “by other mathematicians” and “external” I mean “by people who aren’t able to understand the work itself”). Who would be so stupid as to do so? Well, the same people that use the impact factor of the journal in which my article appears as a measure of the worth of my article. Let me underline that: if there is no other system for evaluation (both internal and external) then some measure using comments will get used.

So: first sort out the evaluation problem, then centralise comments.

The problem with that is that making a place for comments on articles is really easy. Give me five minutes and I could throw together a forum like this one for it (it really does take me just five minutes). “We can haz maths $a^2 + b^2 = c^2$”. In fact, why do we need a centralised place at all? The internet has plenty of places for discussion, and a search by arxiv identifier can easily find all the discussions on a particular paper. There you are, job solved. Furthermore, unless your system for aggregating and evauating comments is better than pagerank, it seems to me that this is the best way to implement comments on papers.

Let me give a concrete example of why I think it is really important to sort out the evaluation system first. Take my paper The smooth structure of piecewise-smooth loops. This is a paper that uses a fair bit of functional analysis, but the topic is really of interest to topologists. So the number of people who could really read it to understand all of it is not that great. The main result is a “Don’t do this” result. Thus also the majority of people who would be interested in knowing about it will have the response “Okay, I won’t”. It is therefore highly unlikely that it would attract a great number of comments, or even citations - if any at all. Its real success can only be measured by whether or not people stop doing whatever it is I warn against, which is deucedly hard to measure!

In summary: comments on papers is technically easy but unless you can clearly differentiate them from some sort of evaluative process I’d rather not have a centralised system for fear that they will be used as such.

• CommentRowNumber30.
• CommentAuthorNoah Snyder
• CommentTimeMar 29th 2013
Currently the AMS has memoirs (over 50 pages), transactions (15-50 pages), and proceedings (shorter than 10 pages). One could imagine adding an extra layer to that: Notes of the AMS, for things which are 1 or 2 pages. To what extent would or wouldn't that work for what Henry has in mind here?

Scott Morrison was recently telling me about his idea for the "TL;DR Journal of Math", which would consist of at most 1 page explanations of interesting examples, theorems, and facts. In particular, part of the idea was that it would contain better expositions or proofs of published results, which is one of the most important kinds of comment on a paper. Currently the main venue for this sort of thing is department seminars, but that doesn't result in circulation to experts at other schools.
• CommentRowNumber31.
• CommentAuthorHenry Cohn
• CommentTimeMar 30th 2013

Until 2004, Proceedings of the AMS had a special category for 1-2 page papers:

Very short notes not to exceed two printed pages are also accepted, and appear under the heading Shorter Notes. Items deemed suitable include an elegant new proof of an important and well-known theorem, an illuminating example or counterexample, or a new viewpoint on familiar results. New results, if of a brief and striking character, might also be acceptable, though in general a paper which is merely very short will not be suitable for the Shorter Notes department.

It had existed for a long time (at least as far back as the 60’s), but it doesn’t seem to have been very popular. I’m not sure why they killed it in 2004 - does anyone know the backstory?

I suspect one difficulty with the shorter notes section was that there aren’t many 1-2 page papers that really feel like they should appear on a nearly equal footing with longer PAMS papers. Publishing them together pushes the notes section towards having quite high standards.

• CommentRowNumber32.
• CommentAuthorNoah Snyder
• CommentTimeApr 1st 2013
Yeah, publishing notes together with more substantial papers does seem tricky in terms of pushing the standards up too high. But now that I think about it, there's a deeper problem, which is that listing notes as papers on CVs would have the same affect. Publishing notes and listing them as papers on a CV would look like resume padding and so might give anti-credit instead of credit, unless the quality of the notes were quite high. The obvious counterproposal is to have a separate section on CVs of "notes", but then listing them looks "weird" for early adopters.
6. I am not sure whether it is worth putting into notes that would not qualify as short papers the kind of heavy editorial process we use for papers. Somehow, what would be needed is probably not really a journal, but a place where this kind of notes can be stored and made available, possibly with a quick filtering (e.g., the referees would be picked among a pool of volonteer by a computer using the subject classification and number of items recently assigned, and would be asked to check that the note fits some general guideline).

By the way, maybe the pdf is not the best format for this; I will probably turn some short texts written in LaTeX into markdown (or rather its pandoc version) which can easily be turned into html (including math support), pdf (via LaTeX) and other formats.
• CommentRowNumber34.
• CommentAuthorAndrew Stacey
• CommentTimeApr 3rd 2013

Just a minor comment:

By the way, maybe the pdf is not the best format for this; I will probably turn some short texts written in LaTeX into markdown (or rather its pandoc version) which can easily be turned into html (including math support), pdf (via LaTeX) and other formats.

I’m developing a LaTeX class file that allows you to write LaTeX source and output to a variety of formats, including Markdown (for blog posts) and direct to XHTML (for webpages) or ePub3 (for eBooks).

• CommentRowNumber35.
• CommentAuthorScott Morrison
• CommentTimeApr 4th 2013

I’d been thinking about “The Desert Island Journal of Mathematics” (or perhaps just “math, tl;dr”), which sounds very much like what Henry described above in the Proc. AMS. One page papers, not necessarily doing anything “new”, but conveying concisely and clearly an idea that might otherwise be difficult to digest from the literature.

Perhaps a journal with a somewhat silly name could host notes in a way that avoids the previous problems — merely by the name it’s clear that this isn’t a standard journal, and is trying to establish a new category.

• CommentRowNumber36.
• CommentAuthorHenry Cohn
• CommentTimeApr 4th 2013

Perhaps a journal with a somewhat silly name could host notes in a way that avoids the previous problems — merely by the name it’s clear that this isn’t a standard journal, and is trying to establish a new category.

The Unrefereed Journal of Mathematics? That would guarantee the papers couldn’t be confused with refereed journal papers. :-)

Or, in highly questionable Latin, Mathematica Inverificata.

• CommentRowNumber37.
• CommentAuthorDavidRoberts
• CommentTimeApr 5th 2013

Mathematica Inverificata

• CommentRowNumber38.
• CommentAuthorScott Morrison
• CommentTimeApr 8th 2013

I don’t think “unrefereed” is really what we want. Just “short, and not necessarily ’original’”.

• CommentRowNumber39.
• CommentAuthorHenry Cohn
• CommentTimeApr 8th 2013

Yeah, “unrefereed” doesn’t capture exactly the right notion, but I’m not sure how to create an absolutely clear distinction from research papers. Anything with “journal” in the name is probably misleading.

What I’d hope for is that each note would be individually citable in some way (and would have a DOI or permalink), but that CVs or web pages would not list each note individually. It would be more like a blog, where you might note on your CV that you frequently post to a certain blog, and you might even highlight a particular post that was particularly important or influential, but you wouldn’t include a comprehensive list of all your blog posts in your CV. (Actually, now that I wite this I wonder whether some people with academic blogs do that. It seems silly to me, but not inconceivable.)

By contrast, if notes are presented in a sufficiently journal-like format that it looks like each note should be listed individually on your CV, then it will put people off by coming across like publication inflation.

• CommentRowNumber40.
• CommentAuthorKevin Walker
• CommentTimeApr 9th 2013
• (edited Apr 9th 2013)

I don’t understand the concern about how short notes might affect CVs. CV writers have the options of (a) not listing short notes, (b) listing them in a section separate from more traditional journal articles, and/or (c) labeling these CV items (individually) with “a mere short note” or something like that. On the other hand, people who want to puff up their CVs have numerous options, whether or not legitimate journals publish short notes. (See, for example, http://www.nytimes.com/2013/04/08/health/for-scientists-an-exploding-world-of-pseudo-academia.html .)

• CommentRowNumber41.
• CommentAuthorHenry Cohn
• CommentTimeApr 9th 2013

I think it’s a matter of how short notes are perceived by others. There are plenty of blog posts that are interesting and worth archiving for future generations, but where it would look bad for the authors to try to publish them as traditional papers. You could presumably publish them in lower-ranked expository journals, but it would come across as pompous (like you think that blog post was really great, worthy of being treated as an academic publication) or it might be interpreted as just a scheme to increase your publication count.

My fear is that if notes are seen as being more like publications than blog posts, then these pressures will discourage people from writing notes on anything but their best ideas, and the whole idea won’t be viable. I’d bet that’s what happened with Proceedings of the AMS: a 1-2 page paper needs to be of a very high standard for it to seem natural to publish it alongside full research papers.

7. Ingrid Daubechies is continuing this discussion on Terry Tao’s blog.