Igor Bogdanoff - Reflections Upon The Peer-review System

Reflections Upon The Peer-review System

During the heyday of this affair, some media coverage cast a negative light on theoretical physics, stating or at least strongly implying that it has become impossible to distinguish a valid paper from a hoax. Overbye's article in the New York Times voiced this opinion, for example, as did Declan Butler's piece in Nature. Posters on blogs and Usenet used the affair to criticize the present status of string theory; for this reason, Woit devoted a chapter of Not Even Wrong, a book strongly critical of string theory, to the affair. On the other hand, George Johnson's report in The New York Times concludes that physicists have generally decided the papers are "probably just the result of fuzzy thinking, bad writing and journal referees more comfortable with correcting typos than challenging thoughts." Furthermore, as physicist Aaron Bergman pointed out while reviewing Not Even Wrong, Woit's conclusion

is undermined by a number of important elisions in the telling of the story, the most important of which is that the writings of the Bogdanovs, to the extent that one can make sense of them, have almost nothing to do with string theory. ... I first learned of the relevant papers in a posting on the internet by Dr. John Baez. Having found a copy of one of the relevant papers available online, I posted that "the referee clearly didn't even glance at it." While the papers were full of rather abstruse prose about a wide variety of technical areas, it was easy to identify outright nonsense in the areas about which I had some expertise. ... A pair of non-string theorists were able to get nonsensical papers generally not about string theory published in journals not generally used by string theorists. This is surely an indictment of something, but its relevance to string theory is marginal at best.

Jacques Distler mused that the tone of the media coverage had more to do with journalistic practices than with physics.

The much-anticipated New York Times article on the Bogdanov scandal has appeared. Alas, it suffers from the usual journalistic conceit that a proper newspaper article must cover a "controversy". There must be two sides to the controversy, and the reporter's job is to elicit quotes from both parties and present them side-by-side. Almost inevitably, this "balanced" approach sheds no light on the matter, and leaves the reader shaking his head, "There they go again..."

Many comments have been made on the possible shortcomings of the referral system for published articles, and also on the criteria for acceptance of a thesis and subsequent delivery of a Doctorate of Philosophy. Frank Wilczek, who edits Annals of Physics (and is now a Nobel laureate), told the press that the scandal motivated him to correct the journal's slipping standards, partly by assigning more reviewing duties to the editorial board.

Prior to the controversy, the reports on the Bogdanov theses and most of the journal referees' reports spoke favorably of their work, describing it as original and containing interesting ideas. This has been the basis of concerns raised about the efficacy of the peer-review system that the scientific community and academia use to determine the merit of submitted manuscripts for publication; one concern is that over-worked and unpaid referees may not be able to thoroughly judge the value of a paper in the little time they can afford to spend on it. Regarding the Bogdanov publications, physicist Steve Carlip remarked:

Referees are volunteers, who as a whole put in a great deal of work for no credit, no money, and little or no recognition, for the good of the community. Sometimes a referee makes a mistake. Sometimes two referees make mistakes at the same time.

I'm a little surprised that anyone is surprised at this. Surely you've seen bad papers published in good journals before this! ... referees give opinions; the real peer review begins after a paper is published.

Similarly, Richard Monastersky, writing in The Chronicle of Higher Education, observed, "There is one way...for physicists to measure the importance of the Bogdanovs' work. If researchers find merit in the twins' ideas, those thoughts will echo in the references of scientific papers for years to come." As of July 2009, the Bogdanovs' six published papers had been cited four times in INSPIRE-HEP, a database of particle physics articles. For comparison, a somewhat controversial cosmological model known as the "ekpyrotic universe" was published in 2001 and had been cited 569 times by July 2009. Before the controversy over their work arose, the scientific community had shown practically no interest in the Bogdanovs' papers; indeed, according to Stony Brook physics professor Jacobus Verbaarschot, who served on Igor Bogdanov's dissertation committee, without the hoax rumors "probably no one would have ever known about their articles." As of 2009, the Bogdanovs have not published any scientific papers since 2003.