The Longest Now


“Fatally Flawed” — Internal Britannica Review Tackles Nature Methods
Thursday March 23rd 2006, 3:22 pm
Filed under: metrics

Below is a letter that Encyclopedia Britannica sent out today to some of its customers, in response to the December Nature article comparing the accuracy of articles in Wikipedia and Britannica.  A more detailed review of the Nature study, including responses to each alleged error and omission, is linked from the front page of www.eb.com; you can also see an HTML version of the review here (thanks to Ben Yates).

 

In one of its recent issues, the science journal Nature published an article
that claimed to compare the accuracy of the online Encyclopædia Britannica
with Wikipedia, the Internet database that allows anyone, regardless of
knowledge or qualifications, to write and edit articles on any subject.
Wikipedia had recently received attention for its alleged inaccuracies, but
Nature’s article claimed that Britannica’s science coverage was only slightly
more accurate than Wikipedia’s.

Arriving amid the revelations of vandalism and errors in Wikipedia, such a
finding was, not surprisingly, big news. Perhaps you even saw the story
yourself. It’s been reported around the world.

Those reports were wrong, however, because Nature’s research was invalid.
As our editors and scholarly advisers have discovered by reviewing the
research in depth, almost everything about the Nature’s investigation was
wrong and misleading. Dozens of inaccuracies attributed to the Britannica
were not inaccuracies at all, and a number of the articles Nature examined
were not even in the Encyclopædia Britannica. The study was so poorly
carried out and its findings so error-laden that it was completely without merit.

Since educators and librarians have been among Britannica’s closest
colleagues for many years, I would like to address you personally with an
explanation of our findings and tell you the truth about the Nature study.

Almost everything Nature did showed carelessness and indifference to basic
research standards. Their numerous errors and spurious procedures included
the following:

*    Rearranging, reediting, and excerpting Britannica articles. Several
of the “articles” Nature sent its outside reviewers were only sections of,
or excerpts from Britannica entries. Some were cut and pasted together
from more than one Britannica article. As a result, Britannica’s coverage
of certain subjects was represented in the study by texts that our editors
never created, approved or even saw.
*    Mistakenly identifying inaccuracies. The journal claimed to have
found dozens of inaccuracies in Britannica that didn’t exist.
*    Reviewing the wrong texts. They reviewed a number of texts that
were not even in the encyclopedia.
*    Failing to check facts. Nature falsely attributed inaccuracies to
Britannica based on statements from its reviewers that were
themselves inaccurate and which Nature’s editors failed to verify.
*    Misrepresenting its findings. Even according to Nature’s own
figures, (which grossly exaggerated the number of inaccuracies in
Britannica) Wikipedia had a third more inaccuracies than Britannica.
Yet the headline of the journal’s report concealed this fact and
implied something very different.

Britannica also made repeated attempts to obtain from Nature the original
data on which the study’s conclusions were based. We invited Nature’s
editors and management to meet with us to discuss our analysis, but they
declined.

The Nature study was thoroughly wrong and represented an unfair affront
to Britannica’s reputation.

Britannica practices the kind of sound scholarship and rigorous editorial
work that few organizations even attempt. This is vital in the age of the
Internet, when there is so much inappropriate material available. Today,
having sources like Britannica is more important than ever, with content
that is reliable, tailored to the age of the user, correlated to curriculum,
and safe for everyone.

Whatever may have prompted Nature to do such careless and sloppy
research, it’s now time for them to uphold their commitment to good
science and retract the study immediately. We have urged them strongly
to do so.

Nature responded with a polite declination.

1 Comment

1 Comment so far
Leave a comment

Here’s a response from Nature:

Encyclopaedia Britannica and Nature: a response

In our issue of 15 December 2005 we published a news article that compared the
Internet offerings of Encyclopaedia Britannica and Wikipedia on scientific topics
(“Internet encyclopaedias go head to head”, Nature 438 (7070) p900-901).
Encyclopaedia Britannica has now posted a lengthy response to this article on its
website, accusing Nature of misrepresentation, sloppiness and indifference to
scholarly standards, and calling on us to retract our article. We reject those
accusations, and are confident our comparison was fair.
Our original article made clear the basis of our comparison. Conducted by our news
staff, it consisted of asking independent scholars to review 50 pairs of articles from
the Wikipedia and Encyclopaedia Britannica websites. The reviewers were not
informed which of their pair of articles came from which source; the subjects of the
articles were chosen in advance to represent a wide range of scientific disciplines. Our
staff compiled lists of factual errors, omissions and misleading statements that the
reviewers pointed to (we had 42 usable responses) and tallied up the total number for
each encyclopaedia: 123 for Britannica, 162 for Wikipedia. Turning the reviewers’
comments into numerical scores did require a modicum of judgement, which was
applied diligently and fairly.
Britannica’s general objections to this article were first made to us in private some
months ago, at which point we willingly sent them every comment by a reviewer that
served as the basis for our assessing something as an inaccuracy. While we were quite
willing to discuss the issues, the company failed to provide specific details of its
complaints when we asked for them in order to be able to assess its allegations. We
did not receive any further correspondence until the publication of its open letter on
22 March 2006. It is regrettable that Britannica chose to make its objections public
without first informing us of them and giving us a chance to respond.
The company claims that our article gave a misleading impression of Encyclopaedia
Britannica’s accuracy. Specifically, the company objects to our headline, which says
that Wikipedia “comes close” to Encyclopaedia Britannica in its coverage of
scientific topics. We feel this was a reasonable characterization, and the full figures
featured prominently in the text of the article. The company also objects to the fact
that in some cases we took material from Britannica’s Book of the Year and its
Student Encyclopedia. This was done in a few cases when the Britannica website
provided articles from these sources when queried on the pre-determined topics; as we
said, the survey compared the content of the websites. In a small number of cases, to
ensure comparable lengths, we provided reviewers with chosen excerpts, not full
articles; this was done with entries from both Encyclopaedia Britannica and
Wikipedia.
In one instance Britannica alleges that we provided a reviewer with material that was
not from the Britannica website. We have checked and are confident that this was not
the case.
Britannica objects that Nature did not check the assertions of its reviewers. This is
true; nor did we claim to. We realised that in some cases our reviewers’ criticisms
would be open to debate, and in some cases might be wrong. But this applied as
much to criticisms of Wikipedia as of Encyclopaedia Britannica. Because the
reviewers were blind to the source of the material they were evaluating, and material
from both sources was treated the same way, there is absolutely no reason to think
that any errors they made would have systematically altered the results of our inquiry.
We note that Britannica has taken issue with less than half the points our reviewers
raised. Both encyclopaedias have made corrections to some of the relevant entries
since our article was published.
We do not intend to retract our article.

Comment by sj 03.24.06 @ 1:33 am

RSS feed for comments on this post. TrackBack URI


Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>




Bad Behavior has blocked 480 access attempts in the last 7 days.