“Fatally Flawed” — Internal Britannica Review Tackles Nature Methods

Below is a letter that Encyclopedia Britannica sent out today to some of its customers, in response to the December Nature article comparing the accuracy of articles in Wikipedia and Britannica.  A more detailed review of the Nature study, including responses to each alleged error and omission, is linked from the front page of www.eb.com; you can also see an HTML version of the review here (thanks to Ben Yates).

In one of its recent issues, the science journal Nature published an article
that claimed to compare the accuracy of the online Encyclopædia Britannica
with Wikipedia, the Internet database that allows anyone, regardless of
knowledge or qualifications, to write and edit articles on any subject.
Wikipedia had recently received attention for its alleged inaccuracies, but
Nature’s article claimed that Britannica’s science coverage was only
slightly more accurate than Wikipedia’s.

Arriving amid the revelations of vandalism and errors in Wikipedia, such a
finding was, not surprisingly, big news. Perhaps you even saw the story
yourself. It’s been reported around the world.

Those reports were wrong, however, because Nature’s research was invalid. As
our editors and scholarly advisers have discovered by reviewing the research
in depth, almost everything about the Nature’s investigation was wrong and
misleading. Dozens of inaccuracies attributed to the Britannica were not
inaccuracies at all, and a number of the articles Nature examined were not
even in the Encyclopædia Britannica. The study was so poorly carried out and
its findings so error-laden that it was completely without merit.

Since educators and librarians have been among Britannica’s closest
colleagues for many years, I would like to address you personally with an
explanation of our findings and tell you the truth about the Nature study.

Almost everything Nature did showed carelessness and indifference to basic
research standards. Their numerous errors and spurious procedures included
the following:

*       Rearranging, reediting, and excerpting Britannica articles. Several
of the “articles” Nature sent its outside reviewers were only sections of,
or excerpts from Britannica entries. Some were cut and pasted together from
more than one Britannica article. As a result, Britannica’s coverage of
certain subjects was represented in the study by texts that our editors
never created, approved or even saw.
*       Mistakenly identifying inaccuracies. The journal claimed to have
found dozens of inaccuracies in Britannica that didn’t exist.
*       Reviewing the wrong texts. They reviewed a number of texts that were
not even in the encyclopedia.
*       Failing to check facts. Nature falsely attributed inaccuracies to
Britannica based on statements from its reviewers that were themselves
inaccurate and which Nature’s editors failed to verify.
*       Misrepresenting its findings. Even according to Nature’s own
figures, (which grossly exaggerated the number of inaccuracies in
Britannica) Wikipedia had a third more inaccuracies than Britannica. Yet the
headline of the journal’s report concealed this fact and implied something
very different.

Britannica also made repeated attempts to obtain from Nature the original
data on which the study’s conclusions were based. We invited Nature’s
editors and management to meet with us to discuss our analysis, but they
declined.

The Nature study was thoroughly wrong and represented an unfair affront to
Britannica’s reputation.

Britannica practices the kind of sound scholarship and rigorous editorial
work that few organizations even attempt. This is vital in the age of the
Internet, when there is so much inappropriate material available. Today,
having sources like Britannica is more important than ever, with content
that is reliable, tailored to the age of the user, correlated to curriculum,
and safe for everyone.

Whatever may have prompted Nature to do such careless and sloppy research,
it’s now time for them to uphold their commitment to good science and
retract the study immediately. We have urged them strongly to do so.

Nature responded with a polite but firm declination.

1 Comment

  1. sj

    March 24, 2006 @ 1:33 am

    1

    Here’s a response from Nature:

    Encyclopaedia Britannica and Nature: a response

    In our issue of 15 December 2005 we published a news article that compared the
    Internet offerings of Encyclopaedia Britannica and Wikipedia on scientific topics
    (“Internet encyclopaedias go head to head”, Nature 438 (7070) p900-901).
    Encyclopaedia Britannica has now posted a lengthy response to this article on its
    website, accusing Nature of misrepresentation, sloppiness and indifference to
    scholarly standards, and calling on us to retract our article. We reject those
    accusations, and are confident our comparison was fair.
    Our original article made clear the basis of our comparison. Conducted by our news
    staff, it consisted of asking independent scholars to review 50 pairs of articles from
    the Wikipedia and Encyclopaedia Britannica websites. The reviewers were not
    informed which of their pair of articles came from which source; the subjects of the
    articles were chosen in advance to represent a wide range of scientific disciplines. Our
    staff compiled lists of factual errors, omissions and misleading statements that the
    reviewers pointed to (we had 42 usable responses) and tallied up the total number for
    each encyclopaedia: 123 for Britannica, 162 for Wikipedia. Turning the reviewers’
    comments into numerical scores did require a modicum of judgement, which was
    applied diligently and fairly.
    Britannica’s general objections to this article were first made to us in private some
    months ago, at which point we willingly sent them every comment by a reviewer that
    served as the basis for our assessing something as an inaccuracy. While we were quite
    willing to discuss the issues, the company failed to provide specific details of its
    complaints when we asked for them in order to be able to assess its allegations. We
    did not receive any further correspondence until the publication of its open letter on
    22 March 2006. It is regrettable that Britannica chose to make its objections public
    without first informing us of them and giving us a chance to respond.
    The company claims that our article gave a misleading impression of Encyclopaedia
    Britannica’s accuracy. Specifically, the company objects to our headline, which says
    that Wikipedia “comes close” to Encyclopaedia Britannica in its coverage of
    scientific topics. We feel this was a reasonable characterization, and the full figures
    featured prominently in the text of the article. The company also objects to the fact
    that in some cases we took material from Britannica’s Book of the Year and its
    Student Encyclopedia. This was done in a few cases when the Britannica website
    provided articles from these sources when queried on the pre-determined topics; as we
    said, the survey compared the content of the websites. In a small number of cases, to
    ensure comparable lengths, we provided reviewers with chosen excerpts, not full
    articles; this was done with entries from both Encyclopaedia Britannica and
    Wikipedia.
    In one instance Britannica alleges that we provided a reviewer with material that was
    not from the Britannica website. We have checked and are confident that this was not
    the case.
    Britannica objects that Nature did not check the assertions of its reviewers. This is
    true; nor did we claim to. We realised that in some cases our reviewers’ criticisms
    would be open to debate, and in some cases might be wrong. But this applied as
    much to criticisms of Wikipedia as of Encyclopaedia Britannica. Because the
    reviewers were blind to the source of the material they were evaluating, and material
    from both sources was treated the same way, there is absolutely no reason to think
    that any errors they made would have systematically altered the results of our inquiry.
    We note that Britannica has taken issue with less than half the points our reviewers
    raised. Both encyclopaedias have made corrections to some of the relevant entries
    since our article was published.
    We do not intend to retract our article.

Log in