Nature compared Wikipedia and Britannica science articles and sent them to experts in the field. The number of "factual errors, critical omissions and misleading statements" were recorded.
From their blog:
I also received a private email from them in response to a request for more information. I hope they don't mind me posting it below:
Update: The reviewer reports are now available on the Nature web site, in Microsoft Word format. See above URL.
This review was covered at Wikipedia:Wikipedia Signpost/2005-12-19/Nature study
Of the 42 articles reviewed, 38 were found to have at least one error – Britannica had 40 articles with at least one error. (NOTE: Nature took some of the excerpts for the study from the version of Britannica for children and youths rather than use the official version for adults. Britannica claims the study is invalid as a result because none of the articles taken from Wikipedia were "for children or youths.") — Preceding unsigned comment added by 64.134.40.134 ( talk) 00:02, 25 November 2012 (UTC)
The following articles had the highest number of errors:
The following articles had no errors highlighted:
Nature's special report also noted the following:
Update: the detailed reviewer reports are now available (see above).
Please post below a table of errors/word statistics, based upon the Nature article and the word counts in the corresponding articles, so that we can see a more controlled comparison of error rates. —Steven G. Johnson 02:53, 15 December 2005 (UTC)
Note that, for a fair comparison, we shouldn't include tables of contents, external links, "see also", or references — most Britannica articles do not include these, and the Nature review did not consider referencing quality. Nature refers to "factual errors, omissions or misleading statements", so some of the "errors" listed below may be errors of omission in incomplete articles, rather than factual errors.
Article name | Britannica | Wikipedia | ||||
---|---|---|---|---|---|---|
Word count | Errors | Errors/word | Word count | Errors | Errors/word | |
Acheulean industry | 500 | 1 | 0.002 | 417 | 7 | 0.016787 |
Agent Orange | 252 | 2 | 0.00793 | 1270 | 2 | 0.0015748 |
Aldol reaction | 130 | 4 | 0.030769 | 660 | 3 | 0.0045455 |
Archimedes' principle | 350 | 2 | 0.0057143 | 607 | 2 | 0.0032949 |
Australopithecus africanus | 235 | 1 | 0.0042553 | 496 | 1 | 0.0020161 |
Bethe, Hans | 658 | 1 | 0.0015198 | 1823 | 2 | 0.0010971 |
Cambrian explosion | 519 | 10 | 0.019268 | 702 (13 Dec.) | 11 | 0.0157 |
Cavity magnetron | 394 | 2 | 0.0050761 | 1121 | 2 | 0.0017841 |
Chandrasekhar, Subrahmanyan | 365 | 4 | 0.010959 | 417 | 0 | 0 |
CJD | 591 | 2 | 0.0033841 | 1373 | 5 | 0.0036417 |
Cloud | 641 | 3 | 0.0046802 | 1689 | 5 | 0.0029603 |
Colloid | 561 | 3 | 0.0053476 | 896 | 6 | 0.0066964 |
Dirac, Paul | 837 | 10 | 0.011947 | 1044 | 9 | 0.0086207 |
Dolly | 1334 | 1 | 0.00074963 | 807 | 4 | 0.0049566 |
Epitaxy | 178 | 5 | 0.028090 | 235 | 2 | 0.0085106 |
Ethanol * | 315 | 3 | 0.0095238 | 2631 | 5 | 0.0019004 |
Field effect transistor | 588 | 3 | 0.0051020 | 933 | 3 | 0.00322 |
Haber process | 241 | 1 | 0.0041494 | 531 | 2 | 0.0037665 |
Kinetic isotope effect | 210 | 1 | 0.0047619 | 569 | 2 | 0.0035149 |
Kin selection | 923 | 3 | 0.0032503 | 404 | 3 | 0.0074257 |
Lipid | 349 | 3 | 0.0085960 | 676 | 0 | 0 |
Lomborg, Bjorn | 518 | 1 | 0.0019305 | 1501 | 1 | 0.00066622 |
Lymphocyte | 479 | 1 | 0.0020877 | 351 | 2 | 0.0056980 |
Mayr, Ernst | 357 | 0 | 0 | 753 | 3 | 0.0039841 |
Meliaceae | 152 | 1 | 0.0065789 | 281 | 3 | 0.010676 |
Mendeleev, Dmitry | 1306 | 8 | 0.0061256 | 1134 | 19 | 0.016755 |
Mutation | 728 | 8 | 0.010989 | 1557 | 6 | 0.0038536 |
Neural network | 557 | 2 | 0.0035907 | 1233 | 7 | 0.0056772 |
Nobel prize | 409 | 4 | 0.0097800 | 2052 | 5 | 0.0024366 |
Pheromone | 313 | 3 | 0.0095847 | 461 | 2 | 0.0043384 |
Prion | 473 | 3 | 0.0063425 | 1583 | 7 | 0.0044220 |
Punctuated equilibrium | 943 | 1 | 0.0010604 | 1265 | 0 | 0 |
Pythagoras' theorem * | 688 | 1 | 0.0014535 | 1899 | 1 | 0.00052659 |
Quark | 1112 | 5 | 0.0044964 | 2060 | 0 | 0 |
Royal Greenwich Observatory | 235 | 3 | 0.012766 | 532 | 5 | 0.0093985 |
Royal Society | 416 | 6 | 0.014423 | 869 | 2 | 0.0023015 |
Synchrotron | 770 | 2 | 0.0025974 | 1590 | 2 | 0.0012579 |
Thyroid | 583 | 4 | 0.0068611 | 1459 | 7 | 0.0047978 |
Vesalius, Andreas | 930 | 2 | 0.0021505 | 1174 | 4 | 0.0034072 |
West Nile Virus | 245 | 1 | 0.0040816 | 1320 | 5 | 0.0037879 |
Wolfram, Stephen | 475 | 2 | 0.0042105 | 559 | 2 | 0.0035778 |
Woodward, Robert Burns | 873 | 0 | 0 | 2320 | 3 | 0.0012931 |
Total | 22733 | 123 | 45254 | 162 | ||
Mean | 541.26 | 2.9286 | 0.0054106 | 1077.5 | 3.8571 | 0.0035798 |
* - Articles marked as good articles
From the editors (see supplementary Nature report above):
So, the criterion was not "vastly different" in length, which would allow e.g. a factor of two difference, and maybe even a factor of 5. —Steven G. Johnson 17:35, 22 December 2005 (UTC)
None of the articles reviewed have featured article status, and none have undergone our internal peer review process. Two articles, Ethanol and Pythagorean theorem, have good article status with the latter being a former featured article (see here, though there is virtually no discussion there). violet/riga (t) 19:48, 15 December 2005 (UTC)
Note that Aldol reaction is now tended by a few PhD Chemistry candidates and is FA.-- SupperBird 22:19, 3 October 2007 (UTC)
Putting a "This article has been identified as possibly containing errors" tag on the front page of each and every articles identified by Nature seemed to some to be a bit of an overreaction. Others were quite comfortable with the practice.
In the long run, it seems that the project survived the tagging of all articles in which Nature's experts found errors. I would suggest that the tagging was even constructive and usefu. I note that we tag tens of thousands of articles as part of the Category:Wikipedia backlog by rank amateurs and teen-agers with éclat. Perhaps our amateurs are simply uncomfortable being reviewed by experts — just as Jimmy Wales finds working with domain experts to be "intimidating".
References:
While some progress was made in the days immediately after the report came out, the effort stalled as the Christmas 2005 holiday approached. When one examines the progress made before Christmas, one can see that much of the work was "low hanging fruit" in the form of easy corrections to superficial criticisms or where specific corrections were offered by the expert reviewers.
To be clear: it took several weeks before even 50% of the listed corrections were made. Unfortunately, some Wikipedia commentators like David Weinberger were under the false impression that "almost all" of the corrections were made within the first 24 hours of the effort. See Web of Ideas: The Authority of Wikipedia Around minute 70 of audio track. March 17, 2006. This further propagates the misconception prompted by the fact that Wikipedia has millions of accounts registered. More realistically, Wikipedia has only a few thousand active editors (or ten thousand if you use a generous definition of "active editor"), with varying levels of knowledge and maturity and many of whom contribute little to mainspace articles outside of reverting obvious vandalism. Refer to Wikipedia:List of Wikipedians by featured article nominations which lists appromixate 1000 users for a realistic list of contributors who have ever contributed substantial amounts of original quality content (with perhaps a factor of 5X for substantial sub unlisted collaborators). There are about 1000 administrators, and less than 100 other users with more elevated access. Also, in the new master plan, there will be only about 2000 "trusted" users, but all users will be rated with new "reliability" software to be implemented. Of course, one's personal relationship with Jimmy Wales or one's popularity within the community will still be able to override this number.
If one examines the status at the start of 2006, one sees the pattern: easy fixes or articles with a small number of errors were addressed at the start of the effort. Many of the easy fixes were biographical or narrative in nature and of a "coffee table book" level of understanding typical of an English major. Note that only the Aldol reaction article was a Featured article at the time of the review and that this article is fortunate to have three PhD chemistry candidates as caretakers (having such expertise involved on an extended basis is a rare situation at Wikipedia). The remaining articles required a conceptual understanding of chemistry, math, physics and engineering at around the college undergraduate level. The criticisms of articles such as Archimedes Principle, Dmitri Mendeleev, Colloid, Epitaxy, Field effect transistor, Kinetic isotope effect, Prion, Thyroid were not just syntactical corrections: they required an understanding at the semantic level on these scientifically non-controversial and well-understood topics for which there are, in an absolute and objective sense, right and wrong answers that are not intuitively obvious but require the ability to master technical concepts and facts.
For the academically honest, these later corrections were not subject to public opinion or Wikipedia's style of consensus building akin to voting since most people (and probably most Wikipedians, including the Board of Trustees and past and present members of the Arbitration committee, other WikiMedia foundation advisers and personnel and the employees of Wikia) do not and never will understand these concepts correctly and with depth. One user in particular, Pinktulip, did demonstrate such intellectual mastery and made a steady single-handed effort during January 2006 to ensure that another month did not pass before all the corrections were accomplished. Pinktulip also then informed Jimmy Wales of the completion of the overall effort on 25 January 2006.
Note that Wikipedia:Expert retention is an ongoing problem at Wikipedia.-- Simongar 20:14, 26 September 2007 (UTC)
Britannica has replied to Nature's article and claim the nature article was itself innacurate in a number of ways. It would probably be good for Wikipedians to check the sources Britannica gives and note needed changes in our articles.
Nature has now issued comments on Britannica's rebuttal, and stands by its original article. [2] (archive).
Jim Giles made a presentation at Wikimania on 2006-08-04.
Nature compared Wikipedia and Britannica science articles and sent them to experts in the field. The number of "factual errors, critical omissions and misleading statements" were recorded.
From their blog:
I also received a private email from them in response to a request for more information. I hope they don't mind me posting it below:
Update: The reviewer reports are now available on the Nature web site, in Microsoft Word format. See above URL.
This review was covered at Wikipedia:Wikipedia Signpost/2005-12-19/Nature study
Of the 42 articles reviewed, 38 were found to have at least one error – Britannica had 40 articles with at least one error. (NOTE: Nature took some of the excerpts for the study from the version of Britannica for children and youths rather than use the official version for adults. Britannica claims the study is invalid as a result because none of the articles taken from Wikipedia were "for children or youths.") — Preceding unsigned comment added by 64.134.40.134 ( talk) 00:02, 25 November 2012 (UTC)
The following articles had the highest number of errors:
The following articles had no errors highlighted:
Nature's special report also noted the following:
Update: the detailed reviewer reports are now available (see above).
Please post below a table of errors/word statistics, based upon the Nature article and the word counts in the corresponding articles, so that we can see a more controlled comparison of error rates. —Steven G. Johnson 02:53, 15 December 2005 (UTC)
Note that, for a fair comparison, we shouldn't include tables of contents, external links, "see also", or references — most Britannica articles do not include these, and the Nature review did not consider referencing quality. Nature refers to "factual errors, omissions or misleading statements", so some of the "errors" listed below may be errors of omission in incomplete articles, rather than factual errors.
Article name | Britannica | Wikipedia | ||||
---|---|---|---|---|---|---|
Word count | Errors | Errors/word | Word count | Errors | Errors/word | |
Acheulean industry | 500 | 1 | 0.002 | 417 | 7 | 0.016787 |
Agent Orange | 252 | 2 | 0.00793 | 1270 | 2 | 0.0015748 |
Aldol reaction | 130 | 4 | 0.030769 | 660 | 3 | 0.0045455 |
Archimedes' principle | 350 | 2 | 0.0057143 | 607 | 2 | 0.0032949 |
Australopithecus africanus | 235 | 1 | 0.0042553 | 496 | 1 | 0.0020161 |
Bethe, Hans | 658 | 1 | 0.0015198 | 1823 | 2 | 0.0010971 |
Cambrian explosion | 519 | 10 | 0.019268 | 702 (13 Dec.) | 11 | 0.0157 |
Cavity magnetron | 394 | 2 | 0.0050761 | 1121 | 2 | 0.0017841 |
Chandrasekhar, Subrahmanyan | 365 | 4 | 0.010959 | 417 | 0 | 0 |
CJD | 591 | 2 | 0.0033841 | 1373 | 5 | 0.0036417 |
Cloud | 641 | 3 | 0.0046802 | 1689 | 5 | 0.0029603 |
Colloid | 561 | 3 | 0.0053476 | 896 | 6 | 0.0066964 |
Dirac, Paul | 837 | 10 | 0.011947 | 1044 | 9 | 0.0086207 |
Dolly | 1334 | 1 | 0.00074963 | 807 | 4 | 0.0049566 |
Epitaxy | 178 | 5 | 0.028090 | 235 | 2 | 0.0085106 |
Ethanol * | 315 | 3 | 0.0095238 | 2631 | 5 | 0.0019004 |
Field effect transistor | 588 | 3 | 0.0051020 | 933 | 3 | 0.00322 |
Haber process | 241 | 1 | 0.0041494 | 531 | 2 | 0.0037665 |
Kinetic isotope effect | 210 | 1 | 0.0047619 | 569 | 2 | 0.0035149 |
Kin selection | 923 | 3 | 0.0032503 | 404 | 3 | 0.0074257 |
Lipid | 349 | 3 | 0.0085960 | 676 | 0 | 0 |
Lomborg, Bjorn | 518 | 1 | 0.0019305 | 1501 | 1 | 0.00066622 |
Lymphocyte | 479 | 1 | 0.0020877 | 351 | 2 | 0.0056980 |
Mayr, Ernst | 357 | 0 | 0 | 753 | 3 | 0.0039841 |
Meliaceae | 152 | 1 | 0.0065789 | 281 | 3 | 0.010676 |
Mendeleev, Dmitry | 1306 | 8 | 0.0061256 | 1134 | 19 | 0.016755 |
Mutation | 728 | 8 | 0.010989 | 1557 | 6 | 0.0038536 |
Neural network | 557 | 2 | 0.0035907 | 1233 | 7 | 0.0056772 |
Nobel prize | 409 | 4 | 0.0097800 | 2052 | 5 | 0.0024366 |
Pheromone | 313 | 3 | 0.0095847 | 461 | 2 | 0.0043384 |
Prion | 473 | 3 | 0.0063425 | 1583 | 7 | 0.0044220 |
Punctuated equilibrium | 943 | 1 | 0.0010604 | 1265 | 0 | 0 |
Pythagoras' theorem * | 688 | 1 | 0.0014535 | 1899 | 1 | 0.00052659 |
Quark | 1112 | 5 | 0.0044964 | 2060 | 0 | 0 |
Royal Greenwich Observatory | 235 | 3 | 0.012766 | 532 | 5 | 0.0093985 |
Royal Society | 416 | 6 | 0.014423 | 869 | 2 | 0.0023015 |
Synchrotron | 770 | 2 | 0.0025974 | 1590 | 2 | 0.0012579 |
Thyroid | 583 | 4 | 0.0068611 | 1459 | 7 | 0.0047978 |
Vesalius, Andreas | 930 | 2 | 0.0021505 | 1174 | 4 | 0.0034072 |
West Nile Virus | 245 | 1 | 0.0040816 | 1320 | 5 | 0.0037879 |
Wolfram, Stephen | 475 | 2 | 0.0042105 | 559 | 2 | 0.0035778 |
Woodward, Robert Burns | 873 | 0 | 0 | 2320 | 3 | 0.0012931 |
Total | 22733 | 123 | 45254 | 162 | ||
Mean | 541.26 | 2.9286 | 0.0054106 | 1077.5 | 3.8571 | 0.0035798 |
* - Articles marked as good articles
From the editors (see supplementary Nature report above):
So, the criterion was not "vastly different" in length, which would allow e.g. a factor of two difference, and maybe even a factor of 5. —Steven G. Johnson 17:35, 22 December 2005 (UTC)
None of the articles reviewed have featured article status, and none have undergone our internal peer review process. Two articles, Ethanol and Pythagorean theorem, have good article status with the latter being a former featured article (see here, though there is virtually no discussion there). violet/riga (t) 19:48, 15 December 2005 (UTC)
Note that Aldol reaction is now tended by a few PhD Chemistry candidates and is FA.-- SupperBird 22:19, 3 October 2007 (UTC)
Putting a "This article has been identified as possibly containing errors" tag on the front page of each and every articles identified by Nature seemed to some to be a bit of an overreaction. Others were quite comfortable with the practice.
In the long run, it seems that the project survived the tagging of all articles in which Nature's experts found errors. I would suggest that the tagging was even constructive and usefu. I note that we tag tens of thousands of articles as part of the Category:Wikipedia backlog by rank amateurs and teen-agers with éclat. Perhaps our amateurs are simply uncomfortable being reviewed by experts — just as Jimmy Wales finds working with domain experts to be "intimidating".
References:
While some progress was made in the days immediately after the report came out, the effort stalled as the Christmas 2005 holiday approached. When one examines the progress made before Christmas, one can see that much of the work was "low hanging fruit" in the form of easy corrections to superficial criticisms or where specific corrections were offered by the expert reviewers.
To be clear: it took several weeks before even 50% of the listed corrections were made. Unfortunately, some Wikipedia commentators like David Weinberger were under the false impression that "almost all" of the corrections were made within the first 24 hours of the effort. See Web of Ideas: The Authority of Wikipedia Around minute 70 of audio track. March 17, 2006. This further propagates the misconception prompted by the fact that Wikipedia has millions of accounts registered. More realistically, Wikipedia has only a few thousand active editors (or ten thousand if you use a generous definition of "active editor"), with varying levels of knowledge and maturity and many of whom contribute little to mainspace articles outside of reverting obvious vandalism. Refer to Wikipedia:List of Wikipedians by featured article nominations which lists appromixate 1000 users for a realistic list of contributors who have ever contributed substantial amounts of original quality content (with perhaps a factor of 5X for substantial sub unlisted collaborators). There are about 1000 administrators, and less than 100 other users with more elevated access. Also, in the new master plan, there will be only about 2000 "trusted" users, but all users will be rated with new "reliability" software to be implemented. Of course, one's personal relationship with Jimmy Wales or one's popularity within the community will still be able to override this number.
If one examines the status at the start of 2006, one sees the pattern: easy fixes or articles with a small number of errors were addressed at the start of the effort. Many of the easy fixes were biographical or narrative in nature and of a "coffee table book" level of understanding typical of an English major. Note that only the Aldol reaction article was a Featured article at the time of the review and that this article is fortunate to have three PhD chemistry candidates as caretakers (having such expertise involved on an extended basis is a rare situation at Wikipedia). The remaining articles required a conceptual understanding of chemistry, math, physics and engineering at around the college undergraduate level. The criticisms of articles such as Archimedes Principle, Dmitri Mendeleev, Colloid, Epitaxy, Field effect transistor, Kinetic isotope effect, Prion, Thyroid were not just syntactical corrections: they required an understanding at the semantic level on these scientifically non-controversial and well-understood topics for which there are, in an absolute and objective sense, right and wrong answers that are not intuitively obvious but require the ability to master technical concepts and facts.
For the academically honest, these later corrections were not subject to public opinion or Wikipedia's style of consensus building akin to voting since most people (and probably most Wikipedians, including the Board of Trustees and past and present members of the Arbitration committee, other WikiMedia foundation advisers and personnel and the employees of Wikia) do not and never will understand these concepts correctly and with depth. One user in particular, Pinktulip, did demonstrate such intellectual mastery and made a steady single-handed effort during January 2006 to ensure that another month did not pass before all the corrections were accomplished. Pinktulip also then informed Jimmy Wales of the completion of the overall effort on 25 January 2006.
Note that Wikipedia:Expert retention is an ongoing problem at Wikipedia.-- Simongar 20:14, 26 September 2007 (UTC)
Britannica has replied to Nature's article and claim the nature article was itself innacurate in a number of ways. It would probably be good for Wikipedians to check the sources Britannica gives and note needed changes in our articles.
Nature has now issued comments on Britannica's rebuttal, and stands by its original article. [2] (archive).
Jim Giles made a presentation at Wikimania on 2006-08-04.