A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.
This article [1] discusses the links between paid efforts and voluntary efforts in the development of Wikipedia, focusing on the question of paid editing. It stresses the fact that Wikipedia is a mixed economy that results partly from paid labor (the technostructure and the people in charge of maintaining it, and those who defend the project in court, i.e. the paid employees of the WF).
The core of the article discusses, based on the debate about Wiki-PR (a company which was paid by firms to "edit" their EN-Wikipedia pages), and the impact it had on Wikipedia policy. It sheds light on the discussion between the Foundation, which expressed a more strict interpretation of the rules, and the contributors, especially from non-English Wikipedias, that took a more "pragmatic" approach. Paid editors provided help to the smaller projects in terms of creation of knowledge. The analysis, which views Wikipedia as a sort of communist organization, is less convincing, as is the fact that the authors did not compare this debate with what happens in FLOSS (free-libre open-source software) or in the non-digital world (the Foundation, or the local community groups), which are other example of the co-existence of voluntary and paid work.
This masters thesis [2] focuses on the Swedish Wikipedia and its gender gap. It quantifies data and provides information about why Swedish women are not contributing to the project. The author collected data through a questionnaire advertised in December 2014 on the Swedish Wikipedia through a project-wide banner (promotion that an average researcher can only dream about when it comes to English Wikipedia). The paper estimates the Swedish Wikipedia gender gap in the form of the percentage of female editors at between 13% to 19%, based on the self-reported data from Wikipedia account profiles and answers to the questionnaire. More interesting is the analysis of the activity of the accounts: the self-declared male accounts are several times more active then the female accounts, with the authors estimating that only about 5% of the site's content is written by women. Contrary to some prior research (most of which focused on the English Wikipedia), the Swedish Wikipedia's editors and readers do not perceive Wikipedia as a place where sexist comments are significant, though about a third agree that general conflicts between editors do take place. Nonetheless, women are less likely than men to think (1) that Wikipedia is welcoming to beginners; (2) that everyone gets treated equally, regardless of gender; (3) that editing means taking on conflicts. Women are more likely than men to acknowledge the existence of sexist comments. In the author's own words, "women have more concerns about the community being sexist and not welcoming, and do not expect conflict as part of editing to the same degree as men", though the author also notes that statistical tests suggest that "the differences in opinion between gender groups do not differ [sic] greatly".
The author concludes that there is no evidence that the Swedish Wikipedia's readers have any preconceived negative notions about the Wikipedia community (such as "it is sexist") that should inhibit potential women contributors from editing and thus contribute to the gender gap. He states: "Significant differences in perceived competence were found. Women report 'I’m not competent enough' as a strong contributing factor to them not editing more than twice as often as men." The author suggests that because women often perceive, whether correctly or not, that they have lower computer skills than men, and see Wikipedia as a website which requires above-average computer skills, this (rather than an unfriendly, sexist community) may be the most significant factor affecting their lack of contributions. (Cf. related coverage: " Mind the skills gap: the role of Internet know-how and gender in differentiated contributions to Wikipedia'", " Does advertising the gender gap help or hurt Wikipedia?")
Four researchers from Dartmouth College have taken the requirement of " verifiability", one of Wikipedia's core content policies, literally. Their preprint [3] examines 295,800 citations from the 5000 most viewed articles on the English Wikipedia (out of a larger set of 23 million citations extracted from a July 2014 dump). These comprised both inline citations (footnotes) and "free citations" (those not related to any particular part of the article). The authors conclude that
Unsurprisingly, the study did not examine whether the cited documents actually match the information in the articles. Rather, it concerns the question whether the citation enables the reader to carry out this verification. The authors argue that
They divide these difficulties into two categories: "technical verifiability" and "practical verifiability."
Technical verifiability is defined as "the extent to which a reference provides supporting information that permits automated technical validation of the existence of the referenced material, based on existing technical standards or conventions," concretely ISBNs, DOIs and Google Books IDs. The study found that:
Practical verifiability is defined as "the extent to which referenced material is accessible to someone encountering the reference." In particular, the authors point out that information supported by a paywalled journal article "is practically unverifiable to someone without the additional means to access the supporting journal article. Similarly, if an ISBN is present but refers to a book that only has one extant copy in a library thousands of miles away, then the information it supports is practically unverifiable to someone without the additional means to access the supporting book." Apparently the authors found it difficult to translate these notions into criteria that would lend themselves to a large scale quantitative analysis, and settled for two rather narrowly defined but still interesting aspects:
The preprint also contains a literature overview about information quality on Wikipedia, which does the topic scant justice (e.g. of the only three mentioned systematic studies of article accuracy, one is the well-known but over a decade old Nature study, another is a 2014 article whose methodology and conclusions have been described as very questionable, see also below).
With some caveats, e.g. that the quality of the 5000 most-viewed English Wikipedia articles might differ from the quality of the average article, the authors conclude that "from the perspective of overall quality of references in Wikipedia, these findings might seem encouraging", but are concerned that many citations are not practically verifiable.
This short (two-page) paper [4] presents "preliminary results that characterize the research done on and using Wikipedia since 2002". It is based on a dataset of 3582 results of a Scopus search in November 2013 (for the term "Wikipedia" in title, abstract and keywords), largely relying on the abstracts of these publications. 641 of them were discarded as unrelated. Of the remaining 2968, the relevance for Wikipedia was judged as "major" for 2301 and as "minor" for 667.
Examining a dichotomy that is familiar to the editors of this newsletter too (which, for example, usually does not cover papers that merely rely on Wikipedia as a text corpus, even though these are numerous in fields such as computer linguistics), the authors write:
defining the latter as employing "Wikipedia either as a source/resource for other research or used Wikipedia to test the feasibility and applicability of tools or methods developed for purposes not directly related to Wikipedia". Those papers only began appearing in 2005, but overtook the "about" category in 2009 and have remained in the majority since." (See also coverage of a presentation at Wikimania 2013 that likewise traced publication numbers over the years – based on Google Scholar instead of Scopus – and dated the first appearance of "Wikipedia as a corpus" research to 2005, too: " Keynote on applicable Wikipedia research")
The researchers classified publications by their methodology, into "social/theoretical" (including "analyses and visualizations of Wikipedia") and "technological" (in the "about" category, this classification was reserved to "tools developed for improving Wikipedia"), and found that:
The authors extended their search beyond Scopus to Web of Science and the ACM Digital Library for an examination of how the overall volume of published Wikipedia research has developed over time. The resulting chart indicates that the fast growth of earlier years leveled off, with even some decrease in 2013, the last year examined.
Three letters to the editor of the Journal of the American Osteopathic Association adds to criticism of an article [supp 1] by Hasty et al. that had appeared in the same journal earlier, and was widely covered in the media with headline phrases such as "90% of [Wikipedia's] medical entries are inaccurate".
Like editors from WikiProject Medicine at the time, the writers of the first letter [5] lament that the paper's authors "have not made their dataset public, so it is impossible to confirm the veracity of their conclusions"; however, "they did share with us a small subset of their dataset on major depressive disorder. We closely examined two statements from Wikipedia that the researchers identified as inaccurate." After outlining that the peer-reviewed literature on these two issues is "rife with debate", and pointing out that some of it supports rather than contradicts the information on Wikipedia, they state that "It seems problematic to conclude that statements made in Wikipedia are wrong based on peer-reviewed literature", also quoting the editors of Nature observing that "peer review per se provides only a minimal assurance of quality". (On another occasion, the lead author had revealed a third Wikipedia statement that according to the study contradicted the peer-reviewed literature and which he described as dangerously wrong; however, it was in agreement with the hypertension guidelines of the UK National Institute for Health and Care Excellence (NICE). [supp 2])
The letter writers highlight the fact that the study relied on "third-year residents with no specific expertise [to] correctly ascertain the accuracy of claims made on Wikipedia" in this way. In a response [6], Hasty et al. acknowledged that the peer-reviewed literature contained diverging viewpoints on the topic, but held that "if Wikipedia articles are considered review articles, then it would be expected that major controversial points would be discussed rather than presented from one perspective."
The second letter [7] criticizes that "Because Hasty et al did not identify a specified number of assertions for each condition and did not measure whether Wikipedia and peer-reviewed literature were correct or not, respectively, their use of the McNemar test to compare Wikipedia vs peer-reviewed medical literature was inappropriate." A third letter also criticized the usage of this statistical test, adding that "I believe that the study here was incorrectly analyzed and inappropriately published through the same peer-review process that Hasty et al are holding to such high esteem. " [8] In their response [6] Hasty et al. defended their method, while acknowledging that "for greater clarity" some tables should have been labeled differently.
With such severe criticism from several independent sources, it is hard not to see this 2014 paper by Hasty et al. as discredited. Unfortunately, it continues to be occasionally cited in the literature (as mentioned in the review of the "verifiability" paper above) and in the media.
A paper [9] in Scientific Reports examined how the public attention to a news topic relates to the pageviews of the Wikipedia article about that topic, and the creation dates of related articles. As proxy for the general attention to the topic, the authors use traffic to pages "neighboring" the main article about the topic itself (i.e. linking to and linked from it), including the time before it was created. From the ( CC BY licensed) paper:
This conference paper [10] states in its abstract an intent to broadly analyze and present all aspects of Wikipedia use in education. Unfortunately, it fails to do so. For the first four and half pages, the paper explains what Wikipedia is, with next to no discussion of the extensive literature on the use of Wikipedia in education or its perceptions in academia. There is a single paragraph of original research, based on the interview of three Swiss Wikipedians; there is little explanation of why those people where interviewed, nor are there any findings beyond description of their brief editing history. The paper ends with some general conclusions. Given the semi-formal style of the paper, this reviewer finds that it resembles an undergraduate student paper of some kind, and it unfortunately adds nothing substantial to the existing literature on Wikipedia, education and academia.
A list of other recent publications that could not be covered in time for this issue – contributions are always welcome for reviewing or summarizing newly published research.
{{
cite journal}}
: Cite journal requires |journal=
(
help)
{{
cite book}}
: |journal=
ignored (
help)
{{
cite journal}}
: Cite journal requires |journal=
(
help)
{{
cite journal}}
: CS1 maint: DOI inactive as of March 2024 (
link)
A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.
This article [1] discusses the links between paid efforts and voluntary efforts in the development of Wikipedia, focusing on the question of paid editing. It stresses the fact that Wikipedia is a mixed economy that results partly from paid labor (the technostructure and the people in charge of maintaining it, and those who defend the project in court, i.e. the paid employees of the WF).
The core of the article discusses, based on the debate about Wiki-PR (a company which was paid by firms to "edit" their EN-Wikipedia pages), and the impact it had on Wikipedia policy. It sheds light on the discussion between the Foundation, which expressed a more strict interpretation of the rules, and the contributors, especially from non-English Wikipedias, that took a more "pragmatic" approach. Paid editors provided help to the smaller projects in terms of creation of knowledge. The analysis, which views Wikipedia as a sort of communist organization, is less convincing, as is the fact that the authors did not compare this debate with what happens in FLOSS (free-libre open-source software) or in the non-digital world (the Foundation, or the local community groups), which are other example of the co-existence of voluntary and paid work.
This masters thesis [2] focuses on the Swedish Wikipedia and its gender gap. It quantifies data and provides information about why Swedish women are not contributing to the project. The author collected data through a questionnaire advertised in December 2014 on the Swedish Wikipedia through a project-wide banner (promotion that an average researcher can only dream about when it comes to English Wikipedia). The paper estimates the Swedish Wikipedia gender gap in the form of the percentage of female editors at between 13% to 19%, based on the self-reported data from Wikipedia account profiles and answers to the questionnaire. More interesting is the analysis of the activity of the accounts: the self-declared male accounts are several times more active then the female accounts, with the authors estimating that only about 5% of the site's content is written by women. Contrary to some prior research (most of which focused on the English Wikipedia), the Swedish Wikipedia's editors and readers do not perceive Wikipedia as a place where sexist comments are significant, though about a third agree that general conflicts between editors do take place. Nonetheless, women are less likely than men to think (1) that Wikipedia is welcoming to beginners; (2) that everyone gets treated equally, regardless of gender; (3) that editing means taking on conflicts. Women are more likely than men to acknowledge the existence of sexist comments. In the author's own words, "women have more concerns about the community being sexist and not welcoming, and do not expect conflict as part of editing to the same degree as men", though the author also notes that statistical tests suggest that "the differences in opinion between gender groups do not differ [sic] greatly".
The author concludes that there is no evidence that the Swedish Wikipedia's readers have any preconceived negative notions about the Wikipedia community (such as "it is sexist") that should inhibit potential women contributors from editing and thus contribute to the gender gap. He states: "Significant differences in perceived competence were found. Women report 'I’m not competent enough' as a strong contributing factor to them not editing more than twice as often as men." The author suggests that because women often perceive, whether correctly or not, that they have lower computer skills than men, and see Wikipedia as a website which requires above-average computer skills, this (rather than an unfriendly, sexist community) may be the most significant factor affecting their lack of contributions. (Cf. related coverage: " Mind the skills gap: the role of Internet know-how and gender in differentiated contributions to Wikipedia'", " Does advertising the gender gap help or hurt Wikipedia?")
Four researchers from Dartmouth College have taken the requirement of " verifiability", one of Wikipedia's core content policies, literally. Their preprint [3] examines 295,800 citations from the 5000 most viewed articles on the English Wikipedia (out of a larger set of 23 million citations extracted from a July 2014 dump). These comprised both inline citations (footnotes) and "free citations" (those not related to any particular part of the article). The authors conclude that
Unsurprisingly, the study did not examine whether the cited documents actually match the information in the articles. Rather, it concerns the question whether the citation enables the reader to carry out this verification. The authors argue that
They divide these difficulties into two categories: "technical verifiability" and "practical verifiability."
Technical verifiability is defined as "the extent to which a reference provides supporting information that permits automated technical validation of the existence of the referenced material, based on existing technical standards or conventions," concretely ISBNs, DOIs and Google Books IDs. The study found that:
Practical verifiability is defined as "the extent to which referenced material is accessible to someone encountering the reference." In particular, the authors point out that information supported by a paywalled journal article "is practically unverifiable to someone without the additional means to access the supporting journal article. Similarly, if an ISBN is present but refers to a book that only has one extant copy in a library thousands of miles away, then the information it supports is practically unverifiable to someone without the additional means to access the supporting book." Apparently the authors found it difficult to translate these notions into criteria that would lend themselves to a large scale quantitative analysis, and settled for two rather narrowly defined but still interesting aspects:
The preprint also contains a literature overview about information quality on Wikipedia, which does the topic scant justice (e.g. of the only three mentioned systematic studies of article accuracy, one is the well-known but over a decade old Nature study, another is a 2014 article whose methodology and conclusions have been described as very questionable, see also below).
With some caveats, e.g. that the quality of the 5000 most-viewed English Wikipedia articles might differ from the quality of the average article, the authors conclude that "from the perspective of overall quality of references in Wikipedia, these findings might seem encouraging", but are concerned that many citations are not practically verifiable.
This short (two-page) paper [4] presents "preliminary results that characterize the research done on and using Wikipedia since 2002". It is based on a dataset of 3582 results of a Scopus search in November 2013 (for the term "Wikipedia" in title, abstract and keywords), largely relying on the abstracts of these publications. 641 of them were discarded as unrelated. Of the remaining 2968, the relevance for Wikipedia was judged as "major" for 2301 and as "minor" for 667.
Examining a dichotomy that is familiar to the editors of this newsletter too (which, for example, usually does not cover papers that merely rely on Wikipedia as a text corpus, even though these are numerous in fields such as computer linguistics), the authors write:
defining the latter as employing "Wikipedia either as a source/resource for other research or used Wikipedia to test the feasibility and applicability of tools or methods developed for purposes not directly related to Wikipedia". Those papers only began appearing in 2005, but overtook the "about" category in 2009 and have remained in the majority since." (See also coverage of a presentation at Wikimania 2013 that likewise traced publication numbers over the years – based on Google Scholar instead of Scopus – and dated the first appearance of "Wikipedia as a corpus" research to 2005, too: " Keynote on applicable Wikipedia research")
The researchers classified publications by their methodology, into "social/theoretical" (including "analyses and visualizations of Wikipedia") and "technological" (in the "about" category, this classification was reserved to "tools developed for improving Wikipedia"), and found that:
The authors extended their search beyond Scopus to Web of Science and the ACM Digital Library for an examination of how the overall volume of published Wikipedia research has developed over time. The resulting chart indicates that the fast growth of earlier years leveled off, with even some decrease in 2013, the last year examined.
Three letters to the editor of the Journal of the American Osteopathic Association adds to criticism of an article [supp 1] by Hasty et al. that had appeared in the same journal earlier, and was widely covered in the media with headline phrases such as "90% of [Wikipedia's] medical entries are inaccurate".
Like editors from WikiProject Medicine at the time, the writers of the first letter [5] lament that the paper's authors "have not made their dataset public, so it is impossible to confirm the veracity of their conclusions"; however, "they did share with us a small subset of their dataset on major depressive disorder. We closely examined two statements from Wikipedia that the researchers identified as inaccurate." After outlining that the peer-reviewed literature on these two issues is "rife with debate", and pointing out that some of it supports rather than contradicts the information on Wikipedia, they state that "It seems problematic to conclude that statements made in Wikipedia are wrong based on peer-reviewed literature", also quoting the editors of Nature observing that "peer review per se provides only a minimal assurance of quality". (On another occasion, the lead author had revealed a third Wikipedia statement that according to the study contradicted the peer-reviewed literature and which he described as dangerously wrong; however, it was in agreement with the hypertension guidelines of the UK National Institute for Health and Care Excellence (NICE). [supp 2])
The letter writers highlight the fact that the study relied on "third-year residents with no specific expertise [to] correctly ascertain the accuracy of claims made on Wikipedia" in this way. In a response [6], Hasty et al. acknowledged that the peer-reviewed literature contained diverging viewpoints on the topic, but held that "if Wikipedia articles are considered review articles, then it would be expected that major controversial points would be discussed rather than presented from one perspective."
The second letter [7] criticizes that "Because Hasty et al did not identify a specified number of assertions for each condition and did not measure whether Wikipedia and peer-reviewed literature were correct or not, respectively, their use of the McNemar test to compare Wikipedia vs peer-reviewed medical literature was inappropriate." A third letter also criticized the usage of this statistical test, adding that "I believe that the study here was incorrectly analyzed and inappropriately published through the same peer-review process that Hasty et al are holding to such high esteem. " [8] In their response [6] Hasty et al. defended their method, while acknowledging that "for greater clarity" some tables should have been labeled differently.
With such severe criticism from several independent sources, it is hard not to see this 2014 paper by Hasty et al. as discredited. Unfortunately, it continues to be occasionally cited in the literature (as mentioned in the review of the "verifiability" paper above) and in the media.
A paper [9] in Scientific Reports examined how the public attention to a news topic relates to the pageviews of the Wikipedia article about that topic, and the creation dates of related articles. As proxy for the general attention to the topic, the authors use traffic to pages "neighboring" the main article about the topic itself (i.e. linking to and linked from it), including the time before it was created. From the ( CC BY licensed) paper:
This conference paper [10] states in its abstract an intent to broadly analyze and present all aspects of Wikipedia use in education. Unfortunately, it fails to do so. For the first four and half pages, the paper explains what Wikipedia is, with next to no discussion of the extensive literature on the use of Wikipedia in education or its perceptions in academia. There is a single paragraph of original research, based on the interview of three Swiss Wikipedians; there is little explanation of why those people where interviewed, nor are there any findings beyond description of their brief editing history. The paper ends with some general conclusions. Given the semi-formal style of the paper, this reviewer finds that it resembles an undergraduate student paper of some kind, and it unfortunately adds nothing substantial to the existing literature on Wikipedia, education and academia.
A list of other recent publications that could not be covered in time for this issue – contributions are always welcome for reviewing or summarizing newly published research.
{{
cite journal}}
: Cite journal requires |journal=
(
help)
{{
cite book}}
: |journal=
ignored (
help)
{{
cite journal}}
: Cite journal requires |journal=
(
help)
{{
cite journal}}
: CS1 maint: DOI inactive as of March 2024 (
link)
Discuss this story
Test of 300k citations: how verifiable is "verifiable" in practice?
I have checked thousands of ISBNs, and in my experience the fraction with invalid check digits is a lot less than 8%. The paper used the WP dump of 7 July 2014, and refers to the article on Glycerol as having an invalid ISBN. ("... , the greatest gain in article rank was a 3,318 spot jump by “Glycerol” from rank 3,891 to rank 573. This article’s only ISBN was invalid ...") The ISBN was added here, and seems to have been unchanged at 7/7/14, and now, as ISBN 3527306730. It is valid. Am I missing something? Mr Stephen ( talk) 22:19, 27 March 2016 (UTC) reply
PS: the study has just been featured in The Atlantic, where the authors also propose some sort of browser plugin that displays a rating of each citation's practical verifiability. Regards, Tbayer (WMF) ( talk) 19:44, 22 April 2016 (UTC) reply
Paid Labour
The attention economy of Wikipedia articles on news topics
Life Expectancy