Thanks for putting together this report. Regarding "Our suggestions..." -- whose suggestions? As this is in projectspace, it's odd to see a page like this and have it express the suggestions of a particular group of people in projectspace voice, if that makes sense. — Rhododendrites talk \\ 00:37, 14 March 2018 (UTC)
Under 2% of drafts being published is about as far from wiki as can be. Especially when you consider informal policies like the idea that everything in the draft namespace can be deleted if it doesn't eventually make it to the main space.
A more fundamental problem is that we have only a single binary option for keeping or deleting articles (with tiny variations of highlighting good or featured articles; though most readers won't notice at all). We would be much better served by a dynamic spectrum of [highlighted] to [hidden] articles, with deletion not even an option unless articles are harmful to others or illegal. Authors should be able to always see their own articles, even if they've been hidden from view for most readers, whenever those authors return to work on the articles again: be it days or years later.
A reversible "hiding" action, by virtue of being reversible and not so painful/confusing to the authors working on an article, would also require less consensus and less decision-making overhead.
Enabling a "visibility" parameter that could be varied by editor input is the sort of fundamental improvements to MediaWiki that I'd love to see. Experimenting with shifting around steps in the current heavy workflow is extremely interesting, but may not have the same impact over time.
Finally, if results were evident within 2 months, is 6 months the right length of time for such studies or can we do more of them more quickly? – SJ + 01:52, 14 March 2018 (UTC)
Authors should be able to always see their own articles, even if they've been hidden from view for most readers, whenever those authors return to work on the articles again: be it days or years laterwill simply never happen, and that is from the WMF. Access to delete pages requires RfA or an RfA equivalent process. TonyBallioni ( talk) 18:05, 14 March 2018 (UTC)
view-deleted
permission, which would be the exact same thing as you are proposing in practice) the WMF vetoed it because they
feared action from the United States Congress.Flagged revisions/pending changes are completely different because they are 100% viewable by the public by simply going to the revision history. They are not hidden. They are just not rendered on the front facing article. All anyone has to do is hit "view history", and the changes are all there to see. I just tested this on
these revisions that are currently pending while logged out, and I saw them just fine (anyone can test this themselves by simply going to
Special:PendingChanges, picking a diff, logging out, and viewing it). You are proposing the one line in the sand that the WMF has drawn re: unbundling permissions. There is no difference between "content not suitable for the public but only certain logged in users can see" and "deleted content". Pending changes is not near equivalent.
TonyBallioni (
talk) 02:32, 17 March 2018 (UTC)If anyone cares to take a snapshot look at the current numbers, the list of submissions accepted for publication within the last 36 hours is here; the similar list for declines is here. NewYorkActuary ( talk) 21:14, 14 March 2018 (UTC)
There are more than 80,000 accepted (and tagged) AfC submissions (see Category:Accepted AfC submissions), so having only 520 accepted in the 3+ years until 2017 seems wildly improbable. The dataset starts at 1 July 2014: for that date alone, we have Category:AfC submissions by date/01 July 2014 32 accepted submissions. However, many of those were not in the draftspace, but in Wikipedia:Articles for creation... Even so, we have for this single day 1 2 3 4 5 (since merged) 6 7 8 9 10 11 12 13 14 15 16 17 18 articles moved from Draft space to mainspace. Which means, assuming that this is an average day, that the number of 520 accepted articles would be reached after 1 month, not after 42 months. Even when looking at more recent months like Category:AfC submissions by date/September 2016, we get a few hundred accepted submissions from the draft space (AfC space wa no longer in use then). I think it would be best if this study was rechecked thoroughly, as the numbers seem way off. Fram ( talk) 08:04, 15 March 2018 (UTC)
Overall, I think it's clear that ACTRIAL was a success, though the continued backlog at AFC is problematic. I'd feel more comfortable encouraging new users to create pages through AFC if the backlog was around 1 week. I do suspect the percentage of accepted drafts is actually closer to 15% than 1.5%, and for non-promotional editors I assume it would be even higher. As far as
Sj's comment regarding a dynamic spectrum of [highlighted] to [hidden] articles
: I'd love to see (for example) the ability to add content about any book on its
Special:BookSources page without the requirement that the book be "notable". But that's not really relevant here. Articles on non-notable private persons, articles promoting non-notable people, and articles promoting new non-notable corporations/organizations should still be removed from the project entirely, and that's the vast majority of declined drafts.
power~enwiki (
π,
ν) 20:34, 15 March 2018 (UTC)
It will be interesting to see what Nettrom and Kaldari find on actual AfC acceptance rates; as others have pointed out 1.2% seems unlikely. It looks like number of AfC submissions per day is a low 3-digit number and number of acceptances in a low 2-digit number. That would put the acceptance rate somewhere roughly around 10%. But, many of our AfC submissions are actually resubmissions so that's going to make the acceptance rate look lower than it actually is.
There is a problem in AfC with reviewers reluctant to approve underdeveloped submissions on potentially notable topics. Fundamental acceptance criteria is that accepted drafts should not be WP:LIKELY to be deleted. Since the WP:LIKELY threshold is just 50%, if reviewers were to take this to heart, we'd see a healthy number of deletions of accepted drafts. This is another good topic for additional research, but I beleive few accepted AfC submissions get deleted. A low deletion rate is to be both celebrated and cursed, in my opinion. In any case, if we were able to get AfC reviewers to let more marginal drafts through, based on my reviewing experience, I think we would less than double AfC acceptance rate so no huge numbers breakthrough there. ~ Kvng ( talk) 14:46, 16 March 2018 (UTC)
{{sources exist}}
tag on it to prevent someone taking it to AfD). —
Insertcleverphrasehere (
or here) 19:03, 16 March 2018 (UTC)Thank you for all the comments on this thread! I'd like to try to explain what we calculated 1.2% of exactly, what we didn't measure, and try to put that number in context. I hope it's okay that I add a subsection for this and respond to some specific concerns inline below.
The report tries to make it clear how I calculated that 1.2%. Here's the quote: "the publication rate of pages created in the Draft namespace is incredibly low (about 1.2%)" The denominator is "pages created in the Draft namespace" (to the best of our ability, recreation of Wikipedia page histories is difficult and we use the Data Lake for data up until July 21, 2017; see also wikitech:Analytics/Data Lake/Edits#Limitations of the historical datasets). I gathered a dataset of 126,957 pages created in Draft between July 1, 2014 and December 1, 2017. I then had a script go through the edit history of all those pages to identify a move from Draft to Main, and found 1,550 pages had done so (and that's 1.2% of the total).
When I first encountered this, I thought there were errors in my dataset. I therefore did a sanity check by identifying how many pages were live in the Main namespace on enwiki as of February 1. The result was about 33% lower. Given that pages get deleted and/or moved back to Draft, getting a lower (but roughly in the same ballpark) number would make sense. I also interpreted that finding to mean that my result seemed reasonable.
We did not measure the AfC acceptance rate. As many have brought up, the number of accepted submissions at AfC is much higher. There are several things to note about the difference between these calculations. First, I used pages created between a defined timespan in our calculation, partly because I needed reliable deletion data, and partly because we used a consistent end date in our analyses. I do not know what timespan "accepted AfC submissions" covers. Second, there's a large number of accepted redirects (from
Wikipedia:Articles for creation/Redirects, I expect). I made a
SQL query that I ran a couple of days ago, finding that the number of non-redirecting accepted submissions was just shy of 50,000. Third, AfC submissions can originate from several places (e.g.
Legacypac mentioned Most userspace submissions that are not test edits get moved to Draft.
) We only studied pages created in the Draft namespace, partly because that allows us to mine edit histories of both live and deleted pages, and partly because we expected ACTRIAL to result in an increase in AfC submissions of that type of pages. It would've been great to also have looked at AfC submissions from the User namespace, as well as pages moved into Draft from the Main namespace.
This discussion has been a great learning experience for me, making it clear that I should be careful about statistics added to reports. They'll be scrutinized (which is good, btw), so I should make sure that I've done my due diligence.
Insertcleverphrasehere asked us to clarify our numbers early in the thread, at which point I supplied some slightly different numbers, without explaining them very well. That did not make this discussion any clearer, and I'm sorry about that. They also asked Is it possible to crunch the numbers for the past 6 months and see what kind of percentage we are talking about?
Using the same approach and looking at the first five months of ACTRIAL, the rate is 11.1% (ref
this gist for those who want some details). I chose to not use pages created after February 15 to let there be some time for reviews and moves.
I also wanted to add a bit of context to the usage of the 1.2% statistic in the report. It was not part of our main study, so it is not part of the main results either; it's used in the part of report where we make some recommendations to the WMF. The main argument we're making is that creating an article is a difficult task, and that the WMF should look at design improvements of that process. In order to back that argument up a bit, or to exemplify that point, I decided to use two tangential statistics from our work. Due to where in the report this was, I did not expect the scrutiny this got, as I instead thought readers would focus on the main argument. In hindsight, I should've been more careful about throwing numbers around, and/or dug up some research papers to cite instead. Apologies for the wall of text response, but I hope it's been helpful in answering some of the questions about this number and how it came to be. I appreciate the feedback and scrutiny this has gotten, so please do ask follow-up questions as well. Cheers, Nettrom ( talk) 23:01, 16 March 2018 (UTC)
I am concerned that only the first two months of data were examined. There are six months of data and no analysis has been done to validate that the two months examined are representative of the entire trial period. Jbh Talk 14:12, 14 March 2018 (UTC)
Can someone please extend this graph to show the full 6 months of data. Two months does not appear to be enough here to accurately understand the implications for AfC. ~ Kvng ( talk) 14:26, 17 March 2018 (UTC)
@ Kaldari and Nettrom: I've downloaded the code and had a look at running it, but it appears to depend on database access. Does it have to be run on the WMF labs, or is it possible to run it remotely? Thanks. Mike Peel ( talk) 17:58, 24 March 2018 (UTC)
"One thing to keep in mind is that during this period, September to November, we typically see an increase in retention, likely due to the school cycle." If anyone goes forward with an analysis of the entire trial period, it would probably be useful to filter out Wiki Education student editors from the data. (I can provide a list of usernames.) We had about 6000 new users who were students doing Wiki Education-supported assignments during that period (and will have about that many again for early 2018), about 15-30% of whom I estimate would have met the 'survival' metric. Very few of them create drafts, as we guide them them to start in sandboxes and the ones who create new articles end up autoconfirmed by the time they want to move to mainspace; ACTRIAL ended up having very little impact on our program, but removing the student editors from the data might help isolate the effects the trial looked for.-- Sage (Wiki Ed) ( talk) 20:22, 14 March 2018 (UTC)
Here Wikipedia:Autoconfirmed_article_creation_trial/Post-trial_Research_Report#Less_low-quality_content_in_article_space WMF staff talks about increased deletions and says "This increase in deletions is not commensurate with the increase in draft creations, meaning that we see a lot of created drafts that appear to not warrant deletion."
This is not correct - it ignores timing. In Mainspace a delete worthy page will most likely get caught in NPP amd deleted within 90 days (even with the backlogs). In Draft page will not typically face deletion until at least 180 days as a G13 or much longer if it goes into the AfC review, waits for reviews 3 or more times, gets some bot or AWB or disambig edits etc. You can somewhat see this by looking at dates created on this list User:MusikBot/StaleDrafts/Report. In other words, the increased unencyclopedic content created in Draft instead of mainspace during ACTRIAL is mostly still sitting with no move yet to delete it.
It's not that the Drafts do "not warrent deletion" it is that they are not old enough to delete G13, no one is even looking at the non-AfC submitted Drafts yet, and even if we wanted to review those pages Gx CSD criteria is way narrower than Ax CSD criteria is. There is no PROD for Drafts. Further some editors insist that we can't look at Notability in MfD (see WP:NMFD) so sending lots of garbage there will not be accepted. Even editors who are happy to debate notability at MfD prefer to let G13 catch the junk later because there is so much junk, trying to debate it all is hopeless.
I 100% agree that a shift of new creations to Draft space has happened, but that is a wonderful thing. Draft space is no index and it has a clock where an Admin human will look at every Draft page in 6 months or so unless someone first deletes or promotes it. Legacypac ( talk) 22:46, 14 March 2018 (UTC)
I would like to express my personal thanks to Danny, Kaldari, and their team for the enormous effort on this research which I consider to be one of the most objective and useful in recent times. ACTRIAL has been an uphill challenge to me all these years to get it it carried out and I'm enthralled not only by the data it has produced, but also by the resounding success it has been. This kind of work helps close the gap between the Community and the Foundation. I hope that the recommendations will be carried through and I look forward to collaborating closely with the WMF on future developments. Again, my heartfelt thanks. Kudpung กุดผึ้ง ( talk) 19:37, 15 March 2018 (UTC)
I've started a page at WP:ACREQ as a centralized place to assemble the case for making Auto-Confirmed REQuired for new mainspace page creations. This should save us typing the same case over and over and be useful when we run the RfC. We can also collect endorsements from interested editors. Legacypac ( talk) 02:24, 17 March 2018 (UTC)
A note that though it may not be the biggest use case, I and others who run editathons and other events designed to get new editors started have struggled with how to make ACTRIAL work in a setting where new editors want to create articles and are being guided and trained to do so. (For one thing, the fact that new users could no longer create articles was a surprise to many of us, leading to lots of confusion midstream in major events.) Given editors at editathons are already generally having their work checked by others, I suppose the most efficient workflow if ACTRIAL stays in place would be to have them write in draft space and have a trainer or experienced editor move the articles to mainspace there and then; but it would be helpful to have that documented. That workflow also makes a fair number of assumptions about the capacity of the trainer in any given event. -- phoebe / ( talk to me) 04:01, 17 March 2018 (UTC)
This was discussed before ACTRIAL in detail. Since new editors are working in Draft anyway the only actual restriction is that non-auto confirmed users can't use their own account to move their creations to mainspace. Consensus at an RFC was against creating a userright for trainers that would allow then to AC new users directly. Some users felt strongly that editthon users should not get special treatment at all.
There are a number of work arounds including:
Remember new users don't know that before ACTRIAL they could have made the move themselves. Don't make it a negitive thing, present the restriction as a positive small control Wikipedia has to discourage vandals and spammers. Everyone will appreciate the need to discourage bad actors. Legacypac ( talk) 05:16, 17 March 2018 (UTC)
Concern about an increase in AfC backlog and associated "struggle" is mentioned in the lead, introduction and Shift in content creation and review and Suggestions to the Wikimedia Foundation sections. The WMF team has indicated they have limited understanding of AfC and are now working to gain better understanding. AfC has a long history of backlogs. Past history needs to be examined and we probably need to look at the full 6 months of ACTRIAL data to understand the actual impacts here. I beleive the hand wringing about AfC backlog is unsubstantiated and possibly even non-NPOV. I propose to remove "struggle" discussion from the lead, introduction and and Suggestions to the Wikimedia Foundation section and flag any unsubstantiated statements in Shift in content creation and review. ~ Kvng ( talk) 15:22, 17 March 2018 (UTC)
+1 to that. ACTRIAL is not the main reason for increased AfC backlog. Other reasons include:
With such a limited number of really active reviewers, the loss of three of us hurt the backlog. However, the true backlog is not two months. Most pages get processed quite quickly as covered here WP:ACREQ. It's the borderline and complex topics that pile up. We are working on a plan to accept pages that are notable but not perfect. Legacypac ( talk) 15:35, 17 March 2018 (UTC)
Also the entire AfC backlog is only 2400-2500 pages, much less than the much reduced NPR backlog. Legacypac ( talk) 15:41, 17 March 2018 (UTC)
Please tell the WMF that the best way to understand the AFC struggle is to write a new article, submit it, shepherd it through the process, and experience the whole thing for ones' self. There is really no substitute for that. 173.228.123.121 ( talk) 06:26, 24 March 2018 (UTC)
One of the hypotheses was that The rate of article growth will be reduced. This was found to be unsupported. Why is this not mentioned as a finding or key finding in the lead or introduction of the report?
I attempted to add this myself but my change was reverted by Nettrom because, "content changes can be discussed on the talk page, the overview corresponds to the three main areas of the study." Well, the report says, "Our study is organized into three themes corresponding to the main findings above." Perhaps I misunderstand the comment or context but this sounds circular. ~ Kvng ( talk) 20:11, 19 March 2018 (UTC)
Given the potential impact of this, a preliminary 2-month report is not enough. When will the 6-month report be available? Thanks. Mike Peel ( talk) 01:26, 24 March 2018 (UTC)
Thanks for putting together this report. Regarding "Our suggestions..." -- whose suggestions? As this is in projectspace, it's odd to see a page like this and have it express the suggestions of a particular group of people in projectspace voice, if that makes sense. — Rhododendrites talk \\ 00:37, 14 March 2018 (UTC)
Under 2% of drafts being published is about as far from wiki as can be. Especially when you consider informal policies like the idea that everything in the draft namespace can be deleted if it doesn't eventually make it to the main space.
A more fundamental problem is that we have only a single binary option for keeping or deleting articles (with tiny variations of highlighting good or featured articles; though most readers won't notice at all). We would be much better served by a dynamic spectrum of [highlighted] to [hidden] articles, with deletion not even an option unless articles are harmful to others or illegal. Authors should be able to always see their own articles, even if they've been hidden from view for most readers, whenever those authors return to work on the articles again: be it days or years later.
A reversible "hiding" action, by virtue of being reversible and not so painful/confusing to the authors working on an article, would also require less consensus and less decision-making overhead.
Enabling a "visibility" parameter that could be varied by editor input is the sort of fundamental improvements to MediaWiki that I'd love to see. Experimenting with shifting around steps in the current heavy workflow is extremely interesting, but may not have the same impact over time.
Finally, if results were evident within 2 months, is 6 months the right length of time for such studies or can we do more of them more quickly? – SJ + 01:52, 14 March 2018 (UTC)
Authors should be able to always see their own articles, even if they've been hidden from view for most readers, whenever those authors return to work on the articles again: be it days or years laterwill simply never happen, and that is from the WMF. Access to delete pages requires RfA or an RfA equivalent process. TonyBallioni ( talk) 18:05, 14 March 2018 (UTC)
view-deleted
permission, which would be the exact same thing as you are proposing in practice) the WMF vetoed it because they
feared action from the United States Congress.Flagged revisions/pending changes are completely different because they are 100% viewable by the public by simply going to the revision history. They are not hidden. They are just not rendered on the front facing article. All anyone has to do is hit "view history", and the changes are all there to see. I just tested this on
these revisions that are currently pending while logged out, and I saw them just fine (anyone can test this themselves by simply going to
Special:PendingChanges, picking a diff, logging out, and viewing it). You are proposing the one line in the sand that the WMF has drawn re: unbundling permissions. There is no difference between "content not suitable for the public but only certain logged in users can see" and "deleted content". Pending changes is not near equivalent.
TonyBallioni (
talk) 02:32, 17 March 2018 (UTC)If anyone cares to take a snapshot look at the current numbers, the list of submissions accepted for publication within the last 36 hours is here; the similar list for declines is here. NewYorkActuary ( talk) 21:14, 14 March 2018 (UTC)
There are more than 80,000 accepted (and tagged) AfC submissions (see Category:Accepted AfC submissions), so having only 520 accepted in the 3+ years until 2017 seems wildly improbable. The dataset starts at 1 July 2014: for that date alone, we have Category:AfC submissions by date/01 July 2014 32 accepted submissions. However, many of those were not in the draftspace, but in Wikipedia:Articles for creation... Even so, we have for this single day 1 2 3 4 5 (since merged) 6 7 8 9 10 11 12 13 14 15 16 17 18 articles moved from Draft space to mainspace. Which means, assuming that this is an average day, that the number of 520 accepted articles would be reached after 1 month, not after 42 months. Even when looking at more recent months like Category:AfC submissions by date/September 2016, we get a few hundred accepted submissions from the draft space (AfC space wa no longer in use then). I think it would be best if this study was rechecked thoroughly, as the numbers seem way off. Fram ( talk) 08:04, 15 March 2018 (UTC)
Overall, I think it's clear that ACTRIAL was a success, though the continued backlog at AFC is problematic. I'd feel more comfortable encouraging new users to create pages through AFC if the backlog was around 1 week. I do suspect the percentage of accepted drafts is actually closer to 15% than 1.5%, and for non-promotional editors I assume it would be even higher. As far as
Sj's comment regarding a dynamic spectrum of [highlighted] to [hidden] articles
: I'd love to see (for example) the ability to add content about any book on its
Special:BookSources page without the requirement that the book be "notable". But that's not really relevant here. Articles on non-notable private persons, articles promoting non-notable people, and articles promoting new non-notable corporations/organizations should still be removed from the project entirely, and that's the vast majority of declined drafts.
power~enwiki (
π,
ν) 20:34, 15 March 2018 (UTC)
It will be interesting to see what Nettrom and Kaldari find on actual AfC acceptance rates; as others have pointed out 1.2% seems unlikely. It looks like number of AfC submissions per day is a low 3-digit number and number of acceptances in a low 2-digit number. That would put the acceptance rate somewhere roughly around 10%. But, many of our AfC submissions are actually resubmissions so that's going to make the acceptance rate look lower than it actually is.
There is a problem in AfC with reviewers reluctant to approve underdeveloped submissions on potentially notable topics. Fundamental acceptance criteria is that accepted drafts should not be WP:LIKELY to be deleted. Since the WP:LIKELY threshold is just 50%, if reviewers were to take this to heart, we'd see a healthy number of deletions of accepted drafts. This is another good topic for additional research, but I beleive few accepted AfC submissions get deleted. A low deletion rate is to be both celebrated and cursed, in my opinion. In any case, if we were able to get AfC reviewers to let more marginal drafts through, based on my reviewing experience, I think we would less than double AfC acceptance rate so no huge numbers breakthrough there. ~ Kvng ( talk) 14:46, 16 March 2018 (UTC)
{{sources exist}}
tag on it to prevent someone taking it to AfD). —
Insertcleverphrasehere (
or here) 19:03, 16 March 2018 (UTC)Thank you for all the comments on this thread! I'd like to try to explain what we calculated 1.2% of exactly, what we didn't measure, and try to put that number in context. I hope it's okay that I add a subsection for this and respond to some specific concerns inline below.
The report tries to make it clear how I calculated that 1.2%. Here's the quote: "the publication rate of pages created in the Draft namespace is incredibly low (about 1.2%)" The denominator is "pages created in the Draft namespace" (to the best of our ability, recreation of Wikipedia page histories is difficult and we use the Data Lake for data up until July 21, 2017; see also wikitech:Analytics/Data Lake/Edits#Limitations of the historical datasets). I gathered a dataset of 126,957 pages created in Draft between July 1, 2014 and December 1, 2017. I then had a script go through the edit history of all those pages to identify a move from Draft to Main, and found 1,550 pages had done so (and that's 1.2% of the total).
When I first encountered this, I thought there were errors in my dataset. I therefore did a sanity check by identifying how many pages were live in the Main namespace on enwiki as of February 1. The result was about 33% lower. Given that pages get deleted and/or moved back to Draft, getting a lower (but roughly in the same ballpark) number would make sense. I also interpreted that finding to mean that my result seemed reasonable.
We did not measure the AfC acceptance rate. As many have brought up, the number of accepted submissions at AfC is much higher. There are several things to note about the difference between these calculations. First, I used pages created between a defined timespan in our calculation, partly because I needed reliable deletion data, and partly because we used a consistent end date in our analyses. I do not know what timespan "accepted AfC submissions" covers. Second, there's a large number of accepted redirects (from
Wikipedia:Articles for creation/Redirects, I expect). I made a
SQL query that I ran a couple of days ago, finding that the number of non-redirecting accepted submissions was just shy of 50,000. Third, AfC submissions can originate from several places (e.g.
Legacypac mentioned Most userspace submissions that are not test edits get moved to Draft.
) We only studied pages created in the Draft namespace, partly because that allows us to mine edit histories of both live and deleted pages, and partly because we expected ACTRIAL to result in an increase in AfC submissions of that type of pages. It would've been great to also have looked at AfC submissions from the User namespace, as well as pages moved into Draft from the Main namespace.
This discussion has been a great learning experience for me, making it clear that I should be careful about statistics added to reports. They'll be scrutinized (which is good, btw), so I should make sure that I've done my due diligence.
Insertcleverphrasehere asked us to clarify our numbers early in the thread, at which point I supplied some slightly different numbers, without explaining them very well. That did not make this discussion any clearer, and I'm sorry about that. They also asked Is it possible to crunch the numbers for the past 6 months and see what kind of percentage we are talking about?
Using the same approach and looking at the first five months of ACTRIAL, the rate is 11.1% (ref
this gist for those who want some details). I chose to not use pages created after February 15 to let there be some time for reviews and moves.
I also wanted to add a bit of context to the usage of the 1.2% statistic in the report. It was not part of our main study, so it is not part of the main results either; it's used in the part of report where we make some recommendations to the WMF. The main argument we're making is that creating an article is a difficult task, and that the WMF should look at design improvements of that process. In order to back that argument up a bit, or to exemplify that point, I decided to use two tangential statistics from our work. Due to where in the report this was, I did not expect the scrutiny this got, as I instead thought readers would focus on the main argument. In hindsight, I should've been more careful about throwing numbers around, and/or dug up some research papers to cite instead. Apologies for the wall of text response, but I hope it's been helpful in answering some of the questions about this number and how it came to be. I appreciate the feedback and scrutiny this has gotten, so please do ask follow-up questions as well. Cheers, Nettrom ( talk) 23:01, 16 March 2018 (UTC)
I am concerned that only the first two months of data were examined. There are six months of data and no analysis has been done to validate that the two months examined are representative of the entire trial period. Jbh Talk 14:12, 14 March 2018 (UTC)
Can someone please extend this graph to show the full 6 months of data. Two months does not appear to be enough here to accurately understand the implications for AfC. ~ Kvng ( talk) 14:26, 17 March 2018 (UTC)
@ Kaldari and Nettrom: I've downloaded the code and had a look at running it, but it appears to depend on database access. Does it have to be run on the WMF labs, or is it possible to run it remotely? Thanks. Mike Peel ( talk) 17:58, 24 March 2018 (UTC)
"One thing to keep in mind is that during this period, September to November, we typically see an increase in retention, likely due to the school cycle." If anyone goes forward with an analysis of the entire trial period, it would probably be useful to filter out Wiki Education student editors from the data. (I can provide a list of usernames.) We had about 6000 new users who were students doing Wiki Education-supported assignments during that period (and will have about that many again for early 2018), about 15-30% of whom I estimate would have met the 'survival' metric. Very few of them create drafts, as we guide them them to start in sandboxes and the ones who create new articles end up autoconfirmed by the time they want to move to mainspace; ACTRIAL ended up having very little impact on our program, but removing the student editors from the data might help isolate the effects the trial looked for.-- Sage (Wiki Ed) ( talk) 20:22, 14 March 2018 (UTC)
Here Wikipedia:Autoconfirmed_article_creation_trial/Post-trial_Research_Report#Less_low-quality_content_in_article_space WMF staff talks about increased deletions and says "This increase in deletions is not commensurate with the increase in draft creations, meaning that we see a lot of created drafts that appear to not warrant deletion."
This is not correct - it ignores timing. In Mainspace a delete worthy page will most likely get caught in NPP amd deleted within 90 days (even with the backlogs). In Draft page will not typically face deletion until at least 180 days as a G13 or much longer if it goes into the AfC review, waits for reviews 3 or more times, gets some bot or AWB or disambig edits etc. You can somewhat see this by looking at dates created on this list User:MusikBot/StaleDrafts/Report. In other words, the increased unencyclopedic content created in Draft instead of mainspace during ACTRIAL is mostly still sitting with no move yet to delete it.
It's not that the Drafts do "not warrent deletion" it is that they are not old enough to delete G13, no one is even looking at the non-AfC submitted Drafts yet, and even if we wanted to review those pages Gx CSD criteria is way narrower than Ax CSD criteria is. There is no PROD for Drafts. Further some editors insist that we can't look at Notability in MfD (see WP:NMFD) so sending lots of garbage there will not be accepted. Even editors who are happy to debate notability at MfD prefer to let G13 catch the junk later because there is so much junk, trying to debate it all is hopeless.
I 100% agree that a shift of new creations to Draft space has happened, but that is a wonderful thing. Draft space is no index and it has a clock where an Admin human will look at every Draft page in 6 months or so unless someone first deletes or promotes it. Legacypac ( talk) 22:46, 14 March 2018 (UTC)
I would like to express my personal thanks to Danny, Kaldari, and their team for the enormous effort on this research which I consider to be one of the most objective and useful in recent times. ACTRIAL has been an uphill challenge to me all these years to get it it carried out and I'm enthralled not only by the data it has produced, but also by the resounding success it has been. This kind of work helps close the gap between the Community and the Foundation. I hope that the recommendations will be carried through and I look forward to collaborating closely with the WMF on future developments. Again, my heartfelt thanks. Kudpung กุดผึ้ง ( talk) 19:37, 15 March 2018 (UTC)
I've started a page at WP:ACREQ as a centralized place to assemble the case for making Auto-Confirmed REQuired for new mainspace page creations. This should save us typing the same case over and over and be useful when we run the RfC. We can also collect endorsements from interested editors. Legacypac ( talk) 02:24, 17 March 2018 (UTC)
A note that though it may not be the biggest use case, I and others who run editathons and other events designed to get new editors started have struggled with how to make ACTRIAL work in a setting where new editors want to create articles and are being guided and trained to do so. (For one thing, the fact that new users could no longer create articles was a surprise to many of us, leading to lots of confusion midstream in major events.) Given editors at editathons are already generally having their work checked by others, I suppose the most efficient workflow if ACTRIAL stays in place would be to have them write in draft space and have a trainer or experienced editor move the articles to mainspace there and then; but it would be helpful to have that documented. That workflow also makes a fair number of assumptions about the capacity of the trainer in any given event. -- phoebe / ( talk to me) 04:01, 17 March 2018 (UTC)
This was discussed before ACTRIAL in detail. Since new editors are working in Draft anyway the only actual restriction is that non-auto confirmed users can't use their own account to move their creations to mainspace. Consensus at an RFC was against creating a userright for trainers that would allow then to AC new users directly. Some users felt strongly that editthon users should not get special treatment at all.
There are a number of work arounds including:
Remember new users don't know that before ACTRIAL they could have made the move themselves. Don't make it a negitive thing, present the restriction as a positive small control Wikipedia has to discourage vandals and spammers. Everyone will appreciate the need to discourage bad actors. Legacypac ( talk) 05:16, 17 March 2018 (UTC)
Concern about an increase in AfC backlog and associated "struggle" is mentioned in the lead, introduction and Shift in content creation and review and Suggestions to the Wikimedia Foundation sections. The WMF team has indicated they have limited understanding of AfC and are now working to gain better understanding. AfC has a long history of backlogs. Past history needs to be examined and we probably need to look at the full 6 months of ACTRIAL data to understand the actual impacts here. I beleive the hand wringing about AfC backlog is unsubstantiated and possibly even non-NPOV. I propose to remove "struggle" discussion from the lead, introduction and and Suggestions to the Wikimedia Foundation section and flag any unsubstantiated statements in Shift in content creation and review. ~ Kvng ( talk) 15:22, 17 March 2018 (UTC)
+1 to that. ACTRIAL is not the main reason for increased AfC backlog. Other reasons include:
With such a limited number of really active reviewers, the loss of three of us hurt the backlog. However, the true backlog is not two months. Most pages get processed quite quickly as covered here WP:ACREQ. It's the borderline and complex topics that pile up. We are working on a plan to accept pages that are notable but not perfect. Legacypac ( talk) 15:35, 17 March 2018 (UTC)
Also the entire AfC backlog is only 2400-2500 pages, much less than the much reduced NPR backlog. Legacypac ( talk) 15:41, 17 March 2018 (UTC)
Please tell the WMF that the best way to understand the AFC struggle is to write a new article, submit it, shepherd it through the process, and experience the whole thing for ones' self. There is really no substitute for that. 173.228.123.121 ( talk) 06:26, 24 March 2018 (UTC)
One of the hypotheses was that The rate of article growth will be reduced. This was found to be unsupported. Why is this not mentioned as a finding or key finding in the lead or introduction of the report?
I attempted to add this myself but my change was reverted by Nettrom because, "content changes can be discussed on the talk page, the overview corresponds to the three main areas of the study." Well, the report says, "Our study is organized into three themes corresponding to the main findings above." Perhaps I misunderstand the comment or context but this sounds circular. ~ Kvng ( talk) 20:11, 19 March 2018 (UTC)
Given the potential impact of this, a preliminary 2-month report is not enough. When will the 6-month report be available? Thanks. Mike Peel ( talk) 01:26, 24 March 2018 (UTC)