This page documents the development process for a completed project undertaken by the WMF Growth team from April to October 2018. |
The Wikimedia Foundation's Community Tech team and Growth teams are extending the New Pages Feed interface to allow both Articles for Creation (AfC) reviewers and New Page Patrol (NPP) reviewers to prioritize pages for review using quality and copyright violation scores. This work will take place during May and June 2018.
Articles for Creation (AfC) is a process in English Wikipedia by which experienced Wikipedians review draft pages to determine whether they can be added to the main article namespace. Because of the April 2018 policy change known as ACPERM, traffic to the AfC process is expected to increase. In discussion with the AfC community, it has been determined that AfC will be able to promote more high quality pages to the main article namespace more quickly through enhanced prioritization tools. Therefore, the Community Tech team is building an improvement to the existing New Pages Feed interface that will allow AfC reviewers to use predictive models to prioritize drafts by their likelihood of copyright violation and their predicted quality. This improvement will also be available to reviewers in the New Page Patrol (NPP) process, who review pages created directly in the main article space, and are the original users of the New Pages Feed.
For the full background on the motivation of this project and corresponding discussion by members of the AfC and NPP communities, please see the original project page.
Goal: we want AfC reviewers to be able to get high quality articles into the main namespace as quickly as possible.
Given the expected increase of submissions to AfC, we plan to help by increasing the efficiency with which reviewers can review drafts while maintaining their quality standards. Specifically, there are three metrics that we intend to maintain or improve:
Community discussion on these metrics is here.
Technical discussion on the appropriate way to create these numbers is in Phabricator here.
Through about a month of conversation on potential improvements, community members and WMF staff agreed that a way to help AfC reviewers pursue the goals and metrics above would be through an interface to allow reviewers to prioritize drafts based on quality scores, and would allow them to quickly assess copyright violations (copyvio). The New Pages Feed, which was built for the separate NPP process, is an existing interface that can facilitate this. The planned changes are as follows:
Below is a set of user stories that describe in more detail what we plan to implement with these changes. User stories are a software engineering practice that help clearly define what software will need to accomplish from the perspective of the user. The user stories below are based on discussion with AfC and NPP reviewers, and are broken down into four categories. The Community Tech team hopes to address all these stories, but may need to omit stories that prove to be too technically challenging for the limited scope of this project, or that the community judges are not actually important.
In the future, this section will link to visual mockups built on these user stories, so that the community can comment on them before the software is built.
As a reviewer, I need to sort and filter the New Pages Feed by prioritization information:
As a reviewer, I need to see prioritization information next to each page’s entry in the New Pages Feed:
As a reviewer:
As a reviewer:
The Community Tech team started planning this project on May 1, 2018, with involvement from WMF designers and community liaisons. The next steps are as follows, and will contain a more specific timeline in the future:
1. Community Tech software engineers will investigate the feasibility and challenges of the user stories listed above. Specifically, those investigations are tracked in these three Phabricator tasks:
2. The team will develop some initial visual mockups of how the New Pages Feed could change with this work.
3. We will post those mockups here and on the talk pages for the AfC and NPP projects for feedback and input from those communities.
4. Engineering work will begin, and we will strive to roll out incremental improvements for testing as soon as they are ready.
On May 8, five of us on the WMF team got together to think through some of the design details around this feature improvement for the New Pages Feed. We talked through the user stories listed above on this page with two objectives: (1) surface as many unanswered questions as we could, and (2) make sure our designer had enough clarity to begin to mock up some ideas. It's not likely that we'll be able to design or engineer according to all the thoughts below, but I'm posting our full notes so that everyone can follow along with our process. Please comment and chime in on the talk page, especially regarding the "Open questions" list at the bottom. We hope to show some mockups for feedback in the next couple weeks.
Please remember that these ideas have not been thoroughly vetted for engineering feasibility yet. We won't be able to do everything in this list this time, and we may or may not be able to do all of the most interesting ideas.
Over the last week, the WMF team has been following the execution plan above by working in two tracks:
Below, I have notes and questions for community members on both these tracks, as well as our current next steps.
Working from the notes from the design session above, and from the resulting conversation on the talk page, we made some low-fidelity wireframes of how we think the expansion of the New Pages Feed could look. I hope everyone can take a look at these images (and the corresponding annotations), and think through whether this will work for your NPP or AfC reviewing workflow – and if not, how the design could be better (they're just wireframes, after all, so nothing is set in stone).
Some overall notes:
Annotations to the wireframes:
The full detail of the technical investigations can be found in the Phabricator tasks, but a summary is included here.
The biggest challenge is around the best way to incorporate copyvio scores. The engineers here are still discussing and thinking through options, so we unfortunately do not have a complete plan right now. We are eager to hear anything that community members can add to the mix to help us define a path forward. There are three main issues:
Given these three issues, we currently think that the quickest path to adding copyvio to New Pages Feed would be through using Turnitin, which is the more straightforward service to implement. In that vein, there are several open questions for community members that we need some guidance on:
Despite the challenges with copyvio described above, we are moving forward on several fronts. Our expectation is that we'll deliver the changes to the New Pages Feed in segments, as opposed to all at once. I'm going to be making Phabricator tasks for several of the following items.
Thank you, and as always, I'm looking forward to the discussion on the talk page.
For those of you who are interested in following along in Phabricator, I've made tasks for the parts of this project that have reached a level of clarity such that the WMF team can start their engineering work. Thanks to all members of the reviewing community who helped us get to this level of clarity. To that end, we're now excited to say that engineering work is happening over the next four weeks on the following specific work items (all listed under the same epic):
The copyvio parts of this project do not yet have Phabricator tasks created. Since that is such an important part of the work, and the largest engineering challenge, the conversation is still ongoing around the right way to implement it. I'm hoping we'll be able to come to some conclusions and be able to create those Phabricator tasks by next week.
These tasks still have a fair number of open questions in them that I'll be working to settle down this week, and about which I will likely have questions on the talk page. And the tasks may split, merge, or get increased details over the coming weeks. That said, if anyone is interested in reading over them, I know the team here welcomes questions, clarifications, and especially to let us know if we've misunderstood something important about the AfC and NPP workflows.
Importantly, we will be testing and asking for feedback on these changes to the New Pages Feed before they are rolled out to users. We definitely do not want to change the feed in ways that are surprising or unproductive. It's my intention to give as many visual progress reports as possible as this work unfolds, and to collaborate with the NPP and AfC communities to roll out changes deliberately.
In that vein, I'm hoping that it will be possible to roll out parts of the project to reviewers as they are complete, instead of waiting for all the parts to be done. That way, we'll be able to get real feedback early, and hopefully deliver goodness on this project as soon as possible.
Over the past week, the Community Tech team has been working on adding draft pages to the New Pages Feed, and making it possible to filter those drafts based on where they are in the AfC process. That work is progressing well so far, without any surprises. Two additional work items that were detailed during the week are the need to explicitly test that drafts are listed in the feed correctly before making changes to production, and making sure that the whole draft backlog is added to the feed.
With respect to the second part of the project -- adding ORES scores to the feed -- Community Tech engineers have talked with engineers on WMF's Scoring Platform team, who are currently working on adding the relevant ORES models to the Mediawiki database to make them easily queryable by features like the New Pages Feed. We're optimistic that that team's work will mean Community Tech will not have to implement something additional to use ORES scores.
And with respect to the third part of the project -- copyvio -- we are taking some time as the engineers work on the beginning parts of this project to think about the right way to implement this capability in the New Pages Feed, taking into account all the thoughts on the Talk page.
The Community Tech team has made progress over the past week on adding draft pages to the New Pages Feed. We have been working on the same Phabricator tasks as mentioned in last week's update ( phab:T195545 and phab:T195924), and we added one additional task to the to-do list: adding a feature flag. This will allow us to wait until a cohesive set of changes are developed for the New Pages Feed before exposing any of them to reviewers -- as opposed to the feed changing in little, incomplete ways over time.
At this point, we do have a glimpse of how the user interface changes are shaping up -- though it is a still a work in progress. The image below is from a developer's local environment, meaning these changes are not yet available on the actual wikis for anyone to see.
The screenshot shows a couple of important points of progress:
It also shows a few things that are incomplete:
Please take a glance at the screenshot and add any of your reactions to this project's talk page. Sometimes seeing something take shape can inspire thoughts that wouldn't have occurred before, and we definitely want to get a sense for whether this feels like it's on the right track.
Over the past week, the Community Tech team has mostly been working on phab:T195924, which is about making it possible to filter drafts by their state ("Unsubmitted", "Awaiting review", "Under review", "Declined", "All"). The team has stood up the software in a testing environment as they develop it, and we have been noting issues in Phabricator as we test out the changing capabilities.
The next major item the team will be working on is making it possible to sort drafts by their submitted and declined dates.
There are four main topics in this update:
We wanted to let everyone know about a team assignment change that will hopefully help this project be completed sooner. So far, the engineering on this project has been done by the Community Tech team. That team has been working on the first major part of the project, which is to add drafts to the New Pages Feed, and make the feed sortable by state and filterable by date. Starting next week, a different Wikimedia Foundation team, the Collaboration team, will be completing the second and third parts of the project, which are adding ORES scores and adding copyvio detection. The Collaboration team was the team that added ORES scores to the Recent Changes feed, giving them good experience using ORES scores and working with the various feeds in Mediawiki. I ( MMiller (WMF)) will continue to be the product manager for this work. Community Tech has been doing great work so far, and we're being careful to transfer their knowledge to Collaboration so that the project continues smoothly.
The last item that the Community Tech team is working on with this project, before the Collaboration team begins their work, is phab:T195547, which will make it possible to sort AfC drafts by their most recent date of submission or most recent date they were declined, in addition to the original date they were created. That is the main work item currently underway this week and next.
Now that the initial work to add drafts to the New Pages Feed is largely complete, we are setting up our ability to rigorously test the new functionality. We are working to surface the new features in the Test Wiki next week. Once the new features are there, we will post another update asking reviewers to try them out and reply with thoughts and bugs. At that point, reviewers who are testing might determine that the simple addition of drafts to New Pages Feed, even without ORES and copyvio, are enough of an improvement that they could be put into production on English Wikipedia.
As mentioned above, next week the Collaboration team will begin the work to integrate ORES models into the New Pages Feed. Now that the ORES work will be beginning, I wanted to resurface a previous conversation and a decision we've made about how to proceed. In the project update from 2018-05-17, we posted wireframes for what we called "Concept A" and "Concept B".
In the discussion on the talk page, some reviewers preferred Concept A because it gives reviewers more control, and some preferred Concept B because it provides clearer recommendations and less opportunity for mis-using the model scores. We have decided to implement Concept A because we believe it will be a good stepping stone to help reviewers figure out whether Concept B is better, and if so, what rules should be used for the structured options of Concept B. In other words, by implementing Concept A, reviewers will have the opportunity to experiment with the ORES scores, decide whether Concept B is preferred, and then develop the rules for it. From the engineering perspective, having implemented Concept A, it will be relatively easy to subsequently implement Concept B.
Please do post on the talk page with reactions, questions, or any other thoughts.
Now that there are two teams working on this project, Community Tech and Growth, there are a handful of interesting updates:
If any AfC or NPP reviewers will be at Wikimania next week, please let me know! I'm hoping to meet some members of this community in person.
We're now formally testing the components of this project in our testing environments. As I've said in previous updates, as soon as we're technically able to do so, I'll ask community members to take some time to test things out as well.
Over the past week, the Community Tech team has continued the work to make drafts sortable by their submission and declined dates. And the Growth team has been writing the code to incorporate ORES scores into the New Pages Feed, and most of that code is now under review before it makes its way to the testing environment.
The Growth team has also learned a lot about using Google and Turnitin for copyvio detection, and has had multiple architectural conversations this week to narrow in on an approach.
Over the past week, the Growth team has finished writing most of the components necessary for applying ORES scores to pages in the New Pages Feed, and along with the filters for the state of drafts, those components are now in our internal testing environments where QA staff are ironing out bugs.
We also conducted a comparison of the two main services that English Wikipedia uses for copyvio detection: Google search (used by Earwig's Copyvio Detector) and Turnitin (used by CopyPatrol). The objective was to help us understand how different the two services are in terms of their results. I'll be assembling the results and posting that in a coming update. We will use that information to help decide which service to use for New Pages Feed, in addition to considerations around the usage limits for those services.
Starting today, everyone is welcome to test out the Growth team's progress on the New Pages Feed using Test Wiki! This has been a long time coming, and our team is excited that you'll be able to get your hands on the work so far. Please make sure to read the " How to test" section below to configure your account. The idea here is that we want to get the reactions and thoughts of AfC and NPP reviewers on an ongoing basis to make sure that we continue to build something useful. Going forward, we'll continue to push updates of the software to Test Wiki for everyone to try out as soon as possible. I'll post here when there is something new to try.
We've deployed a few changes to the testing environment. Please check them out and let us know what you think!
We have not yet done any work on the formatting of the data presented with each draft in its listing in the feed. Because it's a lot of dense information, we would like to hear any suggestions to make it more readable.
The team has now deployed a major set of work to the testing environment. Please check it out and let us know what you think:
In order to see how the models change with different content, it can be helpful to paste wikitext from other articles in the Test Wiki, noting in the edit summary which article it came from. Feel free to create new drafts with the Article Wizard, and refer to the " How to test" section above for more details (or ask on the talk page).
A few of notes on outstanding work that we're still doing on this front:
As the Growth team has been working on adding AfC drafts and ORES to the New Pages Feed ( now testable in Test Wiki), we have also been planning how to add the first copyvio detection tool to the New Pages Feed. This post is about our plan to use CopyPatrol (and the Turnitin service) to accomplish this. Read below for the plan and background, and please speak up on the talk page with your thoughts and reactions – the point, after all, is to build something that helps reviewers get their work done. We've also posted the brief statistical analysis our team did as a part of this planning process.
Below is a quick mockup of what this might look like.
You can see that in the third draft in the list, next to "Possible issues", "copyvio" is listed in red. This word is a link to the CopyPatrol interface, where reviewers can investigate potential violations. Below is a screenshot from CopyPatrol showing its existing interface.
Back when we were planning this effort in May, reviewers participating in the discussion seemed to agree that pre-checking pages for copyvio would help increase reviewing efficiency. The idea is that reviewers could quickly find those pages that are most likely to have copyvio problems, and would save time by not needing to wait as a copyvio tool runs for each page that a reviewer works on.
As the Growth team has been working on the other two major parts of this New Pages Feed upgrade (adding AfC drafts, and adding ORES scores), we have simultaneously been debating the right way to approach the copyvio part. This has been difficult, because unlike with ORES, we rely on third-party services for copyvio detection, like Google (via Earwig's Copyvio Detector) and Turnitin (via CopyPatrol). Integrating third-parties into the Mediawiki software adds technical complexity and risk to our software, since we won’t be able to completely control the services that we’ll be relying on.
We have put a lot of thought into this, and we’ve decided to add copyvio detection in the New Pages Feed using CopyPatrol / Turnitin. The main alternative we considered is Earwig's Copyvio Detector / Google. There are three main reasons we have decided to build with CopyPatrol / Turnitin.
Now that reviewers have had a few weeks to test out the changing New Pages Feed in Test Wiki, we want to get some of the improvements out into the real world so they can help reviewers. Specifically, we're planning on deploying the first of the three parts of this project to English Wikipedia on September 17: adding the "Articles for Creation" side to the feed. AfC reviewers would then be able to browse drafts in the feed, filter on their states, and sort by submitted and declined dates. This would leave the classic NPP workflow unchanged, except for the toggle button for "AfC".
As community reviewers tested the feed in Test Wiki, a couple of bugs and ideas were surfaced that our team has largely addressed, and it does not seem like there are major blocking issues. That said, we know that there is more to making a new feature successful than simply flipping it on. These are some of the things that I think would be good to address, and I'm looking for thoughts from reviewers about how best to them:
I am happy to help with any of the documentation or screenshots.
And then following September 17, here are some tentative dates for rolling out the second and third parts of this project (these may change, but give a sense of the pace of our work):
Let's discuss on the talk page if there are any concerns, and what the correct order of operations is here so that we can start getting the useful new features into the hands of reviewers!
A couple weeks ago, we posted above on our plans for integrating copyvio detection to the New Pages Feed. It seemed like the plan made sense to reviewers who read it, and so we've implemented it in Test Wiki so that reviewers can try it out. Please check it out and let us know what you think.
Here's how it works:
Some notes:
A previous update laid out our team's schedule for deploying the three parts of the new feature set to the New Pages Feed in English Wikipedia. This is an update on how deployment has gone so far and what the schedule holds going forward.
As planned, we did deploy AfC to the New Pages Feed on September 17. The reason we've not announced that the feed is ready to be used by AfC reviewers is that we discovered a set of bugs and issues that we've been fixing since that date. Since AfC reviewers already have a functioning workflow, we would prefer that they try out a well-functioning new workflow rather than a buggy new workflow, and so I have not yet declared victory at the AfC discussion page. I expect that the feed will be in shape for that at the beginning of next week, and at that time, I'll include brief instructions for how to use the feed for AfC review. However, for those of you who have been following along on this project, you can see that the New Pages Feed now has its "Articles for Creation" side. We are still fixing a few UI bugs having to do with the sorting and filtering menus, but you are welcome to try it out.
Here's what's coming up:
For those interested, here are the details on the first deployment and its challenges:
After deploying the feature itself, we needed to spend a few days actually populating the feed with the 40,000+ drafts in English Wikipedia, along with their states ("Awaiting review", "Declined", etc) and their submitted and declined dates. Those dates have proved to be difficult, because the Mediawiki database does not retain a record of when templates and categories were applied to pages. We've approximated the submitted and declined dates using the most recent edit date, but we have a task open to make them more accurate if reviewers find that the dates that are currently in the feed are not close enough for their work. We are still in the process of fixing a couple of UI bugs having to do with the default and sticky values for sorting and filtering selections: T205168 and T205324.
We fixed the bugs that I mentioned in the previous update, and so now the New Pages Feed is ready to be used for AfC review! Here is the announcement on the AfC discussion page. This is a great milestone for this project -- it's our first of three releases (ORES and copyvio are the next two) and will hopefully give AfC reviewers a tool that helps them prioritize their work, and ultimately get high-quality drafts into the article space faster. Thank you all for weighing in, following along, and helping us get to this point!
Our team is quickly turning our full attention to adding ORES scores to the feed (for both AfC and NPP) this Thursday, October 4, or in the days that follow. I will post additional updates as that initiative unfolds. As always, please comment on the talk page with any thoughts! -- MMiller (WMF) ( talk) 20:52, 1 October 2018 (UTC)
Our deployment to add two sets of ORES scores to the New Pages Feed went smoothly yesterday. All pages have scores for both models, and they seem to be the scores that we expect them to have. We've posted here at AfC talk and here at NPP talk to announce and explain the changes. In terms of the objectives of this project, we're excited about this deployment because AfC reviewers will now be able to do things like filter the New Pages Feed to just those drafts that are predicted to be "B-class" and above, thereby accelerating the rate that high quality content makes it to the article namespace, and potentially accelerating how quickly a good-faith newbie gets some positive feedback.
To see interesting counts of how many pages in the feed ended up with which predictions, feel free to check out this Phabricator task.
Our team will now turn our attention to the third, and final, part of this project: adding copyvio detection to the feed. I will be back with more updates as we plan for that deployment the week of October 15 or October 22.
Over the last two weeks, the team has been working to deploy copyvio detection, the third and final component of this project, to English Wikipedia. This has involved the community-driven bot approval process for a modification to EranBot 3, which backs CopyPatrol and which now serves information to the New Pages Feed. This process precipitated some changes to the software, and the creation of a new log, which shows exactly which pages and revisions are being flagged as potentially violating.
The new feature is now deployed for a trial period in which NPP and AfC reviewers can try out the feature at this special URL (as opposed to the regular URL for the New Pages Feed). This trial period will last into next week, at which point the team will decide when to release the feature at the regular URL. The ability to test has been announced here at AfC talk and here at NPP talk. We'll expect to receive feedback at those talk pages, or on this project's talk page. When the feature is released, we'll post recommendations for how to use it.
Yesterday, the team deployed copyvio detection via CopyPatrol to the New Pages Feed. This is the third and final component of this project (along with adding drafts to the feed, and adding ORES scores to the feed). The trial period for the bot that backs this feature went well, and the results of that bot trial are archived here. Only one minor issue was discovered during the trial period.
Our testing shows that all that pages in the New Pages Feed that have been flagged by CopyPatrol are also flagged in the New Pages Feed, with links that go between them. Reviewers will now be able to use this information to further prioritize and triage pages waiting for NPP and AfC review in the feed.
We've posted here at AfC talk and here at NPP talk to announce and explain the changes.
Since this is the final component of this project, we're going to keep an eye on it for the next week to make sure everything continues to work as expected. Then we will post to wrap-up the project.
Now that all the components of this project have been in production for a month without any unsolved issues, it is time to wrap this project up. The effort to improve the efficiency of the Articles for Creation process began in April 2018 and the work was completed in October 2018, seven months later. During the design process, the project expanded from only relating to Articles for Creation to also involving the New Page Review process through work on the New Pages Feed. We were in close contact with both communities throughout the process, and had great discussions where important consensus was built. We're grateful for the community members who tested the software at every step of the development process so that we could be confident that we were building something valuable.
Along the way to adding AfC, ORES, and copyvio detection to the New Pages Feed, we also fixed many bugs in the feed and improved existing components of the feed to make more sense for the contemporary reviewing processes. This mediawiki.org page has been updated to document the 2018 improvements.
Thank you to the AfC and NPP community members who spent their volunteer time thinking about this project and helping us build it.
This page documents the development process for a completed project undertaken by the WMF Growth team from April to October 2018. |
The Wikimedia Foundation's Community Tech team and Growth teams are extending the New Pages Feed interface to allow both Articles for Creation (AfC) reviewers and New Page Patrol (NPP) reviewers to prioritize pages for review using quality and copyright violation scores. This work will take place during May and June 2018.
Articles for Creation (AfC) is a process in English Wikipedia by which experienced Wikipedians review draft pages to determine whether they can be added to the main article namespace. Because of the April 2018 policy change known as ACPERM, traffic to the AfC process is expected to increase. In discussion with the AfC community, it has been determined that AfC will be able to promote more high quality pages to the main article namespace more quickly through enhanced prioritization tools. Therefore, the Community Tech team is building an improvement to the existing New Pages Feed interface that will allow AfC reviewers to use predictive models to prioritize drafts by their likelihood of copyright violation and their predicted quality. This improvement will also be available to reviewers in the New Page Patrol (NPP) process, who review pages created directly in the main article space, and are the original users of the New Pages Feed.
For the full background on the motivation of this project and corresponding discussion by members of the AfC and NPP communities, please see the original project page.
Goal: we want AfC reviewers to be able to get high quality articles into the main namespace as quickly as possible.
Given the expected increase of submissions to AfC, we plan to help by increasing the efficiency with which reviewers can review drafts while maintaining their quality standards. Specifically, there are three metrics that we intend to maintain or improve:
Community discussion on these metrics is here.
Technical discussion on the appropriate way to create these numbers is in Phabricator here.
Through about a month of conversation on potential improvements, community members and WMF staff agreed that a way to help AfC reviewers pursue the goals and metrics above would be through an interface to allow reviewers to prioritize drafts based on quality scores, and would allow them to quickly assess copyright violations (copyvio). The New Pages Feed, which was built for the separate NPP process, is an existing interface that can facilitate this. The planned changes are as follows:
Below is a set of user stories that describe in more detail what we plan to implement with these changes. User stories are a software engineering practice that help clearly define what software will need to accomplish from the perspective of the user. The user stories below are based on discussion with AfC and NPP reviewers, and are broken down into four categories. The Community Tech team hopes to address all these stories, but may need to omit stories that prove to be too technically challenging for the limited scope of this project, or that the community judges are not actually important.
In the future, this section will link to visual mockups built on these user stories, so that the community can comment on them before the software is built.
As a reviewer, I need to sort and filter the New Pages Feed by prioritization information:
As a reviewer, I need to see prioritization information next to each page’s entry in the New Pages Feed:
As a reviewer:
As a reviewer:
The Community Tech team started planning this project on May 1, 2018, with involvement from WMF designers and community liaisons. The next steps are as follows, and will contain a more specific timeline in the future:
1. Community Tech software engineers will investigate the feasibility and challenges of the user stories listed above. Specifically, those investigations are tracked in these three Phabricator tasks:
2. The team will develop some initial visual mockups of how the New Pages Feed could change with this work.
3. We will post those mockups here and on the talk pages for the AfC and NPP projects for feedback and input from those communities.
4. Engineering work will begin, and we will strive to roll out incremental improvements for testing as soon as they are ready.
On May 8, five of us on the WMF team got together to think through some of the design details around this feature improvement for the New Pages Feed. We talked through the user stories listed above on this page with two objectives: (1) surface as many unanswered questions as we could, and (2) make sure our designer had enough clarity to begin to mock up some ideas. It's not likely that we'll be able to design or engineer according to all the thoughts below, but I'm posting our full notes so that everyone can follow along with our process. Please comment and chime in on the talk page, especially regarding the "Open questions" list at the bottom. We hope to show some mockups for feedback in the next couple weeks.
Please remember that these ideas have not been thoroughly vetted for engineering feasibility yet. We won't be able to do everything in this list this time, and we may or may not be able to do all of the most interesting ideas.
Over the last week, the WMF team has been following the execution plan above by working in two tracks:
Below, I have notes and questions for community members on both these tracks, as well as our current next steps.
Working from the notes from the design session above, and from the resulting conversation on the talk page, we made some low-fidelity wireframes of how we think the expansion of the New Pages Feed could look. I hope everyone can take a look at these images (and the corresponding annotations), and think through whether this will work for your NPP or AfC reviewing workflow – and if not, how the design could be better (they're just wireframes, after all, so nothing is set in stone).
Some overall notes:
Annotations to the wireframes:
The full detail of the technical investigations can be found in the Phabricator tasks, but a summary is included here.
The biggest challenge is around the best way to incorporate copyvio scores. The engineers here are still discussing and thinking through options, so we unfortunately do not have a complete plan right now. We are eager to hear anything that community members can add to the mix to help us define a path forward. There are three main issues:
Given these three issues, we currently think that the quickest path to adding copyvio to New Pages Feed would be through using Turnitin, which is the more straightforward service to implement. In that vein, there are several open questions for community members that we need some guidance on:
Despite the challenges with copyvio described above, we are moving forward on several fronts. Our expectation is that we'll deliver the changes to the New Pages Feed in segments, as opposed to all at once. I'm going to be making Phabricator tasks for several of the following items.
Thank you, and as always, I'm looking forward to the discussion on the talk page.
For those of you who are interested in following along in Phabricator, I've made tasks for the parts of this project that have reached a level of clarity such that the WMF team can start their engineering work. Thanks to all members of the reviewing community who helped us get to this level of clarity. To that end, we're now excited to say that engineering work is happening over the next four weeks on the following specific work items (all listed under the same epic):
The copyvio parts of this project do not yet have Phabricator tasks created. Since that is such an important part of the work, and the largest engineering challenge, the conversation is still ongoing around the right way to implement it. I'm hoping we'll be able to come to some conclusions and be able to create those Phabricator tasks by next week.
These tasks still have a fair number of open questions in them that I'll be working to settle down this week, and about which I will likely have questions on the talk page. And the tasks may split, merge, or get increased details over the coming weeks. That said, if anyone is interested in reading over them, I know the team here welcomes questions, clarifications, and especially to let us know if we've misunderstood something important about the AfC and NPP workflows.
Importantly, we will be testing and asking for feedback on these changes to the New Pages Feed before they are rolled out to users. We definitely do not want to change the feed in ways that are surprising or unproductive. It's my intention to give as many visual progress reports as possible as this work unfolds, and to collaborate with the NPP and AfC communities to roll out changes deliberately.
In that vein, I'm hoping that it will be possible to roll out parts of the project to reviewers as they are complete, instead of waiting for all the parts to be done. That way, we'll be able to get real feedback early, and hopefully deliver goodness on this project as soon as possible.
Over the past week, the Community Tech team has been working on adding draft pages to the New Pages Feed, and making it possible to filter those drafts based on where they are in the AfC process. That work is progressing well so far, without any surprises. Two additional work items that were detailed during the week are the need to explicitly test that drafts are listed in the feed correctly before making changes to production, and making sure that the whole draft backlog is added to the feed.
With respect to the second part of the project -- adding ORES scores to the feed -- Community Tech engineers have talked with engineers on WMF's Scoring Platform team, who are currently working on adding the relevant ORES models to the Mediawiki database to make them easily queryable by features like the New Pages Feed. We're optimistic that that team's work will mean Community Tech will not have to implement something additional to use ORES scores.
And with respect to the third part of the project -- copyvio -- we are taking some time as the engineers work on the beginning parts of this project to think about the right way to implement this capability in the New Pages Feed, taking into account all the thoughts on the Talk page.
The Community Tech team has made progress over the past week on adding draft pages to the New Pages Feed. We have been working on the same Phabricator tasks as mentioned in last week's update ( phab:T195545 and phab:T195924), and we added one additional task to the to-do list: adding a feature flag. This will allow us to wait until a cohesive set of changes are developed for the New Pages Feed before exposing any of them to reviewers -- as opposed to the feed changing in little, incomplete ways over time.
At this point, we do have a glimpse of how the user interface changes are shaping up -- though it is a still a work in progress. The image below is from a developer's local environment, meaning these changes are not yet available on the actual wikis for anyone to see.
The screenshot shows a couple of important points of progress:
It also shows a few things that are incomplete:
Please take a glance at the screenshot and add any of your reactions to this project's talk page. Sometimes seeing something take shape can inspire thoughts that wouldn't have occurred before, and we definitely want to get a sense for whether this feels like it's on the right track.
Over the past week, the Community Tech team has mostly been working on phab:T195924, which is about making it possible to filter drafts by their state ("Unsubmitted", "Awaiting review", "Under review", "Declined", "All"). The team has stood up the software in a testing environment as they develop it, and we have been noting issues in Phabricator as we test out the changing capabilities.
The next major item the team will be working on is making it possible to sort drafts by their submitted and declined dates.
There are four main topics in this update:
We wanted to let everyone know about a team assignment change that will hopefully help this project be completed sooner. So far, the engineering on this project has been done by the Community Tech team. That team has been working on the first major part of the project, which is to add drafts to the New Pages Feed, and make the feed sortable by state and filterable by date. Starting next week, a different Wikimedia Foundation team, the Collaboration team, will be completing the second and third parts of the project, which are adding ORES scores and adding copyvio detection. The Collaboration team was the team that added ORES scores to the Recent Changes feed, giving them good experience using ORES scores and working with the various feeds in Mediawiki. I ( MMiller (WMF)) will continue to be the product manager for this work. Community Tech has been doing great work so far, and we're being careful to transfer their knowledge to Collaboration so that the project continues smoothly.
The last item that the Community Tech team is working on with this project, before the Collaboration team begins their work, is phab:T195547, which will make it possible to sort AfC drafts by their most recent date of submission or most recent date they were declined, in addition to the original date they were created. That is the main work item currently underway this week and next.
Now that the initial work to add drafts to the New Pages Feed is largely complete, we are setting up our ability to rigorously test the new functionality. We are working to surface the new features in the Test Wiki next week. Once the new features are there, we will post another update asking reviewers to try them out and reply with thoughts and bugs. At that point, reviewers who are testing might determine that the simple addition of drafts to New Pages Feed, even without ORES and copyvio, are enough of an improvement that they could be put into production on English Wikipedia.
As mentioned above, next week the Collaboration team will begin the work to integrate ORES models into the New Pages Feed. Now that the ORES work will be beginning, I wanted to resurface a previous conversation and a decision we've made about how to proceed. In the project update from 2018-05-17, we posted wireframes for what we called "Concept A" and "Concept B".
In the discussion on the talk page, some reviewers preferred Concept A because it gives reviewers more control, and some preferred Concept B because it provides clearer recommendations and less opportunity for mis-using the model scores. We have decided to implement Concept A because we believe it will be a good stepping stone to help reviewers figure out whether Concept B is better, and if so, what rules should be used for the structured options of Concept B. In other words, by implementing Concept A, reviewers will have the opportunity to experiment with the ORES scores, decide whether Concept B is preferred, and then develop the rules for it. From the engineering perspective, having implemented Concept A, it will be relatively easy to subsequently implement Concept B.
Please do post on the talk page with reactions, questions, or any other thoughts.
Now that there are two teams working on this project, Community Tech and Growth, there are a handful of interesting updates:
If any AfC or NPP reviewers will be at Wikimania next week, please let me know! I'm hoping to meet some members of this community in person.
We're now formally testing the components of this project in our testing environments. As I've said in previous updates, as soon as we're technically able to do so, I'll ask community members to take some time to test things out as well.
Over the past week, the Community Tech team has continued the work to make drafts sortable by their submission and declined dates. And the Growth team has been writing the code to incorporate ORES scores into the New Pages Feed, and most of that code is now under review before it makes its way to the testing environment.
The Growth team has also learned a lot about using Google and Turnitin for copyvio detection, and has had multiple architectural conversations this week to narrow in on an approach.
Over the past week, the Growth team has finished writing most of the components necessary for applying ORES scores to pages in the New Pages Feed, and along with the filters for the state of drafts, those components are now in our internal testing environments where QA staff are ironing out bugs.
We also conducted a comparison of the two main services that English Wikipedia uses for copyvio detection: Google search (used by Earwig's Copyvio Detector) and Turnitin (used by CopyPatrol). The objective was to help us understand how different the two services are in terms of their results. I'll be assembling the results and posting that in a coming update. We will use that information to help decide which service to use for New Pages Feed, in addition to considerations around the usage limits for those services.
Starting today, everyone is welcome to test out the Growth team's progress on the New Pages Feed using Test Wiki! This has been a long time coming, and our team is excited that you'll be able to get your hands on the work so far. Please make sure to read the " How to test" section below to configure your account. The idea here is that we want to get the reactions and thoughts of AfC and NPP reviewers on an ongoing basis to make sure that we continue to build something useful. Going forward, we'll continue to push updates of the software to Test Wiki for everyone to try out as soon as possible. I'll post here when there is something new to try.
We've deployed a few changes to the testing environment. Please check them out and let us know what you think!
We have not yet done any work on the formatting of the data presented with each draft in its listing in the feed. Because it's a lot of dense information, we would like to hear any suggestions to make it more readable.
The team has now deployed a major set of work to the testing environment. Please check it out and let us know what you think:
In order to see how the models change with different content, it can be helpful to paste wikitext from other articles in the Test Wiki, noting in the edit summary which article it came from. Feel free to create new drafts with the Article Wizard, and refer to the " How to test" section above for more details (or ask on the talk page).
A few of notes on outstanding work that we're still doing on this front:
As the Growth team has been working on adding AfC drafts and ORES to the New Pages Feed ( now testable in Test Wiki), we have also been planning how to add the first copyvio detection tool to the New Pages Feed. This post is about our plan to use CopyPatrol (and the Turnitin service) to accomplish this. Read below for the plan and background, and please speak up on the talk page with your thoughts and reactions – the point, after all, is to build something that helps reviewers get their work done. We've also posted the brief statistical analysis our team did as a part of this planning process.
Below is a quick mockup of what this might look like.
You can see that in the third draft in the list, next to "Possible issues", "copyvio" is listed in red. This word is a link to the CopyPatrol interface, where reviewers can investigate potential violations. Below is a screenshot from CopyPatrol showing its existing interface.
Back when we were planning this effort in May, reviewers participating in the discussion seemed to agree that pre-checking pages for copyvio would help increase reviewing efficiency. The idea is that reviewers could quickly find those pages that are most likely to have copyvio problems, and would save time by not needing to wait as a copyvio tool runs for each page that a reviewer works on.
As the Growth team has been working on the other two major parts of this New Pages Feed upgrade (adding AfC drafts, and adding ORES scores), we have simultaneously been debating the right way to approach the copyvio part. This has been difficult, because unlike with ORES, we rely on third-party services for copyvio detection, like Google (via Earwig's Copyvio Detector) and Turnitin (via CopyPatrol). Integrating third-parties into the Mediawiki software adds technical complexity and risk to our software, since we won’t be able to completely control the services that we’ll be relying on.
We have put a lot of thought into this, and we’ve decided to add copyvio detection in the New Pages Feed using CopyPatrol / Turnitin. The main alternative we considered is Earwig's Copyvio Detector / Google. There are three main reasons we have decided to build with CopyPatrol / Turnitin.
Now that reviewers have had a few weeks to test out the changing New Pages Feed in Test Wiki, we want to get some of the improvements out into the real world so they can help reviewers. Specifically, we're planning on deploying the first of the three parts of this project to English Wikipedia on September 17: adding the "Articles for Creation" side to the feed. AfC reviewers would then be able to browse drafts in the feed, filter on their states, and sort by submitted and declined dates. This would leave the classic NPP workflow unchanged, except for the toggle button for "AfC".
As community reviewers tested the feed in Test Wiki, a couple of bugs and ideas were surfaced that our team has largely addressed, and it does not seem like there are major blocking issues. That said, we know that there is more to making a new feature successful than simply flipping it on. These are some of the things that I think would be good to address, and I'm looking for thoughts from reviewers about how best to them:
I am happy to help with any of the documentation or screenshots.
And then following September 17, here are some tentative dates for rolling out the second and third parts of this project (these may change, but give a sense of the pace of our work):
Let's discuss on the talk page if there are any concerns, and what the correct order of operations is here so that we can start getting the useful new features into the hands of reviewers!
A couple weeks ago, we posted above on our plans for integrating copyvio detection to the New Pages Feed. It seemed like the plan made sense to reviewers who read it, and so we've implemented it in Test Wiki so that reviewers can try it out. Please check it out and let us know what you think.
Here's how it works:
Some notes:
A previous update laid out our team's schedule for deploying the three parts of the new feature set to the New Pages Feed in English Wikipedia. This is an update on how deployment has gone so far and what the schedule holds going forward.
As planned, we did deploy AfC to the New Pages Feed on September 17. The reason we've not announced that the feed is ready to be used by AfC reviewers is that we discovered a set of bugs and issues that we've been fixing since that date. Since AfC reviewers already have a functioning workflow, we would prefer that they try out a well-functioning new workflow rather than a buggy new workflow, and so I have not yet declared victory at the AfC discussion page. I expect that the feed will be in shape for that at the beginning of next week, and at that time, I'll include brief instructions for how to use the feed for AfC review. However, for those of you who have been following along on this project, you can see that the New Pages Feed now has its "Articles for Creation" side. We are still fixing a few UI bugs having to do with the sorting and filtering menus, but you are welcome to try it out.
Here's what's coming up:
For those interested, here are the details on the first deployment and its challenges:
After deploying the feature itself, we needed to spend a few days actually populating the feed with the 40,000+ drafts in English Wikipedia, along with their states ("Awaiting review", "Declined", etc) and their submitted and declined dates. Those dates have proved to be difficult, because the Mediawiki database does not retain a record of when templates and categories were applied to pages. We've approximated the submitted and declined dates using the most recent edit date, but we have a task open to make them more accurate if reviewers find that the dates that are currently in the feed are not close enough for their work. We are still in the process of fixing a couple of UI bugs having to do with the default and sticky values for sorting and filtering selections: T205168 and T205324.
We fixed the bugs that I mentioned in the previous update, and so now the New Pages Feed is ready to be used for AfC review! Here is the announcement on the AfC discussion page. This is a great milestone for this project -- it's our first of three releases (ORES and copyvio are the next two) and will hopefully give AfC reviewers a tool that helps them prioritize their work, and ultimately get high-quality drafts into the article space faster. Thank you all for weighing in, following along, and helping us get to this point!
Our team is quickly turning our full attention to adding ORES scores to the feed (for both AfC and NPP) this Thursday, October 4, or in the days that follow. I will post additional updates as that initiative unfolds. As always, please comment on the talk page with any thoughts! -- MMiller (WMF) ( talk) 20:52, 1 October 2018 (UTC)
Our deployment to add two sets of ORES scores to the New Pages Feed went smoothly yesterday. All pages have scores for both models, and they seem to be the scores that we expect them to have. We've posted here at AfC talk and here at NPP talk to announce and explain the changes. In terms of the objectives of this project, we're excited about this deployment because AfC reviewers will now be able to do things like filter the New Pages Feed to just those drafts that are predicted to be "B-class" and above, thereby accelerating the rate that high quality content makes it to the article namespace, and potentially accelerating how quickly a good-faith newbie gets some positive feedback.
To see interesting counts of how many pages in the feed ended up with which predictions, feel free to check out this Phabricator task.
Our team will now turn our attention to the third, and final, part of this project: adding copyvio detection to the feed. I will be back with more updates as we plan for that deployment the week of October 15 or October 22.
Over the last two weeks, the team has been working to deploy copyvio detection, the third and final component of this project, to English Wikipedia. This has involved the community-driven bot approval process for a modification to EranBot 3, which backs CopyPatrol and which now serves information to the New Pages Feed. This process precipitated some changes to the software, and the creation of a new log, which shows exactly which pages and revisions are being flagged as potentially violating.
The new feature is now deployed for a trial period in which NPP and AfC reviewers can try out the feature at this special URL (as opposed to the regular URL for the New Pages Feed). This trial period will last into next week, at which point the team will decide when to release the feature at the regular URL. The ability to test has been announced here at AfC talk and here at NPP talk. We'll expect to receive feedback at those talk pages, or on this project's talk page. When the feature is released, we'll post recommendations for how to use it.
Yesterday, the team deployed copyvio detection via CopyPatrol to the New Pages Feed. This is the third and final component of this project (along with adding drafts to the feed, and adding ORES scores to the feed). The trial period for the bot that backs this feature went well, and the results of that bot trial are archived here. Only one minor issue was discovered during the trial period.
Our testing shows that all that pages in the New Pages Feed that have been flagged by CopyPatrol are also flagged in the New Pages Feed, with links that go between them. Reviewers will now be able to use this information to further prioritize and triage pages waiting for NPP and AfC review in the feed.
We've posted here at AfC talk and here at NPP talk to announce and explain the changes.
Since this is the final component of this project, we're going to keep an eye on it for the next week to make sure everything continues to work as expected. Then we will post to wrap-up the project.
Now that all the components of this project have been in production for a month without any unsolved issues, it is time to wrap this project up. The effort to improve the efficiency of the Articles for Creation process began in April 2018 and the work was completed in October 2018, seven months later. During the design process, the project expanded from only relating to Articles for Creation to also involving the New Page Review process through work on the New Pages Feed. We were in close contact with both communities throughout the process, and had great discussions where important consensus was built. We're grateful for the community members who tested the software at every step of the development process so that we could be confident that we were building something valuable.
Along the way to adding AfC, ORES, and copyvio detection to the New Pages Feed, we also fixed many bugs in the feed and improved existing components of the feed to make more sense for the contemporary reviewing processes. This mediawiki.org page has been updated to document the 2018 improvements.
Thank you to the AfC and NPP community members who spent their volunteer time thinking about this project and helping us build it.