Late last month, the "Technology report" included a story using code review backlog figures – the only code review figures then available – to construct a rough narrative about the average experience of code contributors. This week, we hope to go one better, by looking directly at code review wait times, and, in particular, median code review times
To this end, the Signpost independently analysed data from the first 23,900 changesets as they stood on September 17, incorporating some 66,500 reviews across 32,100 patchsets. From this base, changes targeted at branches other than the default "master" branch were discarded, as were changesets submitted and reviews performed by bots. Self-reviews were also discarded, but reviews made by a different user in the form of a superseding patch were retained. Finally, users were categorised by hand according to whether they would be best regarded as staff or volunteers. [nb 1] Although this week's article focuses mainly on so-called "core" MediaWiki code, future issues will probe extension-related statistics.
WMF bosses will, on the whole, be pleased with the final figures. 50% of revisions to core MediaWiki code submitted during August was reviewed for the first time in just 3 hours 30 minutes, with 25% being reviewed in 20 minutes and 75% within 27 hours. These figures were similar across both first patchsets and later amendments, and similar with regard to slight changes in what qualified as "a review". [nb 2] The relevant trend over time is considered in the following tables. On the left is review time across all patchsets submitted to core; with the right hand table, just the first patchset in any given changeset is included.
Month | 25% | Median | 75% | Current mean |
---|---|---|---|---|
May | 42 minutes | 4 hours and 25 minutes | 1 day, 11 hours and 27 minutes | 3 days, 3 hours and 38 minutes |
June | 47 minutes | 19 hours and 10 minutes | 3 days, 16 hours and 45 minutes | 5 days, 8 hours and 29 minutes |
July [nb 3] | 39–40 minutes | 7 hours and 4–8 minutes | 2 days, 5–9 hours | 2 days, 16 hours and 38 minutes |
August [nb 3] | 20–21 minutes | 3 hours and 11–29 minutes | 21–24 hours | 1 day, 11 hours and 52 minutes |
Month | 25% | Median | 75% | Current mean |
---|---|---|---|---|
May | 38 minutes | 3 hours and 27 minutes | 1 day, 5 hours and 4 minutes | 2 days, 1 hour and 58 minutes |
June | 45 minutes | 12 hours and 34 minutes | 2 days, 13 hours and 31 minutes | 3 days, 7 hours and 39 minutes |
July | 22 minutes | 3 hours and 16 minutes | 1 day, 7 hours and 21 minutes | 1 day, 17 hours and 18 minutes |
August | 19 minutes | 3 hours and 33 minutes | 19 hours and 50 minutes | 1 day, 1 hour and 26 minutes |
The data show, then, that there has been a marked improvement in getting followup patchsets reviewed quicker, while review times for "first attempt" patchsets have improved less dramatically. Other analyses are more concerning. For example, a volunteer-written patchset waits, on average (either median or mean) twice as long as a staff-written one for its first review, although the gap has closed from three times as long in June and July. Staff provide 86% of first reviews for core, with just five staff members collectively accounting for some 55% of the total. [nb 4] Moreover, even in August, more than 5% of patchsets targeted at core waited a week for their first review.
As with all large datasets, it is difficult to rule out subtle methodological issues and in any case unideal to pinpoint trends over as short a period as four months. The full data set is available upon request.
Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.
Late last month, the "Technology report" included a story using code review backlog figures – the only code review figures then available – to construct a rough narrative about the average experience of code contributors. This week, we hope to go one better, by looking directly at code review wait times, and, in particular, median code review times
To this end, the Signpost independently analysed data from the first 23,900 changesets as they stood on September 17, incorporating some 66,500 reviews across 32,100 patchsets. From this base, changes targeted at branches other than the default "master" branch were discarded, as were changesets submitted and reviews performed by bots. Self-reviews were also discarded, but reviews made by a different user in the form of a superseding patch were retained. Finally, users were categorised by hand according to whether they would be best regarded as staff or volunteers. [nb 1] Although this week's article focuses mainly on so-called "core" MediaWiki code, future issues will probe extension-related statistics.
WMF bosses will, on the whole, be pleased with the final figures. 50% of revisions to core MediaWiki code submitted during August was reviewed for the first time in just 3 hours 30 minutes, with 25% being reviewed in 20 minutes and 75% within 27 hours. These figures were similar across both first patchsets and later amendments, and similar with regard to slight changes in what qualified as "a review". [nb 2] The relevant trend over time is considered in the following tables. On the left is review time across all patchsets submitted to core; with the right hand table, just the first patchset in any given changeset is included.
Month | 25% | Median | 75% | Current mean |
---|---|---|---|---|
May | 42 minutes | 4 hours and 25 minutes | 1 day, 11 hours and 27 minutes | 3 days, 3 hours and 38 minutes |
June | 47 minutes | 19 hours and 10 minutes | 3 days, 16 hours and 45 minutes | 5 days, 8 hours and 29 minutes |
July [nb 3] | 39–40 minutes | 7 hours and 4–8 minutes | 2 days, 5–9 hours | 2 days, 16 hours and 38 minutes |
August [nb 3] | 20–21 minutes | 3 hours and 11–29 minutes | 21–24 hours | 1 day, 11 hours and 52 minutes |
Month | 25% | Median | 75% | Current mean |
---|---|---|---|---|
May | 38 minutes | 3 hours and 27 minutes | 1 day, 5 hours and 4 minutes | 2 days, 1 hour and 58 minutes |
June | 45 minutes | 12 hours and 34 minutes | 2 days, 13 hours and 31 minutes | 3 days, 7 hours and 39 minutes |
July | 22 minutes | 3 hours and 16 minutes | 1 day, 7 hours and 21 minutes | 1 day, 17 hours and 18 minutes |
August | 19 minutes | 3 hours and 33 minutes | 19 hours and 50 minutes | 1 day, 1 hour and 26 minutes |
The data show, then, that there has been a marked improvement in getting followup patchsets reviewed quicker, while review times for "first attempt" patchsets have improved less dramatically. Other analyses are more concerning. For example, a volunteer-written patchset waits, on average (either median or mean) twice as long as a staff-written one for its first review, although the gap has closed from three times as long in June and July. Staff provide 86% of first reviews for core, with just five staff members collectively accounting for some 55% of the total. [nb 4] Moreover, even in August, more than 5% of patchsets targeted at core waited a week for their first review.
As with all large datasets, it is difficult to rule out subtle methodological issues and in any case unideal to pinpoint trends over as short a period as four months. The full data set is available upon request.
Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for several weeks.
Discuss this story
Non-difference in PEF-1 success rate
There was no significant difference in the success rate of editors in the PEF-1 experiment. When I arrived, this report stated that there was an insignificant decrease in quality. This is both false and misleading. There was actually an insignificant increase in quality in the reported statistics. However, this detail is irrelevant since the statistical test failed to identify a meaningful difference and other analyses from the report reversed the comparison. The best we can report to a lay audience is that no change was observed. See meta:Research:Post-edit_feedback/PEF-1. -- EpochFail( talk| work) 19:55, 26 September 2012 (UTC) reply