Activity Analysis of Dispute Resolution
In May, I did an analysis of a handful of dispute resolution forums, which was undertaken to follow up on the results of the survey. Essentially, I had the opinion of the community on dispute resolution, but no actual data as to the state of the processes. Below you will find my methodology as well as the results of the analysis.
The that were analysed were:
WP:3O, WP:DRN, WP:MEDCAB, WP:RFM, WP:WQA and all WP:RFCs filed between 1-31 May.
All of the following measurements were checked against all dispute resolution forums, unless otherwise noted.
- Number of disputes filed: Doing a raw number count based on the history of each page in the period of 1-31 May, I will be able to determine which forums are used the most utilised by the community.
- Number of editors participating: "Editors" were anyone that had made an edit to the respective dispute resolution page. This was determined partially with a database query, and partially by running a query on the pages using this tool.
- Response time: This was defined by the time between a dispute being filed and being responded to by a dispute resolution volunteer.
- Number of volunteers: A key concern from the survey was the insufficient amount of dispute resolution volunteers. Discussions were examined to determine the amount of volunteers.
- Resolution time: As the timeframe for dispute resolution was a concern brought up in the survey, this analysed the average length of time that a dispute was open.
- Success rate: This was be checked by looking at the discussions, and seeing if the dispute had been resolved, based on article and talk page activity.
The results of the forums are listed in the table below. Some data is not provided due to relevance.
| Forum | # of disputes | # of participants | # of volunteers | Average first response time | Average resolution time | Success rate |
|---|---|---|---|---|---|---|
| Third opinion | 31 | Not assessed | N/A | 25 hours | N/A | 52% |
| Dispute resolution noticeboard | 42 | 207 | 25 | 16.6 hours (21.4% of disputes were never looked at) | 8.6 days | 47.61% |
| Mediation Cabal | 4 | 17 | 5 | N/A | 28 days | 100% |
| Formal Mediation | 7 | 25 | 4 | N/A | 15 days | 0% |
| Wikiquette assistance | 17 | 67 | Not assessed | 5.3 hours | 45.5 hours | 21.4% |
| Requests for comment | 15 | 339 (212 to one RFC) | N/A | N/A | N/A | N/A |
August results
| Metric | Result for May | Goals for August | Results for August | Results for August % |
|---|---|---|---|---|
| First response time | 16 hrs 36 mins | <10 hrs | 5h 29 min | 67% reduction in response time |
| # of active volunteers | 25 - 1 to 12 ratio with 207 participants (average 1.47 per thread with 42 threads opened) | 30+ | 20 - 1 to 11 out of 177 total editors (average 2.85 per thread with 52 threads opened) | 20% reduction in volunteers - but the active volunteers were more responsive to disputes |
| Timeframe thread open | 8.6 days | 5 - 7 days | 2d 10 hr - 3 d 11 hr for disputes that were addressed | 60% reduction in discussion time |
| Success rate | 47.61% | 70%+ | 64.29% | 16.6% increase in success |
The results at DRN showed some encouraging stats - a reduction of 67% to first response times, 60% reduction in discussion times, 25% reduction in thread size, an average of 2.85 volunteers to a thread up from 1.5 and a success rate of over 64%. The amount of volunteers decreased by 20%; so from the results it shows that while the disputes were handled in a quicker timeframe by more volunteers per thread, it was from a small core group of volunteers. This emphasises the need for more volunteers - if the existing volunteers burnout, dispute resolution will suffer.
Also, there was a 25% reduction in thread size compared to May.
Full results for May:
- Number of disputes filed: 42
- Number of editors participating: 207
- Response time: 16.6 hours
- Disputes responded to: 78.6%
- Number of volunteers: 25 (ratio of 1 to 12)
- Resolution time: 8.6 days
- Success rate: 47.61%
The changes that were implemented to the dispute resolution noticedboard had a positive impact on the response and resolution time of cases at DRN, the issue of a shortage of volunteers still remains. While a universal form to request dispute resolution is currently being developed, without volunteers to resolve the cases, dispute resolution will not work. Over recent months, the amount of active volunteers has started to drop off, and this has caused dispute resolution to suffer.
To combat this, a second survey will shortly be launched. This survey will target existing volunteers, to learn their experiences, their motivations for starting to resolve disputes, and if they have become inactive, why. Hopefully with this data, we can "unlock" the door to a way forward where dispute resolution has plenty of volunteers, and disputes still get resolved in an efficient manner.
An analysis of the history of the Mediation Committee is also underway, to find the average timeframe a mediator is active at the committee, measure success rates of members, the average time a case takes to resolve, and the average time between acceptance of a case and a mediator taking on the case. It is hoped that with this information, we can find ways to improve mediation.
Activity Analysis of Dispute Resolution
In May, I did an analysis of a handful of dispute resolution forums, which was undertaken to follow up on the results of the survey. Essentially, I had the opinion of the community on dispute resolution, but no actual data as to the state of the processes. Below you will find my methodology as well as the results of the analysis.
The that were analysed were:
WP:3O, WP:DRN, WP:MEDCAB, WP:RFM, WP:WQA and all WP:RFCs filed between 1-31 May.
All of the following measurements were checked against all dispute resolution forums, unless otherwise noted.
- Number of disputes filed: Doing a raw number count based on the history of each page in the period of 1-31 May, I will be able to determine which forums are used the most utilised by the community.
- Number of editors participating: "Editors" were anyone that had made an edit to the respective dispute resolution page. This was determined partially with a database query, and partially by running a query on the pages using this tool.
- Response time: This was defined by the time between a dispute being filed and being responded to by a dispute resolution volunteer.
- Number of volunteers: A key concern from the survey was the insufficient amount of dispute resolution volunteers. Discussions were examined to determine the amount of volunteers.
- Resolution time: As the timeframe for dispute resolution was a concern brought up in the survey, this analysed the average length of time that a dispute was open.
- Success rate: This was be checked by looking at the discussions, and seeing if the dispute had been resolved, based on article and talk page activity.
The results of the forums are listed in the table below. Some data is not provided due to relevance.
| Forum | # of disputes | # of participants | # of volunteers | Average first response time | Average resolution time | Success rate |
|---|---|---|---|---|---|---|
| Third opinion | 31 | Not assessed | N/A | 25 hours | N/A | 52% |
| Dispute resolution noticeboard | 42 | 207 | 25 | 16.6 hours (21.4% of disputes were never looked at) | 8.6 days | 47.61% |
| Mediation Cabal | 4 | 17 | 5 | N/A | 28 days | 100% |
| Formal Mediation | 7 | 25 | 4 | N/A | 15 days | 0% |
| Wikiquette assistance | 17 | 67 | Not assessed | 5.3 hours | 45.5 hours | 21.4% |
| Requests for comment | 15 | 339 (212 to one RFC) | N/A | N/A | N/A | N/A |
August results
| Metric | Result for May | Goals for August | Results for August | Results for August % |
|---|---|---|---|---|
| First response time | 16 hrs 36 mins | <10 hrs | 5h 29 min | 67% reduction in response time |
| # of active volunteers | 25 - 1 to 12 ratio with 207 participants (average 1.47 per thread with 42 threads opened) | 30+ | 20 - 1 to 11 out of 177 total editors (average 2.85 per thread with 52 threads opened) | 20% reduction in volunteers - but the active volunteers were more responsive to disputes |
| Timeframe thread open | 8.6 days | 5 - 7 days | 2d 10 hr - 3 d 11 hr for disputes that were addressed | 60% reduction in discussion time |
| Success rate | 47.61% | 70%+ | 64.29% | 16.6% increase in success |
The results at DRN showed some encouraging stats - a reduction of 67% to first response times, 60% reduction in discussion times, 25% reduction in thread size, an average of 2.85 volunteers to a thread up from 1.5 and a success rate of over 64%. The amount of volunteers decreased by 20%; so from the results it shows that while the disputes were handled in a quicker timeframe by more volunteers per thread, it was from a small core group of volunteers. This emphasises the need for more volunteers - if the existing volunteers burnout, dispute resolution will suffer.
Also, there was a 25% reduction in thread size compared to May.
Full results for May:
- Number of disputes filed: 42
- Number of editors participating: 207
- Response time: 16.6 hours
- Disputes responded to: 78.6%
- Number of volunteers: 25 (ratio of 1 to 12)
- Resolution time: 8.6 days
- Success rate: 47.61%
The changes that were implemented to the dispute resolution noticedboard had a positive impact on the response and resolution time of cases at DRN, the issue of a shortage of volunteers still remains. While a universal form to request dispute resolution is currently being developed, without volunteers to resolve the cases, dispute resolution will not work. Over recent months, the amount of active volunteers has started to drop off, and this has caused dispute resolution to suffer.
To combat this, a second survey will shortly be launched. This survey will target existing volunteers, to learn their experiences, their motivations for starting to resolve disputes, and if they have become inactive, why. Hopefully with this data, we can "unlock" the door to a way forward where dispute resolution has plenty of volunteers, and disputes still get resolved in an efficient manner.
An analysis of the history of the Mediation Committee is also underway, to find the average timeframe a mediator is active at the committee, measure success rates of members, the average time a case takes to resolve, and the average time between acceptance of a case and a mediator taking on the case. It is hoped that with this information, we can find ways to improve mediation.