OK, this is a page for discussion of Moni3's proposed AN/I analysis.
Note: The following collapsed discussions provide all the background to this project. They have been edited (by me) for relevance - please see the linked version for the complete discussions.
please open to read the various discussions that led to starting this project
|
---|
Excerpted from AN_talk discussionData collection to assess how effective ANI is at responding to complaints I apologize right up front here. I will be unable to do this because I simply don't have the time, but I'm hoping this idea will spur on someone else who does. To be better informed about how effective ANI is at responding to editor complaints, those who are participating in this discussion should be aware of how ANI operates on a daily basis. If I had the time, I would chart the success of each thread at ANI going back at least three months. I would rate each thread on a scale of 1 to 5, much like the DEFCON ratings:
A: "Sidetrack" means any instance someone inserts a comment irrelevant to the original complaint, including attempts at humor, comments about responding admin(s), accusations that one or more responding admins should not be participating because they are involved, or something else that does not address a solution to the stated problem. For whoever may take this on, and I really hope someone will, my DEFCON rating system here is based on my experiences at ANI. However, if you see fit to tweak or change the rating method, it should follow that any changes still rates each thread's success rate on:
You may have to ping my talk page to get me to respond here if you have questions. Please consider taking this on, and again, I apologize for not being able to do this myself. -- Moni3 ( talk) 22:43, 10 February 2012 (UTC)
Excerpted from Moni3's talk pageWork proceeds I finished the basic framework yesterday. The schema allows for diffs to be mapped to a user, to a specific thread and to an archive. These individual diffs can then be scored, and these scores can be rolled up to the thread level. I've also made room to expand the system to consider other noticeboards apart from AN/I, but I'm not planning to use that functionality in the short term. Thread data includes who started the thread, how many comments it attracted, the open and close date, the general category (user complaint, sock, vandalism, page prot, etc) and whether it was resolved. User data includes whether they are an admin or an editor. (There are numerous other editor attributes we could include, but this was enough to get started). Diff data includes which user started it, the full text of the diff, the edit summary, and some analytic parameters. The diff analytics consists of a set of 1-10 scores on various parameters. To get started I chose the following parameters: "Tone", "Relevance", "Sarcasm", "Hostility", "Constructiveness". I also have a flag for "Side Topic", and a measure for "Side topic relevance" - part of an attempt to analyze thread drift. Note - It is already apparent to me that these parameters need refinement, but whatever, that's why we have prototypes. I have loaded about 1000 users into the system (based on a scraping of who edited AN/I) and the threads headers from AN/I Archive 729 and 730. I've loaded about 50 diffs as well, but I want to get to 1000 before sending it over to you for review (1000 diffs is only about 3 days worth).It takes about a minute to enter a single diff (this will get faster), so there are quite a few hours of work ahead. I hope to have something to you by the end of the week. Do you have MS Access available to use? I chose Access because it is great for "quick and dirty" development, as there are (no doubt) numerous fundamental changes yet to be made before we have a system we're all happy with. If you don't have Access I can dump the results out to a Google spreadsheet or similar resource. If the system seems useful we can look into making it more permanent in nature, but that's a long way off. Manning ( talk) 03:47, 13 February 2012 (UTC)
Kim - I'm just using Access for prototyping of a data entry system - once I load all the diffs, we'll need to go through and score them and I can't use SPSS for that. Ultimately the best system would be a web front end over a MySql database, but I'm using Access for now because it is ideal for "quick and dirty". If we started with a web-based system then every time we change our mind about the params we'd have to recode the user interface (which gets really tedious). Once we have the data loaded and scored then SPSS would be ideal for analysing it (or Cognos which I work with). I'm not real stressed about the params just yet, I think when you actually have diff data to work with your thinking might change anyway (it did for me). Once I get something out to you with diff data loaded you'll be in a much better position to make design choices. Cheers Manning ( talk) 11:25, 13 February 2012 (UTC)
Manning, et al, when I said I was short on time...that wasn't an exaggeration. This is the earliest I've been able to reply to Manning's original post on my talk page, and the rest of my week(s) are going to go the same. I feel like a dingus suggesting stuff and then having others do it, but it hadn't been suggested and it seems Manning and company know what they're doing--far more than I would. Feel free to use my talk page to discuss it. -- Moni3 ( talk) 21:54, 13 February 2012 (UTC)
|
See Also - this discussion with CBM (a tool server developer).
OK, the first phase of the Access DB prototype is complete. It's got about 500 diffs in it (chiefly related to AN/I Archive 729). Each diff is linked to a parent thread (stored in the thread table) and parent threads are linked to an archive (stored in the archive table). Anyone who wants a copy of this database in Access 97 format please send me an email and I'll return the DB to you in a zip format.
The next step is to decide on the methodology to assess these. I've actually removed all the assessment parameters from the prototype I will send you, as I'd like you guys to look at it without my preconceptions attached.
Things I have learned so far:
That's it for now. Am looking forward to your feedback. Manning ( talk) 02:24, 15 February 2012 (UTC)
CBM just came through with a kick-ass extract. I've got 15000 URLs for diffs, plus the edit summary and user details. From here we'd need to build an interface which sucks in the diff and allows us to score them. Anyway, while I've brought the extracted diffs into the prototype, I haven't done anything else with them (other than format the revision number into a URL).
OK, the prototype database is getting sent out now. There is one form there which has example parameters (although they are disabled at the moment). That's the kind of thing that can be built for the prototype. Once we get a better idea of what we actually want, I can build a new form, and once we are happy with that, we can turn it into a web-based tool. Manning ( talk) 11:10, 15 February 2012 (UTC)
I'm no data analyst and I'm more than happy to leave that to those who are. However I'm not entirely clear on how it's intended to score/categorise/whatever the threads and what the end result will be. So far I think we have Moni's initial DEFCON score and Manning's list ("Tone", "Relevance", "Sarcasm", "Hostility" etc). Assuming we're aiming for an end result that tells us some basic stats like the ratio of resolved to unresolved threads, and then gives a more detailed analysis of what made for a successful or unsuccessful resolution, do we need an agreed set of scoring criteria and scores? EyeSerene talk 13:27, 15 February 2012 (UTC)
The key tables in the DB are the thread and diff tables. The real value of this system will be in the thread analysis obviously. The thread table contains the following:
This last column currently has ten choices: Block|Unblock Review / User Conduct Complaint / Content Dispute /Page Prot Request / Current Event response / Sock related / Admin conduct complaint / Vandalism in progress / Legal Threat / Policy Interpretation.
There may be other categories that need to be created. Based on the discussion above it is clear that the thread table needs more criteria. Some initial ideas could be:
I've got more ideas about the DIFF table, but sadly I have to go to work now. Damn capitalism. Manning ( talk) 21:15, 15 February 2012 (UTC)
As the purpose of this analysis is to review the effectiveness of threads, I've tried a new approach for collecting raw data (again using the randomly chosen Archive 729). This time I've simply scraped the entire text contents of the Archive page, and then mapped it to users (OK that part's still in progress).
The drawbacks of this approach is that we can't directly link back to the diff involved (as the archives are posted by MizsaBot II), and linking to users is a bit trickier. I've also had to replace all the return characters with (BR) tags and I stripped out unicode. (Also the dates and times all end up being Australian EST, due to my user settings). The advantage is it makes looking at the thread as a whole easier, and there is no uncertainty about which comment goes with which thread. I'll have a prototype version out fairly soon for anyone interested. Manning ( talk) 23:42, 15 February 2012 (UTC)
Hello Manning and fellow proponents of ANI improvement. Here's a table with my analysis of the first eight sections from Wikipedia:Administrators' noticeboard/IncidentArchive729. I've tried to assign Moni's DEFCON code numbers. She also gives the meaning of her codes in the green collapsed section near the top of this page, labeled 'please open to read the various discussions..'.
This group of cases is boringly straightforward and most of them seem to be well handled. (These are threads 1-11, neglecting some repeats). They all get code 5, except for one that is code 4 (since no answer was provided). I invite anybody to scan through the remainder of ANI729 to look for cases of Moni's 'bad' codes. If all codes are good, then ANI does not need improving. Perhaps the table should be extended to just include the 'bad' cases.
Name | Start date | Description | Filer | Subject of the complaint | Outcome | Comment on appropriateness | Moni's DEFCON code |
---|---|---|---|---|---|---|---|
AWB usage suspended for Waacstats | 26 November 2011 | Lack of care and excessively rapid edits when using AWB | PBS | Waacstats | Mistakes were acknowledged by the editor. His access to AWB was restored. | ANI was the right place for this. Filer and subject both acted properly. | 5 |
Admin attention requested: continuing unruly page moves | 28 November 2011 | Continuing unruly page moves regarding 5 O'clock (song) | Noetica | Jab7842 and others | No response | ANI was the right place for this. Filer acted properly. A response would have been good. | 4 (since thread was not answered by any admin or experienced person) |
User:Bonowatcher | 26 November 2011 | Repeated attacks on the subject of a BLP article at Chaz Bono | 76.204.91.45 | Bonowatcher | Bonowatcher already blocked as a sock | ANI was the right place for this. Filer acted properly. This was an obvious BLP violation and the sock was also obvious (at least to a checkuser). | 5 |
Ducky socks, page protection | 26 November 2011 | Edit warring at Korean language by apparent socks | Crossmr | KoreanResearchCenter | Socks blocked | Either ANI or SPI would have been OK for this report. Filer acted properly. | 5 |
Persistent spammer | 27 November 2011 | Spamming by User:Niel Mokerjee | Heiro | Niel Mokerjee | User was blocked one month for spamming | ANI was the right place for this, though AIV might also have worked. Obvious spam. | 5 |
User:Swat0120 | 28 November 2011 | Vandalism-only account | Rangoon11 | Swat0210 | Level three warning given | This report should have been filed at AIV instead of ANI. Otherwise, no problem. | 5 |
Personal attack by Hipocrite | 28 November 2011 | Personal attack | TParis | Hipocrite | Editor withdrew the alleged personal attack | ANI was the right place for this. The subject of the complaint continued to make unusual remarks in the ANI thread. An admin boxed the discussion to keep it from continuing. | 5 |
Athletics page vandalism | 28 November 2011 | Personal attack | Trackinfo | 78.148.44.92 | IP blocked three hours | This report could have been filed at AIV instead of ANI. Otherwise, no problem. | 5 |
-- EdJohnston ( talk) 22:59, 22 February 2012 (UTC)
OK, this is a page for discussion of Moni3's proposed AN/I analysis.
Note: The following collapsed discussions provide all the background to this project. They have been edited (by me) for relevance - please see the linked version for the complete discussions.
please open to read the various discussions that led to starting this project
|
---|
Excerpted from AN_talk discussionData collection to assess how effective ANI is at responding to complaints I apologize right up front here. I will be unable to do this because I simply don't have the time, but I'm hoping this idea will spur on someone else who does. To be better informed about how effective ANI is at responding to editor complaints, those who are participating in this discussion should be aware of how ANI operates on a daily basis. If I had the time, I would chart the success of each thread at ANI going back at least three months. I would rate each thread on a scale of 1 to 5, much like the DEFCON ratings:
A: "Sidetrack" means any instance someone inserts a comment irrelevant to the original complaint, including attempts at humor, comments about responding admin(s), accusations that one or more responding admins should not be participating because they are involved, or something else that does not address a solution to the stated problem. For whoever may take this on, and I really hope someone will, my DEFCON rating system here is based on my experiences at ANI. However, if you see fit to tweak or change the rating method, it should follow that any changes still rates each thread's success rate on:
You may have to ping my talk page to get me to respond here if you have questions. Please consider taking this on, and again, I apologize for not being able to do this myself. -- Moni3 ( talk) 22:43, 10 February 2012 (UTC)
Excerpted from Moni3's talk pageWork proceeds I finished the basic framework yesterday. The schema allows for diffs to be mapped to a user, to a specific thread and to an archive. These individual diffs can then be scored, and these scores can be rolled up to the thread level. I've also made room to expand the system to consider other noticeboards apart from AN/I, but I'm not planning to use that functionality in the short term. Thread data includes who started the thread, how many comments it attracted, the open and close date, the general category (user complaint, sock, vandalism, page prot, etc) and whether it was resolved. User data includes whether they are an admin or an editor. (There are numerous other editor attributes we could include, but this was enough to get started). Diff data includes which user started it, the full text of the diff, the edit summary, and some analytic parameters. The diff analytics consists of a set of 1-10 scores on various parameters. To get started I chose the following parameters: "Tone", "Relevance", "Sarcasm", "Hostility", "Constructiveness". I also have a flag for "Side Topic", and a measure for "Side topic relevance" - part of an attempt to analyze thread drift. Note - It is already apparent to me that these parameters need refinement, but whatever, that's why we have prototypes. I have loaded about 1000 users into the system (based on a scraping of who edited AN/I) and the threads headers from AN/I Archive 729 and 730. I've loaded about 50 diffs as well, but I want to get to 1000 before sending it over to you for review (1000 diffs is only about 3 days worth).It takes about a minute to enter a single diff (this will get faster), so there are quite a few hours of work ahead. I hope to have something to you by the end of the week. Do you have MS Access available to use? I chose Access because it is great for "quick and dirty" development, as there are (no doubt) numerous fundamental changes yet to be made before we have a system we're all happy with. If you don't have Access I can dump the results out to a Google spreadsheet or similar resource. If the system seems useful we can look into making it more permanent in nature, but that's a long way off. Manning ( talk) 03:47, 13 February 2012 (UTC)
Kim - I'm just using Access for prototyping of a data entry system - once I load all the diffs, we'll need to go through and score them and I can't use SPSS for that. Ultimately the best system would be a web front end over a MySql database, but I'm using Access for now because it is ideal for "quick and dirty". If we started with a web-based system then every time we change our mind about the params we'd have to recode the user interface (which gets really tedious). Once we have the data loaded and scored then SPSS would be ideal for analysing it (or Cognos which I work with). I'm not real stressed about the params just yet, I think when you actually have diff data to work with your thinking might change anyway (it did for me). Once I get something out to you with diff data loaded you'll be in a much better position to make design choices. Cheers Manning ( talk) 11:25, 13 February 2012 (UTC)
Manning, et al, when I said I was short on time...that wasn't an exaggeration. This is the earliest I've been able to reply to Manning's original post on my talk page, and the rest of my week(s) are going to go the same. I feel like a dingus suggesting stuff and then having others do it, but it hadn't been suggested and it seems Manning and company know what they're doing--far more than I would. Feel free to use my talk page to discuss it. -- Moni3 ( talk) 21:54, 13 February 2012 (UTC)
|
See Also - this discussion with CBM (a tool server developer).
OK, the first phase of the Access DB prototype is complete. It's got about 500 diffs in it (chiefly related to AN/I Archive 729). Each diff is linked to a parent thread (stored in the thread table) and parent threads are linked to an archive (stored in the archive table). Anyone who wants a copy of this database in Access 97 format please send me an email and I'll return the DB to you in a zip format.
The next step is to decide on the methodology to assess these. I've actually removed all the assessment parameters from the prototype I will send you, as I'd like you guys to look at it without my preconceptions attached.
Things I have learned so far:
That's it for now. Am looking forward to your feedback. Manning ( talk) 02:24, 15 February 2012 (UTC)
CBM just came through with a kick-ass extract. I've got 15000 URLs for diffs, plus the edit summary and user details. From here we'd need to build an interface which sucks in the diff and allows us to score them. Anyway, while I've brought the extracted diffs into the prototype, I haven't done anything else with them (other than format the revision number into a URL).
OK, the prototype database is getting sent out now. There is one form there which has example parameters (although they are disabled at the moment). That's the kind of thing that can be built for the prototype. Once we get a better idea of what we actually want, I can build a new form, and once we are happy with that, we can turn it into a web-based tool. Manning ( talk) 11:10, 15 February 2012 (UTC)
I'm no data analyst and I'm more than happy to leave that to those who are. However I'm not entirely clear on how it's intended to score/categorise/whatever the threads and what the end result will be. So far I think we have Moni's initial DEFCON score and Manning's list ("Tone", "Relevance", "Sarcasm", "Hostility" etc). Assuming we're aiming for an end result that tells us some basic stats like the ratio of resolved to unresolved threads, and then gives a more detailed analysis of what made for a successful or unsuccessful resolution, do we need an agreed set of scoring criteria and scores? EyeSerene talk 13:27, 15 February 2012 (UTC)
The key tables in the DB are the thread and diff tables. The real value of this system will be in the thread analysis obviously. The thread table contains the following:
This last column currently has ten choices: Block|Unblock Review / User Conduct Complaint / Content Dispute /Page Prot Request / Current Event response / Sock related / Admin conduct complaint / Vandalism in progress / Legal Threat / Policy Interpretation.
There may be other categories that need to be created. Based on the discussion above it is clear that the thread table needs more criteria. Some initial ideas could be:
I've got more ideas about the DIFF table, but sadly I have to go to work now. Damn capitalism. Manning ( talk) 21:15, 15 February 2012 (UTC)
As the purpose of this analysis is to review the effectiveness of threads, I've tried a new approach for collecting raw data (again using the randomly chosen Archive 729). This time I've simply scraped the entire text contents of the Archive page, and then mapped it to users (OK that part's still in progress).
The drawbacks of this approach is that we can't directly link back to the diff involved (as the archives are posted by MizsaBot II), and linking to users is a bit trickier. I've also had to replace all the return characters with (BR) tags and I stripped out unicode. (Also the dates and times all end up being Australian EST, due to my user settings). The advantage is it makes looking at the thread as a whole easier, and there is no uncertainty about which comment goes with which thread. I'll have a prototype version out fairly soon for anyone interested. Manning ( talk) 23:42, 15 February 2012 (UTC)
Hello Manning and fellow proponents of ANI improvement. Here's a table with my analysis of the first eight sections from Wikipedia:Administrators' noticeboard/IncidentArchive729. I've tried to assign Moni's DEFCON code numbers. She also gives the meaning of her codes in the green collapsed section near the top of this page, labeled 'please open to read the various discussions..'.
This group of cases is boringly straightforward and most of them seem to be well handled. (These are threads 1-11, neglecting some repeats). They all get code 5, except for one that is code 4 (since no answer was provided). I invite anybody to scan through the remainder of ANI729 to look for cases of Moni's 'bad' codes. If all codes are good, then ANI does not need improving. Perhaps the table should be extended to just include the 'bad' cases.
Name | Start date | Description | Filer | Subject of the complaint | Outcome | Comment on appropriateness | Moni's DEFCON code |
---|---|---|---|---|---|---|---|
AWB usage suspended for Waacstats | 26 November 2011 | Lack of care and excessively rapid edits when using AWB | PBS | Waacstats | Mistakes were acknowledged by the editor. His access to AWB was restored. | ANI was the right place for this. Filer and subject both acted properly. | 5 |
Admin attention requested: continuing unruly page moves | 28 November 2011 | Continuing unruly page moves regarding 5 O'clock (song) | Noetica | Jab7842 and others | No response | ANI was the right place for this. Filer acted properly. A response would have been good. | 4 (since thread was not answered by any admin or experienced person) |
User:Bonowatcher | 26 November 2011 | Repeated attacks on the subject of a BLP article at Chaz Bono | 76.204.91.45 | Bonowatcher | Bonowatcher already blocked as a sock | ANI was the right place for this. Filer acted properly. This was an obvious BLP violation and the sock was also obvious (at least to a checkuser). | 5 |
Ducky socks, page protection | 26 November 2011 | Edit warring at Korean language by apparent socks | Crossmr | KoreanResearchCenter | Socks blocked | Either ANI or SPI would have been OK for this report. Filer acted properly. | 5 |
Persistent spammer | 27 November 2011 | Spamming by User:Niel Mokerjee | Heiro | Niel Mokerjee | User was blocked one month for spamming | ANI was the right place for this, though AIV might also have worked. Obvious spam. | 5 |
User:Swat0120 | 28 November 2011 | Vandalism-only account | Rangoon11 | Swat0210 | Level three warning given | This report should have been filed at AIV instead of ANI. Otherwise, no problem. | 5 |
Personal attack by Hipocrite | 28 November 2011 | Personal attack | TParis | Hipocrite | Editor withdrew the alleged personal attack | ANI was the right place for this. The subject of the complaint continued to make unusual remarks in the ANI thread. An admin boxed the discussion to keep it from continuing. | 5 |
Athletics page vandalism | 28 November 2011 | Personal attack | Trackinfo | 78.148.44.92 | IP blocked three hours | This report could have been filed at AIV instead of ANI. Otherwise, no problem. | 5 |
-- EdJohnston ( talk) 22:59, 22 February 2012 (UTC)