I was trying to work out some sort of a proposed remedy to fix the constant problem of people pushing for the inclusion of small studies of dubious standing into articles, but am now stuck. Would the sourcing board sufficiently handle these problems? Does anyone have any ideas on this one? Baegis ( talk) 14:07, 25 April 2008 (UTC)
- We could include language under WP:RS#Scholarship referring to the hierarchy of journals in analogy to the hierarchy of needs. This would require editors to weight progressively less prominently articles in Science / Nature, then in Physical Review Letters / Europhysics Letters, then Physical Review X ... ending with zero consideration for papers in Medical Hypotheses or Creation (except per WP:SPS). There are a great number of journals slushing around at the base of this pyramid who are desperate for submissions and relevancy. Some of their papers are just fine (at least, I have no qualms about citing a paper I for some reason submitted to a journal to which not even LANL subscribes; meh), but the degree of pre-publication oversight is substantially less, so they need to be granted less consideration. If papers in the elite journals provide sufficient depth of coverage, it should not be necessary to bring in the Publish-or-Perish-Pile (PoPP) to this tertiary source.
- Instances where nobody reputable will touch the issue, or even will have heard of it, are more problematic. Perhaps WP:REDFLAG could be elaborated upon to spell out more clearly that, even when low quality sources are the only ones dealing directly with a particular narrowly-defined issue, they may be excluded or minimized based on high quality sources dealing with a broader issue. This is particularly a problem with so-called energy therapies - everyone and their duck is free to suffer a revelation and open a clinic. None of this changes the fact that putative energy does not exist but the first law of thermodynamics does.
- Certainly, publishing journal alone is not sufficient to ensure quality (and claims that everything Elsevier publishes should be weighted per The Lancet are frankly ludicrous)- we must also examine critical acceptance. This needs to be spelled out in WP#Scholarship and enforced. I hesitate to recommend for instance a moratorium on citations for one year from publication date (as a 'challenge and remove' mechanism, presumably - instruction creep, and all that), but we have editors from widely diverse backgrounds. Not all of them (well, us - medical and medical denialist papers read significantly differently from physics papers, and I would be surprised if there are not nuances which I miss) are capable of critically evaluating a study to assign a meaningful trust metric.
- Which, really, is the root of our problem - at least the contribution-weighted majority of the editors in this little fracas can cite WP:UNDUE, WP:FRINGE, WP:REDFLAG, &c. ad nauseam without agreeing on the application. -
Eldereft ~(
s)
talk~
17:32, 25 April 2008 (UTC)
- Small studies certainly have their limitations, but when a body of small studies add up, a pattern can be recognized. It is like the body of research at
Arsenicum album, some of which were blind trials (both animal and human), and some of which were not. And while I recognize that double-blind placebo controlled trials in mice, rat, and human studies are preferred, the unblinded animal studies still provide useful and even reliable data and information, unless you feel that the psychic power of knowing which mouse/rat got the homeopathic doses of Arsenicum was powerful enough to elicit more healing in them and help to eliminate more arsenic through their stools and urine. The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know. By the way, the other important and inadequately appreciated issue is the problem of the large studies. The vast majority of the large clinical human trials in homeopathy give just one medicine to patients with vastly different symptoms even though they may have the same disease. This "test" of homeopathy is only effective in a small percentage of situations. It is not just happenstance that homeopaths respect biochemical/biological individualization. The system is based on it. People who really know statistics know that a good study has to have both internal validity AND external validity (how come so many smart scientifically thinking people here are ignoring this vital NPOV.
DanaUllman
Talk
06:12, 9 May 2008 (UTC)
- A Woody Allen joke comes to mind: Two women are at a restaurant. One says to the other, "My, the food here is awful!" The other says, "Yes, and what small portions!" In case you don't get it, the lesson is that high quantity does not make up for low quality. Aside from that, I'm not going to waste more time explaining the placebo effect to you, Dana. I've tried before and you learned nothing; I don't see what's different now. -- Infophile (Talk) (Contribs) 16:45, 9 May 2008 (UTC)
- Small studies certainly have their limitations, but when a body of small studies add up, a pattern can be recognized. It is like the body of research at
Arsenicum album, some of which were blind trials (both animal and human), and some of which were not. And while I recognize that double-blind placebo controlled trials in mice, rat, and human studies are preferred, the unblinded animal studies still provide useful and even reliable data and information, unless you feel that the psychic power of knowing which mouse/rat got the homeopathic doses of Arsenicum was powerful enough to elicit more healing in them and help to eliminate more arsenic through their stools and urine. The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know. By the way, the other important and inadequately appreciated issue is the problem of the large studies. The vast majority of the large clinical human trials in homeopathy give just one medicine to patients with vastly different symptoms even though they may have the same disease. This "test" of homeopathy is only effective in a small percentage of situations. It is not just happenstance that homeopaths respect biochemical/biological individualization. The system is based on it. People who really know statistics know that a good study has to have both internal validity AND external validity (how come so many smart scientifically thinking people here are ignoring this vital NPOV.
DanaUllman
Talk
06:12, 9 May 2008 (UTC)
Infophile, in serious and due respect, your posting reeks of bad faith. If I were to make a similar statement, you and others would rake me over the coals. At the same time, I love your sense of humor, especially your discussion of the placebo effects. I too have a respect for the placebo effect, except in mice and rats. Can you say with a straight face that the mice and the rats are influenced greatly because the experimenter knows which rat/mouse was given a homeopathic medicine? Because the experiments measure arsenic levels in their bodies, I'm impressed that you feel that the experimenter's belief is powerful enough to influence these levels. I think that you are much more radical in your thinking than I. DanaUllman Talk 03:55, 10 May 2008 (UTC)
- Would you mind explaining exactly how that constitutes bad faith? I'm not assuming that you're intentionally misleading us here, which would indeed be bad faith. Disagreeing with you is in no way a show of bad faith, nor is believing that most of the studies that show positive effects for homeopathy are poor quality.
- On the subject of the placebo effect, I'm not an animal researcher (and I don't believe you are either), so I'm not going to pretend I know more than they do about the reactions of animals. In this case, I just go to the results of the literature. A selection of examples:
[1]
[2]
[3]. Your efforts would be better focused looking at the results there and determining what, if any, flaws might be present in the studies. --
Infophile
(Talk)
(Contribs)
17:56, 10 May 2008 (UTC)
- Infophile, wow, the most recent of the 3 references that you cite above is from 1971! Only 1 of the 3 references included an abstact, and the article/study was nothing (!) like any of the homeopathic trials to which I referenced. The study you cite is when a rat population was given a real drug and then a placebo, and the rats seemed to respond to the placebo in the same way that they responded to the drug because both were given in injection form. I never referred to any similar design as that in my references. To which homeopathic reference are you referring? Please cite and please retract. As for my statement about your "bad faith," I wrote that because you claimed to have tried to explain the placebo effect to me in the past and "learned nothing." This statement assumes that I do not have a knowledge of or an appreciation for the placebo effect, an assumption is that is not based on anything (that is not assuming good faith).
DanaUllman
Talk
00:34, 12 May 2008 (UTC)
- On the bad faith issue, you said, "The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know." You're obviously mischaracterizing the placebo effect as something "psychic" here, and not even recognizing that you're talking about it. That's why I assumed you didn't know about it or appreciate it - at least in the case of animals. As for the studies... I'm baffled as to why you assume they should resemble the homeopathic studies in question. I'm using them to point out that animals can show the placebo effect. They don't need to be of any form other than some that would show this result. On the issue of their age... so? The effect is well accepted by animal researchers. There's simply been no reason to pursue the question recently.
- But really, I think it's about time to drop this. Nothing's going to be gained by continuing this argument here. All of these content disputes around the arbitration case are misplaced at best. Let's save the effort for the homeopathy talk pages, where the results might at least make a difference. -- Infophile (Talk) (Contribs) 03:37, 12 May 2008 (UTC)
- Infophile, wow, the most recent of the 3 references that you cite above is from 1971! Only 1 of the 3 references included an abstact, and the article/study was nothing (!) like any of the homeopathic trials to which I referenced. The study you cite is when a rat population was given a real drug and then a placebo, and the rats seemed to respond to the placebo in the same way that they responded to the drug because both were given in injection form. I never referred to any similar design as that in my references. To which homeopathic reference are you referring? Please cite and please retract. As for my statement about your "bad faith," I wrote that because you claimed to have tried to explain the placebo effect to me in the past and "learned nothing." This statement assumes that I do not have a knowledge of or an appreciation for the placebo effect, an assumption is that is not based on anything (that is not assuming good faith).
DanaUllman
Talk
00:34, 12 May 2008 (UTC)
Wikipedia:Requests_for_arbitration#Request_to_amend:_Wikipedia:Requests_for_arbitration.2FPseudoscience_and_Wikipedia:Requests_for_arbitration.2FMartinphi-ScienceApologist. has the potential to affect editors and topics involved in this case. Vassyana ( talk) 12:55, 2 May 2008 (UTC)
I was trying to work out some sort of a proposed remedy to fix the constant problem of people pushing for the inclusion of small studies of dubious standing into articles, but am now stuck. Would the sourcing board sufficiently handle these problems? Does anyone have any ideas on this one? Baegis ( talk) 14:07, 25 April 2008 (UTC)
- We could include language under WP:RS#Scholarship referring to the hierarchy of journals in analogy to the hierarchy of needs. This would require editors to weight progressively less prominently articles in Science / Nature, then in Physical Review Letters / Europhysics Letters, then Physical Review X ... ending with zero consideration for papers in Medical Hypotheses or Creation (except per WP:SPS). There are a great number of journals slushing around at the base of this pyramid who are desperate for submissions and relevancy. Some of their papers are just fine (at least, I have no qualms about citing a paper I for some reason submitted to a journal to which not even LANL subscribes; meh), but the degree of pre-publication oversight is substantially less, so they need to be granted less consideration. If papers in the elite journals provide sufficient depth of coverage, it should not be necessary to bring in the Publish-or-Perish-Pile (PoPP) to this tertiary source.
- Instances where nobody reputable will touch the issue, or even will have heard of it, are more problematic. Perhaps WP:REDFLAG could be elaborated upon to spell out more clearly that, even when low quality sources are the only ones dealing directly with a particular narrowly-defined issue, they may be excluded or minimized based on high quality sources dealing with a broader issue. This is particularly a problem with so-called energy therapies - everyone and their duck is free to suffer a revelation and open a clinic. None of this changes the fact that putative energy does not exist but the first law of thermodynamics does.
- Certainly, publishing journal alone is not sufficient to ensure quality (and claims that everything Elsevier publishes should be weighted per The Lancet are frankly ludicrous)- we must also examine critical acceptance. This needs to be spelled out in WP#Scholarship and enforced. I hesitate to recommend for instance a moratorium on citations for one year from publication date (as a 'challenge and remove' mechanism, presumably - instruction creep, and all that), but we have editors from widely diverse backgrounds. Not all of them (well, us - medical and medical denialist papers read significantly differently from physics papers, and I would be surprised if there are not nuances which I miss) are capable of critically evaluating a study to assign a meaningful trust metric.
- Which, really, is the root of our problem - at least the contribution-weighted majority of the editors in this little fracas can cite WP:UNDUE, WP:FRINGE, WP:REDFLAG, &c. ad nauseam without agreeing on the application. -
Eldereft ~(
s)
talk~
17:32, 25 April 2008 (UTC)
- Small studies certainly have their limitations, but when a body of small studies add up, a pattern can be recognized. It is like the body of research at
Arsenicum album, some of which were blind trials (both animal and human), and some of which were not. And while I recognize that double-blind placebo controlled trials in mice, rat, and human studies are preferred, the unblinded animal studies still provide useful and even reliable data and information, unless you feel that the psychic power of knowing which mouse/rat got the homeopathic doses of Arsenicum was powerful enough to elicit more healing in them and help to eliminate more arsenic through their stools and urine. The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know. By the way, the other important and inadequately appreciated issue is the problem of the large studies. The vast majority of the large clinical human trials in homeopathy give just one medicine to patients with vastly different symptoms even though they may have the same disease. This "test" of homeopathy is only effective in a small percentage of situations. It is not just happenstance that homeopaths respect biochemical/biological individualization. The system is based on it. People who really know statistics know that a good study has to have both internal validity AND external validity (how come so many smart scientifically thinking people here are ignoring this vital NPOV.
DanaUllman
Talk
06:12, 9 May 2008 (UTC)
- A Woody Allen joke comes to mind: Two women are at a restaurant. One says to the other, "My, the food here is awful!" The other says, "Yes, and what small portions!" In case you don't get it, the lesson is that high quantity does not make up for low quality. Aside from that, I'm not going to waste more time explaining the placebo effect to you, Dana. I've tried before and you learned nothing; I don't see what's different now. -- Infophile (Talk) (Contribs) 16:45, 9 May 2008 (UTC)
- Small studies certainly have their limitations, but when a body of small studies add up, a pattern can be recognized. It is like the body of research at
Arsenicum album, some of which were blind trials (both animal and human), and some of which were not. And while I recognize that double-blind placebo controlled trials in mice, rat, and human studies are preferred, the unblinded animal studies still provide useful and even reliable data and information, unless you feel that the psychic power of knowing which mouse/rat got the homeopathic doses of Arsenicum was powerful enough to elicit more healing in them and help to eliminate more arsenic through their stools and urine. The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know. By the way, the other important and inadequately appreciated issue is the problem of the large studies. The vast majority of the large clinical human trials in homeopathy give just one medicine to patients with vastly different symptoms even though they may have the same disease. This "test" of homeopathy is only effective in a small percentage of situations. It is not just happenstance that homeopaths respect biochemical/biological individualization. The system is based on it. People who really know statistics know that a good study has to have both internal validity AND external validity (how come so many smart scientifically thinking people here are ignoring this vital NPOV.
DanaUllman
Talk
06:12, 9 May 2008 (UTC)
Infophile, in serious and due respect, your posting reeks of bad faith. If I were to make a similar statement, you and others would rake me over the coals. At the same time, I love your sense of humor, especially your discussion of the placebo effects. I too have a respect for the placebo effect, except in mice and rats. Can you say with a straight face that the mice and the rats are influenced greatly because the experimenter knows which rat/mouse was given a homeopathic medicine? Because the experiments measure arsenic levels in their bodies, I'm impressed that you feel that the experimenter's belief is powerful enough to influence these levels. I think that you are much more radical in your thinking than I. DanaUllman Talk 03:55, 10 May 2008 (UTC)
- Would you mind explaining exactly how that constitutes bad faith? I'm not assuming that you're intentionally misleading us here, which would indeed be bad faith. Disagreeing with you is in no way a show of bad faith, nor is believing that most of the studies that show positive effects for homeopathy are poor quality.
- On the subject of the placebo effect, I'm not an animal researcher (and I don't believe you are either), so I'm not going to pretend I know more than they do about the reactions of animals. In this case, I just go to the results of the literature. A selection of examples:
[1]
[2]
[3]. Your efforts would be better focused looking at the results there and determining what, if any, flaws might be present in the studies. --
Infophile
(Talk)
(Contribs)
17:56, 10 May 2008 (UTC)
- Infophile, wow, the most recent of the 3 references that you cite above is from 1971! Only 1 of the 3 references included an abstact, and the article/study was nothing (!) like any of the homeopathic trials to which I referenced. The study you cite is when a rat population was given a real drug and then a placebo, and the rats seemed to respond to the placebo in the same way that they responded to the drug because both were given in injection form. I never referred to any similar design as that in my references. To which homeopathic reference are you referring? Please cite and please retract. As for my statement about your "bad faith," I wrote that because you claimed to have tried to explain the placebo effect to me in the past and "learned nothing." This statement assumes that I do not have a knowledge of or an appreciation for the placebo effect, an assumption is that is not based on anything (that is not assuming good faith).
DanaUllman
Talk
00:34, 12 May 2008 (UTC)
- On the bad faith issue, you said, "The results were either because of the homeopathic medicine OR the psychic "knowing" (if it is the latter, should we report on this in a wiki article on psychic phenomena? Curious minds want to know." You're obviously mischaracterizing the placebo effect as something "psychic" here, and not even recognizing that you're talking about it. That's why I assumed you didn't know about it or appreciate it - at least in the case of animals. As for the studies... I'm baffled as to why you assume they should resemble the homeopathic studies in question. I'm using them to point out that animals can show the placebo effect. They don't need to be of any form other than some that would show this result. On the issue of their age... so? The effect is well accepted by animal researchers. There's simply been no reason to pursue the question recently.
- But really, I think it's about time to drop this. Nothing's going to be gained by continuing this argument here. All of these content disputes around the arbitration case are misplaced at best. Let's save the effort for the homeopathy talk pages, where the results might at least make a difference. -- Infophile (Talk) (Contribs) 03:37, 12 May 2008 (UTC)
- Infophile, wow, the most recent of the 3 references that you cite above is from 1971! Only 1 of the 3 references included an abstact, and the article/study was nothing (!) like any of the homeopathic trials to which I referenced. The study you cite is when a rat population was given a real drug and then a placebo, and the rats seemed to respond to the placebo in the same way that they responded to the drug because both were given in injection form. I never referred to any similar design as that in my references. To which homeopathic reference are you referring? Please cite and please retract. As for my statement about your "bad faith," I wrote that because you claimed to have tried to explain the placebo effect to me in the past and "learned nothing." This statement assumes that I do not have a knowledge of or an appreciation for the placebo effect, an assumption is that is not based on anything (that is not assuming good faith).
DanaUllman
Talk
00:34, 12 May 2008 (UTC)
Wikipedia:Requests_for_arbitration#Request_to_amend:_Wikipedia:Requests_for_arbitration.2FPseudoscience_and_Wikipedia:Requests_for_arbitration.2FMartinphi-ScienceApologist. has the potential to affect editors and topics involved in this case. Vassyana ( talk) 12:55, 2 May 2008 (UTC)