This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page. |
Archive 5 | ← | Archive 9 | Archive 10 | Archive 11 | Archive 12 | Archive 13 | → | Archive 15 |
I would like to get the group’s feedback on a proposed addition to MEDRS. MEDRS currently covers the topics of Evidence (factual data) and Scientific Consensus (largely the interpretation of that data) in separate sections. In current practice, we commonly treat a source that is high quality for Evidence as being of equally high quality as a source for Scientific Consensus. Below I argue that this is not always true and that MEDRS should be slightly modified to recognize this.
What MEDRS currently says about Evidence
What MEDRS currently says about Scientific Consensus
Proposed modification to the section on Scientific Consensus
Justification
Examples of how this is an issue in actual articles
I’d like to emphasize that I’m not suggesting that minority opinions be edited out, but only that treatment guidelines be give some precedence in determining what mainstream opinion is. Because they are consensus documents written by large groups of acknowledged experts, the use of these guidelines as a favored source helps reduce "noise", cherry picking, and helps to assure that the opinions are those of acknowledged experts. The fact that they are issued by large specialist societies or major healthcare agencies helps assure that that they are widely endorsed and do in fact represent Scientific Consensus. Thank you in advance for your considered opinion. Formerly 98 ( talk) 18:16, 18 November 2014 (UTC)
amended proposal based on feedback above.
I would also add the Cochrane collaboration to that list of scientific bodies and than it has my support. Is this replacing some text or simply being added? Doc James ( talk · contribs · email) 01:09, 19 November 2014 (UTC)
It may be that I am slicing things a bit too thinly here by separating out Evidence (data) and opinion on what that data means and how it should be used (my understanding of what Scientific Consensus is). But the problem I keep coming back to is two authors who are fairly junior researchers expressing opinions that did not derive directly from mathematical analysis of the historical trial record, and these opinions being used to rebut or offset consensus treatment guidelines endorsed by professional medical specialty societies. They have a right to their opinions, they may be right. But while they seek to influence Scientific Consensus, they don't necessarily represent it. Formerly 98 ( talk) 01:38, 19 November 2014 (UTC)
funny as I wrote User:Casliber/Fivecrat in November 2008.
@ Doc James: I personally think Cochrane is always an important voice, and would always mention their position on any subject they have covered. But when they disagree with the consensus treatment guidelines that have been endorsed by the 84,000 member APA, they become an important minority viewpoint, and consequently get less weight than the majority opinion. The APA makes a point of generating consensus guidelines. The Cochrane groups's job is not to generate consensus documents, but to investigate and make recommendations. They are in fact at their most valuable when they challenge the current consensus/status quo.
Thats what I think. If you disagree, your thoughts on how your position fits with WP:NPOV when the Cochrane position is in the minority would be helpful here I think. Formerly 98 ( talk) 05:36, 19 November 2014 (UTC)
I absolutely agree on presenting all. I'm just trying to get to weight. Formerly 98 ( talk) 05:55, 19 November 2014 (UTC) My understanding is that NPOV requires weighting according to prevealence of the viewpoint among experts. Is this not correct? — Preceding unsigned comment added by Formerly 98 ( talk • contribs) 05:56, 19 November 2014
I think the second draft above is a real improvement on the current wording. "Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting" ensures that contradictory Cochrane conclusions won't be muscled out. Regarding weight, if one of their conclusions challenges the existing consensus, I hope we'd give it enough prominence to be sure the reader won't miss it. -- Anthonyhcole ( talk · contribs · email) 10:25, 19 November 2014 (UTC)
I have lost faith in Cochrane after seeing what a disaster they made of their review of the ketogenic diet (see Talk:Ketogenic diet#Incorporating 2012 systematic review). The introduction to that review, covering history, makes a number of factual errors. Their analysis and summary of the studies is flawed and very badly presented. The authors, from what I can see, are not experts in the subject and work in local district hospitals (one wonders if they actually have any personal experience of that treatment since their hospitals would be unlikely to offer it). Cochrane's formulaic approach may work when there is sufficient data or when the authors are wise enough to admit they have nothing useful to say. Here is an example of authors over extending themselves and producing a paper that is only fit to line a hamster's cage. The difference with a body of experts (or a combination of experts + other parties such as patient groups such as we see with SIGN) producing evidence-based treatment guidelines is that they are required to produce a document that has some utility. They aren't engaging in a mechanical process, ticking the systematic review check boxes, and churning out something of no practical use. And not all aspects of treatment (drugs, protocol, review) are studied by RCTs so there is still a place for expert consensus opinion -- something Cochrane's work cannot supply.
I think there is a danger that our "treatment" sections become OR when we rely on academic papers that do not actually describe current treatment nor have any authority behind their opinions to guide ideal treatment. We are all fans here of evidence based medicine, but there is a danger our enthusiasm for this leads to a gap with reality. Our readers need to learn how diseases are actually or at least expected to be treated alongside learning about the strength of the evidence behind these treatment options. Therefore I do not support Cochrane being given equal status to official or consensus treatment guidelines for sourcing our "treatment" sections. Btw SIGN is the Scottish equivalent of NICE (England + Wales) and their work is a useful online source. -- Colin° Talk 10:39, 19 November 2014 (UTC)
Cochrane is widely considered one of the best available sources for evidence. This is not to say they're perfect, because "best" is a relative and not absolute term, but they rank at or near the top for evaluation of the current state of evidence and what conclusions can be drawn. Medical associations are as well imperfect, and their positions can be subject to influence from factors other than objective evaluation of the evidence, such as politics. This is a strong reason why Cochrane's position as an independent organization must not be devalued.
Zad
68
16:14, 19 November 2014 (UTC)
Thank you everyone for the input. I hear the concerns about the value of Cochrane and would like to repeat that I never suggested that we include only a statement from a single group in the articles or that we exclude Cochrane, and the proposed language does not suggest this in any way. I do believe that MEDRS should reflect WP:NPOV which is a pillar that cannot be modified or abrogated by the Projects. WP:NPOV requires that we weight viewpoint based on their prominence, and not on our personal opinions as to which source is "best".
To address these concerns, I suggest modifying the language as follows. "Wikipedia policies on the neutral point of view and not using primary research demand that we present the prevailing medical or scientific consensus. Statements and practice guidelines issued by major professional medical societies (For example, the American Diabetes Association, the European Society of Cardiology or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,AHRQ, NICE, USPTDF, and WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, monographs, review articles and meta analyses. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the Cochrane Collaboration).
Thanks all again. I hope this will reduce some concerns and look forward to hearing your thoughts. Formerly 98 ( talk) 17:18, 19 November 2014 (UTC)
MastCell's parachute article is funny but there is a danger that parody is not taken seriously. Let's pick a real example, and I'm afraid it has to remain on the topic of epilepsy since that's something I can actually talk about, albeit with my small lay brain. The first thing our epilepsy article does when mentioning specific drugs is to comment on their efficacy. It cites three Cochrane reviews. The article does go on to list practice guidelines, citing NICE, but the important thing is that three Cochrane reviews are cited and used before anything else. So the first thing a reader wanting to know about the drug treatment for epilepsy learns is that "Phenytoin, carbamazepine and valproate appear to be equally effective in both focal and generalized seizures. Controlled release carbamazepine appears to work as well as immediate release carbamazepine, and may have fewer side effects." This obsession with discussing efficacy trial results permeates our medical articles, and comes from the fact that we over-laud systematic reviews (and others over-use primary research papers) as though our articles should be founded upon them.
So let's consider if one cited Cochrane review actually supports the statement "appear to be equally effective in both focal and generalized seizures" for phenytoin and carbamazepine. The paper is PMID 12076427 and the full text " Carbamazepine versus phenytoin monotherapy for epilepsy". Firstly, the review only looked at the drugs "when used as monotherapy in subjects with partial onset seizures, or generalized onset tonic-clonic seizures with or without other generalized seizure types." -- which is not all "generalized seizures" (e.g. absence seizures and atonic seizures are different) and not all patients are treated with monotherapy. Secondly, consider the primary outcome is "Time to withdrawal of allocated treatment". This is one recommended measure of epilepsy treatment that attempts to balance whether the drug is effective at reducing seizures and whether the side effects are tolerable. However it is a proxy for effectiveness and side-effects and not without its problems. Still, have a good read of the article. As you read, your heart should sink as study after study is thrown away, data is found to be missing, studies looked at different endpoints from this one's, or studies had methods that aren't what we'd wish for, or only examined seizure types that aren't as general as the author's wanted. In the end, the authors decided not to use the studies' findings but to look at the individual patient data where they could obtain it. They got this for just three trials. One trial looked at children and two looked at adults. One trial looked at partial onset seizures only and the other two at patients with either partial onset seizures or generalized tonic-clonic seizures. These are not patient groups that are typically combined for analysis (heterogeneous). Only one was double-blinded. Is "buggered if I know" a fair summary of the review conclusion?
The limitations of their proxy for efficacy+tolerance become apparent when you read the background section of the paper. The experts have already concluded that these drugs are much of a muchness wrt reducing seizures and are more concerned with their side effect profiles. Are these side effects the kind that influence patients asking to stop a medication during a trial? No. Phenytoin has long term problems with facial disfigurement. Both drugs may very rarely kill their patients with a nasty alergic rash. Carbamazepine may cause neural tube defects in babies, though neither drug is without its pregnancy issues. None of these concerns that the author say influence drug choice in clinical practice are ones that a short-term efficacy study of 50 patients is likely to discover.
We have a Cochrane review that has discovered aspects of our ignorance but otherwise does not support the statement that these drugs are equally effective, nor does it even attempt to answer the implied statement that both these drugs are effective.
Before anyone trumpets or recommends Cochrane or indeed any form of literature or sourcing, one must first consider what statement one is trying to say in article text. I recommend that for disease treatment/management sections, the primary focus should be to source information that describes what the current treatment is and what our best expert minds consider is the best clinical practice. For drugs, the primary consideration is for what the drug is licensed; for what it is a recommended treatment by authorities; and for what it might be widely used even if lacking evidence or in the face of disapproval (e.g. cough medicine). An academic paper that effectively says "we cannot answer the question" is not a source for very much at all. -- Colin° Talk 23:11, 19 November 2014 (UTC)
Zad
68
16:23, 20 November 2014 (UTC)
I've been stewing on this Cochrane issue for a while now. I know we have a formal collaboration with them and that is a good thing, for sure! But i have concerns similar to those of Formerly and Colin. The specific proposal by Doc James above, was to add "Cochrane Collaboration" to the list of sample major scientific/medical organizations. I for one struggle with listing it as an organization for several reasons:
So, no we should not list CC as an example organization with NICE, AHRQ, etc. yes it would be great to name them as a producer of meta-analyses, as they are the premier generator of them. that's my view on the question at hand. Thoughts?? Jytdog ( talk) 16:31, 20 November 2014 (UTC)
How much does NICE or the AHRQ or the APA dictate practice? These are just guidelines that physicians may follow to various degrees. They also may or may not be adopted by the mainstream. Look at antibiotics for the acute bronchitis. Little evidence supports their use. No guidelines support their use. Yet they are commonly used. We basically have three things 1) what the evidence shows 2) what the guidelines say 3) what physicians do. Cochrane reviews are not primary research papers. Doc James ( talk · contribs · email) 20:35, 20 November 2014 (UTC)
ok, i took a shot at implementing the proposal. Here is the change to the original language:
Wikipedia policies on the
neutral point of view and
not using original research demand that we present any prevailing
medical or
scientific consensus, which can be found in recent, authoritative
review articles or in textbooks or in some forms of
monographs. Statements and
practice guidelines issued by major professional medical or scientific societies (For example, the
American Diabetes Association, the
European Society of Cardiology, or the
Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,
AHRQ,
USPSTF,
NICE, and
WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, some forms of
monographs,
review articles, and
systematic reviews that include
meta-analysis. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate
weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the
Cochrane Collaboration). Although significant-minority views are welcome in Wikipedia, such views must be presented in the context of their acceptance by experts in the field. Additionally, the views of tiny minorities need not be reported.
Hopefully this will fly... Jytdog ( talk) 20:37, 23 November 2014 (UTC)
Yup likely trying to achieve too much. Reviews by Cochrane are of the same caliber as those from AHRQ and USPSTF. They often represent a more international perspective while AHRQ and the USPSTF represent a more American POV. Have edited to remove this device. Doc James ( talk · contribs · email) 04:18, 28 November 2014 (UTC)
It seems a little unfair to me to suggest that I am raising this issue based on content disputes on a handful of pages. It has the effect of trivializing my position without actually addressing the points that I've raised. Its very effective as a rhetorical technique, but I'd like to think we can address each others concerns and points more directly than this based on the mutual respect we have developed working together.
James asks a very good question: "Why do you think this change is needed? What issues do you see on Wikipedia that need this change to be addressed?" Let me try to answer that.
With respect to Colin's comments, I'd be interested to know if he has a specific policy position that he is supporting here. I've seen a lot of fairly harsh criticism of both my comments and James, which has sometimes bordered on ridicule, but if you have offered a specific alternative, I somehow missed it. Criticism in the absence of offering a meaningful alternative doesn't really help us reach a consensus.
Clinical guidelines from organizations the ultimate? For my two cents, I too agree that clinical guidelines from groups of experts are probably the best we’re going to be able to do as a simple algorithm for what is representative of NPOV in clinical medicine. You can secondarily mention the Cochrane Library and database and use these, of course, but remember that many clinical guideline formal recommendations use Cochrane data also, so you get that for free. For example, NICE has cited Cochrane many times as evidence which they used to formulate their guidelines. Not as much as Cochrane wishes, no doubt, but they do. [5]. So cite Cochrane particularly if your guidelines miss some nice Cochrane review you found (perhaps a later one), or it contradicts the guidelines. And while we are at it, we should remember that Cochrane is not the only source of clinical trial meta analysis! Sometimes something just doesn’t exist as a Cochrane meta analysis, like (say) the effect of fibrates on cardiovascular outcomes. No such review exists (and yes, that’s shocking, but illustrative). You CAN get a meta analysis that used the Cochrane library of fibrate trials, as well as MEDLINE and Embase, BUT using the library of trials kept by Cochrane does not make this a Cochrane review, with the Cochrane label. But this one is published in LANCET and is perfectly useable as a meta analysis. [6] The problem is knowing how to translate it into guidelines. NICE guidelines really don’t recommend fibrates for anything (maybe horrible primary triglyceridemia), but the meta analysis says they lower severe MI incidence by 10%, without impact on mortality. So now what? I’ll save that for later.
Cochrane not enough. Guidelines often utilize data from epidemiology and even animal and pathophysiology data, which Cochrane (as a strictly clinical trial evidence entity) does not. This is good and bad. The good is that it lets you include recommendations for having a parachute when you jump from a plane (very funny example above). The downside is that pathology and proxy marker data can fool clinicians into prescribing drugs that are no good (examples on request if you don’t have favorite ones of your own—mine is encainide). This problem is why some docs used niacin as a cardiac preventive up until mid-July, but there were also problems of underpowered studies with niacin, and other studies just contradicted each other.
Other meta analysis available. We need to remember that what Cochrane brings to the table is a sort of “guarantee” (warrantee?) that they don’t let chemists do reviews of surgery articles. I had more confidence in them before reading user:Colin's Cochrane complaints above, but what can you do? Again, product quality problems cannot be detected until you are a sophisticated user; sometimes just the GG Gucci label on something is not enough, because it get stamped on crap sometimes. You can use impact factors, but good journals print crap, too. And are biased. And print biased stuff. The routine male circumcision wars in JAMA a few years ago were hilarious, partly because each side took itself so medically and scientifically seriously, except the angry letters and reviews were always urologists Smith, Jones and Lee vs. urologists Cohen, Cantor and Mohel on the opposing view.
Primary studies often contain secondary material of high quality. When you have an article that is from a very large and long clinical study that is nearly definitive and is unlikely to be repeated soon, I think it’s a little silly not to simply go to that primary article’s “discussion” and “conclusion” sections for your secondary material (even primary science articles have secondary-discussion and synthesis parts, per the meanings of “secondary” in WP:RS). That’s another topic for discussion in MEDRS. It’s ridiculous to wait until an article in the NEJM is picked up and reviewed and compared to previous studies by the Lake Wobegon Family Practice Quarterly, when the authors in the NEJM almost surely did a better job in the original, biased though they may have been. And the editorial NEJM staff probably oversaw their conclusions with a more jaundiced eye than the Lake Wobegon editor, who was glad to just have enough material to make his page count.
Epistemology always bites when on the internet, nobody can tell you’re a dog. Ultimately we came back to the epistemological question I complained of several years ago, which is that you cannot write a good NPOV science article without appeal to authority, and that appeal rests on quicksand at some point, since (without ability to do a background check of any type) you very quickly run out of authoritative views on who is an authority. And if you make it that far, you won’t be able to find authorities on who is the authority on finding authorities. At some point, you cannot avoid just taking polls of the schmucks who post here. From our mother’s basements. Ultimately, it’s impossible to write a technical WP article with a reasonable NPOV without breaking guidelines on whether or not you should trust certain editors who write the things. You watch to see if the editor fixing the Lie Algebras article looks like their LaTeX is up to snuff (or broken and ugly), and you sense from their TALK input whether or not they know what they are talking about, mathematically. Something cannot be done by cites, or cites of cites, or cites of meta reviews of reviews of primary cites. The other way (editor trust) is actually how most good articles on WP get written, though we deny it. So I’m here (as usual) to remind you all of the fundamental hypocrisy that is Wikipedia. If Wikipedia were actually written the way that it is supposed to be (no recognizable editors involved—we randomize all usernames with each edit, so there are no possible reputations), Wikipedia would be a lot crappier than it is. However, we use longstanding usernames, and we get to know them (and can look up their contribution histories). And that works as a better proxy than we’d like to admit, for the editor CVs we lack. S B H arris 00:57, 26 November 2014 (UTC)
====Re-poll====
OK, I think that @
Yobol: has just backed us up to a version that is less supported than the version that we had before his edit, so I'm going to do a poll. Please let me know if the following is roughly correct:
I have the following people having expressed the opinions below at some point. Let me know if this is correct or not:
Mostly agree that treatment guidelines are an "ideal" source for determining Scientific Consensus, and by implication, should get more weight (in the sense outlined in
WP:NPOV) than other secondary sources, which are not consensus documents. As near as I can tell, this group opposes calling out Cochrane as a special source that should always be called out alongside the consensus treatment guidelines.
The following appear to feel that Cochrane should still get mention on an least an equal footing with consensus treatment guidelines, but seem to support emphasizing these (Consensus guidelines and Cochrane) over other secondary sources for determining Scientific Consensus:
Opinion unclear or difficult to categorize:
Is this a correct count?
I assumed that we were in agreement that the garden variety meta analyses and reviews should be weighted a little less than consensus treatment guidelines for determining Scientific Consensus, and were split on Cochrane. So I tried to split the difference on Cochrane, calling them an important minority voice when not in agreement with consensus treatment guidelines. Let me know what you all think.
Formerly 98 (
talk) 02:58, 25 November 2014 (UTC)
Please don't do this. We discuss and come to a consensus. We don't vote or tally opinions. If you believe "a slight plurality felt that Cochrane's views always deserved mention even when they were out of synch" then I couldn't really care if a slight plurality felt the moon was made of cheese. They would be wrong. Some policy reading required. -- Colin° Talk 12:55, 25 November 2014 (UTC)
The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
User:Jytdog has proposed changing:
to
Zad
68
02:50, 30 November 2014 (UTC)Meanwhile, why is this still in the guideline? SandyGeorgia ( Talk) 16:04, 5 December 2014 (UTC)
This proposal was initiated by my concern that meta analyses are being inappropriately used to indicate Scientific Consensus, when that is not their purpose. They play an important role, but due to the large number of subjective decisions made in their preparation and interpretation, invariably reflect the pre-existing biases of the authors. Per published meta analyses:
Treatment guidelines consider and integrate all these results, help provide expert analysis of the conflicting results, and reduce cherry picking. Formerly 98 ( talk) 12:58, 29 November 2014 (UTC)
I think it's a false choice, to say that we have to go with or prioritize a practice guideline against high-quality (sorry, Colin, yes I know there are a few exceptions to every rule), independent source of evidence like Cochrane. As mentioned above, practice guidelines can recommend a course of action in cases where the evidence is less than excellent, and individual organizations can be subject to political pressure or other biases that an independent reviewer might not be subject to. So, why wouldn't we use both? The article can say, "The evidence says this. NICE recommends that." if that's the case. If we're going to name names of organizations that produce practice guidelines, we should also names well-respected, influential sources of evidence. Can anyone argue Cochrane is any less well-regarded than the named bodies producing practice guidelines?
Zad
68
03:05, 30 November 2014 (UTC)
Zad
68
03:16, 30 November 2014 (UTC)
Am I meeting your point or still talking past?
Zad
68
19:52, 30 November 2014 (UTC)
It appears that the question is how should we balance EBM and expert opinion.--Yes, exactly how I see it, and:
Some feel one should get greater weight over the other and vise versa. My position is they should often have equal weight or that at least weight should be determined on a case by case basis.--Yes, exactly. I still think there's an incorrect assumption underlying some of the comments in this discussion that it's either one or the other, or that one has to be prioritized over the other. I really see a top-tier source of evidence like Cochrane and a top-tier authority's practice guidelines as right there next to each other, neither should be prescriptively prioritized over the other, and the use of each, including the weight, needs to be handled on a situation-by-situation basis.
Sbharris wrote (above) a really awesome, laying-the-truth-bare mini-essay on how the sausage really gets made here, and I'm disappointed it really didn't get any traction.
Zad
68
19:20, 30 November 2014 (UTC)
Guidelines and systematic reviews are different things typically. Guidelines often address a much broader question while systematic reviews address a much narrower one. Thus while guidelines often have more authors all these authors many not be looking extensively at each question. Thus it may be less authors than a systematic review. Not that I am suggesting we determine weight by how many authors a review or guideline has. Doc James ( talk · contribs · email) 03:11, 1 December 2014 (UTC)
The wiki ganfyd states here that activated protein C is grade A recommendation for pneumonia based on a 2008 guideline from British Society for Antimicrobial Chemotherapy. The ref is here [16]. However a 2011 Cochrane review found that it was associated with worse outcomes. [17] It took the FDA 6 months to pull the drug at which time UpToDate updated their recommendations. Wikipedia reflected the best available evidence within a couple of days of it coming out. Doc James ( talk · contribs · email) 12:44, 30 November 2014 (UTC)
In addition to the embarrassing nonsense about quaternary sources, the above cherry-picked example could (a) be repeated (with other studies/papers) as an example of why Wikipedia should cite primary research papers rather than wait around for reviews or textbooks to pick up on the latest findings and (b) merely shows that new publications tend to include more up-to-date information, which is a tautology. We are looking for sources that reliably indicate medical scientific consensus on a topic, because we are an encyclopaedia not a newspaper. We are limited by the speed at which reliable publications pick up and report on this consensus. If we were written by named expert authorities in the subject, then we could be both up-to-date and reliably cite the latest research findings. We aren't and we don't. -- 11:46, 7 December 2014 (UTC)
The primary/secondary sourcing terminology for science differs to that for humanities. The first publication of new research based on previously unpublished data is a primary source. So if (as in the Epilepsy paper used above) a Cochrane review decides to escape the confines of analysing previously published results and instead sources the raw unpublished patient records upon which to perform novel analysis, then this is a primary source. In addition, "all sources are primary for something". And since science always builds on the work of others, and this work is frequently referred-to in a research paper, all primary research papers will include some secondary-source material. It really depends what you take from the paper and how you use it on Wikipedia. So, in the words of a notable EBM advocate, " I Think You'll Find It's a Bit More Complicated Than That" -- don't oversimplify the argument.
We all know that if anyone is likely to produce a "There isn't enough high quality data available" result in their publication it is a Cochrane review. That's why sceptic bloggers battling against pseudo science and snake oil merchants love to cite them. But back in the hospital ward, the physician needs to know what drug to give a child for a certain problem but who additionally has a serious metabolic condition, even though the only EBM randomised controlled trial is on young otherwise-healthy men. Nobody conducts randomised controlled trials on pregnant women to see which drugs cause the most defects in their babies. We have an RCT on refractory epilepsy treated with the ketogenic diet but absolutely no RCT studies on refractory epilepsy (AFAIK, PMID 22170887) treated with drugs (after all, what manufacturer would choose that patient group). Yet the physician, faced with a child who has failed three drugs already, must choose a fourth or some other treatment. To claim, as proposed below, that several RCTs combined in systematic reviews is required before making any guidance or reporting something on WP, is just on another planet.
The old chestnut of "expert opinion" vs "evidence based medicine" is repeated and misused. When these are compared like this, the implication is that the "opinion" is "mere opinion", i.e., not based on good evidence, but rather on either prejudice, limited personal experience or plain old guesswork. If Wikipedia was based only on evidence, then we'd cite nothing but primary research papers and systematic reviews and discard literature reviews, textbooks, clinical guidelines and consensus statements. But then we'd need to be authored by respected named experts who could interpret that evidence properly. Because the evidence is not as general as we need for WP article. We have evidence that of a 200 patients treated in a certain trial by a certain hospital according to a certain protocol and given a certain dose of a drug, that 100 got better. And our science may tell us to what confidence we can then attribute that to the drug. And we can gather data on the side effects seen in this short-term trial. Is "half the patients getting better" an effective drug? Depends on the condition. I wouldn't be happy if my optician only prescribed glasses that worked half the time. On the other hand, some cancers never respond anything like that well. If one in a hundred of those patients got liver failure and died, that might be acceptable for fatal cancer but not for irritable bowel syndrome. The decision about what makes a good drug and therefore worthy of mention in the "treatment" or "management" section of our disease articles, is one of expert opinion. So it always ultimately comes down to the opinion of experts and what they have written in reliable publications. We trust that the best experts and best publications will ensure that opinion is based as much on high-quality evidence as possible, but it is not Wikipedia's job to fix that if we disagree with their conclusions. And WP:WEIGHT requires us to respect the (scientific) consensus, which requires secondary sources to determine.
It is clear that Cochrane papers can be a good source of evidence. But where the review eliminates so much and find that what is left is insufficient to make a positive statement, we must treat it with caution. It could simply be that this "we know nothing => can't recommend" is an artefact of their rather extreme methodology, and actual clinical practice says otherwise. Where their papers turn out to be written by non-experts or deviate from performing a systematic review of the published literature, we must treat it with great caution. But ultimately, their papers are written to influence medical opinion and be a good source of evidence for experts to make decisions. But it is that medical opinion and expert decision-making that we report. -- Colin° Talk 11:46, 7 December 2014 (UTC)
- A1candidate ( talk) 10:06, 30 November 2014 (UTC)
- A1candidate ( talk) 10:06, 30 November 2014 (UTC)
Proposed changes are probably more concise and informative. - A1candidate ( talk) 10:06, 30 November 2014 (UTC)
Zad
68
19:07, 30 November 2014 (UTC)
MastCell pointed to a well-known humorous essay Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. There is of course a grain of truth in the humor, but to see how we might handle it: Every medical society around the world strongly recommends parachute use for those who jump out of airplanes(cite review of practice guidelines worldwide), although no trials of the type that can produce the best kinds of evidence have been conducted into parachute use, due to ethical concerns in conducting those kinds of trials.(cite Cochrane)
Yes, taking reductio ad absurdum for face value is even more absurdum but as before there's a grain of truth in it.
Zad
68
19:34, 30 November 2014 (UTC)
This is an archive of past discussions. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page. |
Archive 5 | ← | Archive 9 | Archive 10 | Archive 11 | Archive 12 | Archive 13 | → | Archive 15 |
I would like to get the group’s feedback on a proposed addition to MEDRS. MEDRS currently covers the topics of Evidence (factual data) and Scientific Consensus (largely the interpretation of that data) in separate sections. In current practice, we commonly treat a source that is high quality for Evidence as being of equally high quality as a source for Scientific Consensus. Below I argue that this is not always true and that MEDRS should be slightly modified to recognize this.
What MEDRS currently says about Evidence
What MEDRS currently says about Scientific Consensus
Proposed modification to the section on Scientific Consensus
Justification
Examples of how this is an issue in actual articles
I’d like to emphasize that I’m not suggesting that minority opinions be edited out, but only that treatment guidelines be give some precedence in determining what mainstream opinion is. Because they are consensus documents written by large groups of acknowledged experts, the use of these guidelines as a favored source helps reduce "noise", cherry picking, and helps to assure that the opinions are those of acknowledged experts. The fact that they are issued by large specialist societies or major healthcare agencies helps assure that that they are widely endorsed and do in fact represent Scientific Consensus. Thank you in advance for your considered opinion. Formerly 98 ( talk) 18:16, 18 November 2014 (UTC)
amended proposal based on feedback above.
I would also add the Cochrane collaboration to that list of scientific bodies and than it has my support. Is this replacing some text or simply being added? Doc James ( talk · contribs · email) 01:09, 19 November 2014 (UTC)
It may be that I am slicing things a bit too thinly here by separating out Evidence (data) and opinion on what that data means and how it should be used (my understanding of what Scientific Consensus is). But the problem I keep coming back to is two authors who are fairly junior researchers expressing opinions that did not derive directly from mathematical analysis of the historical trial record, and these opinions being used to rebut or offset consensus treatment guidelines endorsed by professional medical specialty societies. They have a right to their opinions, they may be right. But while they seek to influence Scientific Consensus, they don't necessarily represent it. Formerly 98 ( talk) 01:38, 19 November 2014 (UTC)
funny as I wrote User:Casliber/Fivecrat in November 2008.
@ Doc James: I personally think Cochrane is always an important voice, and would always mention their position on any subject they have covered. But when they disagree with the consensus treatment guidelines that have been endorsed by the 84,000 member APA, they become an important minority viewpoint, and consequently get less weight than the majority opinion. The APA makes a point of generating consensus guidelines. The Cochrane groups's job is not to generate consensus documents, but to investigate and make recommendations. They are in fact at their most valuable when they challenge the current consensus/status quo.
Thats what I think. If you disagree, your thoughts on how your position fits with WP:NPOV when the Cochrane position is in the minority would be helpful here I think. Formerly 98 ( talk) 05:36, 19 November 2014 (UTC)
I absolutely agree on presenting all. I'm just trying to get to weight. Formerly 98 ( talk) 05:55, 19 November 2014 (UTC) My understanding is that NPOV requires weighting according to prevealence of the viewpoint among experts. Is this not correct? — Preceding unsigned comment added by Formerly 98 ( talk • contribs) 05:56, 19 November 2014
I think the second draft above is a real improvement on the current wording. "Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting" ensures that contradictory Cochrane conclusions won't be muscled out. Regarding weight, if one of their conclusions challenges the existing consensus, I hope we'd give it enough prominence to be sure the reader won't miss it. -- Anthonyhcole ( talk · contribs · email) 10:25, 19 November 2014 (UTC)
I have lost faith in Cochrane after seeing what a disaster they made of their review of the ketogenic diet (see Talk:Ketogenic diet#Incorporating 2012 systematic review). The introduction to that review, covering history, makes a number of factual errors. Their analysis and summary of the studies is flawed and very badly presented. The authors, from what I can see, are not experts in the subject and work in local district hospitals (one wonders if they actually have any personal experience of that treatment since their hospitals would be unlikely to offer it). Cochrane's formulaic approach may work when there is sufficient data or when the authors are wise enough to admit they have nothing useful to say. Here is an example of authors over extending themselves and producing a paper that is only fit to line a hamster's cage. The difference with a body of experts (or a combination of experts + other parties such as patient groups such as we see with SIGN) producing evidence-based treatment guidelines is that they are required to produce a document that has some utility. They aren't engaging in a mechanical process, ticking the systematic review check boxes, and churning out something of no practical use. And not all aspects of treatment (drugs, protocol, review) are studied by RCTs so there is still a place for expert consensus opinion -- something Cochrane's work cannot supply.
I think there is a danger that our "treatment" sections become OR when we rely on academic papers that do not actually describe current treatment nor have any authority behind their opinions to guide ideal treatment. We are all fans here of evidence based medicine, but there is a danger our enthusiasm for this leads to a gap with reality. Our readers need to learn how diseases are actually or at least expected to be treated alongside learning about the strength of the evidence behind these treatment options. Therefore I do not support Cochrane being given equal status to official or consensus treatment guidelines for sourcing our "treatment" sections. Btw SIGN is the Scottish equivalent of NICE (England + Wales) and their work is a useful online source. -- Colin° Talk 10:39, 19 November 2014 (UTC)
Cochrane is widely considered one of the best available sources for evidence. This is not to say they're perfect, because "best" is a relative and not absolute term, but they rank at or near the top for evaluation of the current state of evidence and what conclusions can be drawn. Medical associations are as well imperfect, and their positions can be subject to influence from factors other than objective evaluation of the evidence, such as politics. This is a strong reason why Cochrane's position as an independent organization must not be devalued.
Zad
68
16:14, 19 November 2014 (UTC)
Thank you everyone for the input. I hear the concerns about the value of Cochrane and would like to repeat that I never suggested that we include only a statement from a single group in the articles or that we exclude Cochrane, and the proposed language does not suggest this in any way. I do believe that MEDRS should reflect WP:NPOV which is a pillar that cannot be modified or abrogated by the Projects. WP:NPOV requires that we weight viewpoint based on their prominence, and not on our personal opinions as to which source is "best".
To address these concerns, I suggest modifying the language as follows. "Wikipedia policies on the neutral point of view and not using primary research demand that we present the prevailing medical or scientific consensus. Statements and practice guidelines issued by major professional medical societies (For example, the American Diabetes Association, the European Society of Cardiology or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,AHRQ, NICE, USPTDF, and WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, monographs, review articles and meta analyses. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the Cochrane Collaboration).
Thanks all again. I hope this will reduce some concerns and look forward to hearing your thoughts. Formerly 98 ( talk) 17:18, 19 November 2014 (UTC)
MastCell's parachute article is funny but there is a danger that parody is not taken seriously. Let's pick a real example, and I'm afraid it has to remain on the topic of epilepsy since that's something I can actually talk about, albeit with my small lay brain. The first thing our epilepsy article does when mentioning specific drugs is to comment on their efficacy. It cites three Cochrane reviews. The article does go on to list practice guidelines, citing NICE, but the important thing is that three Cochrane reviews are cited and used before anything else. So the first thing a reader wanting to know about the drug treatment for epilepsy learns is that "Phenytoin, carbamazepine and valproate appear to be equally effective in both focal and generalized seizures. Controlled release carbamazepine appears to work as well as immediate release carbamazepine, and may have fewer side effects." This obsession with discussing efficacy trial results permeates our medical articles, and comes from the fact that we over-laud systematic reviews (and others over-use primary research papers) as though our articles should be founded upon them.
So let's consider if one cited Cochrane review actually supports the statement "appear to be equally effective in both focal and generalized seizures" for phenytoin and carbamazepine. The paper is PMID 12076427 and the full text " Carbamazepine versus phenytoin monotherapy for epilepsy". Firstly, the review only looked at the drugs "when used as monotherapy in subjects with partial onset seizures, or generalized onset tonic-clonic seizures with or without other generalized seizure types." -- which is not all "generalized seizures" (e.g. absence seizures and atonic seizures are different) and not all patients are treated with monotherapy. Secondly, consider the primary outcome is "Time to withdrawal of allocated treatment". This is one recommended measure of epilepsy treatment that attempts to balance whether the drug is effective at reducing seizures and whether the side effects are tolerable. However it is a proxy for effectiveness and side-effects and not without its problems. Still, have a good read of the article. As you read, your heart should sink as study after study is thrown away, data is found to be missing, studies looked at different endpoints from this one's, or studies had methods that aren't what we'd wish for, or only examined seizure types that aren't as general as the author's wanted. In the end, the authors decided not to use the studies' findings but to look at the individual patient data where they could obtain it. They got this for just three trials. One trial looked at children and two looked at adults. One trial looked at partial onset seizures only and the other two at patients with either partial onset seizures or generalized tonic-clonic seizures. These are not patient groups that are typically combined for analysis (heterogeneous). Only one was double-blinded. Is "buggered if I know" a fair summary of the review conclusion?
The limitations of their proxy for efficacy+tolerance become apparent when you read the background section of the paper. The experts have already concluded that these drugs are much of a muchness wrt reducing seizures and are more concerned with their side effect profiles. Are these side effects the kind that influence patients asking to stop a medication during a trial? No. Phenytoin has long term problems with facial disfigurement. Both drugs may very rarely kill their patients with a nasty alergic rash. Carbamazepine may cause neural tube defects in babies, though neither drug is without its pregnancy issues. None of these concerns that the author say influence drug choice in clinical practice are ones that a short-term efficacy study of 50 patients is likely to discover.
We have a Cochrane review that has discovered aspects of our ignorance but otherwise does not support the statement that these drugs are equally effective, nor does it even attempt to answer the implied statement that both these drugs are effective.
Before anyone trumpets or recommends Cochrane or indeed any form of literature or sourcing, one must first consider what statement one is trying to say in article text. I recommend that for disease treatment/management sections, the primary focus should be to source information that describes what the current treatment is and what our best expert minds consider is the best clinical practice. For drugs, the primary consideration is for what the drug is licensed; for what it is a recommended treatment by authorities; and for what it might be widely used even if lacking evidence or in the face of disapproval (e.g. cough medicine). An academic paper that effectively says "we cannot answer the question" is not a source for very much at all. -- Colin° Talk 23:11, 19 November 2014 (UTC)
Zad
68
16:23, 20 November 2014 (UTC)
I've been stewing on this Cochrane issue for a while now. I know we have a formal collaboration with them and that is a good thing, for sure! But i have concerns similar to those of Formerly and Colin. The specific proposal by Doc James above, was to add "Cochrane Collaboration" to the list of sample major scientific/medical organizations. I for one struggle with listing it as an organization for several reasons:
So, no we should not list CC as an example organization with NICE, AHRQ, etc. yes it would be great to name them as a producer of meta-analyses, as they are the premier generator of them. that's my view on the question at hand. Thoughts?? Jytdog ( talk) 16:31, 20 November 2014 (UTC)
How much does NICE or the AHRQ or the APA dictate practice? These are just guidelines that physicians may follow to various degrees. They also may or may not be adopted by the mainstream. Look at antibiotics for the acute bronchitis. Little evidence supports their use. No guidelines support their use. Yet they are commonly used. We basically have three things 1) what the evidence shows 2) what the guidelines say 3) what physicians do. Cochrane reviews are not primary research papers. Doc James ( talk · contribs · email) 20:35, 20 November 2014 (UTC)
ok, i took a shot at implementing the proposal. Here is the change to the original language:
Wikipedia policies on the
neutral point of view and
not using original research demand that we present any prevailing
medical or
scientific consensus, which can be found in recent, authoritative
review articles or in textbooks or in some forms of
monographs. Statements and
practice guidelines issued by major professional medical or scientific societies (For example, the
American Diabetes Association, the
European Society of Cardiology, or the
Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,
AHRQ,
USPSTF,
NICE, and
WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, some forms of
monographs,
review articles, and
systematic reviews that include
meta-analysis. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate
weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the
Cochrane Collaboration). Although significant-minority views are welcome in Wikipedia, such views must be presented in the context of their acceptance by experts in the field. Additionally, the views of tiny minorities need not be reported.
Hopefully this will fly... Jytdog ( talk) 20:37, 23 November 2014 (UTC)
Yup likely trying to achieve too much. Reviews by Cochrane are of the same caliber as those from AHRQ and USPSTF. They often represent a more international perspective while AHRQ and the USPSTF represent a more American POV. Have edited to remove this device. Doc James ( talk · contribs · email) 04:18, 28 November 2014 (UTC)
It seems a little unfair to me to suggest that I am raising this issue based on content disputes on a handful of pages. It has the effect of trivializing my position without actually addressing the points that I've raised. Its very effective as a rhetorical technique, but I'd like to think we can address each others concerns and points more directly than this based on the mutual respect we have developed working together.
James asks a very good question: "Why do you think this change is needed? What issues do you see on Wikipedia that need this change to be addressed?" Let me try to answer that.
With respect to Colin's comments, I'd be interested to know if he has a specific policy position that he is supporting here. I've seen a lot of fairly harsh criticism of both my comments and James, which has sometimes bordered on ridicule, but if you have offered a specific alternative, I somehow missed it. Criticism in the absence of offering a meaningful alternative doesn't really help us reach a consensus.
Clinical guidelines from organizations the ultimate? For my two cents, I too agree that clinical guidelines from groups of experts are probably the best we’re going to be able to do as a simple algorithm for what is representative of NPOV in clinical medicine. You can secondarily mention the Cochrane Library and database and use these, of course, but remember that many clinical guideline formal recommendations use Cochrane data also, so you get that for free. For example, NICE has cited Cochrane many times as evidence which they used to formulate their guidelines. Not as much as Cochrane wishes, no doubt, but they do. [5]. So cite Cochrane particularly if your guidelines miss some nice Cochrane review you found (perhaps a later one), or it contradicts the guidelines. And while we are at it, we should remember that Cochrane is not the only source of clinical trial meta analysis! Sometimes something just doesn’t exist as a Cochrane meta analysis, like (say) the effect of fibrates on cardiovascular outcomes. No such review exists (and yes, that’s shocking, but illustrative). You CAN get a meta analysis that used the Cochrane library of fibrate trials, as well as MEDLINE and Embase, BUT using the library of trials kept by Cochrane does not make this a Cochrane review, with the Cochrane label. But this one is published in LANCET and is perfectly useable as a meta analysis. [6] The problem is knowing how to translate it into guidelines. NICE guidelines really don’t recommend fibrates for anything (maybe horrible primary triglyceridemia), but the meta analysis says they lower severe MI incidence by 10%, without impact on mortality. So now what? I’ll save that for later.
Cochrane not enough. Guidelines often utilize data from epidemiology and even animal and pathophysiology data, which Cochrane (as a strictly clinical trial evidence entity) does not. This is good and bad. The good is that it lets you include recommendations for having a parachute when you jump from a plane (very funny example above). The downside is that pathology and proxy marker data can fool clinicians into prescribing drugs that are no good (examples on request if you don’t have favorite ones of your own—mine is encainide). This problem is why some docs used niacin as a cardiac preventive up until mid-July, but there were also problems of underpowered studies with niacin, and other studies just contradicted each other.
Other meta analysis available. We need to remember that what Cochrane brings to the table is a sort of “guarantee” (warrantee?) that they don’t let chemists do reviews of surgery articles. I had more confidence in them before reading user:Colin's Cochrane complaints above, but what can you do? Again, product quality problems cannot be detected until you are a sophisticated user; sometimes just the GG Gucci label on something is not enough, because it get stamped on crap sometimes. You can use impact factors, but good journals print crap, too. And are biased. And print biased stuff. The routine male circumcision wars in JAMA a few years ago were hilarious, partly because each side took itself so medically and scientifically seriously, except the angry letters and reviews were always urologists Smith, Jones and Lee vs. urologists Cohen, Cantor and Mohel on the opposing view.
Primary studies often contain secondary material of high quality. When you have an article that is from a very large and long clinical study that is nearly definitive and is unlikely to be repeated soon, I think it’s a little silly not to simply go to that primary article’s “discussion” and “conclusion” sections for your secondary material (even primary science articles have secondary-discussion and synthesis parts, per the meanings of “secondary” in WP:RS). That’s another topic for discussion in MEDRS. It’s ridiculous to wait until an article in the NEJM is picked up and reviewed and compared to previous studies by the Lake Wobegon Family Practice Quarterly, when the authors in the NEJM almost surely did a better job in the original, biased though they may have been. And the editorial NEJM staff probably oversaw their conclusions with a more jaundiced eye than the Lake Wobegon editor, who was glad to just have enough material to make his page count.
Epistemology always bites when on the internet, nobody can tell you’re a dog. Ultimately we came back to the epistemological question I complained of several years ago, which is that you cannot write a good NPOV science article without appeal to authority, and that appeal rests on quicksand at some point, since (without ability to do a background check of any type) you very quickly run out of authoritative views on who is an authority. And if you make it that far, you won’t be able to find authorities on who is the authority on finding authorities. At some point, you cannot avoid just taking polls of the schmucks who post here. From our mother’s basements. Ultimately, it’s impossible to write a technical WP article with a reasonable NPOV without breaking guidelines on whether or not you should trust certain editors who write the things. You watch to see if the editor fixing the Lie Algebras article looks like their LaTeX is up to snuff (or broken and ugly), and you sense from their TALK input whether or not they know what they are talking about, mathematically. Something cannot be done by cites, or cites of cites, or cites of meta reviews of reviews of primary cites. The other way (editor trust) is actually how most good articles on WP get written, though we deny it. So I’m here (as usual) to remind you all of the fundamental hypocrisy that is Wikipedia. If Wikipedia were actually written the way that it is supposed to be (no recognizable editors involved—we randomize all usernames with each edit, so there are no possible reputations), Wikipedia would be a lot crappier than it is. However, we use longstanding usernames, and we get to know them (and can look up their contribution histories). And that works as a better proxy than we’d like to admit, for the editor CVs we lack. S B H arris 00:57, 26 November 2014 (UTC)
====Re-poll====
OK, I think that @
Yobol: has just backed us up to a version that is less supported than the version that we had before his edit, so I'm going to do a poll. Please let me know if the following is roughly correct:
I have the following people having expressed the opinions below at some point. Let me know if this is correct or not:
Mostly agree that treatment guidelines are an "ideal" source for determining Scientific Consensus, and by implication, should get more weight (in the sense outlined in
WP:NPOV) than other secondary sources, which are not consensus documents. As near as I can tell, this group opposes calling out Cochrane as a special source that should always be called out alongside the consensus treatment guidelines.
The following appear to feel that Cochrane should still get mention on an least an equal footing with consensus treatment guidelines, but seem to support emphasizing these (Consensus guidelines and Cochrane) over other secondary sources for determining Scientific Consensus:
Opinion unclear or difficult to categorize:
Is this a correct count?
I assumed that we were in agreement that the garden variety meta analyses and reviews should be weighted a little less than consensus treatment guidelines for determining Scientific Consensus, and were split on Cochrane. So I tried to split the difference on Cochrane, calling them an important minority voice when not in agreement with consensus treatment guidelines. Let me know what you all think.
Formerly 98 (
talk) 02:58, 25 November 2014 (UTC)
Please don't do this. We discuss and come to a consensus. We don't vote or tally opinions. If you believe "a slight plurality felt that Cochrane's views always deserved mention even when they were out of synch" then I couldn't really care if a slight plurality felt the moon was made of cheese. They would be wrong. Some policy reading required. -- Colin° Talk 12:55, 25 November 2014 (UTC)
The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.
User:Jytdog has proposed changing:
to
Zad
68
02:50, 30 November 2014 (UTC)Meanwhile, why is this still in the guideline? SandyGeorgia ( Talk) 16:04, 5 December 2014 (UTC)
This proposal was initiated by my concern that meta analyses are being inappropriately used to indicate Scientific Consensus, when that is not their purpose. They play an important role, but due to the large number of subjective decisions made in their preparation and interpretation, invariably reflect the pre-existing biases of the authors. Per published meta analyses:
Treatment guidelines consider and integrate all these results, help provide expert analysis of the conflicting results, and reduce cherry picking. Formerly 98 ( talk) 12:58, 29 November 2014 (UTC)
I think it's a false choice, to say that we have to go with or prioritize a practice guideline against high-quality (sorry, Colin, yes I know there are a few exceptions to every rule), independent source of evidence like Cochrane. As mentioned above, practice guidelines can recommend a course of action in cases where the evidence is less than excellent, and individual organizations can be subject to political pressure or other biases that an independent reviewer might not be subject to. So, why wouldn't we use both? The article can say, "The evidence says this. NICE recommends that." if that's the case. If we're going to name names of organizations that produce practice guidelines, we should also names well-respected, influential sources of evidence. Can anyone argue Cochrane is any less well-regarded than the named bodies producing practice guidelines?
Zad
68
03:05, 30 November 2014 (UTC)
Zad
68
03:16, 30 November 2014 (UTC)
Am I meeting your point or still talking past?
Zad
68
19:52, 30 November 2014 (UTC)
It appears that the question is how should we balance EBM and expert opinion.--Yes, exactly how I see it, and:
Some feel one should get greater weight over the other and vise versa. My position is they should often have equal weight or that at least weight should be determined on a case by case basis.--Yes, exactly. I still think there's an incorrect assumption underlying some of the comments in this discussion that it's either one or the other, or that one has to be prioritized over the other. I really see a top-tier source of evidence like Cochrane and a top-tier authority's practice guidelines as right there next to each other, neither should be prescriptively prioritized over the other, and the use of each, including the weight, needs to be handled on a situation-by-situation basis.
Sbharris wrote (above) a really awesome, laying-the-truth-bare mini-essay on how the sausage really gets made here, and I'm disappointed it really didn't get any traction.
Zad
68
19:20, 30 November 2014 (UTC)
Guidelines and systematic reviews are different things typically. Guidelines often address a much broader question while systematic reviews address a much narrower one. Thus while guidelines often have more authors all these authors many not be looking extensively at each question. Thus it may be less authors than a systematic review. Not that I am suggesting we determine weight by how many authors a review or guideline has. Doc James ( talk · contribs · email) 03:11, 1 December 2014 (UTC)
The wiki ganfyd states here that activated protein C is grade A recommendation for pneumonia based on a 2008 guideline from British Society for Antimicrobial Chemotherapy. The ref is here [16]. However a 2011 Cochrane review found that it was associated with worse outcomes. [17] It took the FDA 6 months to pull the drug at which time UpToDate updated their recommendations. Wikipedia reflected the best available evidence within a couple of days of it coming out. Doc James ( talk · contribs · email) 12:44, 30 November 2014 (UTC)
In addition to the embarrassing nonsense about quaternary sources, the above cherry-picked example could (a) be repeated (with other studies/papers) as an example of why Wikipedia should cite primary research papers rather than wait around for reviews or textbooks to pick up on the latest findings and (b) merely shows that new publications tend to include more up-to-date information, which is a tautology. We are looking for sources that reliably indicate medical scientific consensus on a topic, because we are an encyclopaedia not a newspaper. We are limited by the speed at which reliable publications pick up and report on this consensus. If we were written by named expert authorities in the subject, then we could be both up-to-date and reliably cite the latest research findings. We aren't and we don't. -- 11:46, 7 December 2014 (UTC)
The primary/secondary sourcing terminology for science differs to that for humanities. The first publication of new research based on previously unpublished data is a primary source. So if (as in the Epilepsy paper used above) a Cochrane review decides to escape the confines of analysing previously published results and instead sources the raw unpublished patient records upon which to perform novel analysis, then this is a primary source. In addition, "all sources are primary for something". And since science always builds on the work of others, and this work is frequently referred-to in a research paper, all primary research papers will include some secondary-source material. It really depends what you take from the paper and how you use it on Wikipedia. So, in the words of a notable EBM advocate, " I Think You'll Find It's a Bit More Complicated Than That" -- don't oversimplify the argument.
We all know that if anyone is likely to produce a "There isn't enough high quality data available" result in their publication it is a Cochrane review. That's why sceptic bloggers battling against pseudo science and snake oil merchants love to cite them. But back in the hospital ward, the physician needs to know what drug to give a child for a certain problem but who additionally has a serious metabolic condition, even though the only EBM randomised controlled trial is on young otherwise-healthy men. Nobody conducts randomised controlled trials on pregnant women to see which drugs cause the most defects in their babies. We have an RCT on refractory epilepsy treated with the ketogenic diet but absolutely no RCT studies on refractory epilepsy (AFAIK, PMID 22170887) treated with drugs (after all, what manufacturer would choose that patient group). Yet the physician, faced with a child who has failed three drugs already, must choose a fourth or some other treatment. To claim, as proposed below, that several RCTs combined in systematic reviews is required before making any guidance or reporting something on WP, is just on another planet.
The old chestnut of "expert opinion" vs "evidence based medicine" is repeated and misused. When these are compared like this, the implication is that the "opinion" is "mere opinion", i.e., not based on good evidence, but rather on either prejudice, limited personal experience or plain old guesswork. If Wikipedia was based only on evidence, then we'd cite nothing but primary research papers and systematic reviews and discard literature reviews, textbooks, clinical guidelines and consensus statements. But then we'd need to be authored by respected named experts who could interpret that evidence properly. Because the evidence is not as general as we need for WP article. We have evidence that of a 200 patients treated in a certain trial by a certain hospital according to a certain protocol and given a certain dose of a drug, that 100 got better. And our science may tell us to what confidence we can then attribute that to the drug. And we can gather data on the side effects seen in this short-term trial. Is "half the patients getting better" an effective drug? Depends on the condition. I wouldn't be happy if my optician only prescribed glasses that worked half the time. On the other hand, some cancers never respond anything like that well. If one in a hundred of those patients got liver failure and died, that might be acceptable for fatal cancer but not for irritable bowel syndrome. The decision about what makes a good drug and therefore worthy of mention in the "treatment" or "management" section of our disease articles, is one of expert opinion. So it always ultimately comes down to the opinion of experts and what they have written in reliable publications. We trust that the best experts and best publications will ensure that opinion is based as much on high-quality evidence as possible, but it is not Wikipedia's job to fix that if we disagree with their conclusions. And WP:WEIGHT requires us to respect the (scientific) consensus, which requires secondary sources to determine.
It is clear that Cochrane papers can be a good source of evidence. But where the review eliminates so much and find that what is left is insufficient to make a positive statement, we must treat it with caution. It could simply be that this "we know nothing => can't recommend" is an artefact of their rather extreme methodology, and actual clinical practice says otherwise. Where their papers turn out to be written by non-experts or deviate from performing a systematic review of the published literature, we must treat it with great caution. But ultimately, their papers are written to influence medical opinion and be a good source of evidence for experts to make decisions. But it is that medical opinion and expert decision-making that we report. -- Colin° Talk 11:46, 7 December 2014 (UTC)
- A1candidate ( talk) 10:06, 30 November 2014 (UTC)
- A1candidate ( talk) 10:06, 30 November 2014 (UTC)
Proposed changes are probably more concise and informative. - A1candidate ( talk) 10:06, 30 November 2014 (UTC)
Zad
68
19:07, 30 November 2014 (UTC)
MastCell pointed to a well-known humorous essay Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. There is of course a grain of truth in the humor, but to see how we might handle it: Every medical society around the world strongly recommends parachute use for those who jump out of airplanes(cite review of practice guidelines worldwide), although no trials of the type that can produce the best kinds of evidence have been conducted into parachute use, due to ethical concerns in conducting those kinds of trials.(cite Cochrane)
Yes, taking reductio ad absurdum for face value is even more absurdum but as before there's a grain of truth in it.
Zad
68
19:34, 30 November 2014 (UTC)