- Email: [email protected]
Digging for evidence: Is the answer available?
Accepted Manuscript Digging for Evidence: Is the answer available? Ann Merete Møller PII:
S2210-8440(14)00026-4
DOI:
10.1016/j.tacc.2014.04.010
Reference:
TACC 189
To appear in:
Trends in Anaesthesia and Critical Care
Please cite this article as: Møller AM, Digging for Evidence: Is the answer available?, Trends in Anaesthesia and Critical Care (2014), doi: 10.1016/j.tacc.2014.04.010. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 3. april 2014
Corresponding author:
EP
Ann Merete Møller
TE D
M AN U
SC
RI PT
Digging for Evidence: Is the answer available?
Professor in Anaesthesia and Intensive Care
AC C
Department of Anaesthesia and Intensive Care Herlev University Hospital Herlev Ringvej 75 2730 Herlev Denmark Telephone: +45 38683578 E-mail: [email protected]
1
ACCEPTED MANUSCRIPT 3. april 2014
Introduction
RI PT
As clinical work becomes increasingly complex and the number of possible interventions continues to evolve, and as patients become increasingly aware of possibilities and start to surf the internet on their own, clinicians need to be able to find answers to their clinical questions. But what is a clinical question and is it possible to find answers to all kinds of problems? No doubt the last decades have made it easy to access information (1). However, are clinicians in general educated in the process of searching for evidence and even more importantly, to filter and evaluate the yielded results? After more than 20 years practising evidence based medicine as an integrated part of clinical practice, and increasing quality of research methodology the question still remains: Is the answer out there, and if so, can it be trusted?
AC C
EP
TE D
M AN U
SC
The aim of this paper is to explain a way to pose the clinical question, to perform a comprehensive search and to point out some of the problems that arise in the process of interpreting the search results.
2
ACCEPTED MANUSCRIPT 3. april 2014
Defining the clinical question
RI PT
Before starting the search process, it is important to be absolutely sure what you are looking for. Defining the clinical question may be the most important part of the search process. The clinical question consists of three parts; the population, the intervention/ comparison and the outcomes. This is sometimes called the PICO, as a short name for the parts of the question. Time spent on the process of defining and limiting the question will facilitate the rest of the process by clearing your mind and focusing on what it really is, you want to know. The clinical question can be about a single patient with a specific condition, or it can be about a group of patients. It can be narrow and very specific or it can be wide and sensitive. A welldefined question will later help in designing the search strategy. (2)
M AN U
SC
Defining the population is an important part of the process. If the population is very precisely defined, a potential answer to the question can be very precise. However, the chances are that no answer is out there for this narrow population and even if there are trials looking at the narrow population, it may not be appropriate to extrapolate the results to other patients groups. If, on the contrary the group is widely defined, the chances of finding relevant literature increase. On the flip side is the risk that the population contains a subgroup of patients, for which the results can be different. The subset of patients can “hide” within the broadly defined group.
TE D
The intervention is what is done to the patient. This can be a surgical procedure, a type of anaesthetic, preoperative smoking counselling or whatever else is in focus. The key point is, whether it is well defined and feasible. The comparison depends on the intervention. Examples of relevant comparisons can be no treatment, standard treatment, placebo medication or two types of surgery that can be compared to each other. If the intervention is well described from the beginning the process of finding relevant papers is facilitated.
EP
The outcomes must be something the patient feels, functions or survives in order to be clinically relevant. Examples of clinically relevant outcomes in anaesthesia are pain, postoperative nausea and vomiting, complications after surgery, postoperative cognitive dysfunction and mortality. Outcomes such as length of stay in the postanaesthetic care unit, length of hospital stay and intensive care admittance are relevant, but difficult to deal with, as they are dependent on a wide range of organizational factors, not necessarily related to the question in focus.
AC C
It is important to beware of surrogate outcomes in research. Surrogate outcomes are sometimes believed to “reflect” real outcomes, to be a marker of serious disease or complications. A surrogate outcome is most often a biochemical marker or a physiological measurement that can be obtained with relative ease. However, surrogate outcome measures are not necessarily a true surrogate for what is really sought and there is a high risk that the believed correlation is unreliable and they should be avoided or used with extreme care.
3
ACCEPTED MANUSCRIPT 3. april 2014
Searching for answers
SC
RI PT
In principle, all available information is considered evidence. The obvious place to look for scientific papers is via a database on the internet, but in fact books, journals and newspapers may hold valuable information. Before starting the search it is important to define a search strategy. The clinical question defined above is an integral part of this strategy. How much time and energy put into the development of the search strategy depends on the purpose. If you are going to need a fast but imprecise idea of what the answer may be, it is possible to do a quick and dirty search, using a few keywords and a related article search. However, if you are going to use the answers to write a clinical guideline or a systematic review, or if you want a comprehensive answer to your question, you will have to perform a thorough and comprehensive search. Building the search strategy will typically include the words from the clinical question as well as any synonym for any of the words.
M AN U
The search should be rather sensitive than specific in order not to miss valuable information. The search strategy should therefore be comprehensive, sensitive and reproducible. It is often useful to include the search strategy in the resulting document. There are a wide range of medical databases available across specialties and countries. The most commonly used are the following:
TE D
PubMed, which is a free web based database building on the Medline database of references and abstract of life sciences and biomedical topics. The United States National Library of Medicine (NLM) at the National Institute of Health maintains the database as part of the Entrez system of information retrieval. Although the database contains millions of references, its focus is on major medical journals and journals that publish in English. Limiting the search to PubMed will increase the risk of missing important papers, especially in languages other than English.
AC C
EP
The Cochrane Library holds the large collection of Cochrane Systematic reviews, but it also holds many other resources. The most important of these is CENTRAL, which is a database of clinically controlled trials from almost all existing clinical databases. So searching CENTRAL is an absolute must in order to make the search comprehensive. Other papers available in the Cochrane Library are non-Cochrane systematic reviews, reports of Health Technology Assessment papers on scientific methodology and economic evaluations. Embase is produced by Elsevier, is European in origin and is NOT free. Most institutions, however, have purchased access, so normally there are no problems searching Embase. Embase covers around 7600 journals and has more European journals compared to PubMed. Search methods and the building of search strategies are slightly different in Embase compared to Pubmed and the Cochrane Library, so it is sometimes necessary to slightly remodel the search strategy. Searching Embase also reduces language bias – see later. Other databases that might be searched include “Lilacs” (Literatura Latino-Americana e do Caribe em Ciências da Saúde), which is an on-line bibliographic database in medicine and health sciences, maintained by the Latin American and Caribbean Center on Health Sciences Information (also known as BIREME, located in São Paulo, Brazil. It contains bibliographic references to papers that have been published in a set of scientific and medical journals of the region, and that are often not covered by MEDLINE. “Cinahl”, 4
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
(Cumulative Index to Nursing and Allied Health Literature) is an index of English-language and selected other-language journal articles about nursing, allied health, biomedicine and healthcare, “Biosis”, is an English-language, bibliographic database service, with abstracts and citation indexing. It is part of the Thomson Reuters Web of Knowledge suite. BIOSIS indexes data from 1926 to the present. Besides journal articles, Biosis also covers meeting abstracts, conferences, literature reviews, U.S. patents, books, software, book chapters, notes and letters. There are more than 500 other, smaller and often narrower databases that can be sought, depending on the topic of interest. How to search
M AN U
SC
Use the search strategy, based on the clinical question, with all available synonyms. The strategy ensures reproducibility and transparency. Do not make time restrictions. For some relevant clinical interventions, the original papers are old. As a general rule, older papers have less advanced methodology, but this is not always the case and too much may be missed, when applying time restrictions. If the intervention is modern, there will not be any old papers anyway. When evaluating the paper, it is always important to be aware of the details of the intervention and make sure it is applicable.
Evaluating search results
TE D
Other restrictions and limitations may also lead you astray. Language restriction will surely lead to language bias (see later). Including papers in foreign languages can be a challenge, but very often the paper includes an abstract in English, that will let you know whether the paper is relevant for your clinical question or not. And often it is possible to find a colleague or a friend who can help extract the data from a paper. If not, the least you can do is to point out that the paper exists, even if you cannot include it. The databases have a multitude of filter functions, which can be very useful for a “quick and dirty” search, but which is not always sensitive enough for a thorough search. For example, PubMed has a filter for randomised controlled trials. However, if the trial has been tagged as such, which is often the case; you may miss an important paper.
AC C
EP
When the search is completed, you have (perhaps virtually) a pile of papers that deal with the clinical question you set out to answer. It is important to perform a critical evaluation of each paper in order to assess the methodological quality of the research process, identify potential bias and how this may influence the results. The themes that should be evaluated include: Trial design. Is this a randomised controlled trial, and if so was randomisation performed correctly? How was the allocation concealment achieved? How large was the sample size and how was it calculated? The inclusion and exclusion criteria must be appropriate and the patients lost to follow up must be few. Another important theme is whether the correct statistics were used and whether the results were transparent and logical. Trials can be blinded in many ways and should be, when possible. How large was the treatment effect and are the results altogether relevant for the group of patients in focus. Each trial will most likely have a certain amount of bias. However, not least important are the biases that arise across trials. Publication bias Publication bias is defined as a bias that arises when the publication of research depends on the nature and direction of the study results. This means that studies with positive, significant results are much more likely to be published. And not just that, they are likely to be published faster and in journals with a better impact
5
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
factor. This may lead to the wrong conclusions when using comprehensive searched to try and establish an overview of the existing knowledge in the field and publication bias is one of the largest problems in performing evidence based practice. Publication bias will almost always tend towards overestimating the treatment effect, due to the simple fact that trials that shows insignificant results, e.g. no difference or are in favour of the “old” intervention will have difficulties being published or will be published significantly later than “positive” significant results. See figure 1. Publication bias has nothing to do with the quality of the research methodology. On the contrary, positive results often arise from smaller trials with broad confidence intervals, whereas larger trials that are more reliable are much less likely to find positive significant results, unless the treatment effect is substantial.
M AN U
SC
The reasons that non-significant results are less likely to be published are partly because many journals are focussed on presenting new and interesting results, that will be cited often and thereby increase the impact factor. Other reasons may be that the authors chose not to publish results that may not support their initial hypothesis (the file drawer problem). Finally, if they pursue publication, they may have to try many journals, with decreasing impact factors, which takes time and postpones publication. It may also result in another, related bias, e.g. language bias. Language bias
TE D
Language bias is defined as the bias that is introduced when papers are published in non-English journals. Non-English journals are read by fewer persons, the articles are cited less often and there is a tendency for many readers and authors to filter their searches, so that they only yield papers in English. It can rightly be argued that language bias is a subgroup of publication bias, as the results are the same for the overall interpretation of the evidence underlying the clinical question in focus. In an effort to overcome this problem, the trial registration databases have been established, so that it has become more difficult to withhold unfavourable research results. Many large journals only accept trials that have been registered in such a database prior to the conduct of the trial. The effect of these databases is still not validated. An example of a trial registration database is clinicaltrials.gov.
EP
Outcome reporting bias
AC C
Outcome reporting bias arises, when the researchers measure a number of outcomes, but choose only to report some of them. This can be because they find some outcomes more interesting than others, but it may also be because they are “chasing significance” – a process not entirely unknown for most researchers. However, this is a problem because 1. The non-significant results will remain unknown to the public and because there is an increased risk of type 2 errors due to multiple testing.
Evidence based medicine Modern medicine is evidence based. The definition of evidence based medicine (EBM) is: “the judicious use of the best current evidence in making decisions about the care of the individual patient”. In practise, it consists of the five elements: Defining the clinical question, searching for literature, appraising the literature, implementing the results and finally, evaluation of the process, and then the circle can start
6
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
again. Evidence based medicine is used in daily clinical practice, when writing clinical guidelines and when performing systematic reviews, such as for example Cochrane Reviews.(3) Systematic reviews are characterised by a well formulated question, a published protocol, methods described before the search, a comprehensive data search, unbiased selection and abstraction process, critical appraisal of data, synthesis of data, a structured report and for Cochrane reviews; they are regularly updated. Many systematic reviews include a meta-analysis. Meta-analysis
SC
A meta-analysis is a statistical tool used to pool data from several trials in order to increase the pooled sample size and increase the possibility of finding a “true effect”. However, meta-analysis is a risky business. If the results of a meta-analysis are interpreted without critical sense, it may lead to harmful clinical decisions, putting patients at risk. A common saying about meta-analysis is “garbage in- garbage out” meaning that a result of a meta-analysis depends almost entirely on the quality of the included trials.
M AN U
An important problem with meta-analysis is the heterogeneity of the included trials. Heterogeneity can arise in all 3 parts of the clinical question, e.g. the population can be defined differently, the interventions and controls can be different and the outcomes of the trials can be either entirely different or defined differently. A certain amount of heterogeneity is to be expected, but if it is too large it does not make sense to pool the results. There are no particular cut offs for heterogeneity. It can be evaluated clinically, visually on the Forest plot or by using the I2 test (3) See figure 2.
EP
TE D
Another problem with meta-analysis is publication bias. As mentioned earlier, trials with so-called “positive” or “significant” results are, for a number of reasons, more likely to be published altogether, more likely to be published in English and published earlier. Authors of systematic reviews have tried in many ways to extend the search for getting more trials. Some advocate the use of abstracts from conference books, as many results are presented at conferences, although they may never reach publication. Another possibility is to contact authors of related trials and experts in the field in order to find trials that have suffered the “file drawer problem”, finally, pharmaceutical companies can be contacted and asked for unpublished material. However, these efforts are, almost always, fruitless. Trials that ended in the drawer seem to stay there.
AC C
The fact that “negative” trials are published several years later than “positive” trials adds to the problems with meta-analysis. Some have advocated that only trials started before a certain date should be included, but authors do not seem to not want to include the newest, positive trials and the number of unpublished evidence is completely unknown. Impact factor
The impact factor is a measure that reflects the number of citations to recent articles published in the journal. Journals strive to continuously increase the impact factor and this is the most important point of competition between journals. Authors of clinical trials will often try to publish with the highest possible impact factor. Trials with new and positive results are more likely to be cited than “negative” and inconclusive trials and will thereby help increase the impact factor. This mechanism adds to the problems of publication bias (4)
7
ACCEPTED MANUSCRIPT 3. april 2014
Conclusion Searching for evidence is a difficult process. During the process there are many decisions to be made and it is easy to be discouraged. However, giving up will not improve patient care. Instead, it is important to continue the work for better and evidence based patient care.
RI PT
Focus should be on the development of the research question, the comprehensive search, the evaluation of the individual paper and last, but perhaps most important on the interpretation of the overall search result. Many problems arise when one is trying to summarize evidence.
SC
Most of these factors will tend to increase the treatment effect of the intervention in question to an unknown degree. This means that the answers we find may not after all reflect the truth, or maybe reflect it very imprecisely. There are a number of initiatives that may help us to find better answers in the future.
3.
4.
EP
5.
effect size increased. Large clinical trials are difficult to perform and expensive. They need trial managers that are project leaders and scientists and have enough funding. Of course, large multicentre trials cannot be performed for every clinical question, so it is important to prioritize the question Enhanced research standards. Although research standards have increasingly improved over the last decades, they can still be further improved. It is important to calculate realistic sample sizes, write and follow detailed protocols and define precise and patient related outcomes. The preregistration of protocols in trial databases is an important step in this direction. That authors and journals become aware of striving for useful answers instead of striving for significance, and that journals publish papers because of high methodology standards and less because of results. That a clinical question can be researched more than once. Often one trial is not enough to give a final answer and repeating a trial in another setting will help in identifying possible bias. Increase the awareness of conflicts of interest, not only financial, but also professional.
TE D
2.
M AN U
1. Larger trials. When a study is better powered, bias seems to be reduced and the estimation of the
AC C
Many initiatives have already been taken along this road, and the future will show how far we can go in order to find reliable evidence for the treatment of our patients.
8
ACCEPTED MANUSCRIPT 3. april 2014
AC C
(5)
EP
TE D
M AN U
SC
RI PT
Figure 1
9
ACCEPTED MANUSCRIPT 3. april 2014
Figure 2.
EP AC C
(6)
TE D
M AN U
SC
RI PT
Example of a Forest plot
10
ACCEPTED MANUSCRIPT 3. april 2014
References Davidoff F, Haynes B, Sackett D, Smith R. Evidence based medicine. BMJ [Internet]. 1995 Apr 29 [cited 2014 Apr 3];310(6987):1085–6. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2549494&tool=pmcentrez&rendertype =abstract
2.
Møller AM. How to map the evidence: the development of the systematic review in anaesthesia. Br J Anaesth [Internet]. 2012 Jul [cited 2014 Mar 19];109(1):32–4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22649185
3.
Higgins JPTT, Green S. Cochrane Handbook for Systematic Reviews of Interventions (Wiley Cochrane Series ). Library. 2008.
4.
Lauritsen J, Moller AM. Publications in anesthesia journals: quality and clinical relevance. Anesth Analg [Internet]. 2004 Nov [cited 2014 Apr 3];99(5):1486–91; table of contents. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15502053
5.
Stern J, Simes R. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640–5.
6.
Thomsen T, Villebro N, Møller AM. Interventions for preoperative smoking cessation. Cochrane database Syst Rev [Internet]. 2014 Mar 27 [cited 2014 Apr 1];3:CD002294. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24671929
AC C
EP
TE D
M AN U
SC
RI PT
1.
11
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
3. april 2014
12
S2210-8440(14)00026-4
DOI:
10.1016/j.tacc.2014.04.010
Reference:
TACC 189
To appear in:
Trends in Anaesthesia and Critical Care
Please cite this article as: Møller AM, Digging for Evidence: Is the answer available?, Trends in Anaesthesia and Critical Care (2014), doi: 10.1016/j.tacc.2014.04.010. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 3. april 2014
Corresponding author:
EP
Ann Merete Møller
TE D
M AN U
SC
RI PT
Digging for Evidence: Is the answer available?
Professor in Anaesthesia and Intensive Care
AC C
Department of Anaesthesia and Intensive Care Herlev University Hospital Herlev Ringvej 75 2730 Herlev Denmark Telephone: +45 38683578 E-mail: [email protected]
1
ACCEPTED MANUSCRIPT 3. april 2014
Introduction
RI PT
As clinical work becomes increasingly complex and the number of possible interventions continues to evolve, and as patients become increasingly aware of possibilities and start to surf the internet on their own, clinicians need to be able to find answers to their clinical questions. But what is a clinical question and is it possible to find answers to all kinds of problems? No doubt the last decades have made it easy to access information (1). However, are clinicians in general educated in the process of searching for evidence and even more importantly, to filter and evaluate the yielded results? After more than 20 years practising evidence based medicine as an integrated part of clinical practice, and increasing quality of research methodology the question still remains: Is the answer out there, and if so, can it be trusted?
AC C
EP
TE D
M AN U
SC
The aim of this paper is to explain a way to pose the clinical question, to perform a comprehensive search and to point out some of the problems that arise in the process of interpreting the search results.
2
ACCEPTED MANUSCRIPT 3. april 2014
Defining the clinical question
RI PT
Before starting the search process, it is important to be absolutely sure what you are looking for. Defining the clinical question may be the most important part of the search process. The clinical question consists of three parts; the population, the intervention/ comparison and the outcomes. This is sometimes called the PICO, as a short name for the parts of the question. Time spent on the process of defining and limiting the question will facilitate the rest of the process by clearing your mind and focusing on what it really is, you want to know. The clinical question can be about a single patient with a specific condition, or it can be about a group of patients. It can be narrow and very specific or it can be wide and sensitive. A welldefined question will later help in designing the search strategy. (2)
M AN U
SC
Defining the population is an important part of the process. If the population is very precisely defined, a potential answer to the question can be very precise. However, the chances are that no answer is out there for this narrow population and even if there are trials looking at the narrow population, it may not be appropriate to extrapolate the results to other patients groups. If, on the contrary the group is widely defined, the chances of finding relevant literature increase. On the flip side is the risk that the population contains a subgroup of patients, for which the results can be different. The subset of patients can “hide” within the broadly defined group.
TE D
The intervention is what is done to the patient. This can be a surgical procedure, a type of anaesthetic, preoperative smoking counselling or whatever else is in focus. The key point is, whether it is well defined and feasible. The comparison depends on the intervention. Examples of relevant comparisons can be no treatment, standard treatment, placebo medication or two types of surgery that can be compared to each other. If the intervention is well described from the beginning the process of finding relevant papers is facilitated.
EP
The outcomes must be something the patient feels, functions or survives in order to be clinically relevant. Examples of clinically relevant outcomes in anaesthesia are pain, postoperative nausea and vomiting, complications after surgery, postoperative cognitive dysfunction and mortality. Outcomes such as length of stay in the postanaesthetic care unit, length of hospital stay and intensive care admittance are relevant, but difficult to deal with, as they are dependent on a wide range of organizational factors, not necessarily related to the question in focus.
AC C
It is important to beware of surrogate outcomes in research. Surrogate outcomes are sometimes believed to “reflect” real outcomes, to be a marker of serious disease or complications. A surrogate outcome is most often a biochemical marker or a physiological measurement that can be obtained with relative ease. However, surrogate outcome measures are not necessarily a true surrogate for what is really sought and there is a high risk that the believed correlation is unreliable and they should be avoided or used with extreme care.
3
ACCEPTED MANUSCRIPT 3. april 2014
Searching for answers
SC
RI PT
In principle, all available information is considered evidence. The obvious place to look for scientific papers is via a database on the internet, but in fact books, journals and newspapers may hold valuable information. Before starting the search it is important to define a search strategy. The clinical question defined above is an integral part of this strategy. How much time and energy put into the development of the search strategy depends on the purpose. If you are going to need a fast but imprecise idea of what the answer may be, it is possible to do a quick and dirty search, using a few keywords and a related article search. However, if you are going to use the answers to write a clinical guideline or a systematic review, or if you want a comprehensive answer to your question, you will have to perform a thorough and comprehensive search. Building the search strategy will typically include the words from the clinical question as well as any synonym for any of the words.
M AN U
The search should be rather sensitive than specific in order not to miss valuable information. The search strategy should therefore be comprehensive, sensitive and reproducible. It is often useful to include the search strategy in the resulting document. There are a wide range of medical databases available across specialties and countries. The most commonly used are the following:
TE D
PubMed, which is a free web based database building on the Medline database of references and abstract of life sciences and biomedical topics. The United States National Library of Medicine (NLM) at the National Institute of Health maintains the database as part of the Entrez system of information retrieval. Although the database contains millions of references, its focus is on major medical journals and journals that publish in English. Limiting the search to PubMed will increase the risk of missing important papers, especially in languages other than English.
AC C
EP
The Cochrane Library holds the large collection of Cochrane Systematic reviews, but it also holds many other resources. The most important of these is CENTRAL, which is a database of clinically controlled trials from almost all existing clinical databases. So searching CENTRAL is an absolute must in order to make the search comprehensive. Other papers available in the Cochrane Library are non-Cochrane systematic reviews, reports of Health Technology Assessment papers on scientific methodology and economic evaluations. Embase is produced by Elsevier, is European in origin and is NOT free. Most institutions, however, have purchased access, so normally there are no problems searching Embase. Embase covers around 7600 journals and has more European journals compared to PubMed. Search methods and the building of search strategies are slightly different in Embase compared to Pubmed and the Cochrane Library, so it is sometimes necessary to slightly remodel the search strategy. Searching Embase also reduces language bias – see later. Other databases that might be searched include “Lilacs” (Literatura Latino-Americana e do Caribe em Ciências da Saúde), which is an on-line bibliographic database in medicine and health sciences, maintained by the Latin American and Caribbean Center on Health Sciences Information (also known as BIREME, located in São Paulo, Brazil. It contains bibliographic references to papers that have been published in a set of scientific and medical journals of the region, and that are often not covered by MEDLINE. “Cinahl”, 4
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
(Cumulative Index to Nursing and Allied Health Literature) is an index of English-language and selected other-language journal articles about nursing, allied health, biomedicine and healthcare, “Biosis”, is an English-language, bibliographic database service, with abstracts and citation indexing. It is part of the Thomson Reuters Web of Knowledge suite. BIOSIS indexes data from 1926 to the present. Besides journal articles, Biosis also covers meeting abstracts, conferences, literature reviews, U.S. patents, books, software, book chapters, notes and letters. There are more than 500 other, smaller and often narrower databases that can be sought, depending on the topic of interest. How to search
M AN U
SC
Use the search strategy, based on the clinical question, with all available synonyms. The strategy ensures reproducibility and transparency. Do not make time restrictions. For some relevant clinical interventions, the original papers are old. As a general rule, older papers have less advanced methodology, but this is not always the case and too much may be missed, when applying time restrictions. If the intervention is modern, there will not be any old papers anyway. When evaluating the paper, it is always important to be aware of the details of the intervention and make sure it is applicable.
Evaluating search results
TE D
Other restrictions and limitations may also lead you astray. Language restriction will surely lead to language bias (see later). Including papers in foreign languages can be a challenge, but very often the paper includes an abstract in English, that will let you know whether the paper is relevant for your clinical question or not. And often it is possible to find a colleague or a friend who can help extract the data from a paper. If not, the least you can do is to point out that the paper exists, even if you cannot include it. The databases have a multitude of filter functions, which can be very useful for a “quick and dirty” search, but which is not always sensitive enough for a thorough search. For example, PubMed has a filter for randomised controlled trials. However, if the trial has been tagged as such, which is often the case; you may miss an important paper.
AC C
EP
When the search is completed, you have (perhaps virtually) a pile of papers that deal with the clinical question you set out to answer. It is important to perform a critical evaluation of each paper in order to assess the methodological quality of the research process, identify potential bias and how this may influence the results. The themes that should be evaluated include: Trial design. Is this a randomised controlled trial, and if so was randomisation performed correctly? How was the allocation concealment achieved? How large was the sample size and how was it calculated? The inclusion and exclusion criteria must be appropriate and the patients lost to follow up must be few. Another important theme is whether the correct statistics were used and whether the results were transparent and logical. Trials can be blinded in many ways and should be, when possible. How large was the treatment effect and are the results altogether relevant for the group of patients in focus. Each trial will most likely have a certain amount of bias. However, not least important are the biases that arise across trials. Publication bias Publication bias is defined as a bias that arises when the publication of research depends on the nature and direction of the study results. This means that studies with positive, significant results are much more likely to be published. And not just that, they are likely to be published faster and in journals with a better impact
5
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
factor. This may lead to the wrong conclusions when using comprehensive searched to try and establish an overview of the existing knowledge in the field and publication bias is one of the largest problems in performing evidence based practice. Publication bias will almost always tend towards overestimating the treatment effect, due to the simple fact that trials that shows insignificant results, e.g. no difference or are in favour of the “old” intervention will have difficulties being published or will be published significantly later than “positive” significant results. See figure 1. Publication bias has nothing to do with the quality of the research methodology. On the contrary, positive results often arise from smaller trials with broad confidence intervals, whereas larger trials that are more reliable are much less likely to find positive significant results, unless the treatment effect is substantial.
M AN U
SC
The reasons that non-significant results are less likely to be published are partly because many journals are focussed on presenting new and interesting results, that will be cited often and thereby increase the impact factor. Other reasons may be that the authors chose not to publish results that may not support their initial hypothesis (the file drawer problem). Finally, if they pursue publication, they may have to try many journals, with decreasing impact factors, which takes time and postpones publication. It may also result in another, related bias, e.g. language bias. Language bias
TE D
Language bias is defined as the bias that is introduced when papers are published in non-English journals. Non-English journals are read by fewer persons, the articles are cited less often and there is a tendency for many readers and authors to filter their searches, so that they only yield papers in English. It can rightly be argued that language bias is a subgroup of publication bias, as the results are the same for the overall interpretation of the evidence underlying the clinical question in focus. In an effort to overcome this problem, the trial registration databases have been established, so that it has become more difficult to withhold unfavourable research results. Many large journals only accept trials that have been registered in such a database prior to the conduct of the trial. The effect of these databases is still not validated. An example of a trial registration database is clinicaltrials.gov.
EP
Outcome reporting bias
AC C
Outcome reporting bias arises, when the researchers measure a number of outcomes, but choose only to report some of them. This can be because they find some outcomes more interesting than others, but it may also be because they are “chasing significance” – a process not entirely unknown for most researchers. However, this is a problem because 1. The non-significant results will remain unknown to the public and because there is an increased risk of type 2 errors due to multiple testing.
Evidence based medicine Modern medicine is evidence based. The definition of evidence based medicine (EBM) is: “the judicious use of the best current evidence in making decisions about the care of the individual patient”. In practise, it consists of the five elements: Defining the clinical question, searching for literature, appraising the literature, implementing the results and finally, evaluation of the process, and then the circle can start
6
ACCEPTED MANUSCRIPT 3. april 2014
RI PT
again. Evidence based medicine is used in daily clinical practice, when writing clinical guidelines and when performing systematic reviews, such as for example Cochrane Reviews.(3) Systematic reviews are characterised by a well formulated question, a published protocol, methods described before the search, a comprehensive data search, unbiased selection and abstraction process, critical appraisal of data, synthesis of data, a structured report and for Cochrane reviews; they are regularly updated. Many systematic reviews include a meta-analysis. Meta-analysis
SC
A meta-analysis is a statistical tool used to pool data from several trials in order to increase the pooled sample size and increase the possibility of finding a “true effect”. However, meta-analysis is a risky business. If the results of a meta-analysis are interpreted without critical sense, it may lead to harmful clinical decisions, putting patients at risk. A common saying about meta-analysis is “garbage in- garbage out” meaning that a result of a meta-analysis depends almost entirely on the quality of the included trials.
M AN U
An important problem with meta-analysis is the heterogeneity of the included trials. Heterogeneity can arise in all 3 parts of the clinical question, e.g. the population can be defined differently, the interventions and controls can be different and the outcomes of the trials can be either entirely different or defined differently. A certain amount of heterogeneity is to be expected, but if it is too large it does not make sense to pool the results. There are no particular cut offs for heterogeneity. It can be evaluated clinically, visually on the Forest plot or by using the I2 test (3) See figure 2.
EP
TE D
Another problem with meta-analysis is publication bias. As mentioned earlier, trials with so-called “positive” or “significant” results are, for a number of reasons, more likely to be published altogether, more likely to be published in English and published earlier. Authors of systematic reviews have tried in many ways to extend the search for getting more trials. Some advocate the use of abstracts from conference books, as many results are presented at conferences, although they may never reach publication. Another possibility is to contact authors of related trials and experts in the field in order to find trials that have suffered the “file drawer problem”, finally, pharmaceutical companies can be contacted and asked for unpublished material. However, these efforts are, almost always, fruitless. Trials that ended in the drawer seem to stay there.
AC C
The fact that “negative” trials are published several years later than “positive” trials adds to the problems with meta-analysis. Some have advocated that only trials started before a certain date should be included, but authors do not seem to not want to include the newest, positive trials and the number of unpublished evidence is completely unknown. Impact factor
The impact factor is a measure that reflects the number of citations to recent articles published in the journal. Journals strive to continuously increase the impact factor and this is the most important point of competition between journals. Authors of clinical trials will often try to publish with the highest possible impact factor. Trials with new and positive results are more likely to be cited than “negative” and inconclusive trials and will thereby help increase the impact factor. This mechanism adds to the problems of publication bias (4)
7
ACCEPTED MANUSCRIPT 3. april 2014
Conclusion Searching for evidence is a difficult process. During the process there are many decisions to be made and it is easy to be discouraged. However, giving up will not improve patient care. Instead, it is important to continue the work for better and evidence based patient care.
RI PT
Focus should be on the development of the research question, the comprehensive search, the evaluation of the individual paper and last, but perhaps most important on the interpretation of the overall search result. Many problems arise when one is trying to summarize evidence.
SC
Most of these factors will tend to increase the treatment effect of the intervention in question to an unknown degree. This means that the answers we find may not after all reflect the truth, or maybe reflect it very imprecisely. There are a number of initiatives that may help us to find better answers in the future.
3.
4.
EP
5.
effect size increased. Large clinical trials are difficult to perform and expensive. They need trial managers that are project leaders and scientists and have enough funding. Of course, large multicentre trials cannot be performed for every clinical question, so it is important to prioritize the question Enhanced research standards. Although research standards have increasingly improved over the last decades, they can still be further improved. It is important to calculate realistic sample sizes, write and follow detailed protocols and define precise and patient related outcomes. The preregistration of protocols in trial databases is an important step in this direction. That authors and journals become aware of striving for useful answers instead of striving for significance, and that journals publish papers because of high methodology standards and less because of results. That a clinical question can be researched more than once. Often one trial is not enough to give a final answer and repeating a trial in another setting will help in identifying possible bias. Increase the awareness of conflicts of interest, not only financial, but also professional.
TE D
2.
M AN U
1. Larger trials. When a study is better powered, bias seems to be reduced and the estimation of the
AC C
Many initiatives have already been taken along this road, and the future will show how far we can go in order to find reliable evidence for the treatment of our patients.
8
ACCEPTED MANUSCRIPT 3. april 2014
AC C
(5)
EP
TE D
M AN U
SC
RI PT
Figure 1
9
ACCEPTED MANUSCRIPT 3. april 2014
Figure 2.
EP AC C
(6)
TE D
M AN U
SC
RI PT
Example of a Forest plot
10
ACCEPTED MANUSCRIPT 3. april 2014
References Davidoff F, Haynes B, Sackett D, Smith R. Evidence based medicine. BMJ [Internet]. 1995 Apr 29 [cited 2014 Apr 3];310(6987):1085–6. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2549494&tool=pmcentrez&rendertype =abstract
2.
Møller AM. How to map the evidence: the development of the systematic review in anaesthesia. Br J Anaesth [Internet]. 2012 Jul [cited 2014 Mar 19];109(1):32–4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22649185
3.
Higgins JPTT, Green S. Cochrane Handbook for Systematic Reviews of Interventions (Wiley Cochrane Series ). Library. 2008.
4.
Lauritsen J, Moller AM. Publications in anesthesia journals: quality and clinical relevance. Anesth Analg [Internet]. 2004 Nov [cited 2014 Apr 3];99(5):1486–91; table of contents. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15502053
5.
Stern J, Simes R. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640–5.
6.
Thomsen T, Villebro N, Møller AM. Interventions for preoperative smoking cessation. Cochrane database Syst Rev [Internet]. 2014 Mar 27 [cited 2014 Apr 1];3:CD002294. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24671929
AC C
EP
TE D
M AN U
SC
RI PT
1.
11
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
3. april 2014
12