- Email: [email protected]
How are research reports evaluated?
Volume
6
l
January/February
Issue 1
2001
Ralph W. Hale, MD vidence-based medicine has become the hallmark for assuring that every treatment modality has been proved. This proof is based on studies that meet the most vigorous tests of evaluation. It would be an ideal world if everything we do in medicine could measure up to this demanding yardstick. Unfortunately, this is not the situation. Much of medical practice is based on less strin-
E
OBSTETRICS . Effects of . l
. . l
. .
epidural analgesia Mild and moderate preterm birth Vertical transmission of HCV Episiotomy risk factors Effect of intrapartum antibiotics on GBS infection Disability after extremely preterm birth Defining preedampsia Management of PROM
l l
:
.
. l l
. . l
l
Menopausal symptoms Predicting CIN using risk assessment and HPV testing
9
9
3 3 4 4 5
5
GENERAL HEALTH l
Surveillance
. Whole grain l l
after polypectomy consumption and
ischemic stroke Osteoporosis screening Effect of changes in diet & lifestyle
on CHD Inhaled corticosteroid use & BMD in asthma patients . Flu vaccine & healthy adults . Sports participation reflected in hip bone density . Exercise & pelvic muscle performance l Oral glucose tolerance test & the preparatory diet . Comparison of CPR outcomes l Effect of ERT on coronary-artery atherosclerosis
IO 10 11 11
.
GYNECOLOGY . Fertility
outcome after unruptured extrauterine pregnancy Reducing risk of multiple pregnancy after ovarian stimulation OCs & risk of breast cancer Cervical cancer screening Glucose metabolism & OC use Postmenopausal HRT use VT among OC users before 1995 Urinaw incontinence & hysterectomy Preserving ovarian function in patients with germ cell tumors
gent, but nonetheless, time-tested processes and procedures. However, this should not deter us from intense evaluation of the medical literature. ACOG Practice Bulletins, as well as other ACOG documents, use an evaluation system to rate each article that is used to establish a guideline. This rating system is designed to assist the reader in assessing the level of evidence on which the recommendation
6
: : :
II
12 12 12
::
13
9 9
HISTORICAL
PERSPECWE
is based. By using this system it is the desire of our committees to make their decisions as scientifically sound as possible. The College is often called upon to explain how a recommendation was determined. When this occurs, the level of evidence is determined by the quality of the references in the questioned document. To assist our members, the traditional editorial usually found in ACOG Clinical Review has been replaced by this explanation. Each reference in our Practice Bulletins includes a rating level based on the method outlined by the US Preventative Services Task Force. Most studies usually can be separated into those that are observational and those that are interventional. Each of these is further subdivided into a level based on the type of study and how the investigation was performed. A recommendation is then made based on these factors. This method of arriving at a recommendation sounds confusing until one actually tries it with an article. Observational Studies: In this type of study, the investigator observes some event that may occur and records its outcome. This event may be a natural or artificial occurrence. The initial point of observation is used as a primary event. These studies (Continued on p. 14)
14 From ACOG,
02001 by the American College 01 Obstetnclans and Gynecologtsts PublIshed by Elsewer Science Inc 1085.6862/01/$6 00
January/February
2001
l
Washington,
ACOG
D.C.
CLINICAL
REVIEW
l
1
ing organisms produces puerperal felater, another ver.“3 One month woman was admitted to the hospital, suffering from caries of the left knee joint. When all of the other patients sharing the same room with this woman acquired puerperal fever, Semmelweis realized that “. . . the foul exhalations from the carious joint were so strong, that the air of the labour-room in which she was confined, was so loaded that all the patients in the same room became infected. . . . The air of the labourroom, loaded with the putrid matter, found its way into the gaping genitals just at the completion of labour.” Semmelweis immediately insisted upon chlorinated lime hand-washings between patient visits, cleanliness of linens, and isolation of sick patients. Unfortunately, Semmelweis delayed publishing his accomplishments despite the success of his prophylactic measures. His excuse: “an inborn loathing for everything that is called writing.“4 Consequently, knowledge of his doctrine was spread only by visiting medical students and collegial letters. By 1848, one colleague, Ferdinand von Hebra, had published two articles in the Journal of the Vienna Society of Physicians concerning Semmelweis’ discovery, but mistakenly implicated cadaveric particles alone, not living materials, as the source of disease.2 One year later, Semmelweis, unpublished and rejected by the hospital, fled from Vienna to Budapest. In Budapest, Semmelweis supervised the obstetrical unit at St. Rochus Hospital without financial compensation. He improved departmental sanitation by instituting hand-washing prophylaxis, thereby reducing the maternal mortality to 0.85% over the next 6 years.2 By 1855, with the death of a faculty member, Semmelweis became Chairman of the Department of Obstetrics at the University of Pest. In this role, Semmelweis achieved one of his greatest victories: decreasing hospital mortality from puerperal fever to 0.39%.2 Despite his innumerable accomplishments, Semmelweis never received praise or acceptance from his peers. Because Semmelweis never published his doctrine, many physicians turned to the erroneous publica(Continued onp. 1G)
EDITORIAL contiM/
erable if the data are from more than one group or center.
fron, p. I
may be subdivided into a cohort study, a case-controlled study, or a cross sectional study. Each of these studies has major flaws in their design so evaluation of the evidence must take these flaws into consideration. Examples of these flaws include selection bias, lack of knowledge of all related factors, time and exposure relationship unknowns, and unrecognized outside influences. These types of studies never reach the level of evidence that intervention studies attain. Interventional Studies: These studies are usually known as clinical trials. The investigator assigns groups to specific intervention or nonintervention (control or placebo) groups and monitors specific results of the intervention compared with nonintervention. The major subcategories in this type of investigation are whether or not the study includes randomization. Did the investigator assign subjects to specific groups or were they selected prospectively by a process over which the investigator had no control? Obviously, randomization is preferable. A further subcategory is whether the investigator knows prospectively which subject is assigned to each group even if the assignment is random. If they are unaware, then this is a blinded study. The study may also be a crossover study. One-half of the randomly assigned subjects have the intervention for part of the study and then become controls while the controls then receive the intervention. The best of all studies is a blinded, randomized, crossover evaluation. Unfortunately, this method is applicable only in a small number of studies. Based on these two types of evidence, the method of evaluation rating that is applied to each study is as follows: I. Evidence is based on one randomized controlled trial. II-l.
Evidence is based on a nonrandomized controlled trial.
11-2. Evidence is based on a cohort or case-controlled study. It is pref_
11-3. Evidence is based on multiple observations with or without intervention. Uncontrolled studies are in this category. III.
Based on the opinion of authorities, usually as a result of clinical experience or committee opinions.
Once the evidence is rated, a recommendation can be provided. The recommendation is rated based on the following criteria: A. Recommendation based on good and consistent scientific evidence. B. Recommendation based on limited or inconsistent scientific evidence. C. Recommendation based on consensus and expert opinion. As with every rating system, this one is not perfect. It is easy to see that Level I and possibly II-1 allow Level A recommendations. Likewise, Level C recommendations occur with Level III and possible II-3 evidence. Level 11-3, however, as well as Level 11-l and II-2 may also be the basis of a Level B recommendation. Therefore, the reader must make subjective judgments in many instances. I hope this brief explanation has not been confusing. As you read the medical literature, you can apply these principles to assess the studies in a more discerning way. To assist in this process, the editors ofACOG Ciinical Review will, whenever possible, establish a level of evidence for the articles reviewed.
6
l
January/February
Issue 1
2001
Ralph W. Hale, MD vidence-based medicine has become the hallmark for assuring that every treatment modality has been proved. This proof is based on studies that meet the most vigorous tests of evaluation. It would be an ideal world if everything we do in medicine could measure up to this demanding yardstick. Unfortunately, this is not the situation. Much of medical practice is based on less strin-
E
OBSTETRICS . Effects of . l
. . l
. .
epidural analgesia Mild and moderate preterm birth Vertical transmission of HCV Episiotomy risk factors Effect of intrapartum antibiotics on GBS infection Disability after extremely preterm birth Defining preedampsia Management of PROM
l l
:
.
. l l
. . l
l
Menopausal symptoms Predicting CIN using risk assessment and HPV testing
9
9
3 3 4 4 5
5
GENERAL HEALTH l
Surveillance
. Whole grain l l
after polypectomy consumption and
ischemic stroke Osteoporosis screening Effect of changes in diet & lifestyle
on CHD Inhaled corticosteroid use & BMD in asthma patients . Flu vaccine & healthy adults . Sports participation reflected in hip bone density . Exercise & pelvic muscle performance l Oral glucose tolerance test & the preparatory diet . Comparison of CPR outcomes l Effect of ERT on coronary-artery atherosclerosis
IO 10 11 11
.
GYNECOLOGY . Fertility
outcome after unruptured extrauterine pregnancy Reducing risk of multiple pregnancy after ovarian stimulation OCs & risk of breast cancer Cervical cancer screening Glucose metabolism & OC use Postmenopausal HRT use VT among OC users before 1995 Urinaw incontinence & hysterectomy Preserving ovarian function in patients with germ cell tumors
gent, but nonetheless, time-tested processes and procedures. However, this should not deter us from intense evaluation of the medical literature. ACOG Practice Bulletins, as well as other ACOG documents, use an evaluation system to rate each article that is used to establish a guideline. This rating system is designed to assist the reader in assessing the level of evidence on which the recommendation
6
: : :
II
12 12 12
::
13
9 9
HISTORICAL
PERSPECWE
is based. By using this system it is the desire of our committees to make their decisions as scientifically sound as possible. The College is often called upon to explain how a recommendation was determined. When this occurs, the level of evidence is determined by the quality of the references in the questioned document. To assist our members, the traditional editorial usually found in ACOG Clinical Review has been replaced by this explanation. Each reference in our Practice Bulletins includes a rating level based on the method outlined by the US Preventative Services Task Force. Most studies usually can be separated into those that are observational and those that are interventional. Each of these is further subdivided into a level based on the type of study and how the investigation was performed. A recommendation is then made based on these factors. This method of arriving at a recommendation sounds confusing until one actually tries it with an article. Observational Studies: In this type of study, the investigator observes some event that may occur and records its outcome. This event may be a natural or artificial occurrence. The initial point of observation is used as a primary event. These studies (Continued on p. 14)
14 From ACOG,
02001 by the American College 01 Obstetnclans and Gynecologtsts PublIshed by Elsewer Science Inc 1085.6862/01/$6 00
January/February
2001
l
Washington,
ACOG
D.C.
CLINICAL
REVIEW
l
1
ing organisms produces puerperal felater, another ver.“3 One month woman was admitted to the hospital, suffering from caries of the left knee joint. When all of the other patients sharing the same room with this woman acquired puerperal fever, Semmelweis realized that “. . . the foul exhalations from the carious joint were so strong, that the air of the labour-room in which she was confined, was so loaded that all the patients in the same room became infected. . . . The air of the labourroom, loaded with the putrid matter, found its way into the gaping genitals just at the completion of labour.” Semmelweis immediately insisted upon chlorinated lime hand-washings between patient visits, cleanliness of linens, and isolation of sick patients. Unfortunately, Semmelweis delayed publishing his accomplishments despite the success of his prophylactic measures. His excuse: “an inborn loathing for everything that is called writing.“4 Consequently, knowledge of his doctrine was spread only by visiting medical students and collegial letters. By 1848, one colleague, Ferdinand von Hebra, had published two articles in the Journal of the Vienna Society of Physicians concerning Semmelweis’ discovery, but mistakenly implicated cadaveric particles alone, not living materials, as the source of disease.2 One year later, Semmelweis, unpublished and rejected by the hospital, fled from Vienna to Budapest. In Budapest, Semmelweis supervised the obstetrical unit at St. Rochus Hospital without financial compensation. He improved departmental sanitation by instituting hand-washing prophylaxis, thereby reducing the maternal mortality to 0.85% over the next 6 years.2 By 1855, with the death of a faculty member, Semmelweis became Chairman of the Department of Obstetrics at the University of Pest. In this role, Semmelweis achieved one of his greatest victories: decreasing hospital mortality from puerperal fever to 0.39%.2 Despite his innumerable accomplishments, Semmelweis never received praise or acceptance from his peers. Because Semmelweis never published his doctrine, many physicians turned to the erroneous publica(Continued onp. 1G)
EDITORIAL contiM/
erable if the data are from more than one group or center.
fron, p. I
may be subdivided into a cohort study, a case-controlled study, or a cross sectional study. Each of these studies has major flaws in their design so evaluation of the evidence must take these flaws into consideration. Examples of these flaws include selection bias, lack of knowledge of all related factors, time and exposure relationship unknowns, and unrecognized outside influences. These types of studies never reach the level of evidence that intervention studies attain. Interventional Studies: These studies are usually known as clinical trials. The investigator assigns groups to specific intervention or nonintervention (control or placebo) groups and monitors specific results of the intervention compared with nonintervention. The major subcategories in this type of investigation are whether or not the study includes randomization. Did the investigator assign subjects to specific groups or were they selected prospectively by a process over which the investigator had no control? Obviously, randomization is preferable. A further subcategory is whether the investigator knows prospectively which subject is assigned to each group even if the assignment is random. If they are unaware, then this is a blinded study. The study may also be a crossover study. One-half of the randomly assigned subjects have the intervention for part of the study and then become controls while the controls then receive the intervention. The best of all studies is a blinded, randomized, crossover evaluation. Unfortunately, this method is applicable only in a small number of studies. Based on these two types of evidence, the method of evaluation rating that is applied to each study is as follows: I. Evidence is based on one randomized controlled trial. II-l.
Evidence is based on a nonrandomized controlled trial.
11-2. Evidence is based on a cohort or case-controlled study. It is pref_
11-3. Evidence is based on multiple observations with or without intervention. Uncontrolled studies are in this category. III.
Based on the opinion of authorities, usually as a result of clinical experience or committee opinions.
Once the evidence is rated, a recommendation can be provided. The recommendation is rated based on the following criteria: A. Recommendation based on good and consistent scientific evidence. B. Recommendation based on limited or inconsistent scientific evidence. C. Recommendation based on consensus and expert opinion. As with every rating system, this one is not perfect. It is easy to see that Level I and possibly II-1 allow Level A recommendations. Likewise, Level C recommendations occur with Level III and possible II-3 evidence. Level 11-3, however, as well as Level 11-l and II-2 may also be the basis of a Level B recommendation. Therefore, the reader must make subjective judgments in many instances. I hope this brief explanation has not been confusing. As you read the medical literature, you can apply these principles to assess the studies in a more discerning way. To assist in this process, the editors ofACOG Ciinical Review will, whenever possible, establish a level of evidence for the articles reviewed.