A retrospective comparison of systematic reviews with same-topic rapid reviews

Accepted Manuscript A retrospective comparison of systematic reviews with same-topic rapid reviews Dr. Emily Reynen, Reid Robson, John Ivory, Jeremiah Hwee, Dr. Sharon E. Straus, Dr. Ba’ Pham, Dr. Andrea C. Tricco PII:

S0895-4356(17)30547-4

DOI:

10.1016/j.jclinepi.2017.12.001

Reference:

JCE 9550

To appear in:

Journal of Clinical Epidemiology

Received Date: 18 May 2017 Revised Date:

23 November 2017

Accepted Date: 6 December 2017

Please cite this article as: Reynen E, Robson R, Ivory J, Hwee J, Straus SE, Pham B’, Tricco AC, A retrospective comparison of systematic reviews with same-topic rapid reviews, Journal of Clinical Epidemiology (2018), doi: 10.1016/j.jclinepi.2017.12.001. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

A retrospective comparison of systematic reviews with same-topic rapid reviews a

2

Dr. Emily Reynen

Email: [email protected]

3

Reid Robson

4

John Ivoryb

5

Jeremiah Hwee

6

Dr. Sharon E. Straus

7

Dr. Ba’ Pham

8

Dr. Andrea C. Tricco

9

a

b

Email: [email protected] Email: [email protected] c

RI PT

1

Email: [email protected] b,d

Email: [email protected]

b

Email: [email protected] b,c,*

SC

Email: [email protected]

M AN U

Department of Medicine, Queen’s University, Etherington Hall, Rooms 3032-3043, 94 Stuart Street,

10

Queen's University, Kingston, Ontario K7L 3N6, Canada

11

b

12

Toronto, Ontario M5B 1 W8, Canada

13

c

14

floor, Toronto, Ontario M5T 3M7, Canada

15

d

16

1A1, Canada

17

*Corresponding Author:

18

Dr. Andrea C. Tricco, PhD

19

Scientist, Knowledge Translation Program

20

Li Ka Shing Knowledge Institute, St. Michael’s Hospital

21

209 Victoria Street, East Building, Toronto, Ontario M5B 1W8, Canada

22

Phone: 1-416-864-6060 ext. 77521, Fax: 1-416-864-5805, Email: [email protected]

Li Ka Shing Knowledge Institute of St Michael’s Hospital, 209 Victoria Street, East Building, Room 716,

Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, 6th

AC C

EP

TE D

Department of Geriatric Medicine, University of Toronto, 27 Kings College Circle, Toronto, Ontario M5S

ACCEPTED MANUSCRIPT

ABSTRACT

24

Objective: To compare rapid reviews to same-topic systematic reviews for methods, studies included,

25

and conclusions.

26

Study Design and Setting: A retrospective comparison of studies comparing rapid reviews and

27

systematic reviews by searching four scoping reviews published between 2007-2016. Reports were

28

included if literature searches were conducted within 24 months of each other, and had common research

29

questions. Reviews were compared for duration, studies included, population, intervention, comparisons,

30

outcomes, study designs, quality, methods, and conclusions.

31

Results: Six studies containing 16 review pairs were included, covering nine topics. Overall, rapid

32

reviews more often used abbreviated methods: no search of grey literature, employing one reviewer to

33

screen studies, engaging fewer experts, including fewer studies, and providing shorter reports, with

34

poorer reporting quality and faster completion. Reviews reported similar conclusions, with two exceptions:

35

one systematic review did not include a key study; separately, two rapid reviews failed to highlight an

36

association with early mortality identified by the systematic review. Rapid reviews tended to provide less

37

detail and fewer considerations.

38

Conclusion: Rapid reviews used several methodological shortcuts compared to systematic reviews on

39

the same topic. It was challenging to discern methodological differences because of retrospective

40

assessment and substantial non-reporting, particularly for rapid reviews.

41

KEYWORDS: systematic review, rapid review, methodological, retrospective, comparison

42

Running Title: A Retrospective Comparison of Systematic Reviews with Same-Topic Rapid Reviews

43

Word Count: 199 (abstract – max. 200), 4374 (main text – max. 5000), 5 tables, 1 figure, 1

44

supplementary file

AC C

EP

TE D

M AN U

SC

RI PT

23

ACCEPTED MANUSCRIPT

What is new?

RI PT

Key findings • Generally, rapid reviews used several methodological shortcuts, engaged fewer experts, included fewer studies, had shorter and poorer quality reports, and were completed faster, than systematic reviews on the same topic conducted in a similar time frame. Rapid review conclusions were generally similar to those for systematic reviews, but two rapid reviews failed to report important findings reported by the systematic reviews, which may have been driven in part by missed relevant studies.

•

Some systematic reviews were not well-reported, and one reached a different and inappropriate conclusion compared to the rapid reviews. Systematic reviews generally provided more nuanced conclusions than rapid reviews, and tended to be slightly more conservative.

SC

•

M AN U

What this adds to what is known? • Rapid reviews are not easily assessed against the gold standard systematic review using retrospective comparisons.

EP AC C

46

TE D

What is the implication and what should change now? • A prospective design comparing rapid reviews and systematic reviews concurrently is the more appropriate approach for addressing questions regarding a rapid review’s validity, efficiency, and general suitability as an alternative to a systematic review.

ACCEPTED MANUSCRIPT

47

1. INTRODUCTION While systematic reviews are a comprehensive form of data synthesis, they represent a significant

49

investment in time [1] and resources [2]. Systematic reviews are defined by a structured rigorous

50

approach to obtaining, extracting, appraising and synthesizing individual studies [2]. A review may take

51

up to two years and cost upwards of $100,000 [3]. Rapid reviews have arisen as a timelier, cheaper

52

alternative, and are often requested by healthcare, public health, and governmental organizations [4] to

53

address the needs of clinical, health promotion, preventive medicine, and policy decision makers [2,5].

54

rapid reviews have been found to range from five days to eight months [6]. Tricco and colleagues

55

describe rapid reviews as “a form of knowledge synthesis in which components of the systematic review

56

process are simplified or omitted to produce information in a timely manner” [2]. However, a commonly

57

accepted definition does not yet exist in the literature, because, unlike systematic reviews, rapid reviews

58

lack a standard methodological approach [2], leaving uncertainty regarding the validity of their use.

59

In various scoping reviews examining the application of rapid reviews, researchers have identified studies

60

comparing rapid reviews to systematic reviews [2,6,7]. The comparisons have been examined with

61

varying levels of detail and emphasis, and for different sets of studies. This presents an opportunity to

62

consolidate and comprehensively compare rapid reviews with corresponding systematic reviews of the

63

same topic.

AC C

EP

TE D

M AN U

SC

RI PT

48

ACCEPTED MANUSCRIPT

64

2. METHODS

65

2.1 Objectives The objective of our review was to conduct a retrospective comparison of pairs of systematic reviews and

67

rapid reviews on the same topic by examining their characteristics, methods, conclusions, and quality, as

68

well as studies included. An a priori study design and protocol were used to guide our research

69

(Appendices A, B).

72

The unit of analysis was a systematic review-rapid review “pair”. Eligible pairs required: •

the review question(s) or objective(s) of both rapid reviews and systematic reviews to be clearly stated;

73 74

SC

71

2.2 Eligibility criteria

•

the review questions/objectives were similarly specified in terms of patient population, intervention, control and outcomes;

75

M AN U

70

RI PT

66

•

the literature searches executed within 24 months of one another; and

77

•

the rapid review and systematic review reports must be available and methods described.

78

2.3 Information sources

TE D

76

Systematic review-rapid review pairs were identified from the following sources: a scoping review of

80

methods for developing rapid reviews [8]; a literature review and key-informant interview to identify

81

methods and understand the context for the production of rapid reviews [6]; a scoping review of rapid

82

review methods [2], and an inventory of methods and practice of rapid reviews versus full systematic

83

reviews [7]. These sources were published between 2007 and 2016, and included studies that compared

84

rapid reviews and systematic reviews on the same topic. Their literature searches were last updated in

85

December 2015.

AC C

86

EP

79

2.4 Study selection

87

Based on the information sources above, two reviewers (ER, JI) independently identified studies

88

comparing rapid reviews with systematic reviews, and identified potential pairs. If a study included more

89

than one systematic review or rapid review, for example two systematic reviews and two rapid reviews, it

ACCEPTED MANUSCRIPT

provided multiple potential pairs for our study, in this case four. Reviews were classified as rapid or

91

systematic based on their classification in the information source. Review reports were retrieved based on

92

the information source study reference list. Authors were contacted in cases where reports could not be

93

directly retrieved, and publications were used if reports were not available. Once the reports were

94

reviewed, systematic review-rapid review pairs were determined according to the eligibility criteria.

95

Discrepancies were resolved by discussion. These reports formed the basis for data abstraction.

96

2.5 Data abstraction and quality assessment

RI PT

90

A standardized data abstraction form was used to abstract study characteristics and review methods. A

98

calibration exercise was undertaken by four authors (ER, JI, BP, ACT) prior to full data abstraction. This

99

exercise involved abstraction of one systematic review-rapid review pair, discussion of disagreements for

M AN U

SC

97

abstracted data items, and formation of a standardized set of instructions on how to abstract data. After

101

the calibration exercise, reviews were assigned to two reviewers each (selected from JI, ER, and JH),

102

who abstracted the data independently and resolved disagreements by discussion. A third reviewer (RR)

103

verified the abstracted data. Quality appraisal was undertaken independently by two reviewers (from

104

among JI, ER, JH) using the Assessing the Methodological Quality of Systematic Reviews [9] checklist,

105

after a calibration exercise was undertaken by four authors (ER, JI, BP, ACT). Disagreements were

106

resolved by discussion. Table 1 lists the components abstracted for each review.

2.6 Analysis

EP

107

TE D

100

Results are reported descriptively. Within-pair (systematic review-rapid review pair) comparisons were

109

made for reporting of study characteristics, population, intervention, comparisons, outcomes, study

110

designs, review search methods, methods for study selection and data abstraction, and review

111

conclusions. When comparing rapid reviews and systematic reviews for a particular attribute, for example

112

whether the search language was restricted, the number (%) of pairs where the systematic review

113

reported the search language, and the number (%) of pairs where the systematic review reported a

114

restriction of the search language, were determined. Rapid reviews were summarized similarly, and the

115

results for systematic reviews and rapid reviews were then compared. Quality assessment scores were

116

compared within pairs, as well as on individual items. Review comprehensiveness was compared within

AC C

108

ACCEPTED MANUSCRIPT

pairs by determining the number of studies in each review, and examining why studies were included in

118

one review and not the other.

AC C

EP

TE D

M AN U

SC

RI PT

117

ACCEPTED MANUSCRIPT

119

3. RESULTS

120

3.1 Included studies We identified 12 studies [7,10-20] comparing systematic reviews and rapid reviews, comprising 101

122

systematic review-rapid review pairs across a number of different topics. Six studies [10-15] representing

123

85 systematic review-rapid review pairs were excluded mainly because of insufficient available

124

information and unsuccessful attempts at contacting the authors (Figure 1 and Appendix C). Six studies

125

had accessible data, and were included in our present study, providing 16 systematic review-rapid review

126

pairs involving nine different interventions (Appendices C, D):

Van De Velde and colleagues [16] compared the outcomes of one systematic review [21] and one rapid review [22] that evaluated the medicinal benefit of potato products.

128 129

SC

•

•

M AN U

127

RI PT

121

The Australian Safety
and Efficacy Register of New Interventional Procedures [7] provided a total of seven systematic review-rapid review pairs on four topics: one systematic review and two

131

rapid review’s evaluating drug eluting stents formed two systematic review-rapid review pairs,

132

two pairs evaluating lung volume reduction surgery, two pairs evaluating hip resurfacing, and one

133

pair evaluating living donor liver transplantation. •

healing, and contributed four systematic review-rapid review pairs. •

two pairs.

137 138

•

141 142

The Canadian Agency for Drugs and Technologies in Health 2010 [19] examined aortic valve implantation (one pair).

139 140

Kaltenthaler [18], compared reviews for the use of taxanes in early breast cancer, and contributed

EP

135 136

Peinemann [17] examined the use of negative weighted pressure therapy (NPWT) in wound

•

AC C

134

TE D

130

Lopez 2003 [20] compared an abbreviated and full review on the measurement of temperature in

children (one pair).

3.2 Comparison of Systematic and Rapid Review Characteristics

143

Review characteristics are provided in Table 2 and Appendix E. In 11 of 12 systematic reviews, the term

144

“systematic review” appeared in the title (n=7), summary (n=3) or was found elsewhere in the report (n=1)

145

as a review descriptor. In contrast, “systematic review” did not appear in any titles or descriptors of rapid

ACCEPTED MANUSCRIPT

reviews. Rapid reviews used descriptors such as “a short-cut review”, “rapid report”, “technology

147

assessment”, and “TechNote”.

148

Generally, rapid reviews were described as “rapid” and/or not systematic in their own reports, or in the

149

reports comparing them to systematic reviews (i.e. in our included studies). In examining our six included

150

studies, we see: Van de Velde referred to their own review as “rapid” [16]; the Australian Safety and

151

Efficacy Register of New Interventional Procedures study (7 rapid reviews) defined a rapid review as “any

152

HTA report or systematic review that has taken 1-6 months to produce which contains the elements of a

153

comprehensive literature search” [7]; Peinemann examined a “rapid report” identifying studies published

154

after a full systematic review [17]; Kaltenthaler simply sought to compare a full NICE technology appraisal

155

(which contains a systematic review) to two “rapid appraisals”, known as single technology appraisals,

156

which included a potentially less rigorous review conducted by the submitting manufacturer [18];

157

Canadian Agency for Drugs and Technologies in Health noted that their rapid review was “based on a

158

limited literature search and are not a comprehensive, systematic review” [19]; and the rapid review in

159

Lopez came from a journal series comprised of summaries that are described as “not systematic reviews”

160

[20].

161

The reviews were reported or published between 2002 and 2010, all within 22 months of their paired

162

review, and with just over half of the rapid reviews reported/published after their paired systematic review.

163

The rapid review for wound therapy was an update to one of the four earlier systematic reviews by the

164

same research group, with a search restricted to the subsequent 19 months, intended to determine if

165

there was sufficient new information to warrant a full review.

166

Overall, systematic reviews had more authors, a longer report length, required a longer time frame to

167

complete, and were more likely to use an expert panel/review compared to rapid reviews (Table 2 and

168

Appendix E). For three systematic reviews, only the publication was available, and not the online report.

169

Excluding these three studies, the median systematic review duration was 9.5 months and the median

170

page count was 112. Rapid review median duration and page counts were 3 months and 22 pages

171

respectively.

AC C

EP

TE D

M AN U

SC

RI PT

146

ACCEPTED MANUSCRIPT

172

3.3 Comparison of Systematic and Rapid Review Quality Examining the items on the Assessing the Methodological Quality of Systematic Reviews checklist

174

(Appendix G), no reviews reported on conflict of interest, and nearly all reviews failed to assess

175

publication bias and list included/excluded studies. Nearly all reviews failed to report use of an “a priori”

176

design, and whether there was duplicate study selection and data abstraction. Nonetheless, systematic

177

reviews generally had higher scores. The overall median score was 5.5 for systematic reviews and 2 for

178

rapid reviews with a median difference within pairs of 1.5. If the four wound therapy pairs are excluded,

179

the median difference increases to 2.5.

180

The higher within-pair systematic review scores were driven by multiple items: provision of an a priori

181

design, duplicate selection and abstraction, performance of a comprehensive literature search, grey

182

literature search, listing included and excluded studies, as well as listing included study characteristics. In

183

these cases, the systematic review had the same or higher score than its paired rapid review, with few

184

exceptions. Of these items, three had more within-pair differences than the others: duplicate selection

185

and abstraction, comprehensive search, and grey literature search.

187

SC

M AN U

3.4 Comparison of Systematic Review and Rapid Review - population, intervention, comparisons,

TE D

186

RI PT

173

outcomes, study designs

Of the nine topic areas, there were three topics where systematic reviews had broader objectives than the

189

rapid review: Potato Products, Drug-eluting Stents, and Taxanes Chemotherapy (Appendix F).

190

Some systematic reviews were restricted by eligible study designs: two systematic reviews (Potato

191

products, drug eluting stents) included only randomized controlled trials, while their corresponding rapid

192

reviews also allowed the inclusion of observational studies. Because the systematic review for Potato

193

Products restricted inclusion criteria to randomized controlled trials, this meant the comparator was limited

194

to honey, while honey, gauze, and banana leaf were all comparators in the rapid review. As well, two

195

systematic reviews in Wounds, the systematic review for Aortic Valve Implementation, and one rapid

196

review in Liver Donation, required a minimum number of patients per arm, while this was not required for

197

their corresponding reviews. In two cases (drug eluting stents, and Liver Donation), more outcomes were

198

examined in the systematic review.

AC C

EP

188

ACCEPTED MANUSCRIPT

199

3.5 Comparison of Systematic and Rapid Review Methods – Search The search steps common to systematic reviews are reported in Table 3 for each review. Many of the

201

steps for both review types were not fully reported, and for many search methods, there was less

202

reporting for rapid reviews. Overall, systematic reviews were more likely to report a clear research

203

question compared to rapid reviews. Only one systematic review failed to report a clear search strategy,

204

compared to six rapid reviews. Three of the systematic reviews reported that a published protocol was

205

used, compared to none of the rapid reviews.

206

In assessing limits on search dates, wound therapy (4 pairs) reviews are considered separately because

207

the intent, by design rather than limitation, of the sole rapid review was to examine whether sufficient new

208

information had accumulated since the original systematic review, a year and half earlier. The other

209

wound reviews, all systematic reviews, did not restrict by date. Outside of the topic of wounds, many

210

search date details were not reported. Seven (58%) of the 12 pairs had at least one review not reporting

211

whether the search was limited by date. In six of these seven pairs, the rapid review that did not report

212

this information. Only three of 12 pairs had a review reporting no limits on dates. In nine (75%) of 12

213

pairs, the systematic review reported a date limit, compared to four of 12 (25%) pairs where the rapid

214

review reported a limit. However, in five of the nine pairs, the rapid review did not report whether limits

215

were used. In only one pair did the systematic review not report this information.

216

Search language limits were frequently not reported, where 14 (88%) of 16 pairs had at least one review

217

not reporting this information. In 11 of these 14 pairs, the rapid review did not report, and in three, the

218

systematic review did not report. Of the 16 pairs, six (38%) reported there were no language limits, all

219

solely attributed to the systematic review in the pair. There were no rapid reviews that reported language

220

was not limited. Overall, report of a language limits (usually English) was slightly less common for rapid

221

reviews than systematic reviews (Table 2 and 3).

222

All systematic reviews reported searching more than one database, compared to all but four rapid

223

reviews. One rapid review reported searching one database only, and three rapid reviews did not report

224

this information. In 15 (94%) of 16 pairs, the systematic review in the pair reported searches of the grey

225

literature, while this was the case for rapid reviews in only 7 (44%) pairs. There were six pairs (38%)

AC C

EP

TE D

M AN U

SC

RI PT

200

ACCEPTED MANUSCRIPT

where the rapid review failed to report whether a grey literature search was conducted, compared to one

227

pair (6%) where the systematic review failed to report. The most frequent sources of grey literature were

228

internet sites, conference proceedings, trial registries and regulatory agency websites.

229

Many of the reviews did not report whether the reference lists were scanned. Twelve (75%) pairs had at

230

least one review not reporting this information, and the non-reporting was similar between rapid reviews

231

and systematic reviews (roughly 55%). Reporting that references were scanned was also similar

232

(approximately 45%).

233

All pairs had at least one review that did not report whether authors were contacted. There were more

234

pairs (14) where the systematic review had an absence of reporting (63%), compared to pairs (8) where

235

the rapid review did not report (50%) this information. Four of the six reports of author contact occurred in

236

a rapid review.

SC

M AN U

237

RI PT

226

3.6 Comparison of Systematic Rapid Review Methods – Selection, Abstraction, Appraisal In the screening of titles and abstracts, 15 of 16 (94%) systematic review-rapid review pairs had at least

239

one review not reporting this step. Twelve (75%) and five (31%) pairs had a rapid review and systematic

240

review not reporting respectively, while four (25%) and nine (56%) pairs had a rapid review and

241

systematic review reporting use of two or more independent reviewers respectively. Overall, the

242

systematic reviews reported much more complete study selection, data abstraction and quality appraisal

243

details, and much more commonly reported the use of two independent reviewers, compared to their

244

corresponding rapid review (Table 4).

245

For meta-analyses, nine (56%) of 16 pairs reported one was not done for either review in the pair, while

246

two (13%) pairs reported that both their reviews conducted a meta-analysis. Three (19%) pairs reported a

247

meta-analysis conducted for the systematic review but not the rapid review, and one (6%) pair reported

248

the opposite (Table 4).

249

AC C

EP

TE D

238

3.7 Comparison of Included Studies between Systematic and Rapid Reviews

250

As shown in Appendix H, systematic reviews included more studies than their rapid review pairs for all

251

interventions examined. When considering only studies specific to the rapid review objective, the

ACCEPTED MANUSCRIPT

associated systematic review included more studies in all but two pairs. As well, in the ten pairs where

253

relevant randomized controlled trials were found, systematic review’s included the same number or more

254

randomized controlled trials than their rapid review pair. When systematic reviews did not include the

255

randomized controlled trials included in their paired rapid review, typically this was because the

256

systematic review research question was more restrictive, usually in terms of design criteria. Table 5

257

outlines the reasons systematic review-rapid review pairs did not include the same studies.

258

3.8 Comparing Conclusions between Systematic and Rapid Reviews

RI PT

252

Compared with rapid reviews, systematic review conclusions were generally more detailed and nuanced

260

(drug eluting stents, lung surgery, liver donation, hip resurfacing), providing information on subgroups,

261

health system implications and relevant factors, aspects of implementation, or where further research was

262

needed. Conclusions for the 16 systematic review-rapid review pairs were generally consistent, with some

263

exceptions (Appendix I). In the medicinal application of potato peels, the systematic review did not

264

recommend topical potato peel for burns, because it was less effective than honey, while the rapid review

265

did, because it was more effective than gauze alone. The rapid review authors conjectured the systematic

266

review may have missed a relevant trial comparing to gauze alone. In lung volume reduction surgery, only

267

the systematic review highlighted an increased risk of short-term death with treatment, even though

268

studies highlighting this risk had been included in one of two the rapid reviews. In live donor liver

269

donation, the systematic review highlighted that right lobe donors might not be left with sufficient liver

270

reserve, while the rapid review concluded that the single identified systematic review was outdated, and

271

the current data far from complete. The systematic review also did not include one study that the rapid

272

review did include. No explanation for this could be found.

AC C

EP

TE D

M AN U

SC

259

ACCEPTED MANUSCRIPT

273

4. DISCUSSION As expected, we found that rapid reviews were generally conducted much faster than their systematic

275

review pairs. Overall, rapid reviews had shorter reports, fewer authors, and were less likely to use an

276

expert panel, aspects to our knowledge that have not been directly compared before. In most cases,

277

systematic reviews had more complete reporting compared to rapid reviews. This included the research

278

question, search strategy, whether a protocol was published, as well as details of methods for selection,

279

abstraction, and study appraisal. We observed that rapid reviews limited the search language to English

280

more often, and searched grey literature less often than systematic reviews. Surprisingly, more rapid

281

reviews than systematic reviews reported author contact. The reasons for this are not entirely clear. While

282

this may be an anomaly, it is possible that because of the use of abbreviated methodology, review

283

authors may have more time to spend contacting authors during the review process.

284

Systematic review conclusions in several cases were more detailed and nuanced than those in the rapid

285

review, providing information on implementation and recommendations for further research. Conclusions

286

for the 16 systematic review-rapid review pairs were generally consistent, with two exceptions noted

287

(potato peels, lung volume reduction surgery). Other than the identified inconsistency in the review

288

conclusions for lung volume reduction surgery, our findings are consistent with the original comparative

289

studies from which the reviews were obtained.

290

The overall quality of the included systematic review-rapid review pairs was low on the Assessing the

291

Methodological Quality of Systematic Reviews scale, although systematic reviews were numerically

292

higher compared to rapid reviews. The lower scores for the rapid review indicates that although similar

293

conclusions were drawn compared to the corresponding systematic review, the overall quality of the rapid

294

review is lower and therefore may have a higher risk of bias. Rapid reviews often streamlined the

295

screening of titles and abstracts, and full-text articles, the abstraction of data, and conduct of quality

296

appraisal, likely leading to the lower scores. These various shortcuts, and their permutations, taken by

297

rapid reviews were also described by Tricco et al. [2].

298

Several items related to review conduct were not well reported for either review type (systematic review or

AC C

EP

TE D

M AN U

SC

RI PT

274

ACCEPTED MANUSCRIPT

rapid review). This finding is consistent with the findings of Page et al. who highlighted poor reporting of a

300

large cross-section of systematic reviews in 2016 [23]. Incomplete reporting seen in systematic reviews

301

may represent modifications made to the generally accepted standard to which systematic reviews are

302

held, as outlined in the Cochrane Handbook [24]. This raises the possibility that systematic reviews and

303

rapid reviews exist on a methodological continuum rather than as discrete entities. As such, merely

304

classifying a study based on terminology (either systematic review or rapid review) may not be a true

305

representation of the methodology that was used.

306

In most cases, the systematic reviews included as many, and most often more, studies than their rapid

307

review pairs. However, we found several pairs where the systematic review had applied selection criteria

308

based on study design or a minimum number of patients, whereas the rapid review did not. Not including

309

cases where systematic review selection criteria were narrower than their rapid review pairs, systematic

310

reviews generally included all studies included in the rapid review, with one exception (potato peels; one

311

study), and another that could not be accounted for (liver donation; one study). Rapid reviews often did

312

not include studies that systematic reviews did, because of narrower research questions and eligibility

313

criteria, and in some cases, lack of access to unpublished data or industry submissions. A narrowed

314

focus would be expected for a rapid review that must produce a report in a condensed period with limited

315

resources. Unfortunately, in some rapid reviews, it was difficult to account for absent studies (i.e. studies

316

included in the systematic review, but not the rapid review) because of a lack of reporting details.

317

The literature search from the last of our four key information sources was conducted in May 2013 and

318

last updated in December 2015. No other relevant studies were identified in 2016, as we regularly

319

reviewed alerts on systematic review methods (PUBMED filter sysrev_methods[sb]), and the database of

320

methods publications maintained by the Cochrane Rapid Review Methods group.

321

One limitation of our study is the lack of standard definition of the rapid review. As a result, it is possible

322

that differences in methodology exist within the included rapid reviews. This potential for heterogeneous

323

rapid review methodology makes it difficult to extrapolate our findings to all rapid reviews. Another

324

limitation to our study is the large number of identified pairs we were unable to include because reports

325

were not retrievable despite repeated contact of the review producers. It is unknown what effects this may

AC C

EP

TE D

M AN U

SC

RI PT

299

ACCEPTED MANUSCRIPT

have on our study’s generalizability. Also, to increase consistency, systematic reviews and rapid reviews

327

were classified according to the information source. As such, it is possible some reviews may have been

328

misclassified. We also obtained multiple pairs based on the same systematic review or rapid review.

329

These pairs are thus not completely independent, and this has the potential to bias our findings. There

330

was substantial missing data on the methods used by both systematic reviews and rapid reviews, more

331

so for rapid reviews and in several cases, we were unable to discern differences in application of

332

methods. We had originally intended to appraise the included systematic reviews and rapid reviews using

333

the ROBIS tool [25]. However, due to a lack of resources, we were unable to complete this exercise and

334

instead used the Assessing the Methodological Quality of Systematic Reviews checklist.

335

Overall, the systematic reviews had higher methodological quality scores compared to the rapid reviews,

336

however, the absolute scores were low suggesting that the systematic reviews were of poor quality. What

337

is not clear is whether a rapid review with a high score is a more valuable source of data synthesis

338

compared to a poor-quality systematic review. Our source for rapid review-systematic review pairs

339

comprised studies that compared rapid reviews and systematic reviews on the same topic. An alternative

340

approach would be to identify rapid reviews, and match them, through a search, with systematic reviews

341

on the same topic, conducted within a similar time frame. Depending on the way rapid reviews are

342

identified, such an approach may entail a much larger undertaking, but could yield substantially more

343

pairs if reports are accessible. However, this is unlikely to resolve the issue of substantial missing data

344

regarding the methods used, as we noted in particular for rapid reviews.

345

These important limitations make it clear that a prospective comparative study, having stronger internal

346

validity, and planned data collection, would be much more informative in terms of understanding the

347

differences between systematic reviews and rapid reviews. Such a design would allow and include:

348

proximal study start and search dates, concurrent follow-up using overlapping and comparable

349

population, intervention, comparisons, outcomes, study designs, pre-identified outcomes data collection,

350

and prospective capture of detailed methods and the resources and time required, so that the trade-offs

351

of information with expediency can be assessed. The Systematic Prospective Assessment of Rapid

352

Knowledge Synthesis study has been designed with these features, and is currently underway [26].

AC C

EP

TE D

M AN U

SC

RI PT

326

ACCEPTED MANUSCRIPT

353

5. CONCLUSION Overall, rapid reviews were generally found to use abbreviated methods compared to their systematic

355

review counterparts more often, and commonly did not search the grey literature, or use two reviewers to

356

screen and appraise studies or abstract data. However, it was challenging to discern methodological

357

differences because of substantial non-reporting, particularly for rapid reviews, and the retrospective

358

nature of our study. Nonetheless, similar overall conclusions were reported for rapid reviews and

359

systematic reviews, with two exceptions. One review pair was discordant in their conclusion, because the

360

systematic review did not include a key study, a reminder that all reviews must also be conducted

361

carefully and reported transparently. In another case, two rapid reviews failed to highlight an association

362

with early mortality identified in the systematic review. Rapid reviews tended to provide less detail and

363

fewer considerations than their corresponding systematic reviews. Many of the limitations observed in our

364

study would be addressed in a prospective comparative study. Such a study is underway.

AC C

EP

TE D

M AN U

SC

RI PT

354

ACCEPTED MANUSCRIPT

DECLARATIONS

366

Competing interests

367

Dr. Andrea C. Tricco is an associate editor and Dr. Sharon E. Straus is on the policy advisory board for

368

the journal; all other authors do not have any conflicts of interest to declare.

369

Funding

370

This work was supported by an Ontario Ministry of Research, Innovation, and Science Early Researcher

371

Award that was awarded to ACT. SES is funded by a Tier 1 Canada Research Chair in Knowledge

372

Translation. ACT is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. The funders had

373

no role in the design and conduct of the study; data collection, analysis and interpretation of data; writing

374

of the report; and in the decision to submit the article for publication.

375

Authors' contributions

376

ER, JDI, ACT, BP all made substantial contributions to conception and design, acquisition of data, and

377

analysis and interpretation of data. SES made substantial contributions to conception and design and

378

analysis and interpretation of data. RR made substantial contributions to analysis and interpretation of

379

data, and led the writing of the manuscript. All authors were involved in the drafting and revision of the

380

manuscript and gave final approval of the version submitted for publication.

381

Acknowledgements

382

In addition to the above team, Alissa Epworth (an experienced librarian) conducted our updated literature

383

search. Aline Chhun, Theshani De Silva, Susan Le, and Krystle Amog formatted the manuscript and

384

tables.

385

Authors' information

386

Dr. Reynen is a third-year internal medicine resident at Queen’s University in Kingston, Ontario. Dr.

387

Reynen contributed to this project as part of a research component of her core internal medicine training.

388

She is a graduate of McGill University’s medical school. Emily has also completed an undergraduate and

389

doctorate degree in Pharmacy at the University of Toronto. She practiced as a primary care pharmacist at

AC C

EP

TE D

M AN U

SC

RI PT

365

ACCEPTED MANUSCRIPT

Two Rivers Family Health Team in Cambridge Ontario. Emily has experience in systematic review, rapid

391

review and network meta-analysis methodology. She worked as a clinical research officer for the

392

Canadian Agency for Drugs and Technologies in Health. Emily has worked as a research coordinator at

393

the Li Ka Shing Knowledge Institute of St. Michael’s Hospital since March of 2013.

394

Mr. Reid Robson has a Masters of Mathematics from the University of Waterloo, and has substantial

395

experience in the design and analysis of clinical trials and health economic data, as well as medical

396

writing. Mr. Robson is a part-time research associate at Li Ka Shing Knowledge Institute.

397

John Ivory graduated from the National University of Ireland (Galway) with a BSc in Biochemistry and a

398

MSc in Neuropharmacology. He worked for 3 years as a clinical research coordinator in the Wound Care

399

Clinic of Dr. R. Gary Sibbald in Mississauga ON, and has been working with the Li Ka Shing institute

400

since January 2013, where he has gained experience in systematic review, rapid review and network

401

meta-analysis methodologies.

402

Jeremiah Hwee is a PhD student at the Epidemiology Division, Dalla Lana School of Public Health,

403

University of Toronto, Toronto, Canada. Jeremiah is funded by the Frederick Banting and Charles Best

404

Canada Graduate Scholarship Doctoral Award from the Canadian Institutes for Health Research.

405

Dr. Ba’ Pham is a research associate, health economist, decision modeler, systematic review

406

methodologist, and biostatistician.

407

Dr. Straus is a geriatrician, clinical epidemiologist and professor, and director of the KT Program of SMH

408

and of Geriatric Medicine, University of Toronto. She holds a Tier 1 Canada Research Chair in KT, has

409

been awarded more than $30 million in peer-reviewed grants in the past 10 years, and has published

410

>350 peer-reviewed publications.

411

Dr. Tricco is a Scientist at the Li Ka Shing Knowledge Institute of St. Michael’s Hospital, Associate

412

Professor in the Epidemiology Division of the Dalla Lana School of Public Health, University of Toronto,

413

and holds a Tier 2 Canada Research Chair in Knowledge Synthesis and an Ontario Ministry of Research,

414

Innovation, and Science Early Researcher Award.

415

AC C

EP

TE D

M AN U

SC

RI PT

390

ACCEPTED MANUSCRIPT

416

TABLES

417 418

Table 1. Data abstraction and quality assessment of systematic and rapid reviews in the systematic review-rapid review pairs •

Reported objective of the study, in the abstract, summary, methods, or introduction

Review characteristics

•

Date of report (or publication, if report unavailable), title, journal of publication, country of conduct, produced by, funded by, review duration (derived) Number of authors, search dates, peer review or use of an expert panel Number of report pages, treatments and comparators Review descriptors, e.g. “systematic review”, “rapid review”, “rapid report”

RI PT

Review objectives

• • •

Review search methods

•

Review selection and appraisal methods Studies included

•

Review population, intervention, comparisons, outcomes, eligible study designs Use of a review protocol, clarity of research question, literature search dates, number of data bases searched, language or date restrictions, use of grey literature (literature not formally published in sources such as books or journal articles), scanning of references, contacting authors

SC

•

Methods for screening, data abstraction and quality appraisal, whether a meta-analysis was conducted Total number of included studies, number of studies in the systematic review relevant to the corresponding rapid review objective Number of RCTs (randomized controlled trials) and non-RCTs included Overlap of included studies between SR and RR pairs, reasons studies were included in only one review

M AN U

Review PICOS

•

Data synthesis methods

•

Review conclusion

• •

Quality assessment using the AMSTAR checklist. A calibration exercise was undertaken by four authors (ER, JI, BP, ACT), and after calibration, quality appraisal was undertaken independently by two reviewers (from among JI, ER, JH). Disagreements were resolved by discussion

Abbreviations: AMSTAR = Assessing the Methodological Quality of Systematic Reviews; PICOS = population, intervention, comparison, outcome, study-design; RCT = randomized controlled trials; RR = Rapid Review; SR = Systematic Review.

AC C

419 420

•

EP

Review quality assessment

Method of data synthesis. We categorized the presentation of results into six categories: meta-analysis and narrative summary, narrative or descriptive summary only, meta-analysis or pooling only, clear reasons for not undertaking meta-analysis, unclear or limited data-synthesis, and method not reported Conclusion statements found in the abstract, discussion, summary, or report conclusions Conclusions were compared qualitatively for consistency and emphasis

TE D

• •

ACCEPTED MANUSCRIPT

Table 2. Summary of key comparison of systematic reviews and rapid reviews Report length (pages), range (median)

Expert review panel, n (%)

Review duration, range (median)

Clear research question, n (%) 11 (92) 9 (69)

TE D

M AN U

SC

SR 3−11 (5) 4-673 (55) 9 (75) 2 m−3.5 y (10.5 m) RR 1−12 (1) 2-115 (22) 5 (38) 1−6.5 m (3 m) Abbreviations: m = month; RR = Rapid Review; SR = Systematic Review; y = year.

EP

422

No. of authors, range (median)

Search limits by language, n (%)

RI PT

Study design

AC C

421

44% 31%

Number of included studies, range 5−214 2−24

ACCEPTED MANUSCRIPT

Table 3. Comparison of systematic review and rapid review methods - search Search Limits by Language

NR

Y

NR

NR

Databases searched

Grey literature Searched

RI PT

Search Limits by Date

Search strategy clear

Scanned references

Contacted authors

NR

Y

Limits

English

SR RR RR

Hill (2003) [27] Brophy (2003) [28] Lynch (2003) [29]

Y NR NR

Limits NR NR

English NR NR

Lung Volume Reduction Surgery

SR

Banerjee (2004) [30]

Y

Y Y Unclear/ Inferred Y

Limits

No limit

>1

Y

Y

NR

NR

RR

Mickman (2003) [31]

NR

NR

NR

NR

NR

NR

NR

NR

RR

NICE (2005) [32]

NR

SR RR RR SR RR SR

Vale (2001) [33] Bernath (2002) [34] AHFMR (2002) [35] Middleton (2004) [36] AHFMR (2004) [37] Costa (2005) [38]

NR NR NR NR NR NR

SR SR SR RR SR RR

IQWiG (2006) [39] OHTAC (2006) [40] Pham (2006) [41] IQWiG (2007) [42] Ward (2007) [43] Chilcott (2006) [44]

Y NR NR NR NR NR

RR

Griffin (2006) [45]

NR

SR

McGregor (2009) [46]

RR

Taxanes Chemotherapy

Aortic Valve

Paediatric Thermometry

Y

Unclear

NR

NR

>1

NR

Y

NR

NR

>1 NR 1 only

Y NR NR

Y NR NR

Y NR Y

NR NR Y

M AN U No limit

English

>1

Y

Y

NR

NR

Limits NR Limits Limits Limits No limit

>1 >1 >1 >1 >1 >1

Y NR Y Y Y Y

Y Y Y Y Y Y

NR Y NR Y Y Y

NR NR Y NR Y NR

>1 >1 >1 >1 >1 >1

Y Y Y Y Y Y

Y Y Y Y Y NR

Y NR NR Y NR NR

Y NR NR Y NR NR

Y Y Y Y Y Unclear/ Inferred Y

No limit No limit No limit Limits No limit NR

English NR English No limit English English, French NR English No limit No limit No limit NR

No limit

NR

>1

Y

Y

Y¥

Y

NR

NR

Limits

>1

NR

Y

NR

NR

Murphy (2010) [47]

NR

Y

Limits

English, French English

>1

NR

Unclear

NR

NR

SR

Craig (2002) [48]

NR

Y

Limits

NR

>1

Y

Y

Y

Y

RR

Riddell (2001) [49]

NR

Y

NR

NR

NR

N

NR

NR

NR

TE D

Negative Pressure Wound Therapy

Unclear/ Inferred Unclear/ Inferred Y Y Y Y Y Y

>1

SC

RR Drug Eluting Stents

Liver Donation

424 425 426 427

Review question clear

Vlachojannis (2009) [21] De Buck (2010) [22]

Hip Resurfacing

SR

Published Protocol*

EP

Potato Products

Author (year)

AC C

423

Abbreviations: AHFMR = Alberta Heritage Foundation for Med Research; IQWiG = Institute for Quality and Efficiency in Health Care; NICE = The National Institute for Health and Care Excellence; NR = not reported; OHTAC= Ontario Health Technology Advisory Committee; RR = Rapid Review; SR = Systematic Review. *report refers to a protocol.

ACCEPTED MANUSCRIPT

Table 4. Comparison of systematic review and rapid review methods – selection, abstraction, appraisal

Lung Volume Reduction Surgery

Hip Resurfacing

Liver Donation

SR

Vlachojannis (2009)

RR SR RR RR

De Buck (2010) Hill (2003) Brophy (2003) Lynch (2003)

SR

Banerjee (2004)

≥2 independent reviewers

≥2 independent reviewers

≥2 independent reviewers

RR

Lynch (2003)

NR

NR

NR

RR

NICE (2005)

SR

Vale (2001)

Done but unclear # of reviewers ≥2 independent reviewers

RR

Bernath (2002)

NR

Done but unclear # of reviewers Done but unclear # of reviewers NR

RR SR

AHFMR (2002) Middleton (2004)

NR ≥2 independent reviewers

NR NR

RR

AHFMR (2004)

NR

NR

SR

Costa (2005)

NR

NR

AC C

429 430

≥2 independent reviewers

Done but unclear # of reviewers NR ≥2 independent reviewers NR Done but unclear # of reviewers Done but unclear # of reviewers Done but unclear # of reviewers NR

NR 1 reviewer and 1 verifier NR NR

Done but unclear # of reviewers ≥2 independent reviewers Y; unclear # of reviewers NR Done but unclear # of reviewers Done but unclear # of reviewers Done but unclear # of reviewers 2 independent reviewers

Meta-analysis performed N N Y N N Y Y Y

≥2 independent reviewers

N

Done but unclear # of reviewers NR NR

Unclear / limited data synthesis N N

NR

N

Done but unclear # of Clear reasons for reviewers not doing MA SR IQWiG (2006) 2 independent reviewers 2 independent reviewers Done but unclear # of Y reviewers SR OHTAC (2006) NR NR 1 reviewer and 1 verifier Done but unclear # of Clear reasons for reviewers not doing MA SR Pham (2006) NR NR 1 reviewer and 1 verifier Done but unclear # of N reviewers RR IQWiG (2007) 2 independent reviewers 2 independent reviewers 1 reviewer and 1 verifier 2 independent reviewers N Taxanes SR Ward (2007) 1reviewer and 1verifier 1 reviewer and 1 verifier 2 independent reviewers Done but unclear # of Clear reasons for Chemotherapy reviewers not doing MA RR Chilcott (2006) NR NR NR Done but unclear # of N reviewers RR Griffin (2006) NR NR NR Done but unclear # of N reviewers Aortic Valve SR McGregor (2009) NR NR NR NR N RR CADTH (2010) NR NR NR NR N Paediatric SR Craig (2002) 2 independent reviewers 2 independent reviewers 2 independent reviewers 2 independent reviewers Y Thermometry RR Archimedes (2001) NR NR NR NR N Abbreviations: AHFMR = Alberta Heritage Foundation for Med Research; CADTH = Canadian Agency for Drugs and Technologies in Health; IQWiG = Institute for Quality and Efficiency in Health Care; MA = meta-analyses; NR = not reported; OHTAC = Ontario Health Technology Advisory Committee; RR = Rapid Review; SR= Systematic Review.

EP

Negative Pressure Wound Therapy (2005-7)

Quality Appraisal

RI PT

Drug Eluting Stents

Data Abstraction

SC

Potato Products

Selecting Relevant Studies Titles & Abstracts Full Text Done but unclear # of NR reviewers NR NR ≥2 independent reviewers ≥2 independent reviewers NR NR NR NR

M AN U

Author (year)

TE D

428

ACCEPTED MANUSCRIPT

431

Table 5. Reasons systematic and rapid review pairs did not include the same studies

Simplified search terms, use of a search filter Search restrictions on language or date Focus on new drugs only, or drugs soon to be approved Focus on a particular comparator Pre-specified treatment sequence (oncology) Pre-specified outcomes Published studies only Studies must have had a minimum number of patients SRs had access to industry submissions and unpublished data Trial analysis was not conducted according to randomized groups

SC

• • • • • • • • • •

RI PT

Reasons rapid reviews did not include studies that their paired systematic review did

Reasons systematic reviews did not include studies that their paired rapid review did

EP

TE D

Abbreviations: RCT = Randomized Controlled Trial; RR = Rapid Review; SR = Systematic Review.

AC C

432

More restrictive population or research question Restrictions on study design, e.g. including only RCTs RRs including studies reported after SR search dates SR omitted conference abstract of an interim analysis, but included more recent abstract of the full analysis Reason unknown (1 study)

M AN U

• • • • •

ACCEPTED MANUSCRIPT

FIGURE LEGEND

434

Fig. 1. Flow diagram of included and excluded studies.

435

ADDITIONAL FILES

436

File name: Supplementary File

437

Title of data: Appendices

438

Description of Data: The appendices include all supplemental data and information.

439

Appendix A. Study design and protocol

440

Appendix B. Medline search strategy

441

Appendix C. Excluded studies and reasons for exclusion

442

Appendix D. Included studies and their reviews

443

Appendix E. Comparison of systematic review and rapid review characteristics

444

Appendix F. Comparison of systematic review and rapid review for PICOS

445

Appendix G. Comparison of systematic review and rapid review quality using AMSTAR

446

Appendix H. Comparison of systematic reviews and rapid reviews for studies included

447

Appendix I. Comparison of systematic review and rapid review conclusion

AC C

EP

TE D

M AN U

SC

RI PT

433

ACCEPTED MANUSCRIPT

REFERENCES

449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495

1. Tricco AC, Cardoso R, Thomas SM, Motiwala S, Sullivan S, Kealey MR, et al. Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Implement Sci. 2016;11:4. 2. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224. 3. Petticrew M RH. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006. 4. Tricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open. 2016;6(12):e013929. 5. Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10. 6. Hartling L, Guise JM, Kato E, et al. . EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews. Rockville: Agency for Healthcare Research and Quality (US); 2015. Report No.: 15-EHC008-EF. 7. Cameron A, Watt A, Lathlean T, Sturm T. Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment. Adelaide: Australian Safety and Efficacy Register of New Interventional Procedures – Surgical (ASERNIPS); 2007. Report No.: 60. http://www.surgeons.org/media/297941/rapidvsfull2007_systematicreview.pdf [Last accessed 02.02.2017] 8. Abou-Setta AM, Jeyaraman MM, Attia A, Al-Inany HG, Ferri M, Ansari MT, et al. Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review. PLoS One. 2016;11(12):e0165903. 9. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. 10. Corabian P, Harstall, C. Rapid assessments provide acceptable quality advice. Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet. 2002;18:Abstract 70. 11. Best L, Stevens, A, Colin-Jones, D. Rapid and responsive health technology assessment: the development and evaluation process in the South and West region of England. Journal of Clinical Effectiveness. 1997;2:51-6. 12. Rizzo M, Llewellyn, A, Martin, A. A rapid systematic review versus a Cochrane systematic review: an empirical comparison. 19th Cochrane Colloquium, Supplement. (Suppl(CD000003)); Madrid, Spain. 2011. p. 102. 13. Cairns J. Providing guidance to the NHS: The Scottish Medicines Consortium and the National Institute for Clinical Excellence compared. Health Policy. 2006;76(2):134-43. 14. Warren V. Health technology appraisal of interventional procedures: comparison of rapid and slow methods. J Health Serv Res Policy. 2007;12(3):142-6. 15. Saz Parkinson Z, Lopez-Cuadrado, T, Plama-Ruíz Bouza, C, Amate, JM, Sarria, A. Preliminary study of rapid vs. exhaustive HTA reports: comparison of their usefulness in the decision-making process [abstract]. HTAi 7th Annual Meeting: Maximising the Value of HTA; Dublin. 2010. p. 162-3. 16. Van de Velde S, De Buck E, Dieltjens T, Aertgeerts B. Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review. Phytother Res. 2011;25(5):787-8.

AC C

EP

TE D

M AN U

SC

RI PT

448

ACCEPTED MANUSCRIPT

EP

TE D

M AN U

SC

RI PT

17. Peinemann F, McGauran N, Sauerland S, Lange S. Disagreement in primary study selection between systematic reviews on negative pressure wound therapy. BMC Med Res Methodol. 2008;8:41. 18. Kaltenthaler E, Tappenden P, Booth A, Akehurst R. Comparing methods for full versus single technology appraisal: a case study of docetaxel and paclitaxel for early breast cancer. Health Policy. 2008;87(3):389-400. 19. Health Technology Inquiry Service (HTIS). Transcatheter Aortic Valve Implantation: A Critical Appraisal of a Health Technology Assessment and Comparison with a Rapid Review. (CADTH), Canadian Agency for Drugs and Technologies in Health; 2010. 20. Lopez T. On Archimedes. Arch Dis Child [Internet]. 2003; 88(2):[176-7 pp.]. Available from: http://adc.bmj.com/content/archdischild/88/2/176.2.full.pdf. 21. Vlachojannis JE, Cameron M, Chrubasik S. Medicinal use of potato-derived products: a systematic review. Phytother Res. 2010;24(2):159-62. 22. De Buck E, Van de Velde S. Towards evidence based emergency medicine: best BETs from the Manchester Royal Infirmary. BET 2: potato peel dressings for burn wounds. Emerg Med J. 2010;27(1):55-6. 23. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016;13(5):e1002028. 24. Cochrane Handbook for Systematic Reviews of Interventions, Available from: http://training.cochrane.org/handbook; 2011. 25. Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-34. 26. Tricco A, Clifford TJ, Straus SE. Systematic Prospective Assessment of Rapid Knowledge Synthesis - SPARKS Study. Canadian Research Information System, Available from: http://webapps.cihrirsc.gc.ca/funding/detail_e?pResearchId=7153176&p_version=CIHR&p_language=E&p_sessio n_id=; 2016. 27. Hill R, Bagust A, Bakhai A, Dickson R, Dundar Y, Haycox A, et al. Coronary artery stents: a rapid systematic review and economic evaluation. Health Technol Assess. 2004;8(35):1-242. 28. Brophy J. An evaluation of drug eluting (coated) stents for percutaneous coronary interventions; What should their role be at the McGill University Health Centre (MUHC)? : The Technology Assessment Unit (TAU) of the McGill University Health Centre (MUHC); 2003. Report No.: 10. https://www.mcgill.ca/tau/files/tau/coated.pdf [Last accessed 02.02.2017] 29. Lynch P, Raya T, Mathew V, Soni B, Laroia S, Rahman M Drug-eluting stents for the prevention of restenosis in native coronary arteries. Bloomington, MN: Institute for Clinical Systems Improvement (ICSI) 2003. https://www.mcgill.ca/tau/files/tau/coated.pdf [Last accessed 02.02.2017] 30. Banerjee S, Babidge W, Miller J, Smith J, Noorani HZ, Cuncins-Hearn A, Mensinkai S. Comparison of lung volume reduction surgery with medical management for emphysema [Technology report no 48]. Ottawa: Canadian Coordinating Office for Health Technology Assessment; 2004. https://www.cadth.ca/media/pdf/176_lvrs_tr_e.pdf [Last accessed 02.02.2017] 31. Mickman J, Harmon K, Drage C. Lung Volume Reduction Surgery for Emphysema. Institute for Clinical Systems Improvement (ICSI): Technology Assessment Report; 2003. www.etsad.fr/etsad/afficher_lien.php?id=473 [Last accessed 02.02.2017] 32. National Institute for Clinical Excellence (NICE). Lung volume reduction surgery for advanced emphysema. London: NICE: Information from Interventional Procedure Guidance 114; 2005. https://www.nice.org.uk/guidance/ipg114 [Last accessed 02.02.2017]

AC C

496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546

ACCEPTED MANUSCRIPT

EP

TE D

M AN U

SC

RI PT

33. Vale L, Wyness L, McCormack K, McKenzie L, Brazzelli M, Stearns SC. A systematic review of the effectiveness and cost-effectiveness of metal-on-metal hip resurfacing arthroplasty for treatment of hip disease. Health Technol Assess. 2002;15:1-109. 34. Bernath V. Hip resurfacing in patients with osteoarthritis. Clayton, Victoria: Centre for Clinical Effectiveness (CCE); 2002. 35. Alberta Heritage Foundation for Medical Research (AHFMR). Metal-on-metal hip resurfacing for young, active adults with degenerative hip disease. Edmonton: Alberta Heritage Foundation for Medical Research (AHFMR); Technote TN 33. 2002. 36. Middleton P, Duffield M, Lynch SV, Padbury RT, House T, Stanton P, Verran D, Maddern G. Living donor liver transplantation--adult donor outcomes: a systematic review. 2006. Contract No.: 1. http://www.surgeons.org/media/12482/LDLTreview1004.pdf [Last accessed 02.02.2017] 37. Alberta Heritage Foundation for Medical Research (AHFMR). Adult to adult living donor liver transplantation. Edmonton: Alberta Heritage Foundation for Medical Research (AHFMR); Technote TN 45. 2004. 38. Costa V, Brophy J, McGregor M. Vacuum-Assisted Wound Closure Therapy (V.A.C.®). Technology Assessment Unit (TAU) of the McGill University Health Centre (MUHC); 2005. Report No.: 19. http://www.mcgill.ca/tau/files/tau/VAC_REPORT_FINAL.pdf [Last accessed 02.02.2017] 39. Institute for Quality and Efficiency in Health Care (IQWiG). Negative pressure wound therapy (rapid report). Cologne, Germany: Institute for Quality and Efficiency in Health Care (IQWiG); 2007. https://www.iqwig.de/download/N0602_Executive_summary_Rapid_report_Negative_pressure_wound_therapy.pdf [Last accessed 02.02.2017] 40. Ontario Health Technology Advisory Committee (OHTAC). Negative Pressure Wound Therapy. Ontario Health Technology Advisory Committee (OHTAC); 2006. http://www.hqontario.ca/english/providers/program/ohtac/tech/recommend/rec_npwt_072106.pd f [Last accessed 02.02.2017] 41. Pham C, Middleton PF, Maddern GJ. The safety and efficacy of topical negative pressure in non-healing wounds: a systematic review J Wound Care. 2006;15(6):240-50. 42. Institute for Quality and Efficiency in Health Care (IQWiG). Vacuum therapy of wounds. Cologne, Germany: Institute for Quality and Efficiency in Health Care (IQWiG); 2007. http://www.iqwig.de/download/N0602_Rapid_Report_Vakuumversiegelungstherapie_von_Wunden..pdf [Last accessed 02.02.2017] 43. Ward S, Simpson E, Davis S, Hind D, Rees A, Wilkinson A. Taxanes for the adjuvant treatment of early breast cancer: systematic review and economic evaluation. Health Technol Assess. 2007;11(40):1-144. 44. Chilcott J, Lloyd Jones M, Wilkinson A. Docetaxel for the adjuvant treatment of early node-positive breast cancer: a single technology appraisal. Health Technol Assess. 2009;13(Suppl 1):7-13. 45. Griffin S, Dunn G, Palmer S, Macfarlane K, Brent S, Dyker A, Erhorn S, Humphries C, White S, Horsley W, Ferrie L, Thomas S. The use of paclitaxel in the management of early stage breast cancer. Health Technol Assess. 2009;13(Suppl 1):15-22. 46. McGregor M, Esfandiari S. Transcatheter aortic valve implantation (TAVI) at the MUHC: a health technology assessment. Montreal: Technology Assessment Unit of the McGill University Health Centre; 2009. Report No.: 45. https://www.mcgill.ca/tau/files/tau/TAVI_REPORT.pdf [Last accessed 02.02.2017] 47. Murphy G, Cunningham J. Percutaneous heart valve replacement for valvular heart disease: a review of the clinical effectiveness, cost-effectiveness, and guidelines. Ottawa: Canadian Agency for Drugs and Technologies in Health (CADTH); 2010.

AC C

547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597

ACCEPTED MANUSCRIPT

EP

TE D

M AN U

SC

RI PT

https://www.cadth.ca/media/pdf/l0149_percutaneous_heart_valves_htis-2.pdf [Last accessed 02.02.2017] 48. Craig J, Lancaster GA, Taylor S, Williamson PR, Smyth RL. Infrared ear thermometry compared with rectal thermometry in children: a systematic review. Lancet Infect Dis. 2002;360(9333):603-9. 49. Riddell A, Eppich W. Should tympanic temperature measurement be trusted? Arch Dis Child. 2001;85(5):433-4.

AC C

598 599 600 601 602 603 604 605

ACCEPTED MANUSCRIPT

Fig. 1. Flow diagram of included and excluded studies.

12 Studies (101 SR-RR pairs)

RI PT

Excluded

6 studies (85 SR-RR pairs) Reason for exclusion:

-

Included

AC C

EP

TE D

6 studies (16 SR-RR pairs)

M AN U

-

Insufficient information (2 studies [8 SR-RR pairs]) Unsuccessful attempt at contacting author (1 study [2 SR-RR pairs]) Insufficient data for abstraction (1 study [21 SR-RR pairs]) RR could not be shared publicly (2 studies [55 SR-RR pairs])

SC

-