- Email: [email protected]
Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials?
Journal Pre-proof Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials? Claudio Bianchini, PT, Carola Cosentino, PT, MSc, Matteo Paci, PR, MSc, Marco Baccini, PT, MSc PII:
S0003-9993(20)30024-1
DOI:
https://doi.org/10.1016/j.apmr.2019.12.012
Reference:
YAPMR 57753
To appear in:
ARCHIVES OF PHYSICAL MEDICINE AND REHABILITATION
Received Date: 11 September 2019 Revised Date:
27 November 2019
Accepted Date: 6 December 2019
Please cite this article as: Bianchini C, Cosentino C, Paci M, Baccini M, Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials?, ARCHIVES OF PHYSICAL MEDICINE AND REHABILITATION (2020), doi: https://doi.org/10.1016/j.apmr.2019.12.012. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier Inc. on behalf of the American Congress of Rehabilitation Medicine
Running Title: RCTs in predatory journals
Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials?
Claudio Bianchini1, PT, Carola Cosentino2, PT, MSc, Matteo Paci3 *, PR, MSc, Marco Baccini4, PT, MSc.
1
Nursing Home Le Magnolie, Florence, Italy
2
University of Florence, Italy
3
Unit of Functional Recovery, Azienda USL Toscana Centro, Area Prato, Italy.
4
University of Florence and Azienda Ospedaliero—Universitaria Careggi, Florence, Italy
*
Corresponding author. Unit of Functional Rehabilitation, Azienda USL Toscana Centro, Florence,
Italy; Address: Presidio Ospedaliero Piero Plagi, Viale Michelangiolo 41 Firenze. email: [email protected] 4
University of Florence, Italy
Conflict of Interest: none declared
1
Open access physical therapy journals: do predatory journals publish lower
2
quality Randomized Controlled Trials?
3 4
Abstract
5
Objectives
6
To compare the quality of randomized controlled trials (RCTs) published in predatory and non-
7
predatory journals in the field of physical therapy.
8
Data sources
9
From a list of 18 journals included either on Beall's list (n=9) or in the Directory of Open Access
10
Journals (DOAJ) (n=9), two independent assessors extracted all the RCTs published between 2014
11
and 2017. When journals published more than 40 RCTs, a sample of 40 trials was randomly
12
extracted, preserving the proportions among years. Indexing in PubMed, country of journal
13
publication and dates of submission/acceptance were also recorded for each journal.
14
Main outcome measures
15
The PEDro scale and duration of the peer review.
16
Results
17
Four hundred and ten RCTs were included. The mean PEDro score of articles published in non-
18
Beall, DOAJ journals was higher than those published in Beall journals (5.8±1.7 vs 4.5±1.5; p
19
<.001), with the differences increasing when the indexing in PubMed was also considered (6.5±1.5
20
vs 4.4±1.5; p <.001). The peer review duration was significantly longer in non-Beall than in Beall
21
journals (145.2±92.9 vs 45.4±38.8 days; p<.001) and in journals indexed in PubMed than in non-
22
indexed journals (136.6±100.7 vs 60.4±55.7 days; p<.001). Indexing in PubMed was the strongest
23
independent variable associated with the PEDro score (adjusted R2 = .182), but non-inclusion on
1
24
Beall’s list explained an additional, albeit small portion of the PEDro score variance (cumulative
25
adjusted R2 = .214).
26
Conclusions
27
Potentially predatory journals publish lower-quality trials and have a shorter peer review process
28
than non-Beall journals included in the DOAJ database.
29 30
Keywords: periodicals; peer review; predatory publishers; open access; randomized controlled
31
trials.
32 33
Running Title: RCTs in predatory journals
34 35
Abbreviations
36
PJs = predatory journals
37
NPJs = non-predatory journals
38
RCT = Randomised Controlled Trial
39
DOAJ = Directory of Open Access Journals
40 41 42 43 44 45 46 2
47
Despite debates surrounding the definition of “predatory” journals (PJs) [1], the term generally
48
refers to publications that are suspected of taking fees without providing sufficiently robust editorial
49
or publishing services [2]. PJs recruit articles through aggressive marketing, promising quick
50
reviews and open access publication for a price. Beall [2] defined several criteria for identifying
51
potential PJs and started publishing a list of such journals in 2008, regularly updating it with the
52
addition of new journals and publishers. Due to the growing dispute about the list and the vehement
53
opposition of some publishers, Beall's list was removed in January 2017. However, an archived
54
version is still available (https://beallslist.weebly.com/).
55
Recent studies have investigated characteristics through which PJs might be distinguished from
56
legitimate, non-predatory journals (NPJs) [3]. A lack of a robust peer review process and editorial
57
services seems to be a key feature [3], possibly resulting in the publishing of low quality articles [4-
58
6]. Low quality randomised controlled trials (RCTs), in particular, might have a major negative
59
impact on evidence [6], because RCTs are the gold standard for examining the efficacy of
60
interventions and more likely to be selected as primary studies in systematic reviews and clinical
61
guidelines [7]. Only methodologically sound RCTs can produce reliable evidence and low quality
62
studies may even taint the findings.
63
Few studies have investigated the issue of articles that PJs publish. Authors of papers published in
64
these journals were found to be largely inexperienced researchers (i.e. with no articles published
65
previously) mainly from developing countries [8], but also from developed countries like the USA
66
[9]. Nevertheless, the merit of a journal should be properly determined by assessing the quality of
67
the published articles rather than the authors’ profiles.
68
As regards rehabilitation, Manca et al. [10] identified 59 journals included on Beall's list, showing
69
that the threat of predatory publishing in rehabilitation is real. However, the authors just described
70
the phenomenon without assessing the quality of the publications. The aim of this study was to
71
compare the quality of RCTs published in PJs and NPJs in the field of physical therapy.
72 3
73
Methods
74
Selecting journals and articles
75
The list of rehabilitation journals included on Beall's list as reported by Manca et al. [10] was used
76
to identify PJs. The same keywords used by these authors (rehabilitation OR neurorehabilitation;
77
physical therapy; physiotherapy OR physiotherapies OR physiotherapeutic) were used to identify
78
physical therapy journals included in the Directory of Open Access Journals (DOAJ)
79
(http://www.doaj.org). The search was conducted on September 2017.
80
Physical therapy journals included on Beall’s list (or in the DOAJ) were ordered according to the
81
number of articles published in 2016. Journals were eligible when they were active in the years
82
2014-2017, publishing open access articles in English, mostly (or exclusively) in the field of
83
physical therapy. Only journals that published at least 15 articles in 2016 were included. An equal
84
number of journals was selected from each list; thus, if one list had a greater number of eligible
85
journals, the ones publishing the highest number of papers in 2016 were included.
86
Two independent assessors extracted all the RCTs, even when presented as pilot or feasibility
87
studies, published between 2014 and 2017 in the selected journals. Studies on animals, non-
88
randomised studies and study protocols were excluded.
89
RCTs published in PubMed journals were searched using the strategy "journal title" [jour] AND
90
(random* OR control*). For non-indexed periodicals, the research was carried out manually,
91
through the journal's website. When a journal published more than 40 RCTs from 2014 to 2017, a
92
sample of 40 trials was randomly extracted, preserving the proportions among years, using an
93
online randomization tool (www.randomizer.org).
94 95
Assessment of methodological quality
96
The quality of the selected articles was evaluated using the Physiotherapy Evidence Database
97
(PEDro) scale, one of the most frequently used tools in systematic reviews of interventions in
98
physical therapy and other areas [11]. The PEDro scale rates 11 methodological features as being
4
99
either absent or present. Because the first item (eligibility criteria) is not scored, the total score
100
ranges from 10 (all items satisfied) to 0 (no item satisfied). The PEDro score was shown to be
101
reliable (Intraclass Correlation Coefficient =.68; 95% CI=.57–.76) with a standard error of the
102
measurement 0.70 units [12]. A six-point cut-off [12] is used to separate good/high quality and low
103
quality studies, the latter having so many risks of bias that the truthfulness of the findings is
104
uncertain.
105
The PEDro score of the selected RCTs was extracted from the Pedro.org website when available.
106
The other articles were assessed by two independent raters, randomly selected from a group of 11
107
physical therapists trained in the use of the scale; six of them had completed the PEDro Scale
108
Training Program available on Pedro.org. A randomly selected third rater, blinded to the two
109
original raters and the score of the paper, resolved any disagreements. The assessors’ randomisation
110
was carried out using the same online tool (www.randomizer.org).
111 112
Duration of the peer review process
113
The submission and acceptance dates of the RCTs included, were extracted from the journal website
114
for calculating the duration of the peer review process. When the article failed to provide this
115
information, the journal management team was contacted to retrieve the data.
116 117
Country of journal publication
118
The country of publication was identified as the country of societies/associations that recognize the
119
journal as their own official journal. When a journal was affiliated with two or more
120
societies/associations of different countries or it was not possible to identify any reference
121
society/association, the country of the publisher reported on the website at the time of the selected
122
publication period was chosen. Countries were classified as either developed or developing
123
according
124
web.org/index.php/resources/developing-countries).
5
to
the
International
Statistical
Institute
(https://www.isi-
125 126
Data analysis
127
Three comparisons of the PEDro scores and the peer review duration were conducted using
128
unpaired t-tests: (a) Beall versus non-Beall journals, (b) journals indexed in PubMed versus non-
129
indexed journals, and (c) non-Beall journals listed in PubMed versus the remainder of the overall
130
sample. The association of three features (Beall/non-Beall, indexing/non-indexing in PubMed, non-
131
Beall+indexing in Pubmed/others) with the trial quality (high/low quality) was assessed using
132
Pearson's Chi-square test. The correlation between the PEDro score and peer review duration was
133
analysed with Pearson’s coefficient.
134
Finally, data were put in a stepwise multiple regression model with the PEDro score as the
135
dependent variable and Beall/non-Beall, indexing/non-indexing in PubMed, country of publication
136
(expressed as dichotomous variable, i.e. developed or developing country) and peer review duration
137
as independent variables.
138
The level of statistical significance was set at 0.05. Data analyses were conducted using the package
139
SPSSa, Version 20.0.
140 141
Results
142
Nine journals on Beall’s list and ten journals in the DOAJ met the inclusion criteria. One journal
143
(International Journal of Physiotherapy) found on both lists was excluded from the DOAJ for this
144
study. Information about peer review duration was not available on the Journal of Physiotherapy
145
and Journal of Rehabilitation Medicine websites and was provided for this study by the Editor in
146
Chief and the Editorial Office of these journals, respectively. The main features of the journals
147
included are reported in table 1.
148
A sample of 410 RCTs was analysed; 291 were not scored on PEDro and were assessed for quality
149
for this study (figure 1). The mean PEDro scores, peer review time and quality of the articles
150
published in included journals are reported in table 2.
6
151
The mean PEDro score in non-Beall journals was higher than in Beall journals (5.8±1.7 vs 4.5±1.5;
152
p<.001) (figure 2), in PubMed journals than in non-indexed ones (6.0±1.7 vs 4.5±1.5; p<.001)
153
(figure 3), and in non-Beall+PubMed journals than in others (6.5±1.5 vs 4.4±1.5; p<.001) (figure 4).
154
Accordingly, higher proportions of high quality trials were published in non-Beall than in Beall
155
journals (55.1% vs 23.6%; Chi-square: 39.962; p<.001), in PubMed journals than non-indexed
156
journals (62.1% vs 24.1%; Chi-square: 60.725; p<.001) and in non-Beall+PubMed journals than in
157
others (75.5% vs 22.3%; Chi-square: 111.401; p<.001).
158
Peer review time (days) was significantly longer in non-Beall than Beall journals (145.2±92.9 vs
159
45.4±38.8; p<.001) (figure 5), in PubMed journals than non-indexed journals (136.6±100.7 vs
160
60.4±55.7 days; p<.001) (figure 6) and in non-Beall+PubMed journals than in others (186.9±93.7
161
vs 57.9±52.2; p<.001) (figure 7). The PEDro score was significantly correlated to peer review time
162
(r = .327; p<.001).
163
The multiple regression showed that indexing in PubMed was the strongest independent variable
164
associated with the PEDro score (adjusted R2=0.182). Adding non-inclusion on Beall’s list
165
explained a further 3% of variance (cumulative adjusted R2=0.214).
166 167
Discussion
168
This study is the first to quantify the quality of RCTs published in predatory and non-predatory
169
physical therapy journals using a standardised scale. Its main finding is that quality is significantly
170
lower in potential PJs (included on Beall's list) than in NPJs (not included on Beall's list).
171
Our results are consistent with data reported on PJs in other fields. Oermann et al. [13] assessed 358
172
papers randomly selected from articles published in 103 nursing PJs. The authors examined eight
173
indicators, including presence of an abstract, logical presentation and organization of the article or
174
clear discussion of results, and rated as “poor”, “average” or “excellent” the overall quality of 171,
175
169 and 13 articles respectively. However, the authors included all types of articles and did not use
176
a standardized assessment tool. Data about the quality of RCTs published in a random sample of
7
177
potential PJs and NPJs in orthopaedics have been reported [6]. The authors found that CONSORT
178
scores were lower for RCTs found in PJs than for any RCTs published in NPJs. However, within the
179
studies included in the investigation, there were very few RCTs (n=9). The data of this study, which
180
are extracted from a relatively large sample of published RCTs, extend the finding to the physical
181
therapy area.
182
The data presented strengthen the idea that PJs may encourage the dissemination of unsound
183
research findings that could be translated into clinical practice with risks for the patients’ health [14-
184
16]. The risk of corrupting evidence increases when such publications are included in systematic
185
reviews. Ross-White et al. [17] found that in a sample of 6,302 articles published in PJs in the
186
general field of health and biomedical sciences, 120 articles were cited at least by one systematic
187
review and 157 systematic reviews cited at least one of those articles.
188
The term “predatory journals” was criticised [2,18] because distinguishing good from bad journals
189
is not easy, and Beall adopted the prudent definition of "potential, possible, or probable predatory
190
scholarly open-access publishers and journals" [19]. Indeed, one journal was included both on
191
Beall’s list and in the DOAJ. Moreover, Beall's list was not static, and some publishers and journals
192
were added or removed over time. Nevertheless, the data presented support the doubt that PJs do
193
not adequately check the quality of published RCTs. The mean PEDro score of all Beall journals
194
was lower than 5, except for Physical Medicine and Rehabilitation International which, however,
195
only published two RCTs during the study period. Furthermore, non-inclusion on Beall’s list was an
196
independent predictor of the PEDro score, despite explaining a very small proportion of the
197
variance. Such a small proportion probably depends on the fact that no selected Beall journals were
198
indexed in PubMed, except for the Journal of Physical Therapy Science; thus, the effect of the two
199
factors largely overlaps, and indexing in PubMed is a stronger predictor of the PEDro score than
200
non-inclusion on Beall’s list. Interestingly, the strongest differences in mean PEDro scores and high
201
quality study proportions emerged when periodicals were distinguished according to being/not
202
being non-Beall+Pubmed journals.
8
203
Inclusion of PJs in PubMed is a relevant issue. Several PJs in four important disciplines are indexed
204
in PubMed, namely, rehabilitation 12% [10], neurosciences 11.4% and neurology 20.2% [20], and
205
orthopaedics 8.9% [6]. Among the journals selected in this study, the Journal of Physical Therapy
206
Science was the only Beall journal indexed in PubMed. It published a huge number (over 500) of
207
RCTs in the study period; this accounts for over half the RCTs published in the PJs and is
208
approximately twice the number of RCTs published in the nine NPJs altogether. These numbers
209
indicate that the risk of corrupting evidence is real.
210
The peer review time, which varied among and within the periodicals, was longer on average in
211
non-Beall journals and PubMed journals than in the others. All the PJs showed a shorter peer review
212
duration than all but one NPJ. The NPJ with the shortest acceptance time (Iranian Rehabilitation
213
Journal) had a mean peer review time only just shorter than the PJ with the longest acceptance time
214
(Journal of Yoga & Physical Therapy). Interestingly, the average quality of the RCTs published in
215
the Iranian Rehabilitation Journal was the lowest among the DOAJ journals.
216
These findings regarding peer review time are consistent with the scarce data available in literature.
217
In predatory orthopaedic journals, the reviewing time of PJs was either less than one month or not
218
reported in 36.5% and 56.2% of orthopaedics cases respectively [6], and ranged from 2.7 to 21 days
219
in nursing [5]. In this study, the mean reviewing time of physical therapy PJs was longer than the
220
times reported in other fields, but this difference most likely depends on the type of articles
221
included (only RCTs instead of all types of articles).
222
Edie and Conklin [4] reviewed 53 nursing PJs using 12 quality indicators of peer review processes
223
and showed that language, author-control, and process-transparency indicators were generally poor.
224
Thus, a faster peer review time might be a marker of a less accurate editorial process in PJs. This
225
hypothesis is supported by the finding of a weak but highly significant positive association between
226
peer review time and PEDro score. However, in the multivariate analysis the factor was not an
227
independent predictor of the trial quality. This datum is not surprising, since the peer review
228
duration depends on several variables (e.g., availability and timeliness of reviewers/editors, quality
9
229
of the first submission) and a careful peer reviewing process may also be completed in a short time.
230
Moreover, reducing the time from submission to acceptance is a goal of legitimate journals too.
231
Therefore, although a very quick peer review is a consistent feature of PJs, it cannot be directly
232
linked to quality.
233 234
Study limitations
235
Several methodological biases might have influenced the results. First, inclusion on Beall's list was
236
the only criterion adopted to identify potential PJs. Several criticisms have been advanced regarding
237
Beall’s list, such as the lack of specific or detailed inclusion criteria or the fact of being drafted by a
238
single author [21]. Moreover, the list was deactivated in January 2017 and other lists have taken
239
over since then (for example, https://www2.cabells.com/), with many academic institutions having
240
online listings of what they consider to be predatory journals. Consequently, we cannot exclude that
241
one or more of the journals included in the predatory group were not really predatory at all. Nor
242
does inclusion in the DOAJ guarantee per se that a journal acts honestly, even though the procedure
243
in this case is quite different; a voluntary application by the publisher is needed and a committee
244
might reject a journal if it does not meet the required standards.
245
Moreover, we did not consider the possible influence of the differing longevity of the periodicals on
246
the quality of the published articles, i.e. the fact that a new journal is more likely to receive low
247
quality papers. All the selected Beall journals started their publication in 2011 or later, except for
248
the Journal of Physical Therapy Science. Conversely, all the non-Beall journals were already active
249
before that year, except for the Journal of Rehabilitation Science & Research. However, receiving
250
low quality submissions should lead to a lengthening, rather than a shortening, of the peer review
251
process, so it is unlikely that this factor had a major impact.
252
Although raters were not informed about the inclusion of the journals on Beall’s list or in the DOAJ,
253
they were not blinded with regard to the journal where the articles were published, because
254
removing any references to the journal from each paper would hardly have been possible.
10
255
Therefore, we cannot rule out that the assessors actually knew whether the assigned articles were
256
from a predatory or a non-predatory publisher, and this could have influenced the assessments.
257
Moreover, none of the PEDro scale items offer perfect reliability for the consensus ratings, so the
258
use of a cut-off score for classifying the article for quality (high or low) may lead to some
259
misclassification. Since the standard error of the measurement for the PEDro score is 0.70 units
260
[12], the risk of misclassification is high when the assigned score is close to the selected cut-off
261
(i.e., 5/10 or 6/10), and in this study that was true for 38% of the trials (n=162). Nonetheless, the
262
effect, if any, would have been modest because the analysis led to consistent findings, irrespective
263
of the variable (the PEDro scores or the proportions of high/low quality trials) it was compared
264
with.
265 266
Conclusions
267
Within the physical therapy field, PJs publish RCTs of lower quality and have shorter peer review
268
process times than journals not included on Beall's list. Indexing in PubMed is the strongest
269
independent variable associated with the PEDro score, but non-inclusion on Beall’s list explains an
270
additional small portion of the PEDro score variance. Other types of study that are relevant for
271
clinical decisions and development of scientific findings, such as systematic reviews or validation
272
studies, should be investigated. A citation analysis of articles published in PJs is also advisable in
273
order to open a scenario on the impact of these articles on literature.
274 275 276 277 278 279 280
11
References 1. Eriksson S. Helgesson G. Time to stop talking about ‘predatory journals’. Learn Publ 2018; 31(2): 181-3. DOI: https://doi.org/10.1002/leap.1135
281
2. Beall J. Criteria for determining predatory open-access publishers. 2nd edition. Denver, CO:
282
Scholarly Open Access; 2012. http://scholarlyoa.files.wordpress.com/2012/11/criteria-
283
2012-2.pdf.
284
3. Shamseer L, Moher, D, Maduekwe O, Turner L, Barbour V, Burch R, et al. Potential
285
predatory and legitimate biomedical journals: can you tell the difference? A cross-sectional
286
comparison. BMC Medicine 2017; 15: 1. DOI: 10.1186/s12916-017-0785-9
287 288
4. Edie AH, Conklin JL. Avoiding predatory journals: Quick peer review processes too good to be true. Nurs Forum. 2019 Feb 25. doi: 10.1111/nuf.12333. [Epub ahead of print]
289
5. Oermann MH, Conklin JL, Nicoll LH, Chinn PL, Ashton KS, Edie AH, et al. Study of
290
predatory open access nursing journals. J Nurs Scholarsh 2016; 48(6): 624–632. DOI:
291
10.1111/jnu.12248
292
6. Yan JR, Baldawi H, Lex JR, Simchovich G, Baisi LP, Bozzo A, et al. Predatory publishing
293
in orthopaedic research. J Bone Joint Surg Am 2018; 100(21): e138. doi:
294
10.2106/JBJS.17.01569.
295
7. Olivo et al. SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess
296
the quality of randomized controlled trials: a systematic review. Phys Ther 2008; 88(2):156-
297
175. DOI: 10.2522/ptj.20070147
298
8. Xia J, Harmon J, Connolly K, Donnelly R, Anderson M, Howard H. Who publishes in
299
predatory
300
https://doi.org/10.1002/asi.23265
301 302 303
journals.
J Assoc
Sci Technol
2015;
66(7):
1406-1417.
DOI:
9. Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, et al. Stop this waste of people, animals and money. Nature. 2017; 549: 23-25. doi: 10.1038/549023a. 10. Manca A, Martinez G, Cugusi L, Dragone D, Mercuro G, Deriu F. Predatory Open Access in
304
Rehabilitation.
305
10.1016/j.apmr.2017.01.002
12
Inf
Arch
Phys
Med
Rehabil
2017;
98(5):
1051-1056.
DOI:
306
11. Elkins MR, Moseley AM, Sherrington C, Herbert RD, Maher CG. Growth in the
307
Physiotherapy Evidence Database (PEDro) and use of the PEDro scale. Br J Sports Med
308
2013; 47(4): 188–189. DOI: 10.1136/bjsports-2012-091804
309 310
12. Maher G, Sherrington C, Herbert R et al. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther 2003; 83(8): 713–721.
311
13. Oermann MH, Nicoll LH, Chinn PL, Ashton KS, Conklin JL, Edie AH, et al. Quality of
312
articles published in predatory nursing journals. Nurs Outlook. 2018; 66(1): 4-10. doi:
313
10.1016/j.outlook.2017.05.005.
314 315 316 317 318 319 320 321
14. Beall J. Medical publishing triage - chronicling predatory open access publishers. Ann Med Surg (Lond). 2013; 2(2): 47-9. doi: 10.1016/S2049-0801(13)70035-9 15. Beall J. Predatory journals threaten the quality of published medical research. J Orthop Sports Phys Ther 2017; 47(1): 3-5. doi:10.2519/jospt.2017.0601 16. Cook C. Predatory Journals: The Worst Thing in Publishing, Ever. J Orthop Sports Phys Ther 2017;47(1):1-2. doi:10.2519/jospt.2017.0101 17. Ross-White A, Godfrey CM, Sears KA, Wilson R. Predatory publications in evidence syntheses. J Med Libr Assoc 2019; 107(1): 57–61. doi: 10.5195/jmla.2019.491
322
18. Wager E. Why we should worry less about predatory publishers and more about the quality
323
of research and training at our academic institutions. J Epidemiol 2017; 27(3): 87–88.
324
DOI:10.1016/j.je.2017.01.001
325 326
19. Butler D. Investigating journals: The dark side of publishing. Nature 2013;495(7442):433-5. doi: 10.1038/495433a.
327
20. Manca A, Martinez G, Cugusi L, Dragone D, Dvir Z, Deriu F. The surge of predatory open-
328
access in neurosciences and neurology. Neuroscience 2017;353: 166-173. doi:
329
10.1016/j.neuroscience.2017.04.014.
330 331
13
21. Teixeira da Silva JA. Caution with the continued use of Jeffrey Beall’s “predatory” open access pub-lishing lists. AME Medical Journal 2017; 2: 1–5.
332 333
Figure captions:
334
Figure 1. Flow-chart of the selection procedure.
335
Figure 2. Mean PEDro score in non-Beall and Beall journals.
336
Figure 3. Mean PEDro score in PubMed and non-indexed journals.
337
Figure 4. Mean PEDro score in non-Beall+PubMed journals and the others.
338
Figure 5. Duration of the peer review process in non-Beall and in Beall journals.
339
Figure 6. Duration of the peer review process in PubMed and non-indexed journals.
340
Figure 7. Duration of the peer review process in non-Beall+PubMed journals and the others.
341
14
Contribution of the Paper: •
Predatory physical therapy journals publish RCTs of lower quality and have shorter time of peer review process than journals not included in the Beall's list.
•
Due to the great number of papers published by predatory physical therapy journals, there is a clear risk that predatory publishers substantially contribute to the dissemination of unsound research findings.
Table 1. Main features of included journals.
Journal
Publisher
Country
Journal release
Papers in 2016
RCT in 2014-2017
OMICS International
India
2014
62
14
International Journal of Physical Medicine OMICS International & Rehabilitation
India
2013
92
30
International Journal of Physiotherapy (IJPHI)
India
2014
125
160
International Journal of Physiotherapy and IMED Research Research Publications
India
2014
72
144
Journal of Novel Physiotherapies
OMICS International
India
2011
56
25
Journal of Physical Therapy Science
J Stage
Japan
1990
726
517
Journal of Yoga & Physical Therapy
OMICS International
India
2011
47
4
Open Journal of Therapy and Rehabilitation
Scientific Research Publishing
China
2013
19
9
Physical Medicine and Rehabilitation International
Austin Publishing Group
USA
2014
20
2
BMC Sports Science, Medicine and Rehabilitation
BioMed Central
UK
2009
39
17
Brazilian Journal of Physical Therapy
Associação Brasileira
Brazil
2006
70
25
Beall International Journal of Neurorehabilitation
Standalone journal
Not beall
de Pesquisa e PósGraduação em Fisioterapia Egyptian Rheumatology and Rehabilitation
Wolters Kluwer Egypt Medknow Publications
1974
31
15
Rehabilitation Research and Practice
Hindawi
Egypt
2010
18
11
Iranian Rehabilitation Journal
Negah Institute for Scientific Communication
Iran
2003
34
37
Journal of NeuroEngineering and Rehabilitation
BioMed Central
UK
2004
101
36
Journal of Physiotherapy
Elsevier
Australia
1955
90
30
Journal of Rehabilitation Medicine
Foundation for Rehabilitation Information
Scandinavia
1969
129
60
Journal of Rehabilitation Sciences and Research
Shiraz University of Medical Sciences
Iran
2014
20
16
Table 2. PEDro score, and peer review time (mean ± standard deviation) and quality (low/high) of articles published in included journals. Journal
Pedro Score Mean (SD) (range)
Quality
Peer Review Time (days) Mean (SD) (range)
4.36 (1.8) (0 - 7)
LOW
55.21 (97.6) (3 - 375)
International Journal of Physical Medicine & Rehabilitation 4.97 (1.8) (2 - 9)
LOW
54.70 (34.2) (10 - 117)
International Journal of Physiotherapy (IJPHI)
4.74 (1.7) (2 - 7)
LOW
51.92 (27.2) (13 - 149)
International Journal of Physiotherapy and Research
4.11 (1.6) (1 - 8)
LOW
31.37 (36.2) (11 - 240)
Journal of Novel Physiotherapies
4.40 (1.5) (2 - 7)
LOW
37.08 (33.2) (8 - 150)
Journal of Physical Therapy Science
4.28 (1.2) (2 - 7)
LOW
44.91 (23.3) (18 - 147)
Journal of Yoga & Physical Therapy
4.25 (2.2) (1 - 6)
LOW
70.25 (25.6) (53 - 108)
Open Journal of Therapy and Rehabilitation
4.78 (1.8) (2 - 7)
LOW
50.33 (11.8) (35 - 66)
Physical Medicine and Rehabilitation International
5.50 (3.5) (3 - 8)
HIGH
20.50 (2.1) (19 - 22)
BMC Sports Science, Medicine and Rehabilitation
5.6 (1.1) (4 - 8)
HIGH
156.6 (77.1) (37 - 364)
Brazilian Journal of Physical Therapy
6.9 (1.2) (4 - 9)
HIGH
197.4 (58.7) (124 - 333)
Egyptian Rheumatology and Rehabilitation
4.9] (1.1) (3 - 7)
LOW
124.5 (109.9) (18 - 391)
Rehabilitation Research and Practice
6.7 (1.8) (4 - 9)
HIGH
89.3 (31.3) (55 - 134)
Iranian Rehabilitation Journal
3.8 (1.3) (1 - 7)
LOW
61.5 (45.1) (26 - 258)
Journal of NeuroEngineering and Rehabilitation
5.7 (1.5) (3 - 10)
HIGH
208.0 (110.2) (49 - 543)
Journal of Physiotherapy
7.6 (0.9) (5 - 9)
HIGH
143.4 (93.0) (24 - 341)
Beall International Journal of Neurorehabilitation
Not beall
Journal of Rehabilitation Medicine
6.5 (1.4) (3 - 9)
HIGH
221.9 (81.0) (94 - 477)
Journal of Rehabilitation Sciences and Research
4.69 (1.2) (3 - 7)
LOW
89.3 (31.2) (20 - 211)
(((((rehabilitation) OR neurorehabilitation) OR "physical therapy") OR physiotherapy) OR physiotherapies) OR physiotherapeutic
Journals
Beall’s List 59 journals
DOAJ’s list 29 journals Not focused to physiotherapy=7 Not english language= 3 Less than 15 RCTs in 2016= 9 Present also in Beall’s list= 1
Not active in 2014-2017= 24 Less than 15 RCTs in 2016= 25 Not open access=1
Journals included 9
Journals included 9
RCTs selected 207
RCTs selected 212 6 Excluded: study protocol=4 not randomized=2
RCTs
3 Excluded: not randomized=3
RCTs included 204 Scored on PEDro: 25 Scored for the study: 179
RCTs included 206 Scored on PEDro: 103 Scored for the study: 103
S0003-9993(20)30024-1
DOI:
https://doi.org/10.1016/j.apmr.2019.12.012
Reference:
YAPMR 57753
To appear in:
ARCHIVES OF PHYSICAL MEDICINE AND REHABILITATION
Received Date: 11 September 2019 Revised Date:
27 November 2019
Accepted Date: 6 December 2019
Please cite this article as: Bianchini C, Cosentino C, Paci M, Baccini M, Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials?, ARCHIVES OF PHYSICAL MEDICINE AND REHABILITATION (2020), doi: https://doi.org/10.1016/j.apmr.2019.12.012. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier Inc. on behalf of the American Congress of Rehabilitation Medicine
Running Title: RCTs in predatory journals
Open access physical therapy journals: do predatory journals publish lower quality Randomized Controlled Trials?
Claudio Bianchini1, PT, Carola Cosentino2, PT, MSc, Matteo Paci3 *, PR, MSc, Marco Baccini4, PT, MSc.
1
Nursing Home Le Magnolie, Florence, Italy
2
University of Florence, Italy
3
Unit of Functional Recovery, Azienda USL Toscana Centro, Area Prato, Italy.
4
University of Florence and Azienda Ospedaliero—Universitaria Careggi, Florence, Italy
*
Corresponding author. Unit of Functional Rehabilitation, Azienda USL Toscana Centro, Florence,
Italy; Address: Presidio Ospedaliero Piero Plagi, Viale Michelangiolo 41 Firenze. email: [email protected] 4
University of Florence, Italy
Conflict of Interest: none declared
1
Open access physical therapy journals: do predatory journals publish lower
2
quality Randomized Controlled Trials?
3 4
Abstract
5
Objectives
6
To compare the quality of randomized controlled trials (RCTs) published in predatory and non-
7
predatory journals in the field of physical therapy.
8
Data sources
9
From a list of 18 journals included either on Beall's list (n=9) or in the Directory of Open Access
10
Journals (DOAJ) (n=9), two independent assessors extracted all the RCTs published between 2014
11
and 2017. When journals published more than 40 RCTs, a sample of 40 trials was randomly
12
extracted, preserving the proportions among years. Indexing in PubMed, country of journal
13
publication and dates of submission/acceptance were also recorded for each journal.
14
Main outcome measures
15
The PEDro scale and duration of the peer review.
16
Results
17
Four hundred and ten RCTs were included. The mean PEDro score of articles published in non-
18
Beall, DOAJ journals was higher than those published in Beall journals (5.8±1.7 vs 4.5±1.5; p
19
<.001), with the differences increasing when the indexing in PubMed was also considered (6.5±1.5
20
vs 4.4±1.5; p <.001). The peer review duration was significantly longer in non-Beall than in Beall
21
journals (145.2±92.9 vs 45.4±38.8 days; p<.001) and in journals indexed in PubMed than in non-
22
indexed journals (136.6±100.7 vs 60.4±55.7 days; p<.001). Indexing in PubMed was the strongest
23
independent variable associated with the PEDro score (adjusted R2 = .182), but non-inclusion on
1
24
Beall’s list explained an additional, albeit small portion of the PEDro score variance (cumulative
25
adjusted R2 = .214).
26
Conclusions
27
Potentially predatory journals publish lower-quality trials and have a shorter peer review process
28
than non-Beall journals included in the DOAJ database.
29 30
Keywords: periodicals; peer review; predatory publishers; open access; randomized controlled
31
trials.
32 33
Running Title: RCTs in predatory journals
34 35
Abbreviations
36
PJs = predatory journals
37
NPJs = non-predatory journals
38
RCT = Randomised Controlled Trial
39
DOAJ = Directory of Open Access Journals
40 41 42 43 44 45 46 2
47
Despite debates surrounding the definition of “predatory” journals (PJs) [1], the term generally
48
refers to publications that are suspected of taking fees without providing sufficiently robust editorial
49
or publishing services [2]. PJs recruit articles through aggressive marketing, promising quick
50
reviews and open access publication for a price. Beall [2] defined several criteria for identifying
51
potential PJs and started publishing a list of such journals in 2008, regularly updating it with the
52
addition of new journals and publishers. Due to the growing dispute about the list and the vehement
53
opposition of some publishers, Beall's list was removed in January 2017. However, an archived
54
version is still available (https://beallslist.weebly.com/).
55
Recent studies have investigated characteristics through which PJs might be distinguished from
56
legitimate, non-predatory journals (NPJs) [3]. A lack of a robust peer review process and editorial
57
services seems to be a key feature [3], possibly resulting in the publishing of low quality articles [4-
58
6]. Low quality randomised controlled trials (RCTs), in particular, might have a major negative
59
impact on evidence [6], because RCTs are the gold standard for examining the efficacy of
60
interventions and more likely to be selected as primary studies in systematic reviews and clinical
61
guidelines [7]. Only methodologically sound RCTs can produce reliable evidence and low quality
62
studies may even taint the findings.
63
Few studies have investigated the issue of articles that PJs publish. Authors of papers published in
64
these journals were found to be largely inexperienced researchers (i.e. with no articles published
65
previously) mainly from developing countries [8], but also from developed countries like the USA
66
[9]. Nevertheless, the merit of a journal should be properly determined by assessing the quality of
67
the published articles rather than the authors’ profiles.
68
As regards rehabilitation, Manca et al. [10] identified 59 journals included on Beall's list, showing
69
that the threat of predatory publishing in rehabilitation is real. However, the authors just described
70
the phenomenon without assessing the quality of the publications. The aim of this study was to
71
compare the quality of RCTs published in PJs and NPJs in the field of physical therapy.
72 3
73
Methods
74
Selecting journals and articles
75
The list of rehabilitation journals included on Beall's list as reported by Manca et al. [10] was used
76
to identify PJs. The same keywords used by these authors (rehabilitation OR neurorehabilitation;
77
physical therapy; physiotherapy OR physiotherapies OR physiotherapeutic) were used to identify
78
physical therapy journals included in the Directory of Open Access Journals (DOAJ)
79
(http://www.doaj.org). The search was conducted on September 2017.
80
Physical therapy journals included on Beall’s list (or in the DOAJ) were ordered according to the
81
number of articles published in 2016. Journals were eligible when they were active in the years
82
2014-2017, publishing open access articles in English, mostly (or exclusively) in the field of
83
physical therapy. Only journals that published at least 15 articles in 2016 were included. An equal
84
number of journals was selected from each list; thus, if one list had a greater number of eligible
85
journals, the ones publishing the highest number of papers in 2016 were included.
86
Two independent assessors extracted all the RCTs, even when presented as pilot or feasibility
87
studies, published between 2014 and 2017 in the selected journals. Studies on animals, non-
88
randomised studies and study protocols were excluded.
89
RCTs published in PubMed journals were searched using the strategy "journal title" [jour] AND
90
(random* OR control*). For non-indexed periodicals, the research was carried out manually,
91
through the journal's website. When a journal published more than 40 RCTs from 2014 to 2017, a
92
sample of 40 trials was randomly extracted, preserving the proportions among years, using an
93
online randomization tool (www.randomizer.org).
94 95
Assessment of methodological quality
96
The quality of the selected articles was evaluated using the Physiotherapy Evidence Database
97
(PEDro) scale, one of the most frequently used tools in systematic reviews of interventions in
98
physical therapy and other areas [11]. The PEDro scale rates 11 methodological features as being
4
99
either absent or present. Because the first item (eligibility criteria) is not scored, the total score
100
ranges from 10 (all items satisfied) to 0 (no item satisfied). The PEDro score was shown to be
101
reliable (Intraclass Correlation Coefficient =.68; 95% CI=.57–.76) with a standard error of the
102
measurement 0.70 units [12]. A six-point cut-off [12] is used to separate good/high quality and low
103
quality studies, the latter having so many risks of bias that the truthfulness of the findings is
104
uncertain.
105
The PEDro score of the selected RCTs was extracted from the Pedro.org website when available.
106
The other articles were assessed by two independent raters, randomly selected from a group of 11
107
physical therapists trained in the use of the scale; six of them had completed the PEDro Scale
108
Training Program available on Pedro.org. A randomly selected third rater, blinded to the two
109
original raters and the score of the paper, resolved any disagreements. The assessors’ randomisation
110
was carried out using the same online tool (www.randomizer.org).
111 112
Duration of the peer review process
113
The submission and acceptance dates of the RCTs included, were extracted from the journal website
114
for calculating the duration of the peer review process. When the article failed to provide this
115
information, the journal management team was contacted to retrieve the data.
116 117
Country of journal publication
118
The country of publication was identified as the country of societies/associations that recognize the
119
journal as their own official journal. When a journal was affiliated with two or more
120
societies/associations of different countries or it was not possible to identify any reference
121
society/association, the country of the publisher reported on the website at the time of the selected
122
publication period was chosen. Countries were classified as either developed or developing
123
according
124
web.org/index.php/resources/developing-countries).
5
to
the
International
Statistical
Institute
(https://www.isi-
125 126
Data analysis
127
Three comparisons of the PEDro scores and the peer review duration were conducted using
128
unpaired t-tests: (a) Beall versus non-Beall journals, (b) journals indexed in PubMed versus non-
129
indexed journals, and (c) non-Beall journals listed in PubMed versus the remainder of the overall
130
sample. The association of three features (Beall/non-Beall, indexing/non-indexing in PubMed, non-
131
Beall+indexing in Pubmed/others) with the trial quality (high/low quality) was assessed using
132
Pearson's Chi-square test. The correlation between the PEDro score and peer review duration was
133
analysed with Pearson’s coefficient.
134
Finally, data were put in a stepwise multiple regression model with the PEDro score as the
135
dependent variable and Beall/non-Beall, indexing/non-indexing in PubMed, country of publication
136
(expressed as dichotomous variable, i.e. developed or developing country) and peer review duration
137
as independent variables.
138
The level of statistical significance was set at 0.05. Data analyses were conducted using the package
139
SPSSa, Version 20.0.
140 141
Results
142
Nine journals on Beall’s list and ten journals in the DOAJ met the inclusion criteria. One journal
143
(International Journal of Physiotherapy) found on both lists was excluded from the DOAJ for this
144
study. Information about peer review duration was not available on the Journal of Physiotherapy
145
and Journal of Rehabilitation Medicine websites and was provided for this study by the Editor in
146
Chief and the Editorial Office of these journals, respectively. The main features of the journals
147
included are reported in table 1.
148
A sample of 410 RCTs was analysed; 291 were not scored on PEDro and were assessed for quality
149
for this study (figure 1). The mean PEDro scores, peer review time and quality of the articles
150
published in included journals are reported in table 2.
6
151
The mean PEDro score in non-Beall journals was higher than in Beall journals (5.8±1.7 vs 4.5±1.5;
152
p<.001) (figure 2), in PubMed journals than in non-indexed ones (6.0±1.7 vs 4.5±1.5; p<.001)
153
(figure 3), and in non-Beall+PubMed journals than in others (6.5±1.5 vs 4.4±1.5; p<.001) (figure 4).
154
Accordingly, higher proportions of high quality trials were published in non-Beall than in Beall
155
journals (55.1% vs 23.6%; Chi-square: 39.962; p<.001), in PubMed journals than non-indexed
156
journals (62.1% vs 24.1%; Chi-square: 60.725; p<.001) and in non-Beall+PubMed journals than in
157
others (75.5% vs 22.3%; Chi-square: 111.401; p<.001).
158
Peer review time (days) was significantly longer in non-Beall than Beall journals (145.2±92.9 vs
159
45.4±38.8; p<.001) (figure 5), in PubMed journals than non-indexed journals (136.6±100.7 vs
160
60.4±55.7 days; p<.001) (figure 6) and in non-Beall+PubMed journals than in others (186.9±93.7
161
vs 57.9±52.2; p<.001) (figure 7). The PEDro score was significantly correlated to peer review time
162
(r = .327; p<.001).
163
The multiple regression showed that indexing in PubMed was the strongest independent variable
164
associated with the PEDro score (adjusted R2=0.182). Adding non-inclusion on Beall’s list
165
explained a further 3% of variance (cumulative adjusted R2=0.214).
166 167
Discussion
168
This study is the first to quantify the quality of RCTs published in predatory and non-predatory
169
physical therapy journals using a standardised scale. Its main finding is that quality is significantly
170
lower in potential PJs (included on Beall's list) than in NPJs (not included on Beall's list).
171
Our results are consistent with data reported on PJs in other fields. Oermann et al. [13] assessed 358
172
papers randomly selected from articles published in 103 nursing PJs. The authors examined eight
173
indicators, including presence of an abstract, logical presentation and organization of the article or
174
clear discussion of results, and rated as “poor”, “average” or “excellent” the overall quality of 171,
175
169 and 13 articles respectively. However, the authors included all types of articles and did not use
176
a standardized assessment tool. Data about the quality of RCTs published in a random sample of
7
177
potential PJs and NPJs in orthopaedics have been reported [6]. The authors found that CONSORT
178
scores were lower for RCTs found in PJs than for any RCTs published in NPJs. However, within the
179
studies included in the investigation, there were very few RCTs (n=9). The data of this study, which
180
are extracted from a relatively large sample of published RCTs, extend the finding to the physical
181
therapy area.
182
The data presented strengthen the idea that PJs may encourage the dissemination of unsound
183
research findings that could be translated into clinical practice with risks for the patients’ health [14-
184
16]. The risk of corrupting evidence increases when such publications are included in systematic
185
reviews. Ross-White et al. [17] found that in a sample of 6,302 articles published in PJs in the
186
general field of health and biomedical sciences, 120 articles were cited at least by one systematic
187
review and 157 systematic reviews cited at least one of those articles.
188
The term “predatory journals” was criticised [2,18] because distinguishing good from bad journals
189
is not easy, and Beall adopted the prudent definition of "potential, possible, or probable predatory
190
scholarly open-access publishers and journals" [19]. Indeed, one journal was included both on
191
Beall’s list and in the DOAJ. Moreover, Beall's list was not static, and some publishers and journals
192
were added or removed over time. Nevertheless, the data presented support the doubt that PJs do
193
not adequately check the quality of published RCTs. The mean PEDro score of all Beall journals
194
was lower than 5, except for Physical Medicine and Rehabilitation International which, however,
195
only published two RCTs during the study period. Furthermore, non-inclusion on Beall’s list was an
196
independent predictor of the PEDro score, despite explaining a very small proportion of the
197
variance. Such a small proportion probably depends on the fact that no selected Beall journals were
198
indexed in PubMed, except for the Journal of Physical Therapy Science; thus, the effect of the two
199
factors largely overlaps, and indexing in PubMed is a stronger predictor of the PEDro score than
200
non-inclusion on Beall’s list. Interestingly, the strongest differences in mean PEDro scores and high
201
quality study proportions emerged when periodicals were distinguished according to being/not
202
being non-Beall+Pubmed journals.
8
203
Inclusion of PJs in PubMed is a relevant issue. Several PJs in four important disciplines are indexed
204
in PubMed, namely, rehabilitation 12% [10], neurosciences 11.4% and neurology 20.2% [20], and
205
orthopaedics 8.9% [6]. Among the journals selected in this study, the Journal of Physical Therapy
206
Science was the only Beall journal indexed in PubMed. It published a huge number (over 500) of
207
RCTs in the study period; this accounts for over half the RCTs published in the PJs and is
208
approximately twice the number of RCTs published in the nine NPJs altogether. These numbers
209
indicate that the risk of corrupting evidence is real.
210
The peer review time, which varied among and within the periodicals, was longer on average in
211
non-Beall journals and PubMed journals than in the others. All the PJs showed a shorter peer review
212
duration than all but one NPJ. The NPJ with the shortest acceptance time (Iranian Rehabilitation
213
Journal) had a mean peer review time only just shorter than the PJ with the longest acceptance time
214
(Journal of Yoga & Physical Therapy). Interestingly, the average quality of the RCTs published in
215
the Iranian Rehabilitation Journal was the lowest among the DOAJ journals.
216
These findings regarding peer review time are consistent with the scarce data available in literature.
217
In predatory orthopaedic journals, the reviewing time of PJs was either less than one month or not
218
reported in 36.5% and 56.2% of orthopaedics cases respectively [6], and ranged from 2.7 to 21 days
219
in nursing [5]. In this study, the mean reviewing time of physical therapy PJs was longer than the
220
times reported in other fields, but this difference most likely depends on the type of articles
221
included (only RCTs instead of all types of articles).
222
Edie and Conklin [4] reviewed 53 nursing PJs using 12 quality indicators of peer review processes
223
and showed that language, author-control, and process-transparency indicators were generally poor.
224
Thus, a faster peer review time might be a marker of a less accurate editorial process in PJs. This
225
hypothesis is supported by the finding of a weak but highly significant positive association between
226
peer review time and PEDro score. However, in the multivariate analysis the factor was not an
227
independent predictor of the trial quality. This datum is not surprising, since the peer review
228
duration depends on several variables (e.g., availability and timeliness of reviewers/editors, quality
9
229
of the first submission) and a careful peer reviewing process may also be completed in a short time.
230
Moreover, reducing the time from submission to acceptance is a goal of legitimate journals too.
231
Therefore, although a very quick peer review is a consistent feature of PJs, it cannot be directly
232
linked to quality.
233 234
Study limitations
235
Several methodological biases might have influenced the results. First, inclusion on Beall's list was
236
the only criterion adopted to identify potential PJs. Several criticisms have been advanced regarding
237
Beall’s list, such as the lack of specific or detailed inclusion criteria or the fact of being drafted by a
238
single author [21]. Moreover, the list was deactivated in January 2017 and other lists have taken
239
over since then (for example, https://www2.cabells.com/), with many academic institutions having
240
online listings of what they consider to be predatory journals. Consequently, we cannot exclude that
241
one or more of the journals included in the predatory group were not really predatory at all. Nor
242
does inclusion in the DOAJ guarantee per se that a journal acts honestly, even though the procedure
243
in this case is quite different; a voluntary application by the publisher is needed and a committee
244
might reject a journal if it does not meet the required standards.
245
Moreover, we did not consider the possible influence of the differing longevity of the periodicals on
246
the quality of the published articles, i.e. the fact that a new journal is more likely to receive low
247
quality papers. All the selected Beall journals started their publication in 2011 or later, except for
248
the Journal of Physical Therapy Science. Conversely, all the non-Beall journals were already active
249
before that year, except for the Journal of Rehabilitation Science & Research. However, receiving
250
low quality submissions should lead to a lengthening, rather than a shortening, of the peer review
251
process, so it is unlikely that this factor had a major impact.
252
Although raters were not informed about the inclusion of the journals on Beall’s list or in the DOAJ,
253
they were not blinded with regard to the journal where the articles were published, because
254
removing any references to the journal from each paper would hardly have been possible.
10
255
Therefore, we cannot rule out that the assessors actually knew whether the assigned articles were
256
from a predatory or a non-predatory publisher, and this could have influenced the assessments.
257
Moreover, none of the PEDro scale items offer perfect reliability for the consensus ratings, so the
258
use of a cut-off score for classifying the article for quality (high or low) may lead to some
259
misclassification. Since the standard error of the measurement for the PEDro score is 0.70 units
260
[12], the risk of misclassification is high when the assigned score is close to the selected cut-off
261
(i.e., 5/10 or 6/10), and in this study that was true for 38% of the trials (n=162). Nonetheless, the
262
effect, if any, would have been modest because the analysis led to consistent findings, irrespective
263
of the variable (the PEDro scores or the proportions of high/low quality trials) it was compared
264
with.
265 266
Conclusions
267
Within the physical therapy field, PJs publish RCTs of lower quality and have shorter peer review
268
process times than journals not included on Beall's list. Indexing in PubMed is the strongest
269
independent variable associated with the PEDro score, but non-inclusion on Beall’s list explains an
270
additional small portion of the PEDro score variance. Other types of study that are relevant for
271
clinical decisions and development of scientific findings, such as systematic reviews or validation
272
studies, should be investigated. A citation analysis of articles published in PJs is also advisable in
273
order to open a scenario on the impact of these articles on literature.
274 275 276 277 278 279 280
11
References 1. Eriksson S. Helgesson G. Time to stop talking about ‘predatory journals’. Learn Publ 2018; 31(2): 181-3. DOI: https://doi.org/10.1002/leap.1135
281
2. Beall J. Criteria for determining predatory open-access publishers. 2nd edition. Denver, CO:
282
Scholarly Open Access; 2012. http://scholarlyoa.files.wordpress.com/2012/11/criteria-
283
2012-2.pdf.
284
3. Shamseer L, Moher, D, Maduekwe O, Turner L, Barbour V, Burch R, et al. Potential
285
predatory and legitimate biomedical journals: can you tell the difference? A cross-sectional
286
comparison. BMC Medicine 2017; 15: 1. DOI: 10.1186/s12916-017-0785-9
287 288
4. Edie AH, Conklin JL. Avoiding predatory journals: Quick peer review processes too good to be true. Nurs Forum. 2019 Feb 25. doi: 10.1111/nuf.12333. [Epub ahead of print]
289
5. Oermann MH, Conklin JL, Nicoll LH, Chinn PL, Ashton KS, Edie AH, et al. Study of
290
predatory open access nursing journals. J Nurs Scholarsh 2016; 48(6): 624–632. DOI:
291
10.1111/jnu.12248
292
6. Yan JR, Baldawi H, Lex JR, Simchovich G, Baisi LP, Bozzo A, et al. Predatory publishing
293
in orthopaedic research. J Bone Joint Surg Am 2018; 100(21): e138. doi:
294
10.2106/JBJS.17.01569.
295
7. Olivo et al. SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess
296
the quality of randomized controlled trials: a systematic review. Phys Ther 2008; 88(2):156-
297
175. DOI: 10.2522/ptj.20070147
298
8. Xia J, Harmon J, Connolly K, Donnelly R, Anderson M, Howard H. Who publishes in
299
predatory
300
https://doi.org/10.1002/asi.23265
301 302 303
journals.
J Assoc
Sci Technol
2015;
66(7):
1406-1417.
DOI:
9. Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, et al. Stop this waste of people, animals and money. Nature. 2017; 549: 23-25. doi: 10.1038/549023a. 10. Manca A, Martinez G, Cugusi L, Dragone D, Mercuro G, Deriu F. Predatory Open Access in
304
Rehabilitation.
305
10.1016/j.apmr.2017.01.002
12
Inf
Arch
Phys
Med
Rehabil
2017;
98(5):
1051-1056.
DOI:
306
11. Elkins MR, Moseley AM, Sherrington C, Herbert RD, Maher CG. Growth in the
307
Physiotherapy Evidence Database (PEDro) and use of the PEDro scale. Br J Sports Med
308
2013; 47(4): 188–189. DOI: 10.1136/bjsports-2012-091804
309 310
12. Maher G, Sherrington C, Herbert R et al. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther 2003; 83(8): 713–721.
311
13. Oermann MH, Nicoll LH, Chinn PL, Ashton KS, Conklin JL, Edie AH, et al. Quality of
312
articles published in predatory nursing journals. Nurs Outlook. 2018; 66(1): 4-10. doi:
313
10.1016/j.outlook.2017.05.005.
314 315 316 317 318 319 320 321
14. Beall J. Medical publishing triage - chronicling predatory open access publishers. Ann Med Surg (Lond). 2013; 2(2): 47-9. doi: 10.1016/S2049-0801(13)70035-9 15. Beall J. Predatory journals threaten the quality of published medical research. J Orthop Sports Phys Ther 2017; 47(1): 3-5. doi:10.2519/jospt.2017.0601 16. Cook C. Predatory Journals: The Worst Thing in Publishing, Ever. J Orthop Sports Phys Ther 2017;47(1):1-2. doi:10.2519/jospt.2017.0101 17. Ross-White A, Godfrey CM, Sears KA, Wilson R. Predatory publications in evidence syntheses. J Med Libr Assoc 2019; 107(1): 57–61. doi: 10.5195/jmla.2019.491
322
18. Wager E. Why we should worry less about predatory publishers and more about the quality
323
of research and training at our academic institutions. J Epidemiol 2017; 27(3): 87–88.
324
DOI:10.1016/j.je.2017.01.001
325 326
19. Butler D. Investigating journals: The dark side of publishing. Nature 2013;495(7442):433-5. doi: 10.1038/495433a.
327
20. Manca A, Martinez G, Cugusi L, Dragone D, Dvir Z, Deriu F. The surge of predatory open-
328
access in neurosciences and neurology. Neuroscience 2017;353: 166-173. doi:
329
10.1016/j.neuroscience.2017.04.014.
330 331
13
21. Teixeira da Silva JA. Caution with the continued use of Jeffrey Beall’s “predatory” open access pub-lishing lists. AME Medical Journal 2017; 2: 1–5.
332 333
Figure captions:
334
Figure 1. Flow-chart of the selection procedure.
335
Figure 2. Mean PEDro score in non-Beall and Beall journals.
336
Figure 3. Mean PEDro score in PubMed and non-indexed journals.
337
Figure 4. Mean PEDro score in non-Beall+PubMed journals and the others.
338
Figure 5. Duration of the peer review process in non-Beall and in Beall journals.
339
Figure 6. Duration of the peer review process in PubMed and non-indexed journals.
340
Figure 7. Duration of the peer review process in non-Beall+PubMed journals and the others.
341
14
Contribution of the Paper: •
Predatory physical therapy journals publish RCTs of lower quality and have shorter time of peer review process than journals not included in the Beall's list.
•
Due to the great number of papers published by predatory physical therapy journals, there is a clear risk that predatory publishers substantially contribute to the dissemination of unsound research findings.
Table 1. Main features of included journals.
Journal
Publisher
Country
Journal release
Papers in 2016
RCT in 2014-2017
OMICS International
India
2014
62
14
International Journal of Physical Medicine OMICS International & Rehabilitation
India
2013
92
30
International Journal of Physiotherapy (IJPHI)
India
2014
125
160
International Journal of Physiotherapy and IMED Research Research Publications
India
2014
72
144
Journal of Novel Physiotherapies
OMICS International
India
2011
56
25
Journal of Physical Therapy Science
J Stage
Japan
1990
726
517
Journal of Yoga & Physical Therapy
OMICS International
India
2011
47
4
Open Journal of Therapy and Rehabilitation
Scientific Research Publishing
China
2013
19
9
Physical Medicine and Rehabilitation International
Austin Publishing Group
USA
2014
20
2
BMC Sports Science, Medicine and Rehabilitation
BioMed Central
UK
2009
39
17
Brazilian Journal of Physical Therapy
Associação Brasileira
Brazil
2006
70
25
Beall International Journal of Neurorehabilitation
Standalone journal
Not beall
de Pesquisa e PósGraduação em Fisioterapia Egyptian Rheumatology and Rehabilitation
Wolters Kluwer Egypt Medknow Publications
1974
31
15
Rehabilitation Research and Practice
Hindawi
Egypt
2010
18
11
Iranian Rehabilitation Journal
Negah Institute for Scientific Communication
Iran
2003
34
37
Journal of NeuroEngineering and Rehabilitation
BioMed Central
UK
2004
101
36
Journal of Physiotherapy
Elsevier
Australia
1955
90
30
Journal of Rehabilitation Medicine
Foundation for Rehabilitation Information
Scandinavia
1969
129
60
Journal of Rehabilitation Sciences and Research
Shiraz University of Medical Sciences
Iran
2014
20
16
Table 2. PEDro score, and peer review time (mean ± standard deviation) and quality (low/high) of articles published in included journals. Journal
Pedro Score Mean (SD) (range)
Quality
Peer Review Time (days) Mean (SD) (range)
4.36 (1.8) (0 - 7)
LOW
55.21 (97.6) (3 - 375)
International Journal of Physical Medicine & Rehabilitation 4.97 (1.8) (2 - 9)
LOW
54.70 (34.2) (10 - 117)
International Journal of Physiotherapy (IJPHI)
4.74 (1.7) (2 - 7)
LOW
51.92 (27.2) (13 - 149)
International Journal of Physiotherapy and Research
4.11 (1.6) (1 - 8)
LOW
31.37 (36.2) (11 - 240)
Journal of Novel Physiotherapies
4.40 (1.5) (2 - 7)
LOW
37.08 (33.2) (8 - 150)
Journal of Physical Therapy Science
4.28 (1.2) (2 - 7)
LOW
44.91 (23.3) (18 - 147)
Journal of Yoga & Physical Therapy
4.25 (2.2) (1 - 6)
LOW
70.25 (25.6) (53 - 108)
Open Journal of Therapy and Rehabilitation
4.78 (1.8) (2 - 7)
LOW
50.33 (11.8) (35 - 66)
Physical Medicine and Rehabilitation International
5.50 (3.5) (3 - 8)
HIGH
20.50 (2.1) (19 - 22)
BMC Sports Science, Medicine and Rehabilitation
5.6 (1.1) (4 - 8)
HIGH
156.6 (77.1) (37 - 364)
Brazilian Journal of Physical Therapy
6.9 (1.2) (4 - 9)
HIGH
197.4 (58.7) (124 - 333)
Egyptian Rheumatology and Rehabilitation
4.9] (1.1) (3 - 7)
LOW
124.5 (109.9) (18 - 391)
Rehabilitation Research and Practice
6.7 (1.8) (4 - 9)
HIGH
89.3 (31.3) (55 - 134)
Iranian Rehabilitation Journal
3.8 (1.3) (1 - 7)
LOW
61.5 (45.1) (26 - 258)
Journal of NeuroEngineering and Rehabilitation
5.7 (1.5) (3 - 10)
HIGH
208.0 (110.2) (49 - 543)
Journal of Physiotherapy
7.6 (0.9) (5 - 9)
HIGH
143.4 (93.0) (24 - 341)
Beall International Journal of Neurorehabilitation
Not beall
Journal of Rehabilitation Medicine
6.5 (1.4) (3 - 9)
HIGH
221.9 (81.0) (94 - 477)
Journal of Rehabilitation Sciences and Research
4.69 (1.2) (3 - 7)
LOW
89.3 (31.2) (20 - 211)
(((((rehabilitation) OR neurorehabilitation) OR "physical therapy") OR physiotherapy) OR physiotherapies) OR physiotherapeutic
Journals
Beall’s List 59 journals
DOAJ’s list 29 journals Not focused to physiotherapy=7 Not english language= 3 Less than 15 RCTs in 2016= 9 Present also in Beall’s list= 1
Not active in 2014-2017= 24 Less than 15 RCTs in 2016= 25 Not open access=1
Journals included 9
Journals included 9
RCTs selected 207
RCTs selected 212 6 Excluded: study protocol=4 not randomized=2
RCTs
3 Excluded: not randomized=3
RCTs included 204 Scored on PEDro: 25 Scored for the study: 179
RCTs included 206 Scored on PEDro: 103 Scored for the study: 103