- Email: [email protected]
A checklist to assess the quality of reports on spa therapy and balneotherapy trials was developed using the Delphi consensus method: The SPAC checklist
Complementary Therapies in Medicine (2013) 21, 324—332
Available online at www.sciencedirect.com
journal homepage: www.elsevierhealth.com/journals/ctim
A checklist to assess the quality of reports on spa therapy and balneotherapy trials was developed using the Delphi consensus method: The SPAC checklist Hiroharu Kamioka a,∗, Yoichi Kawamura b, Kiichiro Tsutani c, Masaharu Maeda d, Shinya Hayasaka e, Hiroyasu Okuizum f, Shinpei Okada g, Takuya Honda h, Yuichi Iijima i a
Faculty of Regional Environment Science, Tokyo University of Agriculture, Japan Oyamada Memorial Spa Hospital, Japan c Department of Drug Policy and Management, Graduate School of Pharmaceutical Sciences, The University of Tokyo, Japan d Department of Rehabilitation, International University of Health and Welfare Graduate School, Japan e Faculty of Sports and Health Science, Department of Health Science, Daito Bunka University, Japan f Mimaki Onsen (Spa) Clinic, Tomi City, Japan g Physical Education and Medicine Research Foundation, Japan h Fellow of the Japanese Society for the Promotion of Science, Japan i The Shinano Mainichi Shimbun, Japan Available online 7 June 2013 b
KEYWORDS Spa therapy; Balneotherapy; Interventional trials; Health enhancement; Curative effect
Summary Objective: The purpose of this study was to develop a checklist of items that describes and measures the quality of reports of interventional trials assessing spa therapy. Methods: The Delphi consensus method was used to select the number of items in the checklist. A total of eight individuals participated, including an epidemiologist, a clinical research methodologist, clinical researchers, a medical journalist, and a health fitness programmer. Participants ranked on a 9-point Likert scale whether an item should be included in the checklist. Results: Three rounds of the Delphi method were conducted to achieve consensus. The final checklist contained 19 items, with items related to title, place of implementation (specificity of spa), care provider influence, and additional measures to minimize the potential bias from withdrawals, loss to follow-up, and low treatment adherence. Conclusion: This checklist is simple and quick to complete, and should help clinicians and researchers critically appraise the medical and healthcare literature, reviewers assess the quality of reports included in systematic reviews, and researchers plan interventional trials of spa therapy. © 2013 Elsevier Ltd. All rights reserved.
∗ Corresponding author at: Faculty of Regional Environment Science, Tokyo University of Agriculture, 1-1-1 Sakuragaoka, Setagaya-ku, Tokyo 156-8502, Japan. Tel.: +81 35477 2587; fax: +81 35477 2587. E-mail address: [email protected] (H. Kamioka).
0965-2299/$ — see front matter © 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ctim.2013.05.002
A checklist to report spa trials
325
Introduction
Steering author
Balneotherapy or spa therapy has been frequently used as an alternative to medicine for the treatment of disease. Spa therapy is a very popular form of treatment for all types of arthritis in many European countries, as well as in Israel and Japan.1,2 In addition, some reports have demonstrated that comprehensive health education, which includes lifestyle education and exercise in combination with spa bathing, has positive effects for male white-collar workers3 and middleaged and elderly people.4,5 In a recent study, there was a randomized controlled trials (RCTs) which clarified the effects of treatment and QOL for psoriasis patients using sulphurous water.6 In addition, well planned RCTs for atopic dermatitis in children7 and osteoarthritis of the hands8 were carried out using balneotherapy. RCTs are widely accepted as the most reliable method to assess the efficacy of treatments. Assessing the effectiveness of nonpharmacological treatments (NPTs) such as psychotherapy, behavioral therapy, surgery, or acupuncture presents specific methodological issues.9—11 In NPT trials, it is often difficult to perform sham intervention, and blinding of participants and care providers is frequently impossible.9—13 Intervention with spa therapy also has the same issues. Moreover, unlike pharmacological treatment, the success of spa therapy treatment and health enhancement often depends on environmental and other specific factors such as chemical and thermal character, water temperature, ambient temperature, type of bath, nature, and combined intervention such as walking and stretching exercise. Assessing the quality of study reports is particularly important for researchers’ and clinicians’ critical appraisal of healthcare literature, and for systematic reviews.14,15 The AMSTAR16 and PRISMA statement17 specifies that ‘‘the criteria and process used for validity assessment’’ should be reported. Several checklist tools13,18 take into account specific methodological issues in assessing NPTs, such as influence of care providers, standardization, feasibility of blinding, and the risk of bias in unblinded trials. Moreover, specific and unique tools were developed for acupuncture11 or traditional Chinese medicine.19 However, as one type of NPT, spa therapy and its effectiveness cannot be assessed appropriately regardless of the increased number of reports. If the raw material for these reports is flawed, then the conclusions of systematic reviews are more likely to compound these biases. It therefore seems important to develop a specific tool to assess the quality of these study reports. The purpose of this study was to develop a checklist of items that describe and measure the quality of reports of interventional trials that assess spa therapy.
A steering author (H.K.) was in charge of determining the purpose of the instrument, generating items, and selecting participants for the Delphi consensus method to select and reduce the number of items. The steering author also analyzed and discussed participants’ qualitative and quantitative answers after each round, and decided on the format of the feedback.
Methods We applied the general methods for instrument development used by others.18,20—23 This process consists of three main steps: definition of the purpose of the tool, generating items, and selecting items.
Definition of the purpose of the tool The purpose of the tool was to recommend the description and assess the quality of reports of interventional trials that assessed the effectiveness of spa therapy on cure and health enhancement. In this study, quality was defined as internal validity, which implies that the differences observed between groups of participants are linked to the treatment and bias is avoided. We focused on special and methodological items particularly relevant to spa therapy; we did not include the general items already covered in the CONSORT of NPTs,18 the CONSORT 2010,22 and the TREND statement23 such as method of randomization and intention-to-treat analysis in the selection process. We did not consider other aspects of quality such as reporting, clinical relevance, precision of outcomes, statistical analysis, ethical issues, and the appropriateness of the conclusions. The definition of spa therapy varies according to a country and an area, and there is not the clear consensus. For the definition of spa therapy in this study, we used spa bathing for health enhancement and care prevention in communitydwelling people, as well as cure. The definition included comprehensive health education such as exercise, meal, and other healthcare activities. Moreover, this definition specified that spa therapy included visits to spa facilities, but did not include extended stays in spa hotels.24
Generating items Development of the initial pool of items was performed in two stages. First, we collected and confirmed items from existing quality tools of interventional trials.18,22,23 We decided to focus on special and methodological items based on the definition of the purpose of the tool. Specifically, we decided to build a checklist with the title, background, methods, and results. Second, we enumerated items specific to spa therapy that were identified in reference to systematic reviews based on RCTs.25—29 These items were categorized under the following headings: standardization and quality of the treatments, characteristics of spa, care providers, and treatment adherence. Twenty-seven potential items were identified. Each item was reported with a clear definition of what it meant and with examples, when necessary. For example, the item ‘‘Setting’’ had the following definition: ‘‘Clear descriptions of spa with right condition for intervention of health enhancement and cure.’’
326
H. Kamioka et al. Initial checklist 27 items
ROUND 1
Modification of wording; Combination of redundant items; Addition of new item
Oct 7, 2011
Checklist (2nd edition) 28 items
ROUND 2 Oct 20, 2011
Checklist (3rd edition)
Analysis
28 items
ROUND 3
Confirmation of all items; Confirmation of results and discussion
or resulted from a combination of redundant items, the initial item or items were highlighted in red. The objectives of this survey were to construct a checklist with a limited number of items and to select only items with a high level of agreement for most of the participants. When a participant was absent from a round conference, the complete minutes were sent to the participant, and the participant reranked after reviewing the minutes. Because participants were allowed to fully understand all items, the steering author did not delete the item that had a low score in each round.
Feb 4, 2012
Checklist (Final edition) 19 items
Figure 1 Flowchart of items to compile a checklist to assess the quality of intervention/trials of spa therapy. Ratings are 9-point Likert scale, with 1 = disagree and 9 = agree.
Selecting participants A total of eight experts were invited by e-mail to participate, and seven joined this study. One expert declined the invitation because of transfer from an academic institution to the Ministry of Health, Labour and Welfare of Japan. The steering author also participated in this panel as a member. These participants included an epidemiologist, clinical research methodologist, clinical researchers, a medical journalist, and a health fitness programmer. Among five of them who were medical doctors, four had the qualification of board certified member by the Japanese Society of Balneology, Climatology and Physical Medicine.
Selecting items The Delphi consensus method21,30 was used to select and reduce the number of items. Three rounds of the Delphi method were conducted to achieve consensus (Fig. 1). During the first Delphi round, participants individually ranked, on a 9-point Likert scale (1, disagree, to 9, agree), whether an item should be included in the checklist: ‘‘Do you agree that this item should be included in the checklist for spa bathing trials?’’ Participants had the opportunity to give reasons for their choices, to add some items, or to modify wording of items. During the second and third Delphi rounds, participants reranked their agreement with each item; they had the opportunity to change their score in view of the group’s response to the previous round, but participants were told that they need not conform to the group’s view. For each item, participants could see their choice during the previous round and an anonymous summary of participants’ answers (proportion of participants responding to the question who selected each value). If an item was extensively modified
The responses for each Delphi round were reported as the proportion of participants choosing each value of the 9-point Likert scale. Participants’ commentaries were reported for each item. The steering author assessed participants’ qualitative and quantitative answers after each round. Based on the participants’ comments (i.e., qualitative answers) and quantitative answers, redundant items were combined, categories of items with an insufficient consensus rate were excluded, items proposed by participants were added, and items were modified or expanded. Finally, the steering author accepted items rated more than seven points for the third round.
Results Delphi consensus method The flow of items through each round is summarized in Fig. 1.
First round All participants completed the first round of the survey. The 27 items identified were categorized as follows: the title (n = 1), the background in introduction (n = 1), the setting in intervention (n = 16), the care provider in intervention (n = 2), the detailed information in intervention (n = 4), the flow of participants in results (n = 1), the supplementary analysis in results (n = 1), and the intervention cost in results (n = 1). Participants individually ranked on a 9-point Likert scale (1, disagree, to 9, agree) the 27 items and checked an anonymous summary of participants’ answers and basic statistics (mean, standard deviation and range). Based on their comments (i.e., qualitative answers), redundant items were combined (e.g., scale of bathtub; depth and area of bathtub), items proposed by participants were added (information about the intervention subject; e.g., public health service of administration, research group, etc.), items were modified (changed from ‘‘explanation that why researcher carried out in the spa facility’’ to ‘‘explanation of being the spa facility which is suitable for health enhancement’’), items were expanded (some exemplifications of items), and one item was divided into two items (experience and qualification of care provider). According to the participant ratings and comments, 28 items were therefore identified for the second round.
A checklist to report spa trials Table 1 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Basic statistics of checklist after second round.
The item number of CONSORT2010
The issues which should be included (descriptor)
Mean
SD
Min.
Max.
1a (Title) 2a (Introduction background) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention)
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub and room temperature
8.8 8.4 8.0
0.5 0.9 0.9
8 7 7
9 9 9
6.9
0.8
6
8
7.5
1.3
5
9
pH
7.3
1.5
4
9
Chemical and thermal characteristics of spa
7.6
0.7
7
9
Color and smell
6.5
2.0
3
8
Yield of hot spa water
4.8
2.5
1
8
Existence of free-flowing hot spa
4.4
2.4
1
7
Existence of drinkable spa (when applicable)
6.8
2.1
3
9
Guidance recommended for the bathing. (when applicable, the contents) Type of bath
6.5
1.9
4
9
6.9
1.4
5
9
Scale of bathtub
7.4
1.1
6
9
Information about the intervention subject (e.g., public health service of administration, research group, etc.) Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) Access to the site (shuttle bus service, cars, etc.)
7.4
0.7
7
9
7.0
1.2
5
9
6.5
1.5
5
9
(When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise or meals, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Existence of the reward to participants (when applicable, the details) Number and detail reason of dropout Adherence (the frequency and rate of actual implementation) Total cost of intervention and the cost per person
7.4
1.3
5
9
7.6
0.9
7
9
7.3 8.5
0.9 0.9
6 7
9 9
8.6
0.7
7
9
8.4
0.7
7
9
8.6 7.0
0.7 1.3
7 5
9 9
8.1 8.0
0.6 1.1
7 6
9 9
6.8
1.5
5
9
19
4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 5. (Intervention: care providers)
20 21
5. (Intervention: care providers) 5. (Intervention: care providers)
22
5. (Intervention: care providers)
23
5. (Intervention: care providers)
24 25
5. (Intervention: care providers) 5. (Intervention: care providers)
26 27
13a. (Results: participant flow) 18. (Results: ancillary analysis)
28
(Results: intervention cost)
16 17 18
327
Panel; n = 8.
328 Second round Table 1 shows the basic statistics of the 28 items in the second round. In this round, there were 19 items with a mean of seven or more points. All participants confirmed these results. Third round Table 2 shows the basic statistics of the 28 items. In this round, there were also19 items with a mean of seven or more points. All participants confirmed these results. The final checklist of 19 items is provided in Table 3.
Discussion This study resulted in a checklist of items that measured the quality of interventional studies assessing spa therapy. This checklist is simple and quick (i.e., about 3 min to complete the checklist) to complete, and can be used for critical appraisal of the medical and healthcare literature, the planning of interventional studies assessing spa therapy, and the quality assessment of reports of spa therapy trials included in systematic reviews. According to the scientific standard used to develop quality assessment tools, this instrument resulted from a consensus of a panel of eight participants. These participants had varying experience and backgrounds. The number of panel participants was small, but participants performed ranking and discussion in all rounds. The panel was therefore well informed on specific issues in evaluating spa therapy. Tools used to assess the quality of reports could be a scale with a global score or a checklist of items, with different components assessed separately.13 The use of overall quality scores as an indicator of information on several features has been debated.13—15 We decided to construct a checklist of items that could be used to show the independent influence of each item on the effect of spa treatment. Moreover, we did not include items already covered in the checklists18,22,23 such as method of randomization and intention-to-treat analysis in the selection process, and did not consider other aspects of quality such as reporting, clinical relevance, precision of outcomes, statistical analysis, ethical issues, and the appropriateness of the conclusions. It has been more difficult to standardize spa therapy based on multiple interventions (comprehensive and combined methods). However, the item related to the description of the intervention resulted in one of the highest levels of agreement. A clear description of the intervention is necessary to evaluate the treatment administered and consider whether two interventions are similar enough to be grouped in a meta-analysis. For instance, it would seem unacceptable to have no details about the bathing program (content) and an additional intervention (comprehensive education for participants) of administration of spa therapy, both of which are complex and difficult to standardize. However, we could propose a standard form of description of the intervention based on spa therapy in this study. Specific tools of spa therapy have not been developed despite the increased number of reports. Exposure to warm water is a popular treatment (pain-relief effect) for many patients with painful neurologic or musculoskeletal conditions.31 The warmth and buoyancy of water may block
H. Kamioka et al. nociception by acting on thermal receptors and mechanoreceptors, thus influencing spinal segmental mechanisms.32,33 In addition, the warmth may enhance blood flow, which is thought to help in dissipating pain producing substances, and it may facilitate muscle relaxation. Also, the hydrostatic effect may relieve pain by reducing peripheral edema34 and by dampening sympathetic nervous system activity.35 Systematic reviews (SRs) of RCTs were also performed.25—28 These SRs suggested that RCTs based on appropriate research methodology were needed in spa therapy. Specifically, there was insufficient evidence in studies of aquatic exercise, due to poor methodological and reporting quality, and heterogeneity. Therefore, it was necessary to select items specific to spa therapy. As for the specificity of spa, the following 19 items were extracted: explanation of spa suitable for intervention as health enhancement, location of spa facility, bathtub and room temperature, pH, chemical and thermal characteristics, scale of bathtub, information about intervention subject, and existence of exposure other than bathing. Because readers and reviewers obtain information on interventional trials without exceptions, these items are indispensable for meta-analysis. The individual spas have various characteristics, but minimum information is available for these items. Care providers can have some influence on the treatment effect, whether pharmacological or not; however, this influence is more important for spa therapy such as comprehensive intervention (e.g., exercise and bathing), in which the care provider may be considered part of the treatment.1,3—5 Care providers also have more influence on treatment effect when blinding is impossible; however, there are a small number of existing quality tools related to the influence of care providers.11,13,18,19 Assessing skill or experience generally depends on the trial question.36,37 For pragmatic trials, it is more suitable to provide background information about care providers, such as experience, frequency of performing a procedure, years of practice, qualifications, and certification. For explanatory trials, information about the care provider is necessary to show an appropriate intervention standard. We highlighted that withdrawals, loss to follow-up, and low treatment adherence can also be linked to treatment failure, and could be used as outcomes. Thus, it could be difficult to conclude that a trial had low quality based on these items. When patients and care providers are not blinded, performance bias can occur. Therefore, one would probably have more confidence in the results of a trial in which patients and care providers were not blinded, but in which the number of withdrawals and numbers lost to follow-up were similar in all arms.13 It is important that patients are well informed about their participation in the clinical trial, and that the informed consent process be fully administered. Investigators, site staff, and patients should have open communication throughout the duration of the trial.38 Moreover, a recent study suggested that it is important to recognize and address many factors that commonly lead to higher levels of missing data in order to meaningfully reduce missing data.39 It is important to assess whether blinding was performed because, even if blinding is not feasible, results are more
A checklist to report spa trials Table 2 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Basic statistics of checklist after third round.
The item number of CONSORT2010
The issues which should be included (descriptor)
Mean
SD
Min.
Max.
1a (Title) 2a (Introduction background) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention)
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub temperature
9.0 8.9 7.5
0 0.4 1.2
9 8 5
9 9 9
7.5
1.2
6
9
8.6
0.7
7
9
pH
7.6
1.5
5
9
Chemical and thermal characteristics of spa
8.4
0.9
7
9
Color and smell
6.3
1.6
5
9
Yield of hot spa water
4.5
2.8
1
8
Existence of free-flowing hot spa
4.3
2.5
1
8
Existence of drinkable spa intervention (when applicable) Guidance recommended for the bathing. (when applicable, the contents) Type of bath
6.9
2.5
3
9
6.4
2.1
3
9
6.8
2.1
3
9
Scale of bathtub
7.5
0.9
6
9
Information about the intervention subject (e.g., public health service of administration, research group, etc.) Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) Access to the site (shuttle bus service, cars, etc.)
7.5
1.2
6
9
7.5
1.5
5
9
6.1
1.6
4
8
(When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise, meals, or drinkable spa, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Existence of the reward to participants (when applicable, the details) Number and detail reason of dropout Adherence (the frequency and rate of actual implementation) Total cost of intervention and the cost per person
7.1
1.2
5
9
7.9
1.2
5
9
7.3 8.9
1.2 0.4
6 8
9 9
8.8
0.7
7
9
8.3
0.9
7
9
8.5 6.4
0.8 1.1
7 5
9 8
8.3 8.0
0.7 0.8
7 7
9 9
4.8
2.3
1
8
19
4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 5. (Intervention: care providers)
20 21
5. (Intervention: care providers) 5. (Intervention: care providers)
22
5. (Intervention: care providers)
23
5. (Intervention: care providers)
24 25
5. (Intervention: care providers) 5. (Intervention: care providers)
26 27
13a. (Results: participant flow) 18. (Results: ancillary analysis)
28
(Results: Intervention cost)
16 17 18
329
Panel; n = 8.
330 Table 3 Item no.
H. Kamioka et al. Final checklist of items for interventional trials based on spa therapy: the SPAC checklist. Section/topic
The issues which should be included (descriptor)
1 2 3
Title Introduction Method; place of implementation
4
Method; place of implementation
5 6 7 8 9
Method; Method; Method; Method; Method;
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub temperature pH Chemical and thermal characteristics of spa Scale of bathtub Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) (When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Information about the intervention subject (e.g., public health service of administration, research group, etc.) Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise, meals, or drinkable spa, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Number and detail reason of dropout Adherence (the frequency and rate of actual implementation)
place place place place place
of of of of of
implementation implementation implementation implementation implementation
10
Method; place of implementation
11
Method; care providers
12 13
Method; care providers Method; intervention
14
Method; intervention
15
Method; intervention
16
Method; intervention
17 18 19
Method; intervention Result: participant flow Results: ancillary analysis
likely to be biased if the trial is not blinded. In this important issue, we support the CONSORT of NPTs18 definition of the aim of the tool, as described in the Methods section. Although intervention cost was rejected by experts during the consensus process, the items ‘‘total cost of intervention and the cost per person’’ was discussed in depth. Some argued that this item was a key issue, because enforcers of health care examine feasibility by including an intervention expense as well as an effect. Others, however, argued that this checklist was not the design that established the principal objective for the economic effect and should not be required for comprehensive spa therapy. We decided this item was a different important issue and should be more fully evaluated in the future. We assume that this SPAC checklist could be used for quality evaluation of reported articles up to the present. Assessing the quality is also particularly important for researchers’ and clinicians’ critical appraisal of healthcare literature, and for systematic reviews. It may help progress research methodology as a complementary therapy to identify poor descriptions of Balneotherapy. Additionally, the
Reported on page no.
SPAC may improve the quality of article evaluation when combined with other appropriate guideline a as prompted by the EQUATOR network.40 There were several limitations to the present study. Information bias was a limitation because there were not many participants (eight experts). The use of the Delphi consensus method involves some subjectivity in interpreting experts’ answers and in determining the cutoff number of items. Another limitation was that we restricted the definition of quality to internal validity, and other components of quality, such as external validity and relevance of the trials, were not considered. Moreover, we decided to develop a single quality tool usable for spa therapy. Further, other aspects of quality not included in this tool need to be considered when assessing quality in other items (e.g., items related to the method for handling the clustering of observations in the statistical analysis), and could be added to this list. Like various statements and checklists, this checklist is a work-in-progress that will be updated. We therefore invite and welcome readers to submit comments, critiques, and suggestions for improvement.
A checklist to report spa trials
Conclusion This is the first checklist developed using the Delphi technique with experienced participants that assessed the quality of trial reports evaluating spa therapy. This checklist is simple and quick to complete, and should help clinicians and researchers critically appraise the medical and healthcare literature, reviewers assess the quality of reports included in systematic reviews, and researchers plan interventional trials based on spa therapy.
Contributors HK (steering author) conceived the study and takes responsibility for the quality assessment and summary of included studies and data extraction. YK, KT, YI, MM, SH, HO and SO, and SH ranked all items in each round and participated in the discussion for selecting items, adding some items, or modifying the wording of items. TH performed the statistical analysis. All authors read and approved the manuscript.
Funding This study was supported by the Japan Health and Research Institute Research Grant 2011.
Conflict of interest statement None declared.
Ethical approval Not required.
Acknowledgments We would like to express our appreciation to Ms. Rie Higashino (paperwork) and Ms. Mari Makishi (all searches of studies) for their assistance in this study.
References 1. Nguyen M, Revel M, Dougados M. Prolonged effects of 3 week therapy in a spa resort on lumbar spine, knee and hip osteoarthritis: follow-up after 6 months. A randomized controlled trial. British Journal of Rheumatology 1997;36:77—81. 2. Franke A, Reiner L, Pratzel HG. Long-term efficacy of radon spa therapy in rheumatoid arthritis: a randomized, sham-controlled study and follow-up. Rheumatology 2000;39:894—902. 3. Kamioka H, Nakamura Y, Okada S, Kitayuguchi J, Kamada M, Honda T, et al. Effectiveness of comprehensive health education combining lifestyle education and hot spa bathing for male white-collar employees: a randomized controlled trial with 1year follow-up. Journal of Epidemiology 2009;19:219—30. 4. Kamioka H, Nakamura Y, Yazaki T, Uebaba K, Mutoh Y, Okada S, et al. Effectiveness of comprehensive health education combining hot spa bathing and lifestyle education in middle-aged and elderly women: one-year follow-up on randomized controlled trial of three- and six-month interventions. Journal of Epidemiology 2006;16:35—44.
331 5. Kamioka H, Ohshiro H, Mutoh Y, Honda T, Okada S, Takahashi M, et al. Effect of long-term comprehensive health education on the elderly in a Japanese village: Unnan cohort study. International Journal of Sport and Health Science 2008;6:60—5. 6. Gálvez Galve JJ, Peiro PS, Luca MO, Torres AH, Gil ES, Perez MB. Quality of life and assessment after local application of sulphurous water in the home environment in patients with psoriasis vulgaris: a randomised placebo-controlled pilot study. European Journal of Integrative Medicine 2012;4:e213—8. 7. Farina S, Gisondi P, Zanoni M, Pace M, Rizzoli L, Baldo E, et al. Balneotherapy for atopic dermatitis in children at Comano spa in Trentino, Italy. Journal of Dermatological Treatment (England) 2011;22:66—71. 8. Horvath K, Kulisch A, Nemeth A, Bender T. Evaluation of the effect of balneotherapy in patients with osteoarthritis of the hands: a randomized controlled single-blind follow-up study. Clinical Rehabilitation (England) 2012;26:431—41. 9. Black N. Why we need observational studies to evaluate the effectiveness of health care. British Medical Journal 1996;312:1215—8. 10. McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. Randomised trials in surgery: problems and possible solutions. British Medical Journal 2002;324:1448—58. 11. MacPherson H, White A, Cummings M, Jobst K, Rose K, Niemtzow R. Standards for reporting interventions in controlled trials of acupuncture: the STRICTA recommendations. Complementary Therapies in Medicine 2001;9:246—9. 12. Boutron I, Tubach F, Giraudeau B, Ravaud P. Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. Journal of the American Medical Association 2003;290:1062—70. 13. Boutron I, Moher D, Tugwell P, Giraudeau B, Poiaudeau S, Nizard R, et al. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. Journal of Clinical Epidemiology 2005;58:1233—40. 14. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. British Medical Journal 2001; 323:42—6. 15. Verhagen AP, de Vet HCW, de Bie RA, Boers M. The art of quality assessment of RCTs included in systematic reviews. Journal of Clinical Epidemiology 2001;54:651—4. 16. Sea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic review. BMC Medical Research Methodology 2007;7, http://dx.doi.org/10.1186/1471-2288-7-10. 17. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine 2009;151:W65—94. 18. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, CONSORT group. Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Annals of Internal Medicine 2008; 148:W60—6. 19. Wu T, Shang H, Bian Z, Zhang J, Li T, Li Y, et al. Recommendations for reporting adverse drug reactions and adverse events of traditional Chinese medicine. Journal of Evidence-Based Medicine 2010;3:11—7. 20. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled Clinical Trials 1996;17:1—12. 21. Verhagen AP, de Vet HCW, de Bie RA, Kessels AGH, Boers M, Bouter LM, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting
332
22.
23.
24.
25.
26.
27.
28.
29.
30.
H. Kamioka et al. systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology 1998;51:1235—41. Schulz KF, Altman DG, Moher D, CONSORT group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. British Medical Journal 2010;340:c332, http://dx.doi.org/10.1136/bmj.c332. Des Jarlais DC, Lyles C, Crepaz N, TREND group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. American Journal of Public Health 2004;94:361—6. Wigler I, Elkayam O, Paran D, Yaron M. Spa therapy for ganarthrosis: a prospective study. Rheumatology International 1995;15:65—8. Verhagen AP, Bierma-Zeinstra SMA, Boers M, Cardoso JR, Lameck J, de Bie RA. Balneotherapy for rheumatoid arthritis. Cochrane Database of Systematic Reviews 2008;3:CD000518. Verhagen AP, Bierma-Zeinstra SMA, Boers M, Cardoso JR, Lambeck J, de Bie RA, et al. Balneotherapy for osteoarthritis. Cochrane Database Syst Rev 2007;4:CD006864. Forestier R, Franocon A. Crenobalneotherapy for limb osteoarthritis: systematic literature review and methodological analysis. Joint, Bone, Spine 2008;75:138—48. Pittler MH, Karagülle MZ, Karagülle M, Ernst E. Spa therapy and balneotherapy for treating low back pain: meta-analysis of randomized trials. Rheumatology 2006;45:880—4. Kamioka H, Tsutani K, Okuizumi H, Mutoh Y, Ohta M, Handa S, et al. Effectiveness of aquatic exercise and balneotherapy: a summary of systematic reviews based on randomized controlled trials of water immersion therapies. Journal of Epidemiology 2010;20:2—12. Dalkey NC, Helmer O. An experimental application of the Delphi-method to the use of experts. Management Science 1963;9:458—67.
31. Hall J, Swinkels A, Briddon J, McCabe CS. Does aquatic exercise relieve pain in adults with neurologic or musculoskeletal disease? A systematic review and meta-analysis of randomized controlled trials. Archives of Physical Medicine and Rehabilitation 2008;89:873—83. 32. Bender T, Karaglle Z, Balint GP, Gutenbrunner C, Balint PV, Sukenik S. Hydrotherapy, balneotherapy, and spa treatment in pain management. Rheumatology International 2005;25:220—4. 33. Yamazaki F, Endo Y, Torii S, Sagawa S, Shiraki K. Continuous monitoring of change in hemodilution during water immersion in humans: effect of water temperature. Aviation Space and Environmental Medicine 2000;71: 632—9. 34. Gabrielsen A, Videbaek R, Johansen LB, Warberg J, Christensen NJ, Pump B, et al. Forearm vascular and neuroendocrine responses to graded water immersion in humans. Acta Physiologica Scandinavica 2000;169:87—94. 35. Fam AG. Spa treatment in arthritis: a rheumarologist’s view. Journal of Rheumatology 1991;18:1775—7. 36. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. Journal of Chronic Diseases 1967;20:637—48. 37. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. Journal of Clinical Epidemiology 2009;62:499—505. 38. Gabriel AP, Mercado CP. Data retention after a patient withdraws consent in clinical trials. Open Access Journal of Clinical Trials 2011;3:15—9. 39. Fleming TR. Addressing missing data in clinical trials. Annals of Internal Medicine 2011;154:113—7. 40. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR network (URL http://www.equator-network.org/).
Available online at www.sciencedirect.com
journal homepage: www.elsevierhealth.com/journals/ctim
A checklist to assess the quality of reports on spa therapy and balneotherapy trials was developed using the Delphi consensus method: The SPAC checklist Hiroharu Kamioka a,∗, Yoichi Kawamura b, Kiichiro Tsutani c, Masaharu Maeda d, Shinya Hayasaka e, Hiroyasu Okuizum f, Shinpei Okada g, Takuya Honda h, Yuichi Iijima i a
Faculty of Regional Environment Science, Tokyo University of Agriculture, Japan Oyamada Memorial Spa Hospital, Japan c Department of Drug Policy and Management, Graduate School of Pharmaceutical Sciences, The University of Tokyo, Japan d Department of Rehabilitation, International University of Health and Welfare Graduate School, Japan e Faculty of Sports and Health Science, Department of Health Science, Daito Bunka University, Japan f Mimaki Onsen (Spa) Clinic, Tomi City, Japan g Physical Education and Medicine Research Foundation, Japan h Fellow of the Japanese Society for the Promotion of Science, Japan i The Shinano Mainichi Shimbun, Japan Available online 7 June 2013 b
KEYWORDS Spa therapy; Balneotherapy; Interventional trials; Health enhancement; Curative effect
Summary Objective: The purpose of this study was to develop a checklist of items that describes and measures the quality of reports of interventional trials assessing spa therapy. Methods: The Delphi consensus method was used to select the number of items in the checklist. A total of eight individuals participated, including an epidemiologist, a clinical research methodologist, clinical researchers, a medical journalist, and a health fitness programmer. Participants ranked on a 9-point Likert scale whether an item should be included in the checklist. Results: Three rounds of the Delphi method were conducted to achieve consensus. The final checklist contained 19 items, with items related to title, place of implementation (specificity of spa), care provider influence, and additional measures to minimize the potential bias from withdrawals, loss to follow-up, and low treatment adherence. Conclusion: This checklist is simple and quick to complete, and should help clinicians and researchers critically appraise the medical and healthcare literature, reviewers assess the quality of reports included in systematic reviews, and researchers plan interventional trials of spa therapy. © 2013 Elsevier Ltd. All rights reserved.
∗ Corresponding author at: Faculty of Regional Environment Science, Tokyo University of Agriculture, 1-1-1 Sakuragaoka, Setagaya-ku, Tokyo 156-8502, Japan. Tel.: +81 35477 2587; fax: +81 35477 2587. E-mail address: [email protected] (H. Kamioka).
0965-2299/$ — see front matter © 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ctim.2013.05.002
A checklist to report spa trials
325
Introduction
Steering author
Balneotherapy or spa therapy has been frequently used as an alternative to medicine for the treatment of disease. Spa therapy is a very popular form of treatment for all types of arthritis in many European countries, as well as in Israel and Japan.1,2 In addition, some reports have demonstrated that comprehensive health education, which includes lifestyle education and exercise in combination with spa bathing, has positive effects for male white-collar workers3 and middleaged and elderly people.4,5 In a recent study, there was a randomized controlled trials (RCTs) which clarified the effects of treatment and QOL for psoriasis patients using sulphurous water.6 In addition, well planned RCTs for atopic dermatitis in children7 and osteoarthritis of the hands8 were carried out using balneotherapy. RCTs are widely accepted as the most reliable method to assess the efficacy of treatments. Assessing the effectiveness of nonpharmacological treatments (NPTs) such as psychotherapy, behavioral therapy, surgery, or acupuncture presents specific methodological issues.9—11 In NPT trials, it is often difficult to perform sham intervention, and blinding of participants and care providers is frequently impossible.9—13 Intervention with spa therapy also has the same issues. Moreover, unlike pharmacological treatment, the success of spa therapy treatment and health enhancement often depends on environmental and other specific factors such as chemical and thermal character, water temperature, ambient temperature, type of bath, nature, and combined intervention such as walking and stretching exercise. Assessing the quality of study reports is particularly important for researchers’ and clinicians’ critical appraisal of healthcare literature, and for systematic reviews.14,15 The AMSTAR16 and PRISMA statement17 specifies that ‘‘the criteria and process used for validity assessment’’ should be reported. Several checklist tools13,18 take into account specific methodological issues in assessing NPTs, such as influence of care providers, standardization, feasibility of blinding, and the risk of bias in unblinded trials. Moreover, specific and unique tools were developed for acupuncture11 or traditional Chinese medicine.19 However, as one type of NPT, spa therapy and its effectiveness cannot be assessed appropriately regardless of the increased number of reports. If the raw material for these reports is flawed, then the conclusions of systematic reviews are more likely to compound these biases. It therefore seems important to develop a specific tool to assess the quality of these study reports. The purpose of this study was to develop a checklist of items that describe and measure the quality of reports of interventional trials that assess spa therapy.
A steering author (H.K.) was in charge of determining the purpose of the instrument, generating items, and selecting participants for the Delphi consensus method to select and reduce the number of items. The steering author also analyzed and discussed participants’ qualitative and quantitative answers after each round, and decided on the format of the feedback.
Methods We applied the general methods for instrument development used by others.18,20—23 This process consists of three main steps: definition of the purpose of the tool, generating items, and selecting items.
Definition of the purpose of the tool The purpose of the tool was to recommend the description and assess the quality of reports of interventional trials that assessed the effectiveness of spa therapy on cure and health enhancement. In this study, quality was defined as internal validity, which implies that the differences observed between groups of participants are linked to the treatment and bias is avoided. We focused on special and methodological items particularly relevant to spa therapy; we did not include the general items already covered in the CONSORT of NPTs,18 the CONSORT 2010,22 and the TREND statement23 such as method of randomization and intention-to-treat analysis in the selection process. We did not consider other aspects of quality such as reporting, clinical relevance, precision of outcomes, statistical analysis, ethical issues, and the appropriateness of the conclusions. The definition of spa therapy varies according to a country and an area, and there is not the clear consensus. For the definition of spa therapy in this study, we used spa bathing for health enhancement and care prevention in communitydwelling people, as well as cure. The definition included comprehensive health education such as exercise, meal, and other healthcare activities. Moreover, this definition specified that spa therapy included visits to spa facilities, but did not include extended stays in spa hotels.24
Generating items Development of the initial pool of items was performed in two stages. First, we collected and confirmed items from existing quality tools of interventional trials.18,22,23 We decided to focus on special and methodological items based on the definition of the purpose of the tool. Specifically, we decided to build a checklist with the title, background, methods, and results. Second, we enumerated items specific to spa therapy that were identified in reference to systematic reviews based on RCTs.25—29 These items were categorized under the following headings: standardization and quality of the treatments, characteristics of spa, care providers, and treatment adherence. Twenty-seven potential items were identified. Each item was reported with a clear definition of what it meant and with examples, when necessary. For example, the item ‘‘Setting’’ had the following definition: ‘‘Clear descriptions of spa with right condition for intervention of health enhancement and cure.’’
326
H. Kamioka et al. Initial checklist 27 items
ROUND 1
Modification of wording; Combination of redundant items; Addition of new item
Oct 7, 2011
Checklist (2nd edition) 28 items
ROUND 2 Oct 20, 2011
Checklist (3rd edition)
Analysis
28 items
ROUND 3
Confirmation of all items; Confirmation of results and discussion
or resulted from a combination of redundant items, the initial item or items were highlighted in red. The objectives of this survey were to construct a checklist with a limited number of items and to select only items with a high level of agreement for most of the participants. When a participant was absent from a round conference, the complete minutes were sent to the participant, and the participant reranked after reviewing the minutes. Because participants were allowed to fully understand all items, the steering author did not delete the item that had a low score in each round.
Feb 4, 2012
Checklist (Final edition) 19 items
Figure 1 Flowchart of items to compile a checklist to assess the quality of intervention/trials of spa therapy. Ratings are 9-point Likert scale, with 1 = disagree and 9 = agree.
Selecting participants A total of eight experts were invited by e-mail to participate, and seven joined this study. One expert declined the invitation because of transfer from an academic institution to the Ministry of Health, Labour and Welfare of Japan. The steering author also participated in this panel as a member. These participants included an epidemiologist, clinical research methodologist, clinical researchers, a medical journalist, and a health fitness programmer. Among five of them who were medical doctors, four had the qualification of board certified member by the Japanese Society of Balneology, Climatology and Physical Medicine.
Selecting items The Delphi consensus method21,30 was used to select and reduce the number of items. Three rounds of the Delphi method were conducted to achieve consensus (Fig. 1). During the first Delphi round, participants individually ranked, on a 9-point Likert scale (1, disagree, to 9, agree), whether an item should be included in the checklist: ‘‘Do you agree that this item should be included in the checklist for spa bathing trials?’’ Participants had the opportunity to give reasons for their choices, to add some items, or to modify wording of items. During the second and third Delphi rounds, participants reranked their agreement with each item; they had the opportunity to change their score in view of the group’s response to the previous round, but participants were told that they need not conform to the group’s view. For each item, participants could see their choice during the previous round and an anonymous summary of participants’ answers (proportion of participants responding to the question who selected each value). If an item was extensively modified
The responses for each Delphi round were reported as the proportion of participants choosing each value of the 9-point Likert scale. Participants’ commentaries were reported for each item. The steering author assessed participants’ qualitative and quantitative answers after each round. Based on the participants’ comments (i.e., qualitative answers) and quantitative answers, redundant items were combined, categories of items with an insufficient consensus rate were excluded, items proposed by participants were added, and items were modified or expanded. Finally, the steering author accepted items rated more than seven points for the third round.
Results Delphi consensus method The flow of items through each round is summarized in Fig. 1.
First round All participants completed the first round of the survey. The 27 items identified were categorized as follows: the title (n = 1), the background in introduction (n = 1), the setting in intervention (n = 16), the care provider in intervention (n = 2), the detailed information in intervention (n = 4), the flow of participants in results (n = 1), the supplementary analysis in results (n = 1), and the intervention cost in results (n = 1). Participants individually ranked on a 9-point Likert scale (1, disagree, to 9, agree) the 27 items and checked an anonymous summary of participants’ answers and basic statistics (mean, standard deviation and range). Based on their comments (i.e., qualitative answers), redundant items were combined (e.g., scale of bathtub; depth and area of bathtub), items proposed by participants were added (information about the intervention subject; e.g., public health service of administration, research group, etc.), items were modified (changed from ‘‘explanation that why researcher carried out in the spa facility’’ to ‘‘explanation of being the spa facility which is suitable for health enhancement’’), items were expanded (some exemplifications of items), and one item was divided into two items (experience and qualification of care provider). According to the participant ratings and comments, 28 items were therefore identified for the second round.
A checklist to report spa trials Table 1 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Basic statistics of checklist after second round.
The item number of CONSORT2010
The issues which should be included (descriptor)
Mean
SD
Min.
Max.
1a (Title) 2a (Introduction background) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention)
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub and room temperature
8.8 8.4 8.0
0.5 0.9 0.9
8 7 7
9 9 9
6.9
0.8
6
8
7.5
1.3
5
9
pH
7.3
1.5
4
9
Chemical and thermal characteristics of spa
7.6
0.7
7
9
Color and smell
6.5
2.0
3
8
Yield of hot spa water
4.8
2.5
1
8
Existence of free-flowing hot spa
4.4
2.4
1
7
Existence of drinkable spa (when applicable)
6.8
2.1
3
9
Guidance recommended for the bathing. (when applicable, the contents) Type of bath
6.5
1.9
4
9
6.9
1.4
5
9
Scale of bathtub
7.4
1.1
6
9
Information about the intervention subject (e.g., public health service of administration, research group, etc.) Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) Access to the site (shuttle bus service, cars, etc.)
7.4
0.7
7
9
7.0
1.2
5
9
6.5
1.5
5
9
(When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise or meals, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Existence of the reward to participants (when applicable, the details) Number and detail reason of dropout Adherence (the frequency and rate of actual implementation) Total cost of intervention and the cost per person
7.4
1.3
5
9
7.6
0.9
7
9
7.3 8.5
0.9 0.9
6 7
9 9
8.6
0.7
7
9
8.4
0.7
7
9
8.6 7.0
0.7 1.3
7 5
9 9
8.1 8.0
0.6 1.1
7 6
9 9
6.8
1.5
5
9
19
4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 5. (Intervention: care providers)
20 21
5. (Intervention: care providers) 5. (Intervention: care providers)
22
5. (Intervention: care providers)
23
5. (Intervention: care providers)
24 25
5. (Intervention: care providers) 5. (Intervention: care providers)
26 27
13a. (Results: participant flow) 18. (Results: ancillary analysis)
28
(Results: intervention cost)
16 17 18
327
Panel; n = 8.
328 Second round Table 1 shows the basic statistics of the 28 items in the second round. In this round, there were 19 items with a mean of seven or more points. All participants confirmed these results. Third round Table 2 shows the basic statistics of the 28 items. In this round, there were also19 items with a mean of seven or more points. All participants confirmed these results. The final checklist of 19 items is provided in Table 3.
Discussion This study resulted in a checklist of items that measured the quality of interventional studies assessing spa therapy. This checklist is simple and quick (i.e., about 3 min to complete the checklist) to complete, and can be used for critical appraisal of the medical and healthcare literature, the planning of interventional studies assessing spa therapy, and the quality assessment of reports of spa therapy trials included in systematic reviews. According to the scientific standard used to develop quality assessment tools, this instrument resulted from a consensus of a panel of eight participants. These participants had varying experience and backgrounds. The number of panel participants was small, but participants performed ranking and discussion in all rounds. The panel was therefore well informed on specific issues in evaluating spa therapy. Tools used to assess the quality of reports could be a scale with a global score or a checklist of items, with different components assessed separately.13 The use of overall quality scores as an indicator of information on several features has been debated.13—15 We decided to construct a checklist of items that could be used to show the independent influence of each item on the effect of spa treatment. Moreover, we did not include items already covered in the checklists18,22,23 such as method of randomization and intention-to-treat analysis in the selection process, and did not consider other aspects of quality such as reporting, clinical relevance, precision of outcomes, statistical analysis, ethical issues, and the appropriateness of the conclusions. It has been more difficult to standardize spa therapy based on multiple interventions (comprehensive and combined methods). However, the item related to the description of the intervention resulted in one of the highest levels of agreement. A clear description of the intervention is necessary to evaluate the treatment administered and consider whether two interventions are similar enough to be grouped in a meta-analysis. For instance, it would seem unacceptable to have no details about the bathing program (content) and an additional intervention (comprehensive education for participants) of administration of spa therapy, both of which are complex and difficult to standardize. However, we could propose a standard form of description of the intervention based on spa therapy in this study. Specific tools of spa therapy have not been developed despite the increased number of reports. Exposure to warm water is a popular treatment (pain-relief effect) for many patients with painful neurologic or musculoskeletal conditions.31 The warmth and buoyancy of water may block
H. Kamioka et al. nociception by acting on thermal receptors and mechanoreceptors, thus influencing spinal segmental mechanisms.32,33 In addition, the warmth may enhance blood flow, which is thought to help in dissipating pain producing substances, and it may facilitate muscle relaxation. Also, the hydrostatic effect may relieve pain by reducing peripheral edema34 and by dampening sympathetic nervous system activity.35 Systematic reviews (SRs) of RCTs were also performed.25—28 These SRs suggested that RCTs based on appropriate research methodology were needed in spa therapy. Specifically, there was insufficient evidence in studies of aquatic exercise, due to poor methodological and reporting quality, and heterogeneity. Therefore, it was necessary to select items specific to spa therapy. As for the specificity of spa, the following 19 items were extracted: explanation of spa suitable for intervention as health enhancement, location of spa facility, bathtub and room temperature, pH, chemical and thermal characteristics, scale of bathtub, information about intervention subject, and existence of exposure other than bathing. Because readers and reviewers obtain information on interventional trials without exceptions, these items are indispensable for meta-analysis. The individual spas have various characteristics, but minimum information is available for these items. Care providers can have some influence on the treatment effect, whether pharmacological or not; however, this influence is more important for spa therapy such as comprehensive intervention (e.g., exercise and bathing), in which the care provider may be considered part of the treatment.1,3—5 Care providers also have more influence on treatment effect when blinding is impossible; however, there are a small number of existing quality tools related to the influence of care providers.11,13,18,19 Assessing skill or experience generally depends on the trial question.36,37 For pragmatic trials, it is more suitable to provide background information about care providers, such as experience, frequency of performing a procedure, years of practice, qualifications, and certification. For explanatory trials, information about the care provider is necessary to show an appropriate intervention standard. We highlighted that withdrawals, loss to follow-up, and low treatment adherence can also be linked to treatment failure, and could be used as outcomes. Thus, it could be difficult to conclude that a trial had low quality based on these items. When patients and care providers are not blinded, performance bias can occur. Therefore, one would probably have more confidence in the results of a trial in which patients and care providers were not blinded, but in which the number of withdrawals and numbers lost to follow-up were similar in all arms.13 It is important that patients are well informed about their participation in the clinical trial, and that the informed consent process be fully administered. Investigators, site staff, and patients should have open communication throughout the duration of the trial.38 Moreover, a recent study suggested that it is important to recognize and address many factors that commonly lead to higher levels of missing data in order to meaningfully reduce missing data.39 It is important to assess whether blinding was performed because, even if blinding is not feasible, results are more
A checklist to report spa trials Table 2 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Basic statistics of checklist after third round.
The item number of CONSORT2010
The issues which should be included (descriptor)
Mean
SD
Min.
Max.
1a (Title) 2a (Introduction background) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention)
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub temperature
9.0 8.9 7.5
0 0.4 1.2
9 8 5
9 9 9
7.5
1.2
6
9
8.6
0.7
7
9
pH
7.6
1.5
5
9
Chemical and thermal characteristics of spa
8.4
0.9
7
9
Color and smell
6.3
1.6
5
9
Yield of hot spa water
4.5
2.8
1
8
Existence of free-flowing hot spa
4.3
2.5
1
8
Existence of drinkable spa intervention (when applicable) Guidance recommended for the bathing. (when applicable, the contents) Type of bath
6.9
2.5
3
9
6.4
2.1
3
9
6.8
2.1
3
9
Scale of bathtub
7.5
0.9
6
9
Information about the intervention subject (e.g., public health service of administration, research group, etc.) Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) Access to the site (shuttle bus service, cars, etc.)
7.5
1.2
6
9
7.5
1.5
5
9
6.1
1.6
4
8
(When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise, meals, or drinkable spa, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Existence of the reward to participants (when applicable, the details) Number and detail reason of dropout Adherence (the frequency and rate of actual implementation) Total cost of intervention and the cost per person
7.1
1.2
5
9
7.9
1.2
5
9
7.3 8.9
1.2 0.4
6 8
9 9
8.8
0.7
7
9
8.3
0.9
7
9
8.5 6.4
0.8 1.1
7 5
9 8
8.3 8.0
0.7 0.8
7 7
9 9
4.8
2.3
1
8
19
4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 4b (Place of implementation) 5 (Intervention) 5. (Intervention: care providers)
20 21
5. (Intervention: care providers) 5. (Intervention: care providers)
22
5. (Intervention: care providers)
23
5. (Intervention: care providers)
24 25
5. (Intervention: care providers) 5. (Intervention: care providers)
26 27
13a. (Results: participant flow) 18. (Results: ancillary analysis)
28
(Results: Intervention cost)
16 17 18
329
Panel; n = 8.
330 Table 3 Item no.
H. Kamioka et al. Final checklist of items for interventional trials based on spa therapy: the SPAC checklist. Section/topic
The issues which should be included (descriptor)
1 2 3
Title Introduction Method; place of implementation
4
Method; place of implementation
5 6 7 8 9
Method; Method; Method; Method; Method;
Identification as a spa intervention in the title Description of why do spa intervention Explanation of spa suitable for intervention as health enhancement Locations of spa facility where the data were collected (location and surrounding environment) Bathtub temperature pH Chemical and thermal characteristics of spa Scale of bathtub Presence of facilities (if any, the property; e.g., comfortable resting room, etc.) (When applicable) existence of other exposure than bathing (sauna, steam bath, etc.) Qualification of care provider (specialist in balneotherapy, related experts and health fitness programmer, etc.) Experience of care provider Information about the intervention subject (e.g., public health service of administration, research group, etc.) Details of the contents of bathing instruction (including bath time, frequency in use and period) When combined with a method other than bathing (exercise, meals, or drinkable spa, etc.), detail of the contents Instructions about daily life (the same as usual, increase the step number a day as much as possible, etc.) Details of how to deal with the control group Number and detail reason of dropout Adherence (the frequency and rate of actual implementation)
place place place place place
of of of of of
implementation implementation implementation implementation implementation
10
Method; place of implementation
11
Method; care providers
12 13
Method; care providers Method; intervention
14
Method; intervention
15
Method; intervention
16
Method; intervention
17 18 19
Method; intervention Result: participant flow Results: ancillary analysis
likely to be biased if the trial is not blinded. In this important issue, we support the CONSORT of NPTs18 definition of the aim of the tool, as described in the Methods section. Although intervention cost was rejected by experts during the consensus process, the items ‘‘total cost of intervention and the cost per person’’ was discussed in depth. Some argued that this item was a key issue, because enforcers of health care examine feasibility by including an intervention expense as well as an effect. Others, however, argued that this checklist was not the design that established the principal objective for the economic effect and should not be required for comprehensive spa therapy. We decided this item was a different important issue and should be more fully evaluated in the future. We assume that this SPAC checklist could be used for quality evaluation of reported articles up to the present. Assessing the quality is also particularly important for researchers’ and clinicians’ critical appraisal of healthcare literature, and for systematic reviews. It may help progress research methodology as a complementary therapy to identify poor descriptions of Balneotherapy. Additionally, the
Reported on page no.
SPAC may improve the quality of article evaluation when combined with other appropriate guideline a as prompted by the EQUATOR network.40 There were several limitations to the present study. Information bias was a limitation because there were not many participants (eight experts). The use of the Delphi consensus method involves some subjectivity in interpreting experts’ answers and in determining the cutoff number of items. Another limitation was that we restricted the definition of quality to internal validity, and other components of quality, such as external validity and relevance of the trials, were not considered. Moreover, we decided to develop a single quality tool usable for spa therapy. Further, other aspects of quality not included in this tool need to be considered when assessing quality in other items (e.g., items related to the method for handling the clustering of observations in the statistical analysis), and could be added to this list. Like various statements and checklists, this checklist is a work-in-progress that will be updated. We therefore invite and welcome readers to submit comments, critiques, and suggestions for improvement.
A checklist to report spa trials
Conclusion This is the first checklist developed using the Delphi technique with experienced participants that assessed the quality of trial reports evaluating spa therapy. This checklist is simple and quick to complete, and should help clinicians and researchers critically appraise the medical and healthcare literature, reviewers assess the quality of reports included in systematic reviews, and researchers plan interventional trials based on spa therapy.
Contributors HK (steering author) conceived the study and takes responsibility for the quality assessment and summary of included studies and data extraction. YK, KT, YI, MM, SH, HO and SO, and SH ranked all items in each round and participated in the discussion for selecting items, adding some items, or modifying the wording of items. TH performed the statistical analysis. All authors read and approved the manuscript.
Funding This study was supported by the Japan Health and Research Institute Research Grant 2011.
Conflict of interest statement None declared.
Ethical approval Not required.
Acknowledgments We would like to express our appreciation to Ms. Rie Higashino (paperwork) and Ms. Mari Makishi (all searches of studies) for their assistance in this study.
References 1. Nguyen M, Revel M, Dougados M. Prolonged effects of 3 week therapy in a spa resort on lumbar spine, knee and hip osteoarthritis: follow-up after 6 months. A randomized controlled trial. British Journal of Rheumatology 1997;36:77—81. 2. Franke A, Reiner L, Pratzel HG. Long-term efficacy of radon spa therapy in rheumatoid arthritis: a randomized, sham-controlled study and follow-up. Rheumatology 2000;39:894—902. 3. Kamioka H, Nakamura Y, Okada S, Kitayuguchi J, Kamada M, Honda T, et al. Effectiveness of comprehensive health education combining lifestyle education and hot spa bathing for male white-collar employees: a randomized controlled trial with 1year follow-up. Journal of Epidemiology 2009;19:219—30. 4. Kamioka H, Nakamura Y, Yazaki T, Uebaba K, Mutoh Y, Okada S, et al. Effectiveness of comprehensive health education combining hot spa bathing and lifestyle education in middle-aged and elderly women: one-year follow-up on randomized controlled trial of three- and six-month interventions. Journal of Epidemiology 2006;16:35—44.
331 5. Kamioka H, Ohshiro H, Mutoh Y, Honda T, Okada S, Takahashi M, et al. Effect of long-term comprehensive health education on the elderly in a Japanese village: Unnan cohort study. International Journal of Sport and Health Science 2008;6:60—5. 6. Gálvez Galve JJ, Peiro PS, Luca MO, Torres AH, Gil ES, Perez MB. Quality of life and assessment after local application of sulphurous water in the home environment in patients with psoriasis vulgaris: a randomised placebo-controlled pilot study. European Journal of Integrative Medicine 2012;4:e213—8. 7. Farina S, Gisondi P, Zanoni M, Pace M, Rizzoli L, Baldo E, et al. Balneotherapy for atopic dermatitis in children at Comano spa in Trentino, Italy. Journal of Dermatological Treatment (England) 2011;22:66—71. 8. Horvath K, Kulisch A, Nemeth A, Bender T. Evaluation of the effect of balneotherapy in patients with osteoarthritis of the hands: a randomized controlled single-blind follow-up study. Clinical Rehabilitation (England) 2012;26:431—41. 9. Black N. Why we need observational studies to evaluate the effectiveness of health care. British Medical Journal 1996;312:1215—8. 10. McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. Randomised trials in surgery: problems and possible solutions. British Medical Journal 2002;324:1448—58. 11. MacPherson H, White A, Cummings M, Jobst K, Rose K, Niemtzow R. Standards for reporting interventions in controlled trials of acupuncture: the STRICTA recommendations. Complementary Therapies in Medicine 2001;9:246—9. 12. Boutron I, Tubach F, Giraudeau B, Ravaud P. Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. Journal of the American Medical Association 2003;290:1062—70. 13. Boutron I, Moher D, Tugwell P, Giraudeau B, Poiaudeau S, Nizard R, et al. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. Journal of Clinical Epidemiology 2005;58:1233—40. 14. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. British Medical Journal 2001; 323:42—6. 15. Verhagen AP, de Vet HCW, de Bie RA, Boers M. The art of quality assessment of RCTs included in systematic reviews. Journal of Clinical Epidemiology 2001;54:651—4. 16. Sea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic review. BMC Medical Research Methodology 2007;7, http://dx.doi.org/10.1186/1471-2288-7-10. 17. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine 2009;151:W65—94. 18. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, CONSORT group. Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Annals of Internal Medicine 2008; 148:W60—6. 19. Wu T, Shang H, Bian Z, Zhang J, Li T, Li Y, et al. Recommendations for reporting adverse drug reactions and adverse events of traditional Chinese medicine. Journal of Evidence-Based Medicine 2010;3:11—7. 20. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled Clinical Trials 1996;17:1—12. 21. Verhagen AP, de Vet HCW, de Bie RA, Kessels AGH, Boers M, Bouter LM, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting
332
22.
23.
24.
25.
26.
27.
28.
29.
30.
H. Kamioka et al. systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology 1998;51:1235—41. Schulz KF, Altman DG, Moher D, CONSORT group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. British Medical Journal 2010;340:c332, http://dx.doi.org/10.1136/bmj.c332. Des Jarlais DC, Lyles C, Crepaz N, TREND group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. American Journal of Public Health 2004;94:361—6. Wigler I, Elkayam O, Paran D, Yaron M. Spa therapy for ganarthrosis: a prospective study. Rheumatology International 1995;15:65—8. Verhagen AP, Bierma-Zeinstra SMA, Boers M, Cardoso JR, Lameck J, de Bie RA. Balneotherapy for rheumatoid arthritis. Cochrane Database of Systematic Reviews 2008;3:CD000518. Verhagen AP, Bierma-Zeinstra SMA, Boers M, Cardoso JR, Lambeck J, de Bie RA, et al. Balneotherapy for osteoarthritis. Cochrane Database Syst Rev 2007;4:CD006864. Forestier R, Franocon A. Crenobalneotherapy for limb osteoarthritis: systematic literature review and methodological analysis. Joint, Bone, Spine 2008;75:138—48. Pittler MH, Karagülle MZ, Karagülle M, Ernst E. Spa therapy and balneotherapy for treating low back pain: meta-analysis of randomized trials. Rheumatology 2006;45:880—4. Kamioka H, Tsutani K, Okuizumi H, Mutoh Y, Ohta M, Handa S, et al. Effectiveness of aquatic exercise and balneotherapy: a summary of systematic reviews based on randomized controlled trials of water immersion therapies. Journal of Epidemiology 2010;20:2—12. Dalkey NC, Helmer O. An experimental application of the Delphi-method to the use of experts. Management Science 1963;9:458—67.
31. Hall J, Swinkels A, Briddon J, McCabe CS. Does aquatic exercise relieve pain in adults with neurologic or musculoskeletal disease? A systematic review and meta-analysis of randomized controlled trials. Archives of Physical Medicine and Rehabilitation 2008;89:873—83. 32. Bender T, Karaglle Z, Balint GP, Gutenbrunner C, Balint PV, Sukenik S. Hydrotherapy, balneotherapy, and spa treatment in pain management. Rheumatology International 2005;25:220—4. 33. Yamazaki F, Endo Y, Torii S, Sagawa S, Shiraki K. Continuous monitoring of change in hemodilution during water immersion in humans: effect of water temperature. Aviation Space and Environmental Medicine 2000;71: 632—9. 34. Gabrielsen A, Videbaek R, Johansen LB, Warberg J, Christensen NJ, Pump B, et al. Forearm vascular and neuroendocrine responses to graded water immersion in humans. Acta Physiologica Scandinavica 2000;169:87—94. 35. Fam AG. Spa treatment in arthritis: a rheumarologist’s view. Journal of Rheumatology 1991;18:1775—7. 36. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. Journal of Chronic Diseases 1967;20:637—48. 37. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. Journal of Clinical Epidemiology 2009;62:499—505. 38. Gabriel AP, Mercado CP. Data retention after a patient withdraws consent in clinical trials. Open Access Journal of Clinical Trials 2011;3:15—9. 39. Fleming TR. Addressing missing data in clinical trials. Annals of Internal Medicine 2011;154:113—7. 40. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR network (URL http://www.equator-network.org/).