- Email: [email protected]
Pre-notification did not increase response rate in addition to follow-up: a randomized trial
Journal of Clinical Epidemiology 63 (2010) 1276e1278
Pre-notification did not increase response rate in addition to follow-up: a randomized trial Alice Hammink*, Paul Giesen, Michel Wensing IQ healthcare, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands Accepted 12 March 2010
Abstract Objective: Although there is growing evidence on the effectiveness of pre-notification and follow-up on response rates in patient surveys, no studies report the effectiveness of pre-notification in addition to follow-up. The aim of this study was to determine the effect of a pre-notification by mail on the response rate in a patient survey with follow-up, compared with follow-up or pre-notification only. Study Design and Setting: Randomized trial that compared (1) a combination of pre-notification and follow-up with (2) prenotification only and (3) follow-up only. The trial was integrated in a survey study among patients, which measured their experiences with general practice cooperatives for out-of-hour care. Results: Of the total number of 880 patients who received the questionnaire, 45% returned it. No significant effect was found of the combination of pre-notification and follow-up compared with the two other arms of the trial. Conclusion: Adding pre-notification to follow-up in a patient survey had no additional effect on the response rate. This finding must be interpreted with respect to the questionnaire and the study population. Ó 2010 Elsevier Inc. All rights reserved. Keywords: GP; cooperatives; Randomized trial; Response rate; Survey methodology; Pre-notification; Follow-up
1. Introduction Large variation in response rates in surveys has been observed in health research. A systematic review [1] showed a mean response rate of 60%, with a standard deviation of 21%. Although a postal questionnaire is a feasible method to collect information from samples of patients, nonresponse reduces the effective sample size and may introduce selection bias [2]. Several effective strategies have been developed to improve response rates, such as (non)monetary incentives, an attractive format of the questionnaire, and various procedures of delivering the questionnaire to respondents [1]. A systematic review of the literature showed that a prenotification can improve response rates [3]. Different forms of pre-notification were studied, including pre-notification by telephone or mail and pre-notification by a medical researcher or nonmedical researcher. The pooled odds ratio of pre-notification compared with no pre-notification was 1.50, but significant variation was found across the 39
* Corresponding author. IQ healthcare, Radboud University Nijmegen Medical Centre, 114, Geert Grooteplein 21, Postbus 9101, Nijmegen 6500, The Netherlands. E-mail address: [email protected] (A. Hammink). 0895-4356/$ - see front matter Ó 2010 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2010.03.011
available trials [3]. The same review showed that followup, that is, repeated contact after questionnaires are sent to patients, can also improve response rates among initial nonresponders [3]. A pooled odds ratio for follow-up of 1.56 was found, but again, significant variation was found across studies and evidence of selection bias. Pre-notification and follow-up can enhance response rates in patient surveys, but it remains unclear whether pre-notification has an added value in surveys that use follow-up [3]. The aim of this study was to determine the effect of a pre-notification by mail on the response rate in a patient survey with follow-up, compared with follow-up only or pre-notification only.
2. Method A randomized trial was performed, which was integrated in a survey study among patients of a general practice (GP) cooperative for out-of-hour care. This survey measured patients’ experiences with quality aspects of care on GP cooperatives. The questionnaire focused on patients’ experiences with the telephone triagist, general practitioner, and the organization of the GP cooperative. It contained 91 items (all with prestructured answering categories) on
A. Hammink et al. / Journal of Clinical Epidemiology 63 (2010) 1276e1278
1277
2.2. Study population What is new? Key finding Adding prenotification to follow-up in a patient survey had no additional effect on the response rate. What this adds to what is known? A combination of prenotification and follow-up in a patient survey had no additional effect on the response rate, compared with prenotification only and follow-up only. What is the implication, and what should change now? More research is needed to determine whether this finding is applicable to other study populations.
seven double-printed pages. The questionnaire was accompanied by a cover letter signed by the director of the GP cooperative and a prepaid envelope. 2.1. Design Two different interventions to increase response rates were tested, a pre-notification intervention and a follow-up intervention. Patients were randomly allocated to one of three groups: (1) Pre-notification and follow-up; (2) prenotification only; and (3) follow-up only. Random allocation was done by an independent statistician, who was not involved in the project. In all cases, the first mailing, whether it was a pre-notification or questionnaire, was delivered 2e5 days after contact with the GP cooperative. After 5e7 days, the questionnaire was sent to subjects who received a pre-notification. The follow-up letter, without the inclusion of a copy of the questionnaire, was delivered to nonresponders 7e9 days after mailing of the questionnaire. To detect a minimal relevant increase of 12% (45e57%) response rate (alpha 5 0.05, power 5 0.80), 268 patients were needed in each arm of the trial.
A total number of 1,752 patients from three GP cooperatives were identified by a member of the staff of the GP cooperative. Of these 1,752 patients, 880 respondents were randomly selected to participate in this study. Of these 880 respondents, 288 received a pre-notification and follow-up letter, 295 received a pre-notification only, and 297 received a follow-up only. 2.3. Measures The primary outcome measure was the final response rate defined as total response within 1 month after mailing the questionnaire. A secondary outcome measure was the initial response defined as response rate within 9 days after mailing the questionnaire. Considering delay of response, patients who returned the questionnaire before receiving a followup letter were indicated as respondents. After 9 days, all questionnaires that were returned within 3 weeks were indicated as additional response. Returned questionnaires because of a wrong address (N 5 5) were excluded from this study, in addition to deceased patients (N 5 10) and patients who were unable or unwilling to fill out the questionnaire (N 5 29). 2.4. Analysis Relative risks with 95% confidence intervals were calculated regarding the effectiveness on response rates, using the group with follow-up only as the reference group. We performed two analyses: an analysis that focused on the effect in the total population (intention-to-treat analysis) and an analysis that focused on the effect in initial nonresponders.
3. Results Table 1 shows the response rates in the three study groups. Forty-five percent of the patients who received pre-notification and follow-up, returned the questionnaire, compared with 43% who received a pre-notification only and 46% who received follow-up only. In both the total targeted population and the population of initial nonresponders, none of the interventions had an additional effect compared with follow-up only.
Table 1 Response rates
Patients approached Initial response Additional response Final response Relative risk: total population (95% CI) Relative risk: initial nonresponders (95% CI) Abbreviation: CI, confidence interval.
Follow-up
Pre-notification
Pre-notification and follow-up
297 69 (23%) 67 136 (46%)
295 60 68 128 0.94 0.97
288 67 63 130 0.98 0.96
(20%) (43%) (0.79e1.13) (0.73e1.29)
(23%) (45%) (0.82e1.17) (0.72e1.28)
1278
A. Hammink et al. / Journal of Clinical Epidemiology 63 (2010) 1276e1278
4. Discussion Increasing response rates in patient surveys remains a challenge. A large number of studies have shown that both pre-notification and follow-up can increase response rates considerably [3]. Combining these two strategies was expected to lead to a higher response rate, but the current study did not confirm this. The findings may suggest that pre-notification and follow-up address the same facilitators for completing a questionnaire, such as increased attention and higher interest. On the other hand, a large number of patients did not complete the questionnaire, suggesting that both strategies only address some of the factors that influence response rates. This study was part of a survey among patients of a GP cooperative for out-of-hour care. Therefore, the findings of this study should be interpreted with respect to the questionnaire and study population, which may not be representative for other patient populations. Response rates of other surveys among patients of GP cooperatives are higher than the response rates reported in this study. For example, a study to measure patient satisfaction with quality of care on GP cooperatives showed an average response rate of 52.2% [4]. Differences in response rates may be caused by the length of the questionnaire, which was considerably longer in the current study. This study did not include a control group, which received no response-enhancing intervention, because the
effectiveness of both pre-notification and follow-up has been proved already [3]. However, this implies that this study could not show the effectiveness of the interventions combined compared with no intervention. Further research is needed to determine whether the results found in this study can be applied to other surveys. If the results of this study are confirmed in other surveys, there is an opportunity to reduce the costs of dispatching questionnaires.
Acknowledgments This study was funded by the Dutch organization for health research and development (ZonMw). References [1] Asch DA, Jedrziewski MK, Christaki NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129e36. [2] Armstrong BK, White E, Saracci R. Principles of exposure measurement in epidemiology. In: Kelsey JL, Marmot MG, Stolley PD, Vessey MP, editors. Monographs in epidemiology and biostatistics. 21st ed. New York: Oxford University Press Inc.; 1995. p. 294e321. [3] Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, et al. Methods to increase response rates to postal questionnaires. Cochrane Database Syst Rev 2007;(2): MR000008. [4] Moll van CE, Giesen P, Mokkink H, Oort F, Grol R, Klazinga N, et al. Patient satisfaction with large-scale out-of-hours primary health care in The Netherlands: development of a postal questionnaire. Fam Pract 2006;23:437e43.
Pre-notification did not increase response rate in addition to follow-up: a randomized trial Alice Hammink*, Paul Giesen, Michel Wensing IQ healthcare, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands Accepted 12 March 2010
Abstract Objective: Although there is growing evidence on the effectiveness of pre-notification and follow-up on response rates in patient surveys, no studies report the effectiveness of pre-notification in addition to follow-up. The aim of this study was to determine the effect of a pre-notification by mail on the response rate in a patient survey with follow-up, compared with follow-up or pre-notification only. Study Design and Setting: Randomized trial that compared (1) a combination of pre-notification and follow-up with (2) prenotification only and (3) follow-up only. The trial was integrated in a survey study among patients, which measured their experiences with general practice cooperatives for out-of-hour care. Results: Of the total number of 880 patients who received the questionnaire, 45% returned it. No significant effect was found of the combination of pre-notification and follow-up compared with the two other arms of the trial. Conclusion: Adding pre-notification to follow-up in a patient survey had no additional effect on the response rate. This finding must be interpreted with respect to the questionnaire and the study population. Ó 2010 Elsevier Inc. All rights reserved. Keywords: GP; cooperatives; Randomized trial; Response rate; Survey methodology; Pre-notification; Follow-up
1. Introduction Large variation in response rates in surveys has been observed in health research. A systematic review [1] showed a mean response rate of 60%, with a standard deviation of 21%. Although a postal questionnaire is a feasible method to collect information from samples of patients, nonresponse reduces the effective sample size and may introduce selection bias [2]. Several effective strategies have been developed to improve response rates, such as (non)monetary incentives, an attractive format of the questionnaire, and various procedures of delivering the questionnaire to respondents [1]. A systematic review of the literature showed that a prenotification can improve response rates [3]. Different forms of pre-notification were studied, including pre-notification by telephone or mail and pre-notification by a medical researcher or nonmedical researcher. The pooled odds ratio of pre-notification compared with no pre-notification was 1.50, but significant variation was found across the 39
* Corresponding author. IQ healthcare, Radboud University Nijmegen Medical Centre, 114, Geert Grooteplein 21, Postbus 9101, Nijmegen 6500, The Netherlands. E-mail address: [email protected] (A. Hammink). 0895-4356/$ - see front matter Ó 2010 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2010.03.011
available trials [3]. The same review showed that followup, that is, repeated contact after questionnaires are sent to patients, can also improve response rates among initial nonresponders [3]. A pooled odds ratio for follow-up of 1.56 was found, but again, significant variation was found across studies and evidence of selection bias. Pre-notification and follow-up can enhance response rates in patient surveys, but it remains unclear whether pre-notification has an added value in surveys that use follow-up [3]. The aim of this study was to determine the effect of a pre-notification by mail on the response rate in a patient survey with follow-up, compared with follow-up only or pre-notification only.
2. Method A randomized trial was performed, which was integrated in a survey study among patients of a general practice (GP) cooperative for out-of-hour care. This survey measured patients’ experiences with quality aspects of care on GP cooperatives. The questionnaire focused on patients’ experiences with the telephone triagist, general practitioner, and the organization of the GP cooperative. It contained 91 items (all with prestructured answering categories) on
A. Hammink et al. / Journal of Clinical Epidemiology 63 (2010) 1276e1278
1277
2.2. Study population What is new? Key finding Adding prenotification to follow-up in a patient survey had no additional effect on the response rate. What this adds to what is known? A combination of prenotification and follow-up in a patient survey had no additional effect on the response rate, compared with prenotification only and follow-up only. What is the implication, and what should change now? More research is needed to determine whether this finding is applicable to other study populations.
seven double-printed pages. The questionnaire was accompanied by a cover letter signed by the director of the GP cooperative and a prepaid envelope. 2.1. Design Two different interventions to increase response rates were tested, a pre-notification intervention and a follow-up intervention. Patients were randomly allocated to one of three groups: (1) Pre-notification and follow-up; (2) prenotification only; and (3) follow-up only. Random allocation was done by an independent statistician, who was not involved in the project. In all cases, the first mailing, whether it was a pre-notification or questionnaire, was delivered 2e5 days after contact with the GP cooperative. After 5e7 days, the questionnaire was sent to subjects who received a pre-notification. The follow-up letter, without the inclusion of a copy of the questionnaire, was delivered to nonresponders 7e9 days after mailing of the questionnaire. To detect a minimal relevant increase of 12% (45e57%) response rate (alpha 5 0.05, power 5 0.80), 268 patients were needed in each arm of the trial.
A total number of 1,752 patients from three GP cooperatives were identified by a member of the staff of the GP cooperative. Of these 1,752 patients, 880 respondents were randomly selected to participate in this study. Of these 880 respondents, 288 received a pre-notification and follow-up letter, 295 received a pre-notification only, and 297 received a follow-up only. 2.3. Measures The primary outcome measure was the final response rate defined as total response within 1 month after mailing the questionnaire. A secondary outcome measure was the initial response defined as response rate within 9 days after mailing the questionnaire. Considering delay of response, patients who returned the questionnaire before receiving a followup letter were indicated as respondents. After 9 days, all questionnaires that were returned within 3 weeks were indicated as additional response. Returned questionnaires because of a wrong address (N 5 5) were excluded from this study, in addition to deceased patients (N 5 10) and patients who were unable or unwilling to fill out the questionnaire (N 5 29). 2.4. Analysis Relative risks with 95% confidence intervals were calculated regarding the effectiveness on response rates, using the group with follow-up only as the reference group. We performed two analyses: an analysis that focused on the effect in the total population (intention-to-treat analysis) and an analysis that focused on the effect in initial nonresponders.
3. Results Table 1 shows the response rates in the three study groups. Forty-five percent of the patients who received pre-notification and follow-up, returned the questionnaire, compared with 43% who received a pre-notification only and 46% who received follow-up only. In both the total targeted population and the population of initial nonresponders, none of the interventions had an additional effect compared with follow-up only.
Table 1 Response rates
Patients approached Initial response Additional response Final response Relative risk: total population (95% CI) Relative risk: initial nonresponders (95% CI) Abbreviation: CI, confidence interval.
Follow-up
Pre-notification
Pre-notification and follow-up
297 69 (23%) 67 136 (46%)
295 60 68 128 0.94 0.97
288 67 63 130 0.98 0.96
(20%) (43%) (0.79e1.13) (0.73e1.29)
(23%) (45%) (0.82e1.17) (0.72e1.28)
1278
A. Hammink et al. / Journal of Clinical Epidemiology 63 (2010) 1276e1278
4. Discussion Increasing response rates in patient surveys remains a challenge. A large number of studies have shown that both pre-notification and follow-up can increase response rates considerably [3]. Combining these two strategies was expected to lead to a higher response rate, but the current study did not confirm this. The findings may suggest that pre-notification and follow-up address the same facilitators for completing a questionnaire, such as increased attention and higher interest. On the other hand, a large number of patients did not complete the questionnaire, suggesting that both strategies only address some of the factors that influence response rates. This study was part of a survey among patients of a GP cooperative for out-of-hour care. Therefore, the findings of this study should be interpreted with respect to the questionnaire and study population, which may not be representative for other patient populations. Response rates of other surveys among patients of GP cooperatives are higher than the response rates reported in this study. For example, a study to measure patient satisfaction with quality of care on GP cooperatives showed an average response rate of 52.2% [4]. Differences in response rates may be caused by the length of the questionnaire, which was considerably longer in the current study. This study did not include a control group, which received no response-enhancing intervention, because the
effectiveness of both pre-notification and follow-up has been proved already [3]. However, this implies that this study could not show the effectiveness of the interventions combined compared with no intervention. Further research is needed to determine whether the results found in this study can be applied to other surveys. If the results of this study are confirmed in other surveys, there is an opportunity to reduce the costs of dispatching questionnaires.
Acknowledgments This study was funded by the Dutch organization for health research and development (ZonMw). References [1] Asch DA, Jedrziewski MK, Christaki NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129e36. [2] Armstrong BK, White E, Saracci R. Principles of exposure measurement in epidemiology. In: Kelsey JL, Marmot MG, Stolley PD, Vessey MP, editors. Monographs in epidemiology and biostatistics. 21st ed. New York: Oxford University Press Inc.; 1995. p. 294e321. [3] Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, et al. Methods to increase response rates to postal questionnaires. Cochrane Database Syst Rev 2007;(2): MR000008. [4] Moll van CE, Giesen P, Mokkink H, Oort F, Grol R, Klazinga N, et al. Patient satisfaction with large-scale out-of-hours primary health care in The Netherlands: development of a postal questionnaire. Fam Pract 2006;23:437e43.