An examination of the validity of EPSCALE using factor analysis

Patient Education and Counseling 87 (2012) 120–124

Contents lists available at ScienceDirect

Patient Education and Counseling journal homepage: www.elsevier.com/locate/pateducou

Assessment

An examination of the validity of EPSCALE using factor analysis Daniel P. Edgcumbe a,b,*, Jonathan Silverman b, John Benson a,b a b

General Practice and Primary Care Research Unit, University of Cambridge, Cambridge, UK School of Clinical Medicine, University of Cambridge, Cambridge, UK

A R T I C L E I N F O

A B S T R A C T

Article history: Received 10 April 2011 Received in revised form 10 July 2011 Accepted 12 July 2011

Objective: To examine the validity and utility of the Explanation and Planning Scale (EPSCALE) instrument, a widely used scale for teaching and assessment of explanation and planning skills used by clinicians during the medical interview. Methods: Data obtained across 4 OSCE stations during medical student final MB examinations. Exploratory factor analysis, using a single factor and two factor models (based on prior theory) and a six factor empirical model, suggested by parallel analysis. Participants: 124 medical students sitting final MB examinations at the University of Cambridge. Results: A single factor model represented a very poor fit. A two factor model with factors labelled ‘Explanation’ and ‘Planning’ produced an improved fit, but the best was seen with a six factor model, with factors which broadly corresponded to the domains of the Calgary–Cambridge guide. Conclusions: These factor models provide supportive evidence for the construct validity of EPSCALE. Practice implications: EPSCALE can justifiably be used in the assessment of shared-decision making skills. ß 2011 Elsevier Ireland Ltd. All rights reserved.

Keywords: Medical education Shared decision making Assessment EPSCALE Communication skills

1. Introduction Explanation and Planning Scale (EPSCALE) was derived from the Calgary–Cambridge guide. The Calgary–Cambridge guide is a model for the medical interview that is widely used throughout Europe and North America for teaching and assessment, which was developed by iterative consensus with expert overview [1,2]. EPSCALE is intended to measure the process skills used by clinicians within the context of the medical interview in providing explanations to patients, and planning future management. It includes descriptors for building the relationship between patient and clinician; providing the appropriate information for the patient; aiding accurate recall and understanding; and achieving a shared understanding. The EPSCALE and descriptors are shown in Table 1. There has been a shift in the nature of the relationship between the patient and doctor from one of medical paternalism, to a partnership, so that it is increasingly recognized that shared decision making within a consultation is important [3]. It represents a process of information sharing between doctor and patient, in which both parties take an interest and participate in the decision making process, to arrive at a mutually negotiated

* Corresponding author at: General Practice and Primary Care Research Unit, University of Cambridge, Forvie Site, Robinson Way, Cambridge CB2 0SR, UK. E-mail addresses: [email protected] (D.P. Edgcumbe), [email protected] (J. Benson). 0738-3991/$ – see front matter ß 2011 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.pec.2011.07.011

outcome. Patients who are involved in making decisions about their healthcare experience higher levels of satisfaction[4], and can discuss issues with their doctors in a positive relationship [5]. Despite the acknowledgement of the importance of shared decision making, it is only relatively recently that consideration has been given to measuring the skills required by clinicians to facilitate the process [6]. What exactly constitutes a shared decision for a particular individual is difficult to define [7]. A shared decision is probably one that is most consistent with the patient’s own values [8], but clarifying the values that are important to a particular individual is not an easy task. Nor can the outcome of a decision be used to measure its nature; shared decisions may lead to poor outcomes, or decisions which are not shared may lead to good outcomes. Posthoc analysis of the outcome therefore cannot be used as a way of assessing the extent to which a decision is a shared. One approach has been to seek to measure the shared nature of a decision, for example by using a measure of decisional conflict [9]. Another has been to examine the decision-making process,[10] with the development of instruments such as ‘Elements of SharedDecision Making’[11] and OPTION (observing patient involvement in decision making) [12,13]. EPSCALE does include shared decision making, but in contrast to existing instruments, does not start at the point of decision and includes a more holistic set of skills which facilitate effective shared decision making. The instrument comprises 15 separate elements, each of which has an ordinal scale. The scale ranges from 0, where a particular

Table 1 EPSCALE and descriptors. EPSCALE

0

Building the relationship Respects patient

Discovers what other information No effort to discover what extra would help patient information would help Aiding accurate recall and understanding Organises explanation No organisation of explanation Checks patient’s understanding Uses clear language Achieving a shared understanding: incorporating the patient’s perspective Relates explanations to patient’s illness framework Encourages patient to contribute reactions, feelings and own ideas Picks up &responds to patient’s non-verbal &covert verbal cues Planning: shared decision making Explores management options with patient Involves patient in decision making

Appropriately negotiates mutually acceptable action plan

2

3

Little interest and concern for patient’s well being Minimal (only non-verbal) response to patient’s feelings and predicament Little eye contact OR some inappropriate non-verbal behaviour

Some interest and concern for patient Some verbal response to patient’s feelings and predicament Good eye contact, generally appropriate non-verbal behaviour

Clear interest and concern for patient as a person Sensitive verbal and non-verbal response to patient’s feelings and predicament Good eye contact, substantial and appropriate non-verbal behaviour

Occasional pauses but does not elicit patient’s response Attempts to find out starting point but still gives info as prepared

Pauses, with some effort to gauge patient’s response before proceeding Discovers starting point, some adjustment to info- giving

Little effort to discover or respond to patient’s info needs

Makes some effort to discover and address patient’s info needs

Minimal organisation of explanation

Organises explanation, but no overt Organises explanation, with overt signposting / signposting / summarising summarising Carefully checks that patient has understood Asks patient to restate information given

Does not check patient understanding

Minimal checking that patient has understood Frequent use of unexplained jargon and Some unexplained jargon and confusing language confusing language

Repeatedly chunks and checks, using patient’s response to guide next steps Discovers starting point and patient’s preference for amount of information, carefully tailors explanation Carefully and repeatedly seeks and addresses patient’s needs

Majority of language used clear (1-2 unexplained jargon words only)

Clear language used throughout

No reference at all to patient’s ideas, concerns, expectations No opportunities for patient to contribute No response to patient’s non-verbal and covert verbal cues

Little attempt to relate explanation to pt’s ideas, etc Limited opportunities for patient to contribute but no response Minimal response to patient’s non-verbal and covert verbal cues

Makes reasonable attempt to relate explanation to pt’s ideas, etc Several opportunities for pt to contribute with some response Some response to patient’s non-verbal and covert verbal cues

Sensitively relates explanation to ideas, etc

No exploration of available options, only directives given No involvement or resists involvement of patient in decision making, directives given Presents plan without checking with patient

Offers options in cursory fashion

Carefully explores options with patient

Makes suggestions rather than directives but limits patient involvement in decision making Presents plan with cursory check for patient’s approval

Actively encourages patient involvement in decision making

Fully explores options and dilemmas, signposting position of equipoise or own preferences Establishes level of involvement patient wishes in decision making: if appropriate, fully encourages patient to make choices &decisions Full and appropriate negotiation of plan with patient; final agreement checked

Reasonable and appropriate negotiation of plan with patient

Actively encourages patient to contribute and responds well Sensitively responds to patient’s non-verbal and covert verbal cues

D.P. Edgcumbe et al. / Patient Education and Counseling 87 (2012) 120–124

Shows no interest or concern OR is overtly offensive Empathy Ignores patient’s feelings and predicament Uses appropriate non-verbal behaviour No eye contact OR inappropriate non-verbal behaviour Providing the correct amount/type of information for the individual patient Chunks and checks, using patient’s Gives long, uninterrupted speech response to guide next steps Assesses the patient’s starting point No attempt to gauge patient’s starting point

1

121

122

D.P. Edgcumbe et al. / Patient Education and Counseling 87 (2012) 120–124

skill is not demonstrated or performed very poorly, to 3 where there is excellent demonstration of a skill. Individual descriptors are given to guide examiners. Previous work has demonstrated that EPSCALE has high internal consistency, provided evidence of content validity and defined its generalizability [2]. This paper reports an assessment of the construct validity of EPSCALE. We used a factor analysis of EPSCALE data to achieve this. By examining the interrelationships between the EPSCALE components and comparing the results obtained with the theoretical basis of EPSCALE, we were able to make a judgement about how well the instrument measures what it purports to measure. 2. Methods Data were obtained from 124 clinical medical students at the University of Cambridge, sitting their Final MB examinations in November 2006. There were four stations in the examination which specifically assessed explanation and planning skills. Students were given details of the cases two weeks prior to the examination, to ensure that they could obtain relevant content knowledge, as these stations were designed to assess process skills, rather than clinical knowledge. Students were assessed by direct observation, with one examiner for each station. The examiners were not the same for all stations for all students, consisting of hospital specialists, general practitioners and communications specialist teachers who had been trained in a 2 hour calibration session. The development of EPSCALE itself is described in greater detail in previous work [2]. Students’ scores from each of the four OSCE stations were pooled, giving 496 sets of marks. Cases where EPSCALE data were missing were removed, leaving 460 cases for analysis. All analysis was performed using R 2.10.1 for Mac OS, Leopard build 32-bit (R Foundation for Statistical Computing, 2009). We undertook exploratory factor analysis to look for underlying structure in the data. Factor analysis assumes a model in which the observed data are influenced by hypothetical underlying factors [14]. We set out to examine whether the processes measured by EPSCALE reflect heterogeneous sets of skills, in the context of a preexisting theoretical model (the Calgary–Cambridge guide). Exploratory factor analysis can shed light on the number of distinct factors (or skill sets) that affect the total EPSCALE score, by examining the correlation between the individual variables and the modeled factors. When conducting factor analysis, the Pearson correlation coefficient matrix is traditionally used to quantify the correlation between variables. However for ordinal variables (which have limited range and may be highly skewed) use of the polychoric correlation coefficient is thought to yield more accurate results [15,16]. We therefore undertook analysis using the polychoric correlation coefficients, which were calculated using the polycor package. Exploratory factor analysis was performed using the psych package, which permits a variety of fitting and rotation methods. The output includes a number of summary statistics, including goodness-of-fit measures. We used a maximum likelihood method for factor modeling and rotation. A fundamental question when conducting factor analysis is how many factors should be extracted to model the data. There are a number of methods which have been used, including the K1 method (which retains factors with eigenvalues of greater than one), Cattell’s scree test and parallel analysis. There are also less quantitative approaches such as those which focus on interpretability of extracted factors, or emphasize prior theory in factor selection [17]. In essence the choice of number of factors to use can be based on prior theory, by mathematical methods, or by a

combination of quantitative and qualitative methods. We used two separate approaches, one based on prior theory and the other using a mathematical approach where we employed a refined version[18] of parallel analysis [19]. This approach led us to explore: 1. A single factor model, on the basis of prior theoretical consideration, as EPSCALE is one scale, to explore the possibility that it represents a single construct. 2. A two factor varimax-rotated model, on the basis of prior theoretical consideration. EPSCALE as an ‘Explanation’ and ‘Planning’ scale might be expected to distinguish these dimensions. 3. A six factor oblimin-rotated model, based on the number of factors suggested by the parallel analysis, without recourse to prior theory. We examined the goodness-of-fit of the resulting models, using the Tucker-Lewis index, also known as the non-normed fit index (NNFI). This a goodness-of-fit index that is relatively unaffected by sample size [20]. Conventionally, a good fit is considered to be achieved if the NNFI is 0.9, with an excellent fit represented by an NNFI 0.95. 3. Results 3.1. Single factor analysis Results of the single factor analysis are shown in Table 2. All of the EPSCALE items loaded highly onto the single factor, i.e. the variables all correlated highly with the single factor. The single factor model however only accounted for 42 of the observed variance and the NNFI was 0.62. The single factor model therefore represented a poor fit, suggesting a solution with more than one factor would be appropriate. On the basis that as a scale measuring ‘Explanation’ and ‘Planning’ skills, the instrument might be expected to incorporate these theoretical domains, we proceeded to a two factor model. 3.2. Two factor analysis The two factor model resulted in a better fit, with an NNFI of 0.8. The two factors together accounted for 52 of the variance observed in the data. The factor loadings are shown in Table 3. Content analysis allowed these two factors justifiably to be named: Table 2 Single factor model loadings. Single factor Building the relationship Respects patient 0.776 Empathy 0.760 Non-verbal behaviour 0.689 Providing the correct amount/type of information for the individual patient Chunks and checks 0.680 Assesses starting point 0.638 Discovers other info 0.532 Aiding accurate recall and understanding Organises explanation 0.690 Checks understanding 0.474 Uses clear language 0.627 Achieving a shared understanding: incorporating the patient’s perspective Relates explanation to illness 0.672 Encourages contributions 0.740 Responds to non-verbal and covert cues 0.693 Planning: shared decision making Explores management options 0.508 Involves patient 0.616 Negotiates a plan 0.531

D.P. Edgcumbe et al. / Patient Education and Counseling 87 (2012) 120–124 Table 3 Two factor model loadings. Planning

Explanation

Building the relationship 0.786 0.220 Respects patient Empathy 0.829 0.094 Non-verbal behaviour 0.668 0.210 Providing the correct amount/type of information for the individual patient Chunks and checks 0.712 0.112 Assesses starting point 0.590 0.216 0.501 0.161 Discovers other info Aiding accurate recall and understanding Organises explanation 0.622 0.269 Checks understanding 0.405 0.252 Uses clear language 0.606 0.181 Achieving a shared understanding: incorporating the patient’s perspective Relates explanation to illness 0.606 0.223 0.633 0.349 Encourages contributions Responds to non-verbal and covert cues 0.634 0.233 Planning: shared decision making Explores management options 0.202 0.788 Involves patient 0.288 0.911 Negotiates a plan 0.207 0.829 Bold value shows which factors variables have been assigned to.

1. Explanation 2. Planning Twelve items of the EPSCALE loaded highly onto the ‘Explanation’ factor, and the remaining three loaded highly onto the ‘Planning’ factor. There was little in the way of cross-loading. 3.3. Six factor analysis The NNFI for the six factor model was 0.96, suggesting an excellent fit. The six factor model accounted for 52 of the observed data variance. Content analysis was used to ascribe labels and the factor loadings are shown in Table 4. 4. Discussion and conclusion 4.1. Discussion This study is the first factor analysis performed on EPSCALE. A six factor model resulted in an excellent fit with factors labeled: relationship; organisation; patient’s involvement; non-verbal; checking; and planning.

123

We found that a single factor model did not adequately reflect the data. As a scale purporting to measure ‘Explanation’ and ‘Planning’ skills, it is not surprising that a single factor model fits the data very poorly. This is consistent with EPSCALE encapsulating more than one theoretical construct. A two factor model had a much better fit than the single factor model, producing two readily interpretable factors ‘Explanation’ and ‘Planning’. Although the fit provided by the two factor model is much better than that of the single factor model and represents a substantial proportion of the observed variance, it still did not entirely adequately represent the data; this was reflected by a goodness-of-fit index that fell somewhat short of what would be considered a good fit. This is probably because the ‘Explanation’ domains incorporate a greater range of skills than the ‘Planning’ domain; this is consistent with the Calgary–Cambridge formulation, which divides the ‘Explanation’ skills into four separate domains, whilst ‘Planning’ is represented as a single domain. On the basis of prior empirical parallel analysis, we produced a six factor oblimin-rotated model. The six factor model is more complicated, but a number of features are worth highlighting. The ‘Planning’ factor was preserved from the two factor model, with no cross-loading of the variables onto other factors, thus providing further support to the notion that planning represents a distinct theoretical construct. The extracted factors broadly correspond to the domains of the Calgary–Cambridge guide, albeit with some overlap. Nor is there any significant redundancy, with all of the observed variables loading on to the six factors. Together, these provide further evidence of the instrument’s validity. Clear strengths of our work include: the data for this work were obtained from medical students sitting finals examinations, which is an appropriate context given EPSCALE is frequently used in teaching and assessment; we used a robust method for determining the number of factors and appropriate correlation calculations for the nature of the data. However, a limitation of our work is that it is not possible to make any claims about the generalizability of the observed factor structure: whether a similar structure would be seen in the consultations of experienced practising doctors remains to be seen. In relation to other studies, efforts have been made to develop other measures of shared decision making. Perhaps the most prominent of these is the OPTION scale. OPTION is a reliable scale used to measure shared-decision making processes, which was developed from general practice consultations, with rigorous scale-development procedures [12].

Table 4 Six factor model loadings. Relationship

Organisation

Building the relationship Respects patient 0.669 0.098 Empathy 0.901 0.014 Non-verbal behaviour 0.200 0.046 Providing the correct amount/type of information for the individual patient Chunks and checks 0.238 0.326 Assesses starting point 0.076 0.400 Discovers other info 0.300 0.016 Aiding accurate recall and understanding Organises explanation 0.041 0.894 Checks understanding 0.060 0.171 Uses clear language 0.334 0.227 Achieving a shared understanding: incorporating the patient’s perspective Relates explanation to illness 0.014 0.319 Encourages contributions 0.004 0.019 Responds to non-verbal cues 0.099 0.171 Planning: shared decision making Explores management options 0.032 0.091 Involves patient 0.047 0.061 Negotiates a plan 0.094 0.057 Bold value shows which factors variables have been assigned to.

Patient’s involvement

Non-verbal

Checking

Planning

0.059 0.052 0.024

0.157 0.022 0.728

0.019 0.004 0.063

0.132 0.016 0.065

0.072 0.154 0.581

0.254 0.124 0.212

0.385 0.105 0.092

0.075 0.048 0.031

0.013 0.144 0.179

0.055 0.014 0.040

0.040 0.526 0.064

0.044 0.121 0.061

0.480 0.543 0.352

0.231 0.293 0.385

0.168 0.180 0.145

0.031 0.161 0.064

0.016 0.037 0.046

0.087 0.007 0.065

0.084 0.080 0.057

0.786 0.988 0.811

124

D.P. Edgcumbe et al. / Patient Education and Counseling 87 (2012) 120–124

Attempted factor analysis of the OPTION scale employed a scree plot, which suggested a two factor solution, but it was not possible to determine a pattern to the item loadings. This was interpreted by the authors as the OPTION scale measuring only a single construct of shared decision-making [13]. This contrasts with EPSCALE, where a six factor model was produced, with demonstrable discriminant validity, accounting for a large proportion of the variance This is in keeping with the idea that EPSCALE incorporates a broader range of skills than OPTION, which is very much focused on the clinical decision itself. 4.2. Conclusion The EPSCALE instrument already has demonstrable reliability and generalizability. Our work provides strong further evidence for the construct validity of EPSCALE, justifying its use for the formative and summative assessment of skills in shared decision making. 4.3. Practical implications It is clear that EPSCALE captures multiple dimensions of the explanation and planning process, however, this may only represent a small part of shared decision-making. Patients may consult with friends, relatives and other peers in arriving at a decision. They may take time to weigh up their own values and make judgements about what is most important to them, and they may make use of decision aids outside of the consulting room. There is nevertheless a clear role for doctors in supporting shared decision-making and empowering patients to express their preferences (even if that preference is that they would like their doctor to make all the decisions for them). Medical students therefore need to be equipped with the skills to give appropriate information to patients and support shared decision-making, and then assessed on their ability to put those skills into practice. EPSCALE can be used (and indeed is already used) in both the teaching and assessment of explanation and planning skills as an instrument with proven reliability and generalisability characteristics [2]. Further research could undertake exploratory factor analysis on an independent sample, or attempt to extrapolate results to a different setting (such as practising doctors). Highly skilled, experienced communicators might be expected to score more highly on EPSCALE than medical students and work could be undertaken to examine this, to further assess EPSCALE’s validity.

References [1] Silverman J, Kurtz S, Draper J.In: Skills for communicating with patients2nd ed., Oxford Radcliffe Medical Press; 2005. [2] Silverman J, Archer J, Gillard S, Howells R, Benson J. Initial evaluation of epscale, a rating scale that assesses the process of explanation and planning in the medical interview. Patient Educ Couns 2011;82:89–93. http:// doi:10.1016/j.pec.2010.02.022. http://www.sciencedirect.com/science/ article/B6TBC-4YP16V2-3/2/17e6e9315fd9c5f7d503628538476438. [3] Emanuel EJ, Emanuel LL. Four models of the physician-patient relationship. J Amer Med Assoc 1992;267:2221–6. KIE: KIE BoB Subject Heading: professional patient relationship; KIE: Full author name: Emanuel, Ezekiel J; KIE: Full author name: Emanuel, Linda L. [4] Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, et al. Systematic review of involving patients in the planning and development of health care. Brit Med J 2002;325:1263. KIE: KIE Bib: health care. [5] Thornton H, Edwards A, Elwyn G. Evolving the multiple roles of ‘patients’ in health-care research: reflections after involvement in a trial of shared decision-making. Health Expect 2003;6:189–97. [6] Charles C, Gafni A, Whelan T. Shared decision-making in the medical encounter: what does it mean? (or it takes at least two to tango) Soc Sci Med 1997;44:681–92. http://www.sciencedirect.com/science/article/ B6VBF-3SWT27W-24/2/6e8ecf2da7a9c87cdd63ee9a1fc97f9c. [7] Ratliff A, Angell M, Dow RW, Kuppermann M, Nease RFJ, Fisher R, et al. What is a good decision? Eff Clin Pract 1999;2:185–97. [8] Marteau TM, Dormandy E, Michie S. A measure of informed choice. Health Expect 2001;4:99–108. KIE: 30 refs.; KIE: KIE Bib: informed consent; prenatal diagnosis. [9] O’Connor AM. Validation of a decisional conflict scale. Med Decis Making 1995;15:25–30. http://mdm.sagepub.com/content/15/1/25.abstract. [10] Elwyn G, Elwyn B, Miron-Shatz T.In: Shared Decision-Making in Healthcare. Oxford University Press; 2009. 143–149. [11] Braddock III CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. J Amer Med Assoc 1999;282:2313–20. http://jama.ama-assn.org/cgi/content/abstract/ 282/24/2313. [12] Elwyn G, Edwards A, Wensing M, Hood K, Atwell C, Grol R. Shared decision making: developing the option scale for measuring patient involvement. Qual Saf Health Care 2003;12:93–9. [13] Elwyn G, Hutchings H, Edwards A, Rapport F, Wensing M, Cheung WY, et al. The option scale: measuring the extent that clinicians involve patients in decision-making tasks. Health Expect 2005;8:34–42. http://dx.doi.org/ 10.1111/j.1369-7625.2004.00311.x. [14] Lawley DN, Maxwell AE.In: Factor Analysis as a Statistical Method2nd ed., Butterworths: London; 1971. [15] Gilley WF, Uhlig GE. Factor analysis and ordinal data. Education 1993;114. [16] Holgado-Tello F. Chaco´n-Moscoso S., Barbero-Garcı´ a I., Vila-Abad E.. Polychoric versus pearson correlations in exploratory and confirmatory factor analysis of ordinal variables.. Qual Quant 2010;44:153–66. http://dx.doi.org/ 10.1007/s11135-008-9190-y. [17] Hayton JC, Allen DG, Scarpello V. Factor retention decisions in exploratory factor analysis: a tutorial on parallel analysis. Org Res Methods 2004;7:191–205. [18] Glorfield LW. An improvement on horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educ Psychol Meas 1995;55:377–93. [19] Horn JL. A rationale and test for the number of factors in factor analysis. Psykometrika 1965;32:179–85. [20] Bollen KA. Overall fit in covariance structure models: Two types of sample size effects. Psychol Bull 1990;107:256–9.