- Email: [email protected]
Career HOPES: An Internet-delivered career development intervention
Computers in Human Behavior 26 (2010) 339–344
Contents lists available at ScienceDirect
Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh
Career HOPES: An Internet-delivered career development intervention Steve Herman *,1 Stanford University, Stanford, CA 94305, United States
a r t i c l e
i n f o
Article history: Available online 9 December 2009 Keywords: Career development Intervention Internet Computer applications
a b s t r a c t Career HOPES is an Internet-delivered group counseling intervention designed to facilitate occupational exploration and career decision making. The intervention includes automated interactive lessons and self-assessments, homework assignments, and group discussions in private online forums. A randomized, controlled experiment (N = 64) was conducted to evaluate (a) the efficacy of Career HOPES and (b) the contribution of professional moderation of the online group discussions to outcomes. Participants in two treatment conditions showed greater gains than control condition participants in career decidedness as measured by the Occupational Alternatives Questionnaire (d = .54), occupationally relevant selfknowledge (d = .58), and emission of career exploration behaviors (d = .50). In one of the treatment conditions, the online group discussions were moderated by a psychologist with career counseling experience; in the other treatment condition, the discussions were unmoderated. Professional moderation resulted in better outcomes on several variables and greater overall satisfaction with the intervention. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Meta-analyses of empirical studies of career interventions (Oliver & Spokane, 1988; Whiston, Sexton, & Lasoff, 1998) have demonstrated that, on the whole, these interventions have a beneficial impact on participants’ career development. The weighted mean effect size for treatment–control comparisons in the 47 career interventions included in the Whiston et al. meta-analysis was d = .30 (k = 268). Participants in the included interventions showed greater gains than control condition participants on measures of (a) decidedness about career choices (d = .19, k = 50), (b) career maturity (d = .53, k = 59), (c) the frequency of career information-seeking behaviors (d = .31, k = 18), (d) vocational knowledge (d = .97, k = 2), and (e) self-knowledge (d = 1.30 , k = 1). Internet delivery can help overcome geographical, psychological, physical, and financial obstacles to accessing psychosocial interventions and counseling services (Mallen, Vogel, Rochlen, & Day, 2005). Thus, the development of an effective, disseminable Internet-based career development intervention is a desirable goal. There are currently many Websites that offer career inventories, career counseling, and other types of career and job search assistance, either for free (e.g., http://monster.com) or for a fee (e.g., http://www.career planner.com/). However, the efficacy of the services offered by most
* Present address: University of Hawaii at Hilo, Psychology Department, 200 West Kawili Street, Hilo, HI 96720, United States. Tel.: +1 808 933 3284; fax: +1 206 309 0486. E-mail address: [email protected] 1 This research was based on a doctoral dissertation. 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.11.003
of these Websites has not been empirically demonstrated (Mallen et al., 2005). One noteworthy exception is the Career Key inventory at http://careerkey.org, which has been examined in a number of studies (e.g., Levinson, Zeman, & Ohler, 2002). Over the past decade there have been several studies of Internet-delivered career interventions (Beutell & O’Hare, 2006; Bleier, 2007; Clark, 2001; Clark, Horan, Tompkins-Bjorkman, Kovalski, & Hackett, 2000; Gati, Kleiman, Saka, & Zakai, 2003; Hornyak, 2007; Jones, Harbach, Coker, & Staples, 2002; Kovalski & Horan, 1999; Robinson, Meyer, Prince, McLean, & Low, 2000; Severy, 2008; Šverko, Akik, Babarovic´, Bcˇina, & Šverko, 2002; Tompkins Bjorkman, 2003). On the whole, these studies suggest that Internet delivery of career development interventions is feasible, and that some participants benefit from participating in these interventions. The primary aim of the current study was to evaluate the efficacy of Career HOPES, a four-week Internet-based career development intervention created by the author. Career HOPES is a group counseling intervention that includes automated interactive lessons and self-assessments, homework assignments, and group discussions in private online forums. A secondary aim was to determine if participants benefited from having the online group discussions moderated by a career development professional. The question of what additional benefit, if any, accrues from professional moderation of online discussions is an important one because, if interventions such as Career HOPES do not require intensive professional moderation to achieve desired outcomes, then they will be significantly less costly to implement. The experimental design provided for two treatment conditions that were identical, except that in one condition the online discus-
340
S. Herman / Computers in Human Behavior 26 (2010) 339–344
sions were moderated by a career development professional (the author); in the other condition the discussions were unmoderated. In both treatment conditions participants worked through a series of four interactive online lessons, one per week. The third experimental condition was a minimal-intervention control condition in which participants had access to an annotated, selective Webliography of career resources and an unmoderated online discussion forum. It was hypothesized that participants in the two treatment conditions would show greater gains on measures of constructs related to career development than control condition participants, and that participants in the moderated-group treatment condition would show greater gains than those in the unmoderated-group treatment condition. Changes in seven outcome variables were measured by pre- and posttest administrations of nine self-report scales: career decidedness (two measures), frequency of career exploration behaviors emitted, occupationally relevant self-knowledge, vocational knowledge, career choice salience, satisfaction with current career situation (two measures), and satisfaction with future career prospects. 2. Methods Several different methods were used to recruit participants. Major search engine providers were contacted via their Websites and given the address of a Career HOPES recruitment Web page to add to their databases. Notices about the study were sent to several email lists for mental health professionals and educators. Personal email messages requesting referrals to Career HOPES were sent to several career professionals. The only selection criteria were that participants needed to be at least 21 years old and fluent in English. Over a three-week period, 64 participants were recruited. Of the 64 participants, 59% were women and 41% were men. Ages ranged from 21 to 53 (median = 33). The majority of participants were White (82%); the rest were Asian American (5%), Latino/Hispanic (2%), African American (3%), or Other (8%). Most were residents of the U.S. (86%); 13% were from Canada and one participant was from the United Kingdom. According to responses to the pretest surveys, 21% of participants reported not working at all during the current week; 28% worked part-time (less than 35 h) and the remaining 51% worked full-time (35 or more hours). The mean self-reported income when working full-time was $28,500 per year (SD $17,400). A range of occupations was represented, from waitress to civil engineer: 52% of the participants reported managerial or professional occupations; 33% reported technical, sales, or administrative support occupations; 8% reported service occupations; 2% reported precision production, craft, or repair occupations; and 6% said they were students. Educational attainment was high: 22% of participants reported having obtained a graduate degree, 38% had a bachelor’s degree, 36% had some college or an associate degree, 5% had completed high school, and none were high school dropouts. 2.1. Procedures To sign up for the study, participants completed an online informed consent procedure and the online pretests. The entire study and all communications with participants took place online. There was no compensation for participating in the study. Following the completion of the pretests, participants were randomized to one of three experimental groups: (a) moderated-group treatment, (b) unmoderated-group treatment, or (c) a minimal-intervention control group. The objectives of the moderated-group treatment condition intervention were to help participants to (a) clearly define their current career problem or problems; (b) adopt a rational strategy
for making career decisions; (c) become more aware of their own interests, skills, values, and personality traits; (d) become more familiar with online and offline sources of occupational information; (e) increase the frequency of vocational exploratory behaviors such as talking to friends and family about their career plans; (f) challenge problematic career beliefs; and (g) introduce participants to online job search information and resources. The four techniques used to accomplish these objectives were (a) providing annotated links to career resources elsewhere on the Web, (b) administering four interactive lessons, (c) assigning homework, and (d) promoting online group discussions. Each participant in the moderated-group treatment condition was randomly assigned to one of two 10-member virtual groups. Each participant had access to the private online discussion forum that was dedicated to his or her virtual group. The author served as moderator of the online discussions and posted a reply to every participant post within 24 h. No attempt was made to prescreen or edit participants’ messages. A mass emailing was done each week to inform participants that a new lesson was available. In addition, after the end of the first week, personal email messages were sent out to participants who had not yet signed into the site. These messages encouraged participants to sign into the Website and work through the online lesson. Participants received a series of four automated interactive psychoeducational lessons, administered one at a time over a period of four weeks. By making a new lesson available each week, rather than making all of the lessons available at the outset, it was hoped that the total amount of time invested in the intervention by participants would be increased. Except for the final week, participants also received homework assignments at the end of each weekly lesson. The homework assignments were designed to stimulate vocational exploration and to prompt reflection on career development issues raised in the online lessons and group discussions. One innovative technique used to integrate the weekly lessons with the discussion in the online forum was autoposting, the automatic creation and posting of messages based on participants’ interactions with the weekly lessons. Autoposting of messages to the group forums had two goals: First, autoposting structured the dialogue between the participant and the intervention, and between the participant and other participants, in directions likely to lead to fruitful learning experiences. Second, it reduced lurking by making all participants visible to the rest of the group. Here are brief descriptions of the contents of each week’s lesson: Week 1. The first online lesson served as an introduction to Career HOPES. The importance of career decision making was emphasized. Different ways of making poor career decisions were discussed, and a model for making good career decisions was presented. Participants were asked to complete an online checklist in order to identify some of the different types of career problems that they might be facing and were then asked to type in their thoughts about their current career situation. The concept of dysfunctional career beliefs (Krumboltz & Levin, 2004) was presented along with a checklist of potentially dysfunctional beliefs. Participants were asked to type in reflections on how they may be hindered by their own career beliefs. There was a semi-structured introductions exercise and an Ideal Job exercise. Comments entered in response to all of the questions in the lesson were automatically assembled into a message and posted to the forum. The homework for Week 1 was to begin to generate a list of occupational alternatives by asking friends and family what they thought the participant might be good at and enjoy doing. Participants were also asked to reflect on their career beliefs and to discuss them in the forum.
S. Herman / Computers in Human Behavior 26 (2010) 339–344
Week 2. The second lesson began with a review of the previous week’s homework assignment. Examples of ways to challenge dysfunctional career beliefs were presented. Holland’s RIASEC system for classifying vocational interests and occupations (Holland, 1997) was explained and an online version of the Self-Directed Search (SDS, Holland, 1994b) was administered. Participants received automated feedback consisting of (a) their three-letter Holland code from the SDS and (b) a list of matching occupations from the Occupations Finder (Holland, 1994a). For homework, participants were asked to research three appealing occupations identified as possible matches to their interests. Links to online sources of occupational information such as the Occupational Outlook Handbook (U.S. Bureau of Labor Statistics, 2009) were provided. Week 3. The homework from the previous week was reviewed. They were asked to follow a link to another Website in order to take a free online personality inventory, the Keirsey Temperament Survey (Keirsey & Bates, 1984). Work values and transferable skills were discussed and participants filled out a work values inventory that helped them to prioritize ten common work values. The homework for this week was to conduct a practice informational interview with a friend or family member. Week 4. The homework from the previous week was reviewed. This week consisted primarily of a summary of the prior weeks and a brief introduction to job search resources available on the Web. Participants were encouraged to exchange email addresses so that they could remain in contact with each other following the end of the intervention. Other resources. In addition to the forum and the weekly lessons, participants had access to a ‘‘Career Links” page, an annotated and selective Webliography of career resources. The author’s email address and telephone number were made available to participants in all three conditions in case they were experiencing difficulty using Career HOPES or had any other question or problem that they wished to discuss privately. The unmoderated-group treatment condition. The intervention in this condition was identical to that of the moderated-group treatment conditions, except that participants’ discussions in the online forums were unmoderated and no messages were posted to the forums by the author. There were two virtual groups of 10 members each in this condition. The minimal-intervention control condition. In this condition, participants had access to an unmoderated group discussion forum and to the Career Links page, but did not receive the four interactive lessons that were provided to participants in the two treatment conditions. The purpose of the control condition was to simulate resources already freely available elsewhere on the Web, such as chat rooms and pages of links to career resources. There were two virtual groups of 12 members each in this condition. 2.2. Measures 2.2.1. The Occupational Alternatives Question The Occupational Alternatives Question (OAQ, Zener & Schnuelle, 1976) is a two-question measure of career decision status. The first question asks the participant to ‘‘Please list ALL the occupations you are considering right now in the spaces provided below.” There are a total of 12 text entry fields in the online version of the OAQ developed for this study. The second question is ‘‘Which of the occupations listed above would be your first choice right now? (If you are undecided, or are not considering any occupations right now, please type ‘undecided’.)” A summary score is calculated as follows: If the respondent lists only one alternative and this alternative is also his or her first choice, the score is 1. If the respondent lists more than one alternative with one first choice, the score is 2. If the respondent lists one or more alternatives with no first choice, the score is 3. Finally, no alternatives or first choice is scored as a 4. The OAQ has been
341
used in research and practice as a quick measure of career decidedness, for a comprehensive review of the OAQ see Slaney (1988). 2.2.2. Career Exploratory Behaviors Scale The Career Exploratory Behaviors Scale (CEBS) is a simple selfreport measure of career exploratory behaviors created for this study and based on the work of Krumboltz and Thoresen (1964). The respondent answers six questions about his or her behavior during the past four weeks. All of the questions begin ‘‘During the past four weeks . . .” and then ask about some specific behavior, for example ‘‘. . . how many times have you talked to a family member about your career plans?” The questions on the CEBS refer to behaviors that are either assigned as homework or directly encouraged by the Career HOPES program. 2.2.3. The Career Decision Profile The Career Decision Profile (CDP, Jones, 1989) is a 16-item measure of career decision status. It contains six subscales that assess three dimensions of career decision status: career decidedness, comfort with current decisional status, and reasons for indecision (four subscales). The CDP uses a modified Likert format, with eight possible responses to each statement. The possible responses are anchored only at the end-points with Strongly Agree and Strongly Disagree. The CDP has been subjected to rigorous validation studies and has evolved into a reliable and valid measure of career decision status. The test–retest reliability of each of the six scales of the CDP ranges from .66 to .80, and the internal consistency of the six scales as measured by Cronbach’s alpha ranges from .73 to .85 (Jones, 1989). 2.2.4. Satisfaction with current and future career situation Two face-valid items, ‘‘How do you feel about your current career situation?” and ‘‘How do you feel about your future career prospects?” were used to measure participants’ satisfaction with their current and future career situation. Responses were coded as integers from 1 (Terrible) to 7 (Delighted). 3. Results Participants in the two treatment conditions spent an average of 4.9 h each on the Career HOPES Website, whereas control condition participants spent an average of only 38 min each. Only 20% of the 40 participants randomized to the two treatment conditions completed all four of the weekly lessons, 33% completed at least three lessons, 55% completed at least two lessons, and 80% completed at least one lesson. The posttests were completed by 41 of the 64 participants (62% of the treatment condition participants and 67% of the control condition participants). A total of 68 private email messages were received and responded to by the author, mostly concerning technical difficulties. 3.1. Outcomes Univariate ANOVAs using planned, one-sided, orthogonal comparisons were performed to test for differences in gain scores (a) between participants in either one of the two treatment conditions and control condition participants and (b) between moderated and unmoderated treatment condition participants. Effect sizes (Cohen’s d) were calculated using the pooled pretest standard deviations across all three conditions as the denominators and the differences in gain scores as the numerators. The statistical analyses of the outcome data reported here include only participants who completed both the pretest and the posttest measures. Means and standard deviations for each outcome measure, experimental condition, and time (pre- and posttest) are displayed in Table 1.
342
S. Herman / Computers in Human Behavior 26 (2010) 339–344
Table 1 Means and standard deviations by experimental condition and time. Control n Occupational Alternatives Question Career Decision Profile Scale 1: Career decidedness Scale 2: Career comfort Scale 3: Lack of self-clarity Scale 4: Lack of knowledge about occupations and training Scale 5: Indecisiveness Scale 6: Lack of career Choice Urgency Satisfaction with current career situation Satisfaction with future career prospects Career Exploratory Behaviors Scale Raw frequencies Mean ranks of frequencies
17 16 16 16 16 16 16
Unmoderated
Pretest M (SD) 2.2 (0.5) 11.9 10.1 14.4 14.8 9.1 6.7
(3.7) (4.2) (7.3) (4.5) (5.1) (3.5)
Posttest M (SD) 2.2 (0.5) 11.8 10.5 12.3 10.3 8.3 8.0
n 13
(4.4) (4.4) (7.5) (4.9) (6.0) (3.5)
12 12 12 12 12 12
Pretest M (SD) 2.4 (0.5) 11.6 8.1 16.9 14.3 11.4 8.3
(3.9) (4.3) (6.0) (6.5) (6.5) (6.5)
Moderated Posttest M (SD) 2.1 (0.5) 12.6 9.0 10.3 9.8 8.4 7.3
n 13
(3.9) (5.1) (7.0) (5.1) (4.1) (2.9)
13 13 13 13 13 13
Pretest M (SD) 2.5 (0.7) 10.2 8.8 18.1 13.8 9.9 7.8
(4.4) (4.5) (5.3) (6.4) (5.2) (3.3)
Posttest M (SD) 2.2 (0.6) 11.9 9.8 12.8 11.3 12.5 9.4
(3.9) (3.8) (5.4) (5.7) (6.2) (3.2)
16 16
3.3 (1.6) 4.4 (1.4)
4.2 (1.5) 5.1 (1.2)
13 13
3.6 (1.2) 3.8 (1.6)
3.9 (1.5) 3.8 (1.5)
13 13
2.8 (1.8) 3.6 (1.7)
3.8 (1.5) 4.7 (1.5)
17 17
24.8 (20.9) 22.8 (6.9)
28.2 (36.8) 20.7 (6.2)
13 13
29.7 (33.2) 22.5 (6.7)
22.3 (17.6) 20.9 (6.1)
13 13
18.6 (17.3) 20.5 (7.0)
28.4 (19.5) 24.8 (7.3)
Note. For the Career Exploratory Behavior Scale, this table shows means and standard deviations for the untransformed sums of the six items in the scale. The transformed data based on the ranks is shown as the mean ranks of frequencies with all 43 participants ranked together at pre- and posttests.
An alpha level of .05 was used to control the Type I error rate in all statistical tests. Some statisticians argue that when more than one comparison is performed, then the alpha level that is used to control the per-comparison Type I error rate should be made more stringent in order to control the experiment-wise Type I error rate, regardless of whether or not the comparisons are planned or unplanned, orthogonal or nonorthogonal. However, adjusting the per-comparison Type I error rate can result in insufficient power to detect even large effects given the small to medium sample sizes employed in most studies. As Keppel (1991, p. 165) notes, the most widely used strategy among researchers is to evaluate each comparison at an unadjusted per-comparison alpha level. Although Keppel (1991, pp. 178–179) provides a strong rationale for not adjusting per-comparison alpha levels, which is the approach taken in the current study, cautious researchers should regard the findings presented here as tentative. Treatment condition participants showed greater gains than controls on career decidedness as measured by the Occupational Alternatives Question (OAQ), d = .54, F(1, 40) = 2.88, p < .05. On the Decidedness scale of the Career Decision Profile (CDP) there was an observed effect size, d, of .37, which was consistent with the results from the OAQ, although not statistically significant at the .05 level, F(1, 38) = 1.99, p = .08. Treatment condition participants reported greater gains than control participants in self-knowledge as measured by Scale 3 of the CDP, d = .58, F(1, 38) = 4.35, p = .02. Treatment–control condition comparisons were not statistically significant for gains in (a) vocational knowledge, (b) career choice salience, (c) satisfaction with one’s current career situation, or (d) satisfaction with one’s future career prospects. The impact of the intervention on career exploration behaviors as measured by the CEBS was somewhat difficult to ascertain due to the nonnormality of the observed distribution, which had both heavy tails and skewness. Summing the six items of the CEBS pretest for the 43 participants who completed both the pretests and the posttests yields a CEBS total pretest score. The distribution of these 43 pretest scores has a high coefficient of skewness, g1 = 2.7 and significant kurtosis, g2 = 8.8. Results for the CEBS posttest scores are similar. The distribution of raw gain scores, the CEBS total posttest score minus the CEBS total pretest score, shows less skewness, g1 = .7, but significant kurtosis, g2 = 6.5. An inspection of the data did not reveal the presence of any obvious data entry errors or random response patterns that could account for the nonnormality of the score distributions. For example, one participant reported sending out 100 career-related email messages or letters during the past
month at pretest. Although this figure was high, it was certainly not outside the realm of possibility. Because of these problems, nonparametric methods were used to analyze these data. Data from the CEBS were transformed and analyzed as follows: The pretest scores for each of the six individual items making up the scale were ranked separately. All 43 participants who completed both the pre- and posttests were ranked together on each item. Assigned ranks for each item ranged from 1 to 43, with tied ranks assigned the mean rank of the ties. For each participant, a mean pretest rank was calculated by taking the mean of his or her ranks on the six pretest items. The same procedure was used to calculate mean posttest ranks for each participant. For each participant, a rank gain score was calculated by subtracting the mean pretest rank from the mean posttest rank. Univariate ANOVAs were then conducted on the transformed gain scores. Performing an ANOVA on ranked data with more than two levels of the independent variable is equivalent to the nonparametric Kruskal–Wallis ksample test. Based on this nonparametric analysis, participants in the two treatment conditions reported greater gains in career exploration behaviors than did control condition participants, d = .50, F(1, 40) = 3.27, p = .04. Participants in the moderated treatment condition reported greater gains in satisfaction with their future career prospects, d = .75, F(1, 38) = 3.58, p = .03. There was also a moderate observed superiority in satisfaction with current career situation for moderated treatment condition participants, d = .43, although this effect did not achieve significance at the .05 level, F(1, 38) = 1.56, p = .11. Participants in the moderated treatment condition reported greater gains in career exploration behaviors, d = .86, F(1, 40) = 6.06, p < .01, than participants in the unmoderated treatment condition. No statistically significant differences between moderated and unmoderated condition participants were found in gains on the measures of career decidedness, self-knowledge, vocational knowledge, career choice salience, or satisfaction and comfort with current career situation.
4. Discussion The Career HOPES recruitment Web page emphasized that Career HOPES might be especially helpful to people who were (a) thinking about changing jobs, (b) wondering if their current job was the right one for them, or (c) unemployed or just out of school and not sure what to do next. The implication was that Career
S. Herman / Computers in Human Behavior 26 (2010) 339–344
HOPES would be especially helpful to people who were uncertain about their current career direction. Indeed, one of the most salient positive outcomes for participants in interventions such as Career HOPES is coming to a decision about what occupational alternatives are worth pursuing further. The results of the study indicate that the intervention had a positive impact on career decidedness, helping participants to become more decided about their future career directions. It is an axiom of career counseling that optimal career decisions are based on high quality information – information about one’s own interests, abilities, values, and personality as well as information about the requirements and features of occupational alternatives. Results indicate that the intervention had a positive impact on two indices of information quality: occupationally relevant self-knowledge and the frequency of career exploration behaviors. A secondary purpose of the study was to determine what additional benefit accrues from having a professional group moderator involved in the online discussions. As compared with participants in the unmoderated treatment condition, participants in the moderated treatment condition showed greater gains in satisfaction with future career prospects and in the emission of vocational exploration behaviors. Participants in the moderated treatment condition were more satisfied overall with the intervention. Among participants who completed the posttests, 92% of the moderated treatment participants and 55% of the unmoderated treatment condition subjects rated the intervention as Useful or Very Useful on a 5-point scale. The difference between the two conditions in overall satisfaction was statistically significant, d = .57, F(1, 35) = 3.94, p = .03. Although a positive impact on participants’ career development was observed on some important indices, gains were not observed across the board. In all, 18 statistical tests were performed on nine outcome measures (nine tests for treatment vs. control and nine for moderated vs. unmoderated). The null hypothesis was rejected in only 5 of these 18 tests. One possible partial explanation for these results is that the control condition participants showed unexpected and relatively large gains on some outcome measures. This may have been due to an initial underestimation of the positive impact of the minimal-intervention control condition, which was intended to function as an attention placebo. Spokane (1991, p. 40) notes that dropout rates in group-based career interventions typically exceed 50%. The Career HOPES intervention was not an exception to this rule. There were a total of 40 participants in the two treatment conditions. Of those 40 participants, 32 completed the first lesson, 22 completed the second, 13 completed the third, and 8 completed the fourth. In other words, by the strictest possible criterion, completion of all four online lessons, 80% of the treatment condition participants would be classified as treatment dropouts. The posttest surveys suggest that some participants may have dropped out of the study because of the length or lack of appeal of the weekly lessons. On the posttest intervention evaluation questionnaire, 60% of the respondents opined that the weekly lessons were too long, and none thought the lessons were too short. Lesson difficulty did not appear to be a problem; 83% of unmoderated condition participants and 100% of moderated condition participants rated the difficulty level as ‘‘about right.” The observed impact of the intervention compares favorably with that of other career development interventions on certain outcomes. For example, on the two measures of career decidedness, Career HOPES showed a mean effect size of d = .46. In the Whiston et al. (1998) meta-analysis, there were 50 treatment–control condition comparisons on measures of career decidedness. The mean weighted effect size in these 50 studies on career decidedness was d = .19. The weighted mean effect size for all treatment–control comparisons in Whiston et al. was d = .30. The mean effect size across the nine outcome measures in the current study was d = .19.
343
How efficient was Career HOPES? Career HOPES treatment condition participants spent an average of 4.9 h online and the mean effect size across all measures was d = .19. Efficiency can be quantified as mean effect size change per hour of treatment. For Career HOPES, this was .04 per hour. Whiston et al. (1998) calculated the efficiency of the career interventions in their meta-analysis. Based on their calculations, the efficiency of Career HOPES is comparable to that of career classes (.08/h), group counseling (.06/h), and career workshops (.05/h). Career HOPES was significantly less efficient than individual counseling (.92/h) and computer-assisted career guidance interventions (.23/h). In the current study the author spent 20 h 58 min online responding to messages posted by the 20 participants in the moderatedgroup condition forums. This amounts to an average of about 1 h per participant. If all of these participants had completed all four online lessons, the moderator would have had to spend an estimated 2 h online per participant. The relatively small advantage of the moderated-group treatment as compared with the unmoderatedgroup treatment on the outcome measures suggest that this may not be a cost-effective use of a counselor’s time. On the other hand, the amount of moderator time invested in this first trial of Career HOPES may overestimate the amount of time that would be required to achieve similar results in a more refined version of Career HOPES. One possible disadvantage to interventions such as Career HOPES that rely on automated instruction is that they are inherently less flexible than interventions in which a human being responds directly to an individual’s concerns. Krumboltz (1992) has argued that the goals of counseling should be tailored to the individual. He stressed, for example, that changes in the direction of greater career decidedness are not desirable for all counseling clients, and that there can be ‘‘wisdom in indecision.” These points are well taken and they highlight an important feature of psychoeducational interventions of this type, namely, that the success of such interventions is likely to depend on the homogeneity of the target population’s goals or problems. In focused, group-based interventions such as Career HOPES, some degree of individual tailoring is possible through the use of interactive programmed instruction and the open-ended discussions of individual issues in the online forums. However, such interventions are probably most likely to succeed when clients are carefully selected or self-selected to fit the intervention. To achieve accurate self-selection, the recruitment materials must clearly spell out who is likely to benefit. 4.1. Recommendations A number of recommendations for future studies and for the future development of the Career HOPES project follow from the current analysis. First, the interactive psychoeducational component of Career HOPES needs to be expanded and improved. The current version of Career HOPES does not appear to be as efficient as offline computer-assisted career guidance interventions. The performance of computer-assisted interventions should be considered the minimal standard that Internet-delivered career interventions must achieve. More than half of the participants who completed the posttests thought that the weekly lessons were too long and none thought they were too short. The length of the lessons may have contributed to the high noncompletion rate; therefore, the length of each lesson should be shortened. It may be possible to improve the efficacy of Career HOPES and other similar interventions by shortening the length of each lesson and increasing their number. Increasing the number of lessons would increase the number of weeks, or time span, of the intervention, which may result in improved treatment efficacy due to an increase in the period of sustained attention and involvement by participants. Some of the content of the Career HOPES lessons should be rethought. People do not like read-
344
S. Herman / Computers in Human Behavior 26 (2010) 339–344
ing a lot of text online. Reducing the amount of text that participants need to read and increasing interactivity will likely increase the appeal of the intervention and should be a primary goal of future efforts. A fruitful area for future research and development will be creating and testing new formats for professional involvement in Internet-delivered psychoeducational interventions such as Career HOPES in order to better leverage the investment of time by professional moderators. Some possibilities that should be explored in future studies include creating lists of Frequently Asked Questions (FAQs). Perhaps weekly live group chat sessions would be a more productive use of a professional moderator’s time than reading and responding to each message posted by participants. Or a moderator could be ‘‘on call” – responding only to specific questions and requests for assistance. It may be possible to make use of ‘‘boilerplate” responses to common questions. Finding ways to reduce the need for intensive professional oversight is important because, otherwise, Internet counseling interventions are going to be almost as expensive to deliver as in-person counseling. Increasing participant commitment to, and investment in, online interventions is another area in which improvements need to be made. For example, adding live video or audio chat or posting pictures and personal profiles of participants may facilitate an increased sense of personal relationship and commitment. In future studies of this type, no-treatment or wait list control conditions should be employed. Data from the current study suggest that the control condition intervention may have been more powerful than assumed at the outset. Future studies should also compare online interventions to in-person interventions with similar content whenever possible. 5. Conclusion Debates over the pros and cons of online counseling have usually focused on differences between individual face-to-face counseling and counseling via email or chat (e.g., Mallen et al., 2005). The main advantage of conducting individual counseling sessions online is that the Internet makes it possible to provide needed counseling services to individuals who might not otherwise have access. However, it is not likely that individual online counseling will prove to be significantly more effective, or cost-effective, than face-to-face counseling. Semi-automated, group-based psychoeducational interventions such as Career HOPES, on the other hand, hold out the promise of improving the effectiveness and cost-effectiveness of psychosocial interventions as well as increasing access to needed counseling services. Acknowledgements Author thanks his dissertation chair, mentor, and friend, John Krumboltz, for his many invaluable contributions to the Career HOPES project. Thanks also to Karen Slaton, who helped with the design and implementation of the Career HOPES Website. References Beutell, N. J., & O’Hare, M. M. (2006). Career pathfinders: A qualitative study of career development. Psychological Reports, 98, 517–528.
Bleier, J. K. (2007). The impact of career counseling plus DISCOVER (Internet version) on the academic achievement of high school sophomores at risk for dropping out of school. Dissertation Abstracts International: Section B: The Sciences and Engineering, 68. Clark, G. (2001). Experimental evaluation of an interactive internet career guidance program for parents of gifted girls. Dissertation Abstracts International: Section B: The Sciences and Engineering, 61, 5555. Clark, G., Horan, J. J., Tompkins-Bjorkman, A., Kovalski, T., & Hackett, G. (2000). Interactive career counseling on the internet. Journal of Career Assessment, 8, 85–93. Gati, I., Kleiman, T., Saka, N., & Zakai, A. (2003). Perceived benefits of using an Internet-based interactive career planning system. Journal of Vocational Behavior, 62, 272–286. Holland, J. L. (1994a). The occupations finder (Form R). Odessa, FL: Psychological Assessment Resources. Holland, J. L. (1994b). The self-directed search (Form R). Odessa, FL: Psychological Assessment Resources. Holland, J. L. (1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Odessa, FL, US: Psychological Assessment Resources. Hornyak, D. A. (2007). Utilizing cognitive information processing theory to assess the effectiveness of DISCOVER on college students’ career development. Dissertation Abstracts International: Section A: Humanities and Social Sciences, 68, 2319. Jones, L. K. (1989). Measuring a three-dimensional construct of career indecision among college students: A revision of the Vocational Decision Scale: The Career Decision Profile. Journal of Counseling Psychology, 36, 477–486. Jones, W. P., Harbach, R. L., Coker, J. K., & Staples, P. A. (2002). Web-assisted vocational test interpretation. Journal of Employment Counseling, 39, 127–137. Keirsey, D., & Bates, M. (1984). Please understand me: Character and temperament types. Del Mar, CA: Prometheus Nemesis Book Co. Keppel, G. (1991). Design and analysis: A researcher’s handbook (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. Kovalski, T. M., & Horan, J. J. (1999). The effects of Internet-based cognitive restructuring on the irrational career beliefs of adolescent girls. Journal of Cognitive Psychotherapy, 13, 145–152. Krumboltz, J. D. (1992). The wisdom of indecision. Journal of Vocational Behavior, 41, 239–244. Krumboltz, J. D., & Levin, A. S. (2004). Luck is no accident: Making the most of happenstance in your life and career. Atascadero, CA: Impact Publishers. Krumboltz, J. D., & Thoresen, C. E. (1964). The effect of behavioral counseling in group and individual settings on information-seeking behavior. Journal of Counseling Psychology, 11, 324–335. Levinson, E. M., Zeman, H. L., & Ohler, D. L. (2002). A critical evaluation of the Webbased version of the Career Key. The Career Development Quarterly, 51, 26–35. Mallen, M. J., Vogel, D. L., Rochlen, A. B., & Day, S. X. (2005). Online counseling: Reviewing the literature from a counseling psychology framework. Counseling Psychologist, 33, 819–871. Oliver, L. W., & Spokane, A. R. (1988). Career-intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447–462. Robinson, N. K., Meyer, D., Prince, J. P., McLean, C., & Low, R. (2000). Mining the internet for career information: A model approach for college students. Journal of Career Assessment, 8, 37–54. Severy, L. E. (2008). Analysis of an online career narrative intervention: ‘‘What’s my story?”. The Career Development Quarterly, 56, 268–273. Slaney, R. B. (1988). The assessment of career decision making. In W. B. Walsh & S. H. Osipow (Eds.), Career decision making. Hillsdale, NJ: Erlbaum. Spokane, A. R. (1991). Career intervention (p. 353). Englewood Cliffs, NJ, US: Prentice Hall, Inc. Šverko, B., Akik, N., Babarovic´, T., Bcˇina, A., & Šverko, I. (2002). Validity of e-advice: The evaluation of an Internet-based system for career planning. International Journal for Educational and Vocational Guidance, 2, 193–215. Tompkins Bjorkman, A. B. (2003). Internet-based attributional retraining and career decision making. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63, 5540. U.S. Bureau of Labor Statistics (2009). Occupational Outlook Handbook, 2008–09 edition. Washington, DC: U.S. Bureau of Labor Statistics.
Contents lists available at ScienceDirect
Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh
Career HOPES: An Internet-delivered career development intervention Steve Herman *,1 Stanford University, Stanford, CA 94305, United States
a r t i c l e
i n f o
Article history: Available online 9 December 2009 Keywords: Career development Intervention Internet Computer applications
a b s t r a c t Career HOPES is an Internet-delivered group counseling intervention designed to facilitate occupational exploration and career decision making. The intervention includes automated interactive lessons and self-assessments, homework assignments, and group discussions in private online forums. A randomized, controlled experiment (N = 64) was conducted to evaluate (a) the efficacy of Career HOPES and (b) the contribution of professional moderation of the online group discussions to outcomes. Participants in two treatment conditions showed greater gains than control condition participants in career decidedness as measured by the Occupational Alternatives Questionnaire (d = .54), occupationally relevant selfknowledge (d = .58), and emission of career exploration behaviors (d = .50). In one of the treatment conditions, the online group discussions were moderated by a psychologist with career counseling experience; in the other treatment condition, the discussions were unmoderated. Professional moderation resulted in better outcomes on several variables and greater overall satisfaction with the intervention. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Meta-analyses of empirical studies of career interventions (Oliver & Spokane, 1988; Whiston, Sexton, & Lasoff, 1998) have demonstrated that, on the whole, these interventions have a beneficial impact on participants’ career development. The weighted mean effect size for treatment–control comparisons in the 47 career interventions included in the Whiston et al. meta-analysis was d = .30 (k = 268). Participants in the included interventions showed greater gains than control condition participants on measures of (a) decidedness about career choices (d = .19, k = 50), (b) career maturity (d = .53, k = 59), (c) the frequency of career information-seeking behaviors (d = .31, k = 18), (d) vocational knowledge (d = .97, k = 2), and (e) self-knowledge (d = 1.30 , k = 1). Internet delivery can help overcome geographical, psychological, physical, and financial obstacles to accessing psychosocial interventions and counseling services (Mallen, Vogel, Rochlen, & Day, 2005). Thus, the development of an effective, disseminable Internet-based career development intervention is a desirable goal. There are currently many Websites that offer career inventories, career counseling, and other types of career and job search assistance, either for free (e.g., http://monster.com) or for a fee (e.g., http://www.career planner.com/). However, the efficacy of the services offered by most
* Present address: University of Hawaii at Hilo, Psychology Department, 200 West Kawili Street, Hilo, HI 96720, United States. Tel.: +1 808 933 3284; fax: +1 206 309 0486. E-mail address: [email protected] 1 This research was based on a doctoral dissertation. 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.11.003
of these Websites has not been empirically demonstrated (Mallen et al., 2005). One noteworthy exception is the Career Key inventory at http://careerkey.org, which has been examined in a number of studies (e.g., Levinson, Zeman, & Ohler, 2002). Over the past decade there have been several studies of Internet-delivered career interventions (Beutell & O’Hare, 2006; Bleier, 2007; Clark, 2001; Clark, Horan, Tompkins-Bjorkman, Kovalski, & Hackett, 2000; Gati, Kleiman, Saka, & Zakai, 2003; Hornyak, 2007; Jones, Harbach, Coker, & Staples, 2002; Kovalski & Horan, 1999; Robinson, Meyer, Prince, McLean, & Low, 2000; Severy, 2008; Šverko, Akik, Babarovic´, Bcˇina, & Šverko, 2002; Tompkins Bjorkman, 2003). On the whole, these studies suggest that Internet delivery of career development interventions is feasible, and that some participants benefit from participating in these interventions. The primary aim of the current study was to evaluate the efficacy of Career HOPES, a four-week Internet-based career development intervention created by the author. Career HOPES is a group counseling intervention that includes automated interactive lessons and self-assessments, homework assignments, and group discussions in private online forums. A secondary aim was to determine if participants benefited from having the online group discussions moderated by a career development professional. The question of what additional benefit, if any, accrues from professional moderation of online discussions is an important one because, if interventions such as Career HOPES do not require intensive professional moderation to achieve desired outcomes, then they will be significantly less costly to implement. The experimental design provided for two treatment conditions that were identical, except that in one condition the online discus-
340
S. Herman / Computers in Human Behavior 26 (2010) 339–344
sions were moderated by a career development professional (the author); in the other condition the discussions were unmoderated. In both treatment conditions participants worked through a series of four interactive online lessons, one per week. The third experimental condition was a minimal-intervention control condition in which participants had access to an annotated, selective Webliography of career resources and an unmoderated online discussion forum. It was hypothesized that participants in the two treatment conditions would show greater gains on measures of constructs related to career development than control condition participants, and that participants in the moderated-group treatment condition would show greater gains than those in the unmoderated-group treatment condition. Changes in seven outcome variables were measured by pre- and posttest administrations of nine self-report scales: career decidedness (two measures), frequency of career exploration behaviors emitted, occupationally relevant self-knowledge, vocational knowledge, career choice salience, satisfaction with current career situation (two measures), and satisfaction with future career prospects. 2. Methods Several different methods were used to recruit participants. Major search engine providers were contacted via their Websites and given the address of a Career HOPES recruitment Web page to add to their databases. Notices about the study were sent to several email lists for mental health professionals and educators. Personal email messages requesting referrals to Career HOPES were sent to several career professionals. The only selection criteria were that participants needed to be at least 21 years old and fluent in English. Over a three-week period, 64 participants were recruited. Of the 64 participants, 59% were women and 41% were men. Ages ranged from 21 to 53 (median = 33). The majority of participants were White (82%); the rest were Asian American (5%), Latino/Hispanic (2%), African American (3%), or Other (8%). Most were residents of the U.S. (86%); 13% were from Canada and one participant was from the United Kingdom. According to responses to the pretest surveys, 21% of participants reported not working at all during the current week; 28% worked part-time (less than 35 h) and the remaining 51% worked full-time (35 or more hours). The mean self-reported income when working full-time was $28,500 per year (SD $17,400). A range of occupations was represented, from waitress to civil engineer: 52% of the participants reported managerial or professional occupations; 33% reported technical, sales, or administrative support occupations; 8% reported service occupations; 2% reported precision production, craft, or repair occupations; and 6% said they were students. Educational attainment was high: 22% of participants reported having obtained a graduate degree, 38% had a bachelor’s degree, 36% had some college or an associate degree, 5% had completed high school, and none were high school dropouts. 2.1. Procedures To sign up for the study, participants completed an online informed consent procedure and the online pretests. The entire study and all communications with participants took place online. There was no compensation for participating in the study. Following the completion of the pretests, participants were randomized to one of three experimental groups: (a) moderated-group treatment, (b) unmoderated-group treatment, or (c) a minimal-intervention control group. The objectives of the moderated-group treatment condition intervention were to help participants to (a) clearly define their current career problem or problems; (b) adopt a rational strategy
for making career decisions; (c) become more aware of their own interests, skills, values, and personality traits; (d) become more familiar with online and offline sources of occupational information; (e) increase the frequency of vocational exploratory behaviors such as talking to friends and family about their career plans; (f) challenge problematic career beliefs; and (g) introduce participants to online job search information and resources. The four techniques used to accomplish these objectives were (a) providing annotated links to career resources elsewhere on the Web, (b) administering four interactive lessons, (c) assigning homework, and (d) promoting online group discussions. Each participant in the moderated-group treatment condition was randomly assigned to one of two 10-member virtual groups. Each participant had access to the private online discussion forum that was dedicated to his or her virtual group. The author served as moderator of the online discussions and posted a reply to every participant post within 24 h. No attempt was made to prescreen or edit participants’ messages. A mass emailing was done each week to inform participants that a new lesson was available. In addition, after the end of the first week, personal email messages were sent out to participants who had not yet signed into the site. These messages encouraged participants to sign into the Website and work through the online lesson. Participants received a series of four automated interactive psychoeducational lessons, administered one at a time over a period of four weeks. By making a new lesson available each week, rather than making all of the lessons available at the outset, it was hoped that the total amount of time invested in the intervention by participants would be increased. Except for the final week, participants also received homework assignments at the end of each weekly lesson. The homework assignments were designed to stimulate vocational exploration and to prompt reflection on career development issues raised in the online lessons and group discussions. One innovative technique used to integrate the weekly lessons with the discussion in the online forum was autoposting, the automatic creation and posting of messages based on participants’ interactions with the weekly lessons. Autoposting of messages to the group forums had two goals: First, autoposting structured the dialogue between the participant and the intervention, and between the participant and other participants, in directions likely to lead to fruitful learning experiences. Second, it reduced lurking by making all participants visible to the rest of the group. Here are brief descriptions of the contents of each week’s lesson: Week 1. The first online lesson served as an introduction to Career HOPES. The importance of career decision making was emphasized. Different ways of making poor career decisions were discussed, and a model for making good career decisions was presented. Participants were asked to complete an online checklist in order to identify some of the different types of career problems that they might be facing and were then asked to type in their thoughts about their current career situation. The concept of dysfunctional career beliefs (Krumboltz & Levin, 2004) was presented along with a checklist of potentially dysfunctional beliefs. Participants were asked to type in reflections on how they may be hindered by their own career beliefs. There was a semi-structured introductions exercise and an Ideal Job exercise. Comments entered in response to all of the questions in the lesson were automatically assembled into a message and posted to the forum. The homework for Week 1 was to begin to generate a list of occupational alternatives by asking friends and family what they thought the participant might be good at and enjoy doing. Participants were also asked to reflect on their career beliefs and to discuss them in the forum.
S. Herman / Computers in Human Behavior 26 (2010) 339–344
Week 2. The second lesson began with a review of the previous week’s homework assignment. Examples of ways to challenge dysfunctional career beliefs were presented. Holland’s RIASEC system for classifying vocational interests and occupations (Holland, 1997) was explained and an online version of the Self-Directed Search (SDS, Holland, 1994b) was administered. Participants received automated feedback consisting of (a) their three-letter Holland code from the SDS and (b) a list of matching occupations from the Occupations Finder (Holland, 1994a). For homework, participants were asked to research three appealing occupations identified as possible matches to their interests. Links to online sources of occupational information such as the Occupational Outlook Handbook (U.S. Bureau of Labor Statistics, 2009) were provided. Week 3. The homework from the previous week was reviewed. They were asked to follow a link to another Website in order to take a free online personality inventory, the Keirsey Temperament Survey (Keirsey & Bates, 1984). Work values and transferable skills were discussed and participants filled out a work values inventory that helped them to prioritize ten common work values. The homework for this week was to conduct a practice informational interview with a friend or family member. Week 4. The homework from the previous week was reviewed. This week consisted primarily of a summary of the prior weeks and a brief introduction to job search resources available on the Web. Participants were encouraged to exchange email addresses so that they could remain in contact with each other following the end of the intervention. Other resources. In addition to the forum and the weekly lessons, participants had access to a ‘‘Career Links” page, an annotated and selective Webliography of career resources. The author’s email address and telephone number were made available to participants in all three conditions in case they were experiencing difficulty using Career HOPES or had any other question or problem that they wished to discuss privately. The unmoderated-group treatment condition. The intervention in this condition was identical to that of the moderated-group treatment conditions, except that participants’ discussions in the online forums were unmoderated and no messages were posted to the forums by the author. There were two virtual groups of 10 members each in this condition. The minimal-intervention control condition. In this condition, participants had access to an unmoderated group discussion forum and to the Career Links page, but did not receive the four interactive lessons that were provided to participants in the two treatment conditions. The purpose of the control condition was to simulate resources already freely available elsewhere on the Web, such as chat rooms and pages of links to career resources. There were two virtual groups of 12 members each in this condition. 2.2. Measures 2.2.1. The Occupational Alternatives Question The Occupational Alternatives Question (OAQ, Zener & Schnuelle, 1976) is a two-question measure of career decision status. The first question asks the participant to ‘‘Please list ALL the occupations you are considering right now in the spaces provided below.” There are a total of 12 text entry fields in the online version of the OAQ developed for this study. The second question is ‘‘Which of the occupations listed above would be your first choice right now? (If you are undecided, or are not considering any occupations right now, please type ‘undecided’.)” A summary score is calculated as follows: If the respondent lists only one alternative and this alternative is also his or her first choice, the score is 1. If the respondent lists more than one alternative with one first choice, the score is 2. If the respondent lists one or more alternatives with no first choice, the score is 3. Finally, no alternatives or first choice is scored as a 4. The OAQ has been
341
used in research and practice as a quick measure of career decidedness, for a comprehensive review of the OAQ see Slaney (1988). 2.2.2. Career Exploratory Behaviors Scale The Career Exploratory Behaviors Scale (CEBS) is a simple selfreport measure of career exploratory behaviors created for this study and based on the work of Krumboltz and Thoresen (1964). The respondent answers six questions about his or her behavior during the past four weeks. All of the questions begin ‘‘During the past four weeks . . .” and then ask about some specific behavior, for example ‘‘. . . how many times have you talked to a family member about your career plans?” The questions on the CEBS refer to behaviors that are either assigned as homework or directly encouraged by the Career HOPES program. 2.2.3. The Career Decision Profile The Career Decision Profile (CDP, Jones, 1989) is a 16-item measure of career decision status. It contains six subscales that assess three dimensions of career decision status: career decidedness, comfort with current decisional status, and reasons for indecision (four subscales). The CDP uses a modified Likert format, with eight possible responses to each statement. The possible responses are anchored only at the end-points with Strongly Agree and Strongly Disagree. The CDP has been subjected to rigorous validation studies and has evolved into a reliable and valid measure of career decision status. The test–retest reliability of each of the six scales of the CDP ranges from .66 to .80, and the internal consistency of the six scales as measured by Cronbach’s alpha ranges from .73 to .85 (Jones, 1989). 2.2.4. Satisfaction with current and future career situation Two face-valid items, ‘‘How do you feel about your current career situation?” and ‘‘How do you feel about your future career prospects?” were used to measure participants’ satisfaction with their current and future career situation. Responses were coded as integers from 1 (Terrible) to 7 (Delighted). 3. Results Participants in the two treatment conditions spent an average of 4.9 h each on the Career HOPES Website, whereas control condition participants spent an average of only 38 min each. Only 20% of the 40 participants randomized to the two treatment conditions completed all four of the weekly lessons, 33% completed at least three lessons, 55% completed at least two lessons, and 80% completed at least one lesson. The posttests were completed by 41 of the 64 participants (62% of the treatment condition participants and 67% of the control condition participants). A total of 68 private email messages were received and responded to by the author, mostly concerning technical difficulties. 3.1. Outcomes Univariate ANOVAs using planned, one-sided, orthogonal comparisons were performed to test for differences in gain scores (a) between participants in either one of the two treatment conditions and control condition participants and (b) between moderated and unmoderated treatment condition participants. Effect sizes (Cohen’s d) were calculated using the pooled pretest standard deviations across all three conditions as the denominators and the differences in gain scores as the numerators. The statistical analyses of the outcome data reported here include only participants who completed both the pretest and the posttest measures. Means and standard deviations for each outcome measure, experimental condition, and time (pre- and posttest) are displayed in Table 1.
342
S. Herman / Computers in Human Behavior 26 (2010) 339–344
Table 1 Means and standard deviations by experimental condition and time. Control n Occupational Alternatives Question Career Decision Profile Scale 1: Career decidedness Scale 2: Career comfort Scale 3: Lack of self-clarity Scale 4: Lack of knowledge about occupations and training Scale 5: Indecisiveness Scale 6: Lack of career Choice Urgency Satisfaction with current career situation Satisfaction with future career prospects Career Exploratory Behaviors Scale Raw frequencies Mean ranks of frequencies
17 16 16 16 16 16 16
Unmoderated
Pretest M (SD) 2.2 (0.5) 11.9 10.1 14.4 14.8 9.1 6.7
(3.7) (4.2) (7.3) (4.5) (5.1) (3.5)
Posttest M (SD) 2.2 (0.5) 11.8 10.5 12.3 10.3 8.3 8.0
n 13
(4.4) (4.4) (7.5) (4.9) (6.0) (3.5)
12 12 12 12 12 12
Pretest M (SD) 2.4 (0.5) 11.6 8.1 16.9 14.3 11.4 8.3
(3.9) (4.3) (6.0) (6.5) (6.5) (6.5)
Moderated Posttest M (SD) 2.1 (0.5) 12.6 9.0 10.3 9.8 8.4 7.3
n 13
(3.9) (5.1) (7.0) (5.1) (4.1) (2.9)
13 13 13 13 13 13
Pretest M (SD) 2.5 (0.7) 10.2 8.8 18.1 13.8 9.9 7.8
(4.4) (4.5) (5.3) (6.4) (5.2) (3.3)
Posttest M (SD) 2.2 (0.6) 11.9 9.8 12.8 11.3 12.5 9.4
(3.9) (3.8) (5.4) (5.7) (6.2) (3.2)
16 16
3.3 (1.6) 4.4 (1.4)
4.2 (1.5) 5.1 (1.2)
13 13
3.6 (1.2) 3.8 (1.6)
3.9 (1.5) 3.8 (1.5)
13 13
2.8 (1.8) 3.6 (1.7)
3.8 (1.5) 4.7 (1.5)
17 17
24.8 (20.9) 22.8 (6.9)
28.2 (36.8) 20.7 (6.2)
13 13
29.7 (33.2) 22.5 (6.7)
22.3 (17.6) 20.9 (6.1)
13 13
18.6 (17.3) 20.5 (7.0)
28.4 (19.5) 24.8 (7.3)
Note. For the Career Exploratory Behavior Scale, this table shows means and standard deviations for the untransformed sums of the six items in the scale. The transformed data based on the ranks is shown as the mean ranks of frequencies with all 43 participants ranked together at pre- and posttests.
An alpha level of .05 was used to control the Type I error rate in all statistical tests. Some statisticians argue that when more than one comparison is performed, then the alpha level that is used to control the per-comparison Type I error rate should be made more stringent in order to control the experiment-wise Type I error rate, regardless of whether or not the comparisons are planned or unplanned, orthogonal or nonorthogonal. However, adjusting the per-comparison Type I error rate can result in insufficient power to detect even large effects given the small to medium sample sizes employed in most studies. As Keppel (1991, p. 165) notes, the most widely used strategy among researchers is to evaluate each comparison at an unadjusted per-comparison alpha level. Although Keppel (1991, pp. 178–179) provides a strong rationale for not adjusting per-comparison alpha levels, which is the approach taken in the current study, cautious researchers should regard the findings presented here as tentative. Treatment condition participants showed greater gains than controls on career decidedness as measured by the Occupational Alternatives Question (OAQ), d = .54, F(1, 40) = 2.88, p < .05. On the Decidedness scale of the Career Decision Profile (CDP) there was an observed effect size, d, of .37, which was consistent with the results from the OAQ, although not statistically significant at the .05 level, F(1, 38) = 1.99, p = .08. Treatment condition participants reported greater gains than control participants in self-knowledge as measured by Scale 3 of the CDP, d = .58, F(1, 38) = 4.35, p = .02. Treatment–control condition comparisons were not statistically significant for gains in (a) vocational knowledge, (b) career choice salience, (c) satisfaction with one’s current career situation, or (d) satisfaction with one’s future career prospects. The impact of the intervention on career exploration behaviors as measured by the CEBS was somewhat difficult to ascertain due to the nonnormality of the observed distribution, which had both heavy tails and skewness. Summing the six items of the CEBS pretest for the 43 participants who completed both the pretests and the posttests yields a CEBS total pretest score. The distribution of these 43 pretest scores has a high coefficient of skewness, g1 = 2.7 and significant kurtosis, g2 = 8.8. Results for the CEBS posttest scores are similar. The distribution of raw gain scores, the CEBS total posttest score minus the CEBS total pretest score, shows less skewness, g1 = .7, but significant kurtosis, g2 = 6.5. An inspection of the data did not reveal the presence of any obvious data entry errors or random response patterns that could account for the nonnormality of the score distributions. For example, one participant reported sending out 100 career-related email messages or letters during the past
month at pretest. Although this figure was high, it was certainly not outside the realm of possibility. Because of these problems, nonparametric methods were used to analyze these data. Data from the CEBS were transformed and analyzed as follows: The pretest scores for each of the six individual items making up the scale were ranked separately. All 43 participants who completed both the pre- and posttests were ranked together on each item. Assigned ranks for each item ranged from 1 to 43, with tied ranks assigned the mean rank of the ties. For each participant, a mean pretest rank was calculated by taking the mean of his or her ranks on the six pretest items. The same procedure was used to calculate mean posttest ranks for each participant. For each participant, a rank gain score was calculated by subtracting the mean pretest rank from the mean posttest rank. Univariate ANOVAs were then conducted on the transformed gain scores. Performing an ANOVA on ranked data with more than two levels of the independent variable is equivalent to the nonparametric Kruskal–Wallis ksample test. Based on this nonparametric analysis, participants in the two treatment conditions reported greater gains in career exploration behaviors than did control condition participants, d = .50, F(1, 40) = 3.27, p = .04. Participants in the moderated treatment condition reported greater gains in satisfaction with their future career prospects, d = .75, F(1, 38) = 3.58, p = .03. There was also a moderate observed superiority in satisfaction with current career situation for moderated treatment condition participants, d = .43, although this effect did not achieve significance at the .05 level, F(1, 38) = 1.56, p = .11. Participants in the moderated treatment condition reported greater gains in career exploration behaviors, d = .86, F(1, 40) = 6.06, p < .01, than participants in the unmoderated treatment condition. No statistically significant differences between moderated and unmoderated condition participants were found in gains on the measures of career decidedness, self-knowledge, vocational knowledge, career choice salience, or satisfaction and comfort with current career situation.
4. Discussion The Career HOPES recruitment Web page emphasized that Career HOPES might be especially helpful to people who were (a) thinking about changing jobs, (b) wondering if their current job was the right one for them, or (c) unemployed or just out of school and not sure what to do next. The implication was that Career
S. Herman / Computers in Human Behavior 26 (2010) 339–344
HOPES would be especially helpful to people who were uncertain about their current career direction. Indeed, one of the most salient positive outcomes for participants in interventions such as Career HOPES is coming to a decision about what occupational alternatives are worth pursuing further. The results of the study indicate that the intervention had a positive impact on career decidedness, helping participants to become more decided about their future career directions. It is an axiom of career counseling that optimal career decisions are based on high quality information – information about one’s own interests, abilities, values, and personality as well as information about the requirements and features of occupational alternatives. Results indicate that the intervention had a positive impact on two indices of information quality: occupationally relevant self-knowledge and the frequency of career exploration behaviors. A secondary purpose of the study was to determine what additional benefit accrues from having a professional group moderator involved in the online discussions. As compared with participants in the unmoderated treatment condition, participants in the moderated treatment condition showed greater gains in satisfaction with future career prospects and in the emission of vocational exploration behaviors. Participants in the moderated treatment condition were more satisfied overall with the intervention. Among participants who completed the posttests, 92% of the moderated treatment participants and 55% of the unmoderated treatment condition subjects rated the intervention as Useful or Very Useful on a 5-point scale. The difference between the two conditions in overall satisfaction was statistically significant, d = .57, F(1, 35) = 3.94, p = .03. Although a positive impact on participants’ career development was observed on some important indices, gains were not observed across the board. In all, 18 statistical tests were performed on nine outcome measures (nine tests for treatment vs. control and nine for moderated vs. unmoderated). The null hypothesis was rejected in only 5 of these 18 tests. One possible partial explanation for these results is that the control condition participants showed unexpected and relatively large gains on some outcome measures. This may have been due to an initial underestimation of the positive impact of the minimal-intervention control condition, which was intended to function as an attention placebo. Spokane (1991, p. 40) notes that dropout rates in group-based career interventions typically exceed 50%. The Career HOPES intervention was not an exception to this rule. There were a total of 40 participants in the two treatment conditions. Of those 40 participants, 32 completed the first lesson, 22 completed the second, 13 completed the third, and 8 completed the fourth. In other words, by the strictest possible criterion, completion of all four online lessons, 80% of the treatment condition participants would be classified as treatment dropouts. The posttest surveys suggest that some participants may have dropped out of the study because of the length or lack of appeal of the weekly lessons. On the posttest intervention evaluation questionnaire, 60% of the respondents opined that the weekly lessons were too long, and none thought the lessons were too short. Lesson difficulty did not appear to be a problem; 83% of unmoderated condition participants and 100% of moderated condition participants rated the difficulty level as ‘‘about right.” The observed impact of the intervention compares favorably with that of other career development interventions on certain outcomes. For example, on the two measures of career decidedness, Career HOPES showed a mean effect size of d = .46. In the Whiston et al. (1998) meta-analysis, there were 50 treatment–control condition comparisons on measures of career decidedness. The mean weighted effect size in these 50 studies on career decidedness was d = .19. The weighted mean effect size for all treatment–control comparisons in Whiston et al. was d = .30. The mean effect size across the nine outcome measures in the current study was d = .19.
343
How efficient was Career HOPES? Career HOPES treatment condition participants spent an average of 4.9 h online and the mean effect size across all measures was d = .19. Efficiency can be quantified as mean effect size change per hour of treatment. For Career HOPES, this was .04 per hour. Whiston et al. (1998) calculated the efficiency of the career interventions in their meta-analysis. Based on their calculations, the efficiency of Career HOPES is comparable to that of career classes (.08/h), group counseling (.06/h), and career workshops (.05/h). Career HOPES was significantly less efficient than individual counseling (.92/h) and computer-assisted career guidance interventions (.23/h). In the current study the author spent 20 h 58 min online responding to messages posted by the 20 participants in the moderatedgroup condition forums. This amounts to an average of about 1 h per participant. If all of these participants had completed all four online lessons, the moderator would have had to spend an estimated 2 h online per participant. The relatively small advantage of the moderated-group treatment as compared with the unmoderatedgroup treatment on the outcome measures suggest that this may not be a cost-effective use of a counselor’s time. On the other hand, the amount of moderator time invested in this first trial of Career HOPES may overestimate the amount of time that would be required to achieve similar results in a more refined version of Career HOPES. One possible disadvantage to interventions such as Career HOPES that rely on automated instruction is that they are inherently less flexible than interventions in which a human being responds directly to an individual’s concerns. Krumboltz (1992) has argued that the goals of counseling should be tailored to the individual. He stressed, for example, that changes in the direction of greater career decidedness are not desirable for all counseling clients, and that there can be ‘‘wisdom in indecision.” These points are well taken and they highlight an important feature of psychoeducational interventions of this type, namely, that the success of such interventions is likely to depend on the homogeneity of the target population’s goals or problems. In focused, group-based interventions such as Career HOPES, some degree of individual tailoring is possible through the use of interactive programmed instruction and the open-ended discussions of individual issues in the online forums. However, such interventions are probably most likely to succeed when clients are carefully selected or self-selected to fit the intervention. To achieve accurate self-selection, the recruitment materials must clearly spell out who is likely to benefit. 4.1. Recommendations A number of recommendations for future studies and for the future development of the Career HOPES project follow from the current analysis. First, the interactive psychoeducational component of Career HOPES needs to be expanded and improved. The current version of Career HOPES does not appear to be as efficient as offline computer-assisted career guidance interventions. The performance of computer-assisted interventions should be considered the minimal standard that Internet-delivered career interventions must achieve. More than half of the participants who completed the posttests thought that the weekly lessons were too long and none thought they were too short. The length of the lessons may have contributed to the high noncompletion rate; therefore, the length of each lesson should be shortened. It may be possible to improve the efficacy of Career HOPES and other similar interventions by shortening the length of each lesson and increasing their number. Increasing the number of lessons would increase the number of weeks, or time span, of the intervention, which may result in improved treatment efficacy due to an increase in the period of sustained attention and involvement by participants. Some of the content of the Career HOPES lessons should be rethought. People do not like read-
344
S. Herman / Computers in Human Behavior 26 (2010) 339–344
ing a lot of text online. Reducing the amount of text that participants need to read and increasing interactivity will likely increase the appeal of the intervention and should be a primary goal of future efforts. A fruitful area for future research and development will be creating and testing new formats for professional involvement in Internet-delivered psychoeducational interventions such as Career HOPES in order to better leverage the investment of time by professional moderators. Some possibilities that should be explored in future studies include creating lists of Frequently Asked Questions (FAQs). Perhaps weekly live group chat sessions would be a more productive use of a professional moderator’s time than reading and responding to each message posted by participants. Or a moderator could be ‘‘on call” – responding only to specific questions and requests for assistance. It may be possible to make use of ‘‘boilerplate” responses to common questions. Finding ways to reduce the need for intensive professional oversight is important because, otherwise, Internet counseling interventions are going to be almost as expensive to deliver as in-person counseling. Increasing participant commitment to, and investment in, online interventions is another area in which improvements need to be made. For example, adding live video or audio chat or posting pictures and personal profiles of participants may facilitate an increased sense of personal relationship and commitment. In future studies of this type, no-treatment or wait list control conditions should be employed. Data from the current study suggest that the control condition intervention may have been more powerful than assumed at the outset. Future studies should also compare online interventions to in-person interventions with similar content whenever possible. 5. Conclusion Debates over the pros and cons of online counseling have usually focused on differences between individual face-to-face counseling and counseling via email or chat (e.g., Mallen et al., 2005). The main advantage of conducting individual counseling sessions online is that the Internet makes it possible to provide needed counseling services to individuals who might not otherwise have access. However, it is not likely that individual online counseling will prove to be significantly more effective, or cost-effective, than face-to-face counseling. Semi-automated, group-based psychoeducational interventions such as Career HOPES, on the other hand, hold out the promise of improving the effectiveness and cost-effectiveness of psychosocial interventions as well as increasing access to needed counseling services. Acknowledgements Author thanks his dissertation chair, mentor, and friend, John Krumboltz, for his many invaluable contributions to the Career HOPES project. Thanks also to Karen Slaton, who helped with the design and implementation of the Career HOPES Website. References Beutell, N. J., & O’Hare, M. M. (2006). Career pathfinders: A qualitative study of career development. Psychological Reports, 98, 517–528.
Bleier, J. K. (2007). The impact of career counseling plus DISCOVER (Internet version) on the academic achievement of high school sophomores at risk for dropping out of school. Dissertation Abstracts International: Section B: The Sciences and Engineering, 68. Clark, G. (2001). Experimental evaluation of an interactive internet career guidance program for parents of gifted girls. Dissertation Abstracts International: Section B: The Sciences and Engineering, 61, 5555. Clark, G., Horan, J. J., Tompkins-Bjorkman, A., Kovalski, T., & Hackett, G. (2000). Interactive career counseling on the internet. Journal of Career Assessment, 8, 85–93. Gati, I., Kleiman, T., Saka, N., & Zakai, A. (2003). Perceived benefits of using an Internet-based interactive career planning system. Journal of Vocational Behavior, 62, 272–286. Holland, J. L. (1994a). The occupations finder (Form R). Odessa, FL: Psychological Assessment Resources. Holland, J. L. (1994b). The self-directed search (Form R). Odessa, FL: Psychological Assessment Resources. Holland, J. L. (1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Odessa, FL, US: Psychological Assessment Resources. Hornyak, D. A. (2007). Utilizing cognitive information processing theory to assess the effectiveness of DISCOVER on college students’ career development. Dissertation Abstracts International: Section A: Humanities and Social Sciences, 68, 2319. Jones, L. K. (1989). Measuring a three-dimensional construct of career indecision among college students: A revision of the Vocational Decision Scale: The Career Decision Profile. Journal of Counseling Psychology, 36, 477–486. Jones, W. P., Harbach, R. L., Coker, J. K., & Staples, P. A. (2002). Web-assisted vocational test interpretation. Journal of Employment Counseling, 39, 127–137. Keirsey, D., & Bates, M. (1984). Please understand me: Character and temperament types. Del Mar, CA: Prometheus Nemesis Book Co. Keppel, G. (1991). Design and analysis: A researcher’s handbook (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. Kovalski, T. M., & Horan, J. J. (1999). The effects of Internet-based cognitive restructuring on the irrational career beliefs of adolescent girls. Journal of Cognitive Psychotherapy, 13, 145–152. Krumboltz, J. D. (1992). The wisdom of indecision. Journal of Vocational Behavior, 41, 239–244. Krumboltz, J. D., & Levin, A. S. (2004). Luck is no accident: Making the most of happenstance in your life and career. Atascadero, CA: Impact Publishers. Krumboltz, J. D., & Thoresen, C. E. (1964). The effect of behavioral counseling in group and individual settings on information-seeking behavior. Journal of Counseling Psychology, 11, 324–335. Levinson, E. M., Zeman, H. L., & Ohler, D. L. (2002). A critical evaluation of the Webbased version of the Career Key. The Career Development Quarterly, 51, 26–35. Mallen, M. J., Vogel, D. L., Rochlen, A. B., & Day, S. X. (2005). Online counseling: Reviewing the literature from a counseling psychology framework. Counseling Psychologist, 33, 819–871. Oliver, L. W., & Spokane, A. R. (1988). Career-intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447–462. Robinson, N. K., Meyer, D., Prince, J. P., McLean, C., & Low, R. (2000). Mining the internet for career information: A model approach for college students. Journal of Career Assessment, 8, 37–54. Severy, L. E. (2008). Analysis of an online career narrative intervention: ‘‘What’s my story?”. The Career Development Quarterly, 56, 268–273. Slaney, R. B. (1988). The assessment of career decision making. In W. B. Walsh & S. H. Osipow (Eds.), Career decision making. Hillsdale, NJ: Erlbaum. Spokane, A. R. (1991). Career intervention (p. 353). Englewood Cliffs, NJ, US: Prentice Hall, Inc. Šverko, B., Akik, N., Babarovic´, T., Bcˇina, A., & Šverko, I. (2002). Validity of e-advice: The evaluation of an Internet-based system for career planning. International Journal for Educational and Vocational Guidance, 2, 193–215. Tompkins Bjorkman, A. B. (2003). Internet-based attributional retraining and career decision making. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63, 5540. U.S. Bureau of Labor Statistics (2009). Occupational Outlook Handbook, 2008–09 edition. Washington, DC: U.S. Bureau of Labor Statistics.