- Email: [email protected]
Student focus groups as an assessment technique: A case study
Pergamon
Journal of Accounting Education, Vol. 15, No. 3, pp. 457-468, 1997 © 1997 Elsevier Science Ltd. All rights reserved Printed in Great Britain 0748-5751/97 $17.00 + 0.00
PII: S0748-5751(97)00016-X
STUDENT
FOCUS GROUPS AS AN ASSESSMENT TECHNIQUE: A CASE STUDY Gadis J. Dillon and Lizabeth A. Barclay
OAKLAND UNIVERSITY Abstract: Assessment is increasingly important in the academy. This paper presents a case study of one assessment technique: focus groups. The methodology of focus groups is discussed, and general guidelines for operationalization are presented via the case study. Focus groups are a useful assessment tool for m a n y aspects of accounting programs. © 1997 Elsevier Science Ltd
INTRODUCTION Outcomes assessment is increasingly important at most universities. Traditionally, measures such as student grades, course pass rates, student evaluations of faculty, and a limited number of external measures such as CPA Examination results have been used to determine whether successful educational outcomes occurred. Recently external accrediting bodies, such as the American Assembly of Collegiate Schools of Business, have advocated additional assessment processes. On the authors' campus, most faculty and administrators agree that increased assessment emphasis is appropriate. A variety of assessment techniques is encouraged, since different assessment techniques may provide different information. DeMong et al. (1994) discuss several assessment techniques for Accounting programs. However, their discussion does not include the focus groups technique. The use of focus groups is a useful method which can provide insights not available through other techniques. This paper discusses and illustrates how focus groups may be used as part of an assessment program. A case study is used to present an experience using focus groups to gather and assess information about the perceptions of students who had enrolled in an Introductory Financial Accounting course. Student perceptions may not necessarily reflect the factual situation, but those perceptions influence student attitudes toward a course and, quite likely, influence outcomes. 457
458
G.J. Dillon and L. A. Barclay FOCUS G R O U P S
"A focus group is a carefully planned discussion designed to obtain perceptions on a defined area in a permissive, nonthreatening environment" (Krueger, 1994, p. 6). In a focus group, a small number of participants address a series of questions provided by a moderator. This format permits the moderator to do additional probing based on the discussion. Such probing is not possible, for example, in the normal endof-term written faculty evaluations, nor in traditional survey research. The focus group technique was developed because researchers were concerned that closed-ended questions unduly influenced respondents (Krueger, 1994). In an attempt to develop new ways of information gathering, they went beyond traditional data-collection methods. These initial efforts were called 'nondirective interviewing'. As the procedures developed, these interviews were conducted in groups and came to be known as focused interviews (Krueger, 1994). Merton (1987), who is considered the father of focus groups (Vaughn et al., 1996), indicates that such groups were used as early as the 1940s. Focus groups are widely used by consumer groups and industry to gather information. According to Lederman (1990), the focus group is based on the assumption that people will talk about their problems when in a group of similarly concerned individuals. Lederman also points out that this technique allows for generation of data about the 'why' behind behavior, asking questions that go beyond traditional surveys and exploring lines of thought that likely would not be followed in a more structured information--collection situation. This supports Trachtenberg's (1987) contention that many organizations use focus groups to get insights into consumers that are generally not available from traditional statistical analysis. This is accomplished in part by the structure of the technique. Besides providing an opportunity to probe, focus groups may also be valuable in developing or refining survey instruments (Buttram, 1990). This in the long run may be very cost effective. Inglis (1987) also states that focus groups are good at issue-defining. This is not to suggest that focus group data fully replace more traditional data gathering methods. One reason that researchers would not rely solely on focus group data has to do with group size and the timeintensiveness of the technique. The literature indicates that focus groups typically have eight to twelve members (O'Donnell, 1988, p. 71). The size of the group is important since smaller groups may be intimidating and larger groups can prevent optimal interaction. This interaction is extremely important according to Sink (1991), since comments by one participant can generate useful discussion and idea exploration. Multiple groups are often necessary. At some point, additional groups become redundant because of the targeted focus of the technique. Churchill (1991) indicates
Student Focus Groups
459
that focus groups should be homogeneous; all members o f a particular group should have common experience. Dean & Evans (1994), in a discussion of Total Quality Management, indicate that focus groups can help organizations understand customers, Hence, focus groups are a way of providing information for continuous improvement. In fact, Seymour (1992, p. 54) specifically addresses higher education quality initiatives, indicating that, "It is incumbent upon the college or the program to research the needs of its specific customer groups". While some universities have been hesitant to embrace the student as customer model per se, continuous improvement efforts can assist in delivering course objectives more effectively. Broadly speaking, focus groups can be used with student, alumni and employer groups when assessing the outcomes of accounting programs. This study deals with only one customer group, but the focus group technique can be applied to all. T H E CASE STUDY The description of a case study using the focus group technique is presented next. All the focus groups described here were conducted within approximately one month, shortly after the conclusion of a semester. All participants had been enrolled in Introductory Financial Accounting during that semester. This case study provides a vehicle for presenting a general description of the steps in developing and utilizing the focus group technique. After each step's general description, it is illustrated with specifics from the case study.
Case Background The case was conducted at a university with a traditional two-semester sophomore level Introductory Accounting sequence: one semester of Introductory Financial Accounting and one semester of Introductory Managerial Accounting. All sections used a common text and covered common material. At the time the focus groups were conducted, all students took common examinations administered at a common time. A significant proportion of the students were commuter students; about onethird of the sections were taught in the evening. A mixture of full-time and adjunct faculty taught the introductory courses, with a majority taught by full-time faculty. Historically, a significant percentage of students who enrolled in the Introductory Financial Accounting did not successfully complete the course with a grade of 'C' or better. For example, in a recent semester prior to the one which provided the focus groups, some 49% of the
460
G . J . Dillon and L. A. Barclay
students who registered for the course either formally withdrew with a 'W' (28%) or received a grade of 'F' (9%) or 'D' (12%). The withdrawal rate in particular was considerably higher than in similarly placed courses in the curriculum (e.g., Introductory Microeconomics). Both the Accounting faculty and the administrators in the School of Business Administration were (and are) concerned about the proportion of students not successfully completing the course. These concerns reflect some of the issues discussed in the recent Wingspread Report (1993) which focused on a cultural change needed within universities: a change from 'weeding out' students to assisting students in learning development and developing strategies for success (Wingspread Group on Higher Education, 1993). A range of opinions was expressed as causes for the lack of Success.
Some efforts had been made to improve performance, such as the use of 'Supplemental Instruction' (Martin & Arendale, 1992, p. 3) and free oneon-one tutoring. Unfortunately, few students took advantage of the support efforts available. Course evaluation forms completed at the end of the semester by stillenrolled students were the primary source of systematically gathered information. These evaluations, which were common to all courses in the School of Business Administration, did not reach students who had already withdrawn, or many 'failed' students who no longer attended class (a policy prohibiting withdrawals after mid-semester was strictly enforced). In addition, there were no items concerning student investment in the course (such as the extent of effort devoted to the course and the decision whether to use available assistance). To address those issues, focus groups with successful, unsuccessful and withdrawn students were conducted.
M E T H O D O L O G Y OF FOCUS GROUPS
Design and Recruitment of Focus Groups Although focus groups are a type of qualitative rather than the more traditional quantitative research method, care must still be taken to clarify the purpose underlying the effort. Specifically, the objectives of the research should be defined, and the participants selected consistent with those objectives. It would be inappropriate to merely 'collect' eight to twelve individuals and have them chat about a topic. Identified goals associated with the focus group sessions assist the moderator with his/her probing. The moderator may then put participants at ease and draw them out with respect to the specific information desired. Recruitment of individuals for the sessions must be based on the underlying issues. In the case study, the purpose was to explore students' perceptions for their lack of success in introductory accounting, and to
Student Focus Groups
461
gather information about the students' approach to the course. For example, were there specific structural elements in the course which students perceived as being discouraging? Did these students use tutors or other available assistance resources? How much time and effort did they put into the course? What course-related factors influenced the students actions and perceptions? Based on class lists and grade reports from the most recently completed semester, representative groups of students were selected and invited to participate in the focus groups. Students were categorized as falling within one of three groups: those who withdrew from the course (withdraw group); those who received a grade below 'C' (nonpass group); and those who received a grade of 'C' or higher (pass group). The 'successful' students (pass group) were included in order to obtain control information. The selection process was not completely random. Stratification was used to ensure that the participants were representative of all students. Because the university has a large number of nontraditional students, attempts were made to include students from both day and night sections in each group. In addition, the groups were constructed to balance participation across instructors. Within each section of the course, a random process was used to select specific student participants. After selecting a representative sample, the researchers corresponded with the prospective students via telephone and letter. Not every student contacted was willing to participate. On the other hand, several students seemed quite interested in having their views heard. To further encourage participation, a small cash incentive of $25.00 was offered to each participating student. This was funded by a grant supported by Oakland University. While the payment was somewhat less than a business would pay to focus group participants (the local minimum is $75.00), the incentive did help ensure that students took the task seriously. Once a student expressed willingness to participate in the focus group, a follow-up written confirmation of the focus group's location, date and format was provided. A list of possible discussion topics/questions was also provided; the students were asked to review them prior to their session. This list of questions was intended as the starting point to get people involved. The questions were generally quite structured and had straightforward answers. The researchers fully expected that the discussions in the sessions would go well beyond an answer to these questions. Appendix A lists the questions distributed. The objective of this project was to obtain information from students about perceived impediments to their learning. Course content was not addressed, since it was believed that students did not have the level of experience such that they could appropriately judge course content.
462
G.J. Dillon and L. A. Barclay
Rather, the sessions concentrated on students' experiences and their attitudes toward the course delivery. By having students relate their experiences, ways of improving course delivery could be addressed. To avoid having participants feel inhibited in the groups, the services of an external marketing consultant who was not affiliated to the university in any way but who was experienced in conducting focus groups were secured. Marketing faculty at the university helped make this contact. The consultant, a recent graduate of an MBA program, had an understanding of the introductory accounting course. In the confirmation letter sent to student participants, the moderator was described as an external expert, not a faculty member.
Conduct of the Focus Groups It is best to conduct the focus groups in a nonthreatening location. Refreshments are desirable. The session should he begun with nonthreatening questions. The moderator generally begins each focus group with a brief introduction of himself/herself, introduction of participants by first name only, and restating the purpose that was previously communicated to the participants by phone and letter contacts. As the session continues, the moderator attempts to involve individuals who are not active participants. The moderator may take notes and/or the session may be recorded. If recording is to be used, participants should be informed at initial contact. In structuring each of the focus group sessions (pass, nonpass, withdraw), the researchers attempted to put the participants at ease. The sessions were held in a room in the student center, using a large square table that could accommodate ten to twelve participants and the moderator. Refreshments were available for early arrivals and throughout the session. Participants had been told at the initial contact that the session would be taped; the tape recorder was placed in the center of the table. One of the researchers opened each session by introducing the moderator and explaining the format again. Administrative details (arrangements for payments, completion of an 'informed consent' form by each participant to comply with university policy) were completed. The researchers then left the room. There was no observation of the session; only the moderator and participants remained. Each session lasted approximately two hours (Churchill, 1991, p. 138), which was long enough to give every participant an opportunity to become comfortable in participating. Before and after each session, the researchers met with the moderator. Prior to the sessions, the broad topics that should be addressed were discussed. Strategies for building on the distributed questions were also considered. After each session, the moderator was
Student Focus Groups
463
debriefed as he noted his impressions from the discussions. The researchers also independently reviewed the tape recording of each session.
Evaluating and Reporting Results Both Krueger (1994) and Vaughn et al. (1996) provide detailed discussions of data analysis related to focus groups. Often, analysis requires consideration of many pages of transcripts. Approaches to analysis include coding statements into predetermined categories, and developing data categories (or 'themes') based on observed responses and then using the themes to generate summary statements. It is important to note that when analyzing the data, groups may not have a consensus opinion. Focus groups deal with perceptions. Groups may have majority and minority positions (Vaughn et al., 1996) and this must be considered in developing action recommendations. With respect to the case described here, it is recognized that specific findings may be unique to each institution. Accordingly, only a brief summary of the findings for this specific case are presented. As noted above, focus group data can be reported in different ways. In this case 'themes' (Krueger, 1994, p. 166) or the "essence of the interviewees' responses" (Lederman, 1990, p. 124) was used. In order to help identify themes, it is useful to count the number of times an issue is raised (Vaughn et al., 1996). An issue only raised once and not taken up by other focus group participants may not be as salient as an issue discussed by many participants. Once issues are collected, they can be grouped into themes or overarching categories. For example issues such as "this course is hard," "lots of people flunk this course," and "this course keeps people out of the business school" are all 'course' comments, while a comment such as "I study alone" is not in this category. The topics discussed and the student perceptions generally fell into five themes': 1. 2. 3. 4. 5.
Preconceptions about the course; Study techniques; Use of available resources; Attitudes toward the course structure; and Interactions with faculty.
The researchers agreed with these 'themes' after they had independently reviewed the tapes. A rather high level of consistency between the perceptions of the 'pass group' and the 'nonpass group' was observed. The 'withdraw' group was somewhat different, possibly based on more limited experiences with the course. Further, withdrawal was not always related to academic difficulties. Every participant obviously did not have the same experiences
464
G.J. Dillon and L. A. Barclay
and opinions; the descriptions below summarize frequently-expressed positions. Preconceptions about the Course: Most students had heard that the course was difficult and required lots of work. Further, many had heard it was a 'weed-out' course for the School of Business Administration. Study Techniques: 'Nonpass group' students were less likely to study with fellow students. (While this study did not address the issue, we believe this lack of 'peer group support' to be an important determinant of lack of success.) The 'pass group' students claimed extensive time spent in course preparation. The 'nonpass group' and 'withdraw group' members were much more likely to fall behind early in the semester. Use of Available Resources: Many students, especially in the 'nonpass group', reported that they did not know what resources were available. The 'pass group' was more likely to use Supplemental Instruction. The 'nonpass group' perceived that the Supplemental Instruction student leader was not responsive to their needs. The 'nonpass group' stated that tutors "were not worth the effort"; they felt the learning gained did not justify the time spent. Attitudes Toward Course Structure: Most students disliked the common examinations. They were concerned that "someone other than their instructor was preparing the examination". Further, most felt the exams, especially the multiple choice questions, were too 'tricky'. Some felt class sessions were 'Right out of the book' and that attendance was not a valueadded experience. Interactions with Faculty: Students seemed unwilling to initiate contact with faculty, although no frequent reason was given. The 'nonpass group' perceived that faculty were predisposed to failing students. Some faculty members had quoted failure and drop rates in class. Some faculty members advised students doing poorly on the first exam to withdraw; students then believed that the faculty member was biased toward those so advised who did not withdraw.
Developing Action Recommendations Not all focus group research reports contain recommendations. As discussed earlier focus groups are often exploratory in nature, providing insights into additional, more focused research efforts. When recommendations or suggestions are made, the intended audience may need to be briefed on the focus group technique in order to better discuss options. In particular, the audience should be reminded that the perceptions of the focus group participants are being reported. Student perceptions may not be consistent with faculty perceptions.
Student Focus Groups
465
Information from the focus groups in the case was summarized and presented, along with recommendations, to the Accounting faculty for discussion. Recommendations included: • An expanded syllabus, to include more information for students about how to obtain support services available on campus, and .frequent reference to that syllabus throughout the semester. • Communication to students that all faculty review exams and suggest modifications before each common exam. (Subsequent to conducting the focus groups, the Accounting faculty decided to discontinue common examinations. Focus group information was only one of several factors considered in this decision.) • Justifying the importance of apparently minor wording differences in the 'real world'. A single word can dramatically impact the meaning of a contract. Explain to students that apparently 'tricky' questions have an underlying rationale. • A review and discussion of Supplemental Instruction with the Academic Skills Center. In particular, monitoring and immediate feedback for the student 'leaders' who conduct the Supplemental Instruction sessions. • Faculty should not directly recommend that students withdraw from the course. • Faculty should inform the students that the course will require considerable effort, but it is counterproductive to quote failure and withdrawal rates. The last two recommendations are consistent with Bandura's (1986) selfefficacy theory, which holds that self-efficacy can be influenced by enactive attainment, vicarious experience, verbal persuasion and physiological state. Of these four sources, two are relevant to the recommendations based on information secured from the focus groups. First, the faculty style and tone of communication can either increase or decrease student self-efficacy. Messages which indicate failure are likely to reduce efficacy and lead to increased likelihood o f not passing the course. While verbal persuasion is not the strongest source of self-efficacy, it can be very important in the early stages of a class as it can set up a self-fulfilling prophecy. The second influence, vicarious experience, has specific tie-ins with supplemental instruction and peer tutoring. "Seeing or visualizing other similar people perform successfully can raise self-precepts of efficacy in observers that they too possess the capabilities to master comparable activities." (Bandura, 1986, p. 399) Encouraging students to avail themselves of Supplemental Instruction and other support systems can lead to increased self-efficacy and a willingness to continue in the class. The recommendations for improved communication with students are consistent with research on improving communication effectiveness.
466
G.J. Dillon and L. A. Barclay
Cherrington (1994) specifically details strategies to help improve communication. Repetition of communication episodes in particular is relevant to this situation. Students need to be reminded of support services throughout the term since they may forget such information. Because this was the researchers' first experience making recommendations based on focus group data, it was unclear how the information would be received. In a systematic program assessment, it would be essential for the faculty to commit to the evaluation process before it took place. As with more traditional educational assessment, such as student evaluations of faculty, the researchers found there to be variability in the acceptance of the recommendations within the faculty group.
T R A N S F E R A B I L I T Y TO O T H E R I N S T I T U T I O N S As noted above, this methodology should transfer well to other accounting programs. The specific information gathered from focus groups in this case may be unique to this institution, and the specific recommendations to the Accounting faculty may not be appropriate to others. However, the use of focus groups will allow other institutions to systematically obtain information from their students, alumni and employers that is likely richer than that provided by other techniques, and thus will likely result in sound recommendations appropriate for that program. As a result of the experience, the following recommendations are offered to anyone using focus groups: 1. Do some initial research on focus groups. Vaughn et al. (1996) is an excellent source and contains a check list for helping those unfamiliar with the technique. Also, contact with marketing colleagues may provide additional insights. 2. Decide on the target issues to be addressed and the purpose and use of the focus groups. This definition will help you decide on the types and number of groups to conduct. 3. Do not use Accounting faculty as focus group moderators. This may tend to inhibit participants and perhaps reduce the validity of the information gathered. Further, Accounting faculty may have difficuRy remaining objective. The purpose of the moderator is to gather information regardless of its objectivity, not argue the facts. 4. Do guarantee confidentiality to participants. Students may be inhibited in responses if they believe that faculty will hear their responses, especially if they expect to have the same instructors in the future. Accordingly, students need to be assured that no one besides the project leaders will hear their individual responses. Only a group
Student Focus Groups
5.
6.
7.
8.
467
summary should be presented to the Accounting faculty. In fact, individual participants should not be identified. While in the case study a small honorarium was provided to participants, the nature of the student body at other institutions and the timing of the sessions may make this unnecessary. On a commuter campus, it is difficult to attract students back to campus when they do not have classes scheduled. Know your audience. For example, alumni and employer groups may be more readily available for evening sessions. In those cases, dinner could be provided. Do follow up with reminders before each session. In the case study, three contacts were made with each participant, including a phone call one to three days before the session. There were still some noshows. Be aware o f policies at your institution concerning human subjects research. For the case study, approval was obtained from the institution, and then each participant signed an 'informed consent' form. You may wish to publicize the results of such efforts in student and alumni communication media (newsletters, newspapers, etc.).
CONCLUSION The use of focus groups in the case study provided additional assessment information from students. This information was not readily available on student course evaluations. Based on the data collected, information and recommendations were provided to the Accounting faculty and the Academic Skills Center. This article also identifies several sources (in particular, Vaughn et al., 1996, and Krueger, 1994) that provide a useful description of the focus group technique. This technique can be usefully applied to other research questions and at other institutions.
REFERENCES Bandura, Albert. (1986) Social foundations oJ Thought and Action, a Social Cognitive Theoo', Englewood Cliffs, NJ: Prentice-Hall, Inc. Buttram, Joan L. (1990). Focus Groups: A Starting Point for Needs Assessment, Evaluation Practice, II(3), 207-212. Cherrington, David J. (1994) Organizational Behavior (2nd edition), Allyn & Bacon. Churchill, Gilbert A. (1991) Marketing Research (5th edition), Dryden Press, Chicago. Dean, James W. Jr., & Evans, James R. (1994) Total Quality: Management, Organization, and Strategy, St. Paul: West Publishing Co. DeMong, Richard F., Lindgren, John H. Jr., & Perry, Susan E. (1994). Designing an Assessment Program for Accounting Issues in Accounting Education, 9(1), 11 27.
468
G . J . Dillon and L. A. Barclay
Inglis, Robert C. (1987) In-depth Data: Using Focus Groups to Study Industrial Markets, Business Marketing, November, pp. 79-82. Krueger, Richard A. (1994) Focus Groups." ,4 Practical Guide Jor Applied Research (2nd edition), Thousand Oaks, CA: Sage Publications. Lederman, Linda C. (1990). Assessing Educational Effectiveness: The Focus Group Interview as a Technique for Data Collection, Commun. Educ., 38, 117-127. Martin, Deanna C., & Arendale, David C. (1992) Supplemental Instruction: Improving FirstYear Student Success in High-Risk Courses, National Center for the Freshman Year Experience, University of South Carolina. Merton, Robert K. (1987). The Focused Interview and Focus Groups, Continuities and Discontinuities, Public Opinion Quarterly, 51, 550-566. O'Donnell, Judith M. (1988). Focus Groups: A Habit-Forming Evaluation Technique Training and Development Journal, 42(7), 71-73. Seymour, Daniel T. (1992) On Q, Causing Quality in Higher Education, American Council on Higher Education, MacMillan Publishing Company. Sink, David W. (1991). Focus Groups as an Approach to Outcomes Assessment Am. Rev. Public Administration, 2•(3), 197-204. Trachtenberg, Jeffrey A. (1987) Listening the Old Fashioned Way, Forbes, October 5, pp. 202204. Vaughn, Sharon, Schumm, Jeanne Shay, & Sinagub, Jane (1996) Focus Group Interviews in Education and Psychology, Thousand Oaks, CA: Sage Publications. Wingspread Group on Higher Education (1993) An American Imperative." Higher Expectations .for Higher Education, The Johnson Foundation, Inc., Racine, WI.
APPENDIX
A
Possible Topics of Discussion at Focus Group 1. Had you talked about ACC 200 with friends and peers before you started the class? What were your expectations of the class going in? 2. Did you have a study group or did you study alone? 3. Were you in the Supplemental Instruction sections? 4. Did you use a tutor? (private, Beta Alpha Psi) 5. How much time did you spend on homework outside class? 6. How often did you see your instructor during his/her office hours? 7. Did you sit in on sections of ACC 200 other than your own? 8. Did you have friends in class you could call on if you missed a session'? 9. How did you feel about your progress prior to the first exam? 10. Did exams cover material from lecture and text?
Journal of Accounting Education, Vol. 15, No. 3, pp. 457-468, 1997 © 1997 Elsevier Science Ltd. All rights reserved Printed in Great Britain 0748-5751/97 $17.00 + 0.00
PII: S0748-5751(97)00016-X
STUDENT
FOCUS GROUPS AS AN ASSESSMENT TECHNIQUE: A CASE STUDY Gadis J. Dillon and Lizabeth A. Barclay
OAKLAND UNIVERSITY Abstract: Assessment is increasingly important in the academy. This paper presents a case study of one assessment technique: focus groups. The methodology of focus groups is discussed, and general guidelines for operationalization are presented via the case study. Focus groups are a useful assessment tool for m a n y aspects of accounting programs. © 1997 Elsevier Science Ltd
INTRODUCTION Outcomes assessment is increasingly important at most universities. Traditionally, measures such as student grades, course pass rates, student evaluations of faculty, and a limited number of external measures such as CPA Examination results have been used to determine whether successful educational outcomes occurred. Recently external accrediting bodies, such as the American Assembly of Collegiate Schools of Business, have advocated additional assessment processes. On the authors' campus, most faculty and administrators agree that increased assessment emphasis is appropriate. A variety of assessment techniques is encouraged, since different assessment techniques may provide different information. DeMong et al. (1994) discuss several assessment techniques for Accounting programs. However, their discussion does not include the focus groups technique. The use of focus groups is a useful method which can provide insights not available through other techniques. This paper discusses and illustrates how focus groups may be used as part of an assessment program. A case study is used to present an experience using focus groups to gather and assess information about the perceptions of students who had enrolled in an Introductory Financial Accounting course. Student perceptions may not necessarily reflect the factual situation, but those perceptions influence student attitudes toward a course and, quite likely, influence outcomes. 457
458
G.J. Dillon and L. A. Barclay FOCUS G R O U P S
"A focus group is a carefully planned discussion designed to obtain perceptions on a defined area in a permissive, nonthreatening environment" (Krueger, 1994, p. 6). In a focus group, a small number of participants address a series of questions provided by a moderator. This format permits the moderator to do additional probing based on the discussion. Such probing is not possible, for example, in the normal endof-term written faculty evaluations, nor in traditional survey research. The focus group technique was developed because researchers were concerned that closed-ended questions unduly influenced respondents (Krueger, 1994). In an attempt to develop new ways of information gathering, they went beyond traditional data-collection methods. These initial efforts were called 'nondirective interviewing'. As the procedures developed, these interviews were conducted in groups and came to be known as focused interviews (Krueger, 1994). Merton (1987), who is considered the father of focus groups (Vaughn et al., 1996), indicates that such groups were used as early as the 1940s. Focus groups are widely used by consumer groups and industry to gather information. According to Lederman (1990), the focus group is based on the assumption that people will talk about their problems when in a group of similarly concerned individuals. Lederman also points out that this technique allows for generation of data about the 'why' behind behavior, asking questions that go beyond traditional surveys and exploring lines of thought that likely would not be followed in a more structured information--collection situation. This supports Trachtenberg's (1987) contention that many organizations use focus groups to get insights into consumers that are generally not available from traditional statistical analysis. This is accomplished in part by the structure of the technique. Besides providing an opportunity to probe, focus groups may also be valuable in developing or refining survey instruments (Buttram, 1990). This in the long run may be very cost effective. Inglis (1987) also states that focus groups are good at issue-defining. This is not to suggest that focus group data fully replace more traditional data gathering methods. One reason that researchers would not rely solely on focus group data has to do with group size and the timeintensiveness of the technique. The literature indicates that focus groups typically have eight to twelve members (O'Donnell, 1988, p. 71). The size of the group is important since smaller groups may be intimidating and larger groups can prevent optimal interaction. This interaction is extremely important according to Sink (1991), since comments by one participant can generate useful discussion and idea exploration. Multiple groups are often necessary. At some point, additional groups become redundant because of the targeted focus of the technique. Churchill (1991) indicates
Student Focus Groups
459
that focus groups should be homogeneous; all members o f a particular group should have common experience. Dean & Evans (1994), in a discussion of Total Quality Management, indicate that focus groups can help organizations understand customers, Hence, focus groups are a way of providing information for continuous improvement. In fact, Seymour (1992, p. 54) specifically addresses higher education quality initiatives, indicating that, "It is incumbent upon the college or the program to research the needs of its specific customer groups". While some universities have been hesitant to embrace the student as customer model per se, continuous improvement efforts can assist in delivering course objectives more effectively. Broadly speaking, focus groups can be used with student, alumni and employer groups when assessing the outcomes of accounting programs. This study deals with only one customer group, but the focus group technique can be applied to all. T H E CASE STUDY The description of a case study using the focus group technique is presented next. All the focus groups described here were conducted within approximately one month, shortly after the conclusion of a semester. All participants had been enrolled in Introductory Financial Accounting during that semester. This case study provides a vehicle for presenting a general description of the steps in developing and utilizing the focus group technique. After each step's general description, it is illustrated with specifics from the case study.
Case Background The case was conducted at a university with a traditional two-semester sophomore level Introductory Accounting sequence: one semester of Introductory Financial Accounting and one semester of Introductory Managerial Accounting. All sections used a common text and covered common material. At the time the focus groups were conducted, all students took common examinations administered at a common time. A significant proportion of the students were commuter students; about onethird of the sections were taught in the evening. A mixture of full-time and adjunct faculty taught the introductory courses, with a majority taught by full-time faculty. Historically, a significant percentage of students who enrolled in the Introductory Financial Accounting did not successfully complete the course with a grade of 'C' or better. For example, in a recent semester prior to the one which provided the focus groups, some 49% of the
460
G . J . Dillon and L. A. Barclay
students who registered for the course either formally withdrew with a 'W' (28%) or received a grade of 'F' (9%) or 'D' (12%). The withdrawal rate in particular was considerably higher than in similarly placed courses in the curriculum (e.g., Introductory Microeconomics). Both the Accounting faculty and the administrators in the School of Business Administration were (and are) concerned about the proportion of students not successfully completing the course. These concerns reflect some of the issues discussed in the recent Wingspread Report (1993) which focused on a cultural change needed within universities: a change from 'weeding out' students to assisting students in learning development and developing strategies for success (Wingspread Group on Higher Education, 1993). A range of opinions was expressed as causes for the lack of Success.
Some efforts had been made to improve performance, such as the use of 'Supplemental Instruction' (Martin & Arendale, 1992, p. 3) and free oneon-one tutoring. Unfortunately, few students took advantage of the support efforts available. Course evaluation forms completed at the end of the semester by stillenrolled students were the primary source of systematically gathered information. These evaluations, which were common to all courses in the School of Business Administration, did not reach students who had already withdrawn, or many 'failed' students who no longer attended class (a policy prohibiting withdrawals after mid-semester was strictly enforced). In addition, there were no items concerning student investment in the course (such as the extent of effort devoted to the course and the decision whether to use available assistance). To address those issues, focus groups with successful, unsuccessful and withdrawn students were conducted.
M E T H O D O L O G Y OF FOCUS GROUPS
Design and Recruitment of Focus Groups Although focus groups are a type of qualitative rather than the more traditional quantitative research method, care must still be taken to clarify the purpose underlying the effort. Specifically, the objectives of the research should be defined, and the participants selected consistent with those objectives. It would be inappropriate to merely 'collect' eight to twelve individuals and have them chat about a topic. Identified goals associated with the focus group sessions assist the moderator with his/her probing. The moderator may then put participants at ease and draw them out with respect to the specific information desired. Recruitment of individuals for the sessions must be based on the underlying issues. In the case study, the purpose was to explore students' perceptions for their lack of success in introductory accounting, and to
Student Focus Groups
461
gather information about the students' approach to the course. For example, were there specific structural elements in the course which students perceived as being discouraging? Did these students use tutors or other available assistance resources? How much time and effort did they put into the course? What course-related factors influenced the students actions and perceptions? Based on class lists and grade reports from the most recently completed semester, representative groups of students were selected and invited to participate in the focus groups. Students were categorized as falling within one of three groups: those who withdrew from the course (withdraw group); those who received a grade below 'C' (nonpass group); and those who received a grade of 'C' or higher (pass group). The 'successful' students (pass group) were included in order to obtain control information. The selection process was not completely random. Stratification was used to ensure that the participants were representative of all students. Because the university has a large number of nontraditional students, attempts were made to include students from both day and night sections in each group. In addition, the groups were constructed to balance participation across instructors. Within each section of the course, a random process was used to select specific student participants. After selecting a representative sample, the researchers corresponded with the prospective students via telephone and letter. Not every student contacted was willing to participate. On the other hand, several students seemed quite interested in having their views heard. To further encourage participation, a small cash incentive of $25.00 was offered to each participating student. This was funded by a grant supported by Oakland University. While the payment was somewhat less than a business would pay to focus group participants (the local minimum is $75.00), the incentive did help ensure that students took the task seriously. Once a student expressed willingness to participate in the focus group, a follow-up written confirmation of the focus group's location, date and format was provided. A list of possible discussion topics/questions was also provided; the students were asked to review them prior to their session. This list of questions was intended as the starting point to get people involved. The questions were generally quite structured and had straightforward answers. The researchers fully expected that the discussions in the sessions would go well beyond an answer to these questions. Appendix A lists the questions distributed. The objective of this project was to obtain information from students about perceived impediments to their learning. Course content was not addressed, since it was believed that students did not have the level of experience such that they could appropriately judge course content.
462
G.J. Dillon and L. A. Barclay
Rather, the sessions concentrated on students' experiences and their attitudes toward the course delivery. By having students relate their experiences, ways of improving course delivery could be addressed. To avoid having participants feel inhibited in the groups, the services of an external marketing consultant who was not affiliated to the university in any way but who was experienced in conducting focus groups were secured. Marketing faculty at the university helped make this contact. The consultant, a recent graduate of an MBA program, had an understanding of the introductory accounting course. In the confirmation letter sent to student participants, the moderator was described as an external expert, not a faculty member.
Conduct of the Focus Groups It is best to conduct the focus groups in a nonthreatening location. Refreshments are desirable. The session should he begun with nonthreatening questions. The moderator generally begins each focus group with a brief introduction of himself/herself, introduction of participants by first name only, and restating the purpose that was previously communicated to the participants by phone and letter contacts. As the session continues, the moderator attempts to involve individuals who are not active participants. The moderator may take notes and/or the session may be recorded. If recording is to be used, participants should be informed at initial contact. In structuring each of the focus group sessions (pass, nonpass, withdraw), the researchers attempted to put the participants at ease. The sessions were held in a room in the student center, using a large square table that could accommodate ten to twelve participants and the moderator. Refreshments were available for early arrivals and throughout the session. Participants had been told at the initial contact that the session would be taped; the tape recorder was placed in the center of the table. One of the researchers opened each session by introducing the moderator and explaining the format again. Administrative details (arrangements for payments, completion of an 'informed consent' form by each participant to comply with university policy) were completed. The researchers then left the room. There was no observation of the session; only the moderator and participants remained. Each session lasted approximately two hours (Churchill, 1991, p. 138), which was long enough to give every participant an opportunity to become comfortable in participating. Before and after each session, the researchers met with the moderator. Prior to the sessions, the broad topics that should be addressed were discussed. Strategies for building on the distributed questions were also considered. After each session, the moderator was
Student Focus Groups
463
debriefed as he noted his impressions from the discussions. The researchers also independently reviewed the tape recording of each session.
Evaluating and Reporting Results Both Krueger (1994) and Vaughn et al. (1996) provide detailed discussions of data analysis related to focus groups. Often, analysis requires consideration of many pages of transcripts. Approaches to analysis include coding statements into predetermined categories, and developing data categories (or 'themes') based on observed responses and then using the themes to generate summary statements. It is important to note that when analyzing the data, groups may not have a consensus opinion. Focus groups deal with perceptions. Groups may have majority and minority positions (Vaughn et al., 1996) and this must be considered in developing action recommendations. With respect to the case described here, it is recognized that specific findings may be unique to each institution. Accordingly, only a brief summary of the findings for this specific case are presented. As noted above, focus group data can be reported in different ways. In this case 'themes' (Krueger, 1994, p. 166) or the "essence of the interviewees' responses" (Lederman, 1990, p. 124) was used. In order to help identify themes, it is useful to count the number of times an issue is raised (Vaughn et al., 1996). An issue only raised once and not taken up by other focus group participants may not be as salient as an issue discussed by many participants. Once issues are collected, they can be grouped into themes or overarching categories. For example issues such as "this course is hard," "lots of people flunk this course," and "this course keeps people out of the business school" are all 'course' comments, while a comment such as "I study alone" is not in this category. The topics discussed and the student perceptions generally fell into five themes': 1. 2. 3. 4. 5.
Preconceptions about the course; Study techniques; Use of available resources; Attitudes toward the course structure; and Interactions with faculty.
The researchers agreed with these 'themes' after they had independently reviewed the tapes. A rather high level of consistency between the perceptions of the 'pass group' and the 'nonpass group' was observed. The 'withdraw' group was somewhat different, possibly based on more limited experiences with the course. Further, withdrawal was not always related to academic difficulties. Every participant obviously did not have the same experiences
464
G.J. Dillon and L. A. Barclay
and opinions; the descriptions below summarize frequently-expressed positions. Preconceptions about the Course: Most students had heard that the course was difficult and required lots of work. Further, many had heard it was a 'weed-out' course for the School of Business Administration. Study Techniques: 'Nonpass group' students were less likely to study with fellow students. (While this study did not address the issue, we believe this lack of 'peer group support' to be an important determinant of lack of success.) The 'pass group' students claimed extensive time spent in course preparation. The 'nonpass group' and 'withdraw group' members were much more likely to fall behind early in the semester. Use of Available Resources: Many students, especially in the 'nonpass group', reported that they did not know what resources were available. The 'pass group' was more likely to use Supplemental Instruction. The 'nonpass group' perceived that the Supplemental Instruction student leader was not responsive to their needs. The 'nonpass group' stated that tutors "were not worth the effort"; they felt the learning gained did not justify the time spent. Attitudes Toward Course Structure: Most students disliked the common examinations. They were concerned that "someone other than their instructor was preparing the examination". Further, most felt the exams, especially the multiple choice questions, were too 'tricky'. Some felt class sessions were 'Right out of the book' and that attendance was not a valueadded experience. Interactions with Faculty: Students seemed unwilling to initiate contact with faculty, although no frequent reason was given. The 'nonpass group' perceived that faculty were predisposed to failing students. Some faculty members had quoted failure and drop rates in class. Some faculty members advised students doing poorly on the first exam to withdraw; students then believed that the faculty member was biased toward those so advised who did not withdraw.
Developing Action Recommendations Not all focus group research reports contain recommendations. As discussed earlier focus groups are often exploratory in nature, providing insights into additional, more focused research efforts. When recommendations or suggestions are made, the intended audience may need to be briefed on the focus group technique in order to better discuss options. In particular, the audience should be reminded that the perceptions of the focus group participants are being reported. Student perceptions may not be consistent with faculty perceptions.
Student Focus Groups
465
Information from the focus groups in the case was summarized and presented, along with recommendations, to the Accounting faculty for discussion. Recommendations included: • An expanded syllabus, to include more information for students about how to obtain support services available on campus, and .frequent reference to that syllabus throughout the semester. • Communication to students that all faculty review exams and suggest modifications before each common exam. (Subsequent to conducting the focus groups, the Accounting faculty decided to discontinue common examinations. Focus group information was only one of several factors considered in this decision.) • Justifying the importance of apparently minor wording differences in the 'real world'. A single word can dramatically impact the meaning of a contract. Explain to students that apparently 'tricky' questions have an underlying rationale. • A review and discussion of Supplemental Instruction with the Academic Skills Center. In particular, monitoring and immediate feedback for the student 'leaders' who conduct the Supplemental Instruction sessions. • Faculty should not directly recommend that students withdraw from the course. • Faculty should inform the students that the course will require considerable effort, but it is counterproductive to quote failure and withdrawal rates. The last two recommendations are consistent with Bandura's (1986) selfefficacy theory, which holds that self-efficacy can be influenced by enactive attainment, vicarious experience, verbal persuasion and physiological state. Of these four sources, two are relevant to the recommendations based on information secured from the focus groups. First, the faculty style and tone of communication can either increase or decrease student self-efficacy. Messages which indicate failure are likely to reduce efficacy and lead to increased likelihood o f not passing the course. While verbal persuasion is not the strongest source of self-efficacy, it can be very important in the early stages of a class as it can set up a self-fulfilling prophecy. The second influence, vicarious experience, has specific tie-ins with supplemental instruction and peer tutoring. "Seeing or visualizing other similar people perform successfully can raise self-precepts of efficacy in observers that they too possess the capabilities to master comparable activities." (Bandura, 1986, p. 399) Encouraging students to avail themselves of Supplemental Instruction and other support systems can lead to increased self-efficacy and a willingness to continue in the class. The recommendations for improved communication with students are consistent with research on improving communication effectiveness.
466
G.J. Dillon and L. A. Barclay
Cherrington (1994) specifically details strategies to help improve communication. Repetition of communication episodes in particular is relevant to this situation. Students need to be reminded of support services throughout the term since they may forget such information. Because this was the researchers' first experience making recommendations based on focus group data, it was unclear how the information would be received. In a systematic program assessment, it would be essential for the faculty to commit to the evaluation process before it took place. As with more traditional educational assessment, such as student evaluations of faculty, the researchers found there to be variability in the acceptance of the recommendations within the faculty group.
T R A N S F E R A B I L I T Y TO O T H E R I N S T I T U T I O N S As noted above, this methodology should transfer well to other accounting programs. The specific information gathered from focus groups in this case may be unique to this institution, and the specific recommendations to the Accounting faculty may not be appropriate to others. However, the use of focus groups will allow other institutions to systematically obtain information from their students, alumni and employers that is likely richer than that provided by other techniques, and thus will likely result in sound recommendations appropriate for that program. As a result of the experience, the following recommendations are offered to anyone using focus groups: 1. Do some initial research on focus groups. Vaughn et al. (1996) is an excellent source and contains a check list for helping those unfamiliar with the technique. Also, contact with marketing colleagues may provide additional insights. 2. Decide on the target issues to be addressed and the purpose and use of the focus groups. This definition will help you decide on the types and number of groups to conduct. 3. Do not use Accounting faculty as focus group moderators. This may tend to inhibit participants and perhaps reduce the validity of the information gathered. Further, Accounting faculty may have difficuRy remaining objective. The purpose of the moderator is to gather information regardless of its objectivity, not argue the facts. 4. Do guarantee confidentiality to participants. Students may be inhibited in responses if they believe that faculty will hear their responses, especially if they expect to have the same instructors in the future. Accordingly, students need to be assured that no one besides the project leaders will hear their individual responses. Only a group
Student Focus Groups
5.
6.
7.
8.
467
summary should be presented to the Accounting faculty. In fact, individual participants should not be identified. While in the case study a small honorarium was provided to participants, the nature of the student body at other institutions and the timing of the sessions may make this unnecessary. On a commuter campus, it is difficult to attract students back to campus when they do not have classes scheduled. Know your audience. For example, alumni and employer groups may be more readily available for evening sessions. In those cases, dinner could be provided. Do follow up with reminders before each session. In the case study, three contacts were made with each participant, including a phone call one to three days before the session. There were still some noshows. Be aware o f policies at your institution concerning human subjects research. For the case study, approval was obtained from the institution, and then each participant signed an 'informed consent' form. You may wish to publicize the results of such efforts in student and alumni communication media (newsletters, newspapers, etc.).
CONCLUSION The use of focus groups in the case study provided additional assessment information from students. This information was not readily available on student course evaluations. Based on the data collected, information and recommendations were provided to the Accounting faculty and the Academic Skills Center. This article also identifies several sources (in particular, Vaughn et al., 1996, and Krueger, 1994) that provide a useful description of the focus group technique. This technique can be usefully applied to other research questions and at other institutions.
REFERENCES Bandura, Albert. (1986) Social foundations oJ Thought and Action, a Social Cognitive Theoo', Englewood Cliffs, NJ: Prentice-Hall, Inc. Buttram, Joan L. (1990). Focus Groups: A Starting Point for Needs Assessment, Evaluation Practice, II(3), 207-212. Cherrington, David J. (1994) Organizational Behavior (2nd edition), Allyn & Bacon. Churchill, Gilbert A. (1991) Marketing Research (5th edition), Dryden Press, Chicago. Dean, James W. Jr., & Evans, James R. (1994) Total Quality: Management, Organization, and Strategy, St. Paul: West Publishing Co. DeMong, Richard F., Lindgren, John H. Jr., & Perry, Susan E. (1994). Designing an Assessment Program for Accounting Issues in Accounting Education, 9(1), 11 27.
468
G . J . Dillon and L. A. Barclay
Inglis, Robert C. (1987) In-depth Data: Using Focus Groups to Study Industrial Markets, Business Marketing, November, pp. 79-82. Krueger, Richard A. (1994) Focus Groups." ,4 Practical Guide Jor Applied Research (2nd edition), Thousand Oaks, CA: Sage Publications. Lederman, Linda C. (1990). Assessing Educational Effectiveness: The Focus Group Interview as a Technique for Data Collection, Commun. Educ., 38, 117-127. Martin, Deanna C., & Arendale, David C. (1992) Supplemental Instruction: Improving FirstYear Student Success in High-Risk Courses, National Center for the Freshman Year Experience, University of South Carolina. Merton, Robert K. (1987). The Focused Interview and Focus Groups, Continuities and Discontinuities, Public Opinion Quarterly, 51, 550-566. O'Donnell, Judith M. (1988). Focus Groups: A Habit-Forming Evaluation Technique Training and Development Journal, 42(7), 71-73. Seymour, Daniel T. (1992) On Q, Causing Quality in Higher Education, American Council on Higher Education, MacMillan Publishing Company. Sink, David W. (1991). Focus Groups as an Approach to Outcomes Assessment Am. Rev. Public Administration, 2•(3), 197-204. Trachtenberg, Jeffrey A. (1987) Listening the Old Fashioned Way, Forbes, October 5, pp. 202204. Vaughn, Sharon, Schumm, Jeanne Shay, & Sinagub, Jane (1996) Focus Group Interviews in Education and Psychology, Thousand Oaks, CA: Sage Publications. Wingspread Group on Higher Education (1993) An American Imperative." Higher Expectations .for Higher Education, The Johnson Foundation, Inc., Racine, WI.
APPENDIX
A
Possible Topics of Discussion at Focus Group 1. Had you talked about ACC 200 with friends and peers before you started the class? What were your expectations of the class going in? 2. Did you have a study group or did you study alone? 3. Were you in the Supplemental Instruction sections? 4. Did you use a tutor? (private, Beta Alpha Psi) 5. How much time did you spend on homework outside class? 6. How often did you see your instructor during his/her office hours? 7. Did you sit in on sections of ACC 200 other than your own? 8. Did you have friends in class you could call on if you missed a session'? 9. How did you feel about your progress prior to the first exam? 10. Did exams cover material from lecture and text?