- Email: [email protected]
Focus groups: An important research technique for internal evaluation units
Articles
Focus Groups: An Important Research Technique for Internal Evaluation Units BARBARA POITRAS DUFFY
INTRODUCTION
The FBI’s Office of Planning, Evaluation and Audits (OPEA) has had the primary responsibility for conducting internal program and policy reviews through management studies and evaluations for the past twenty years. These reviews are an important management tool as the FBI seeks to maximize the efficiency and effectiveness of the operations of 56 field divisions and headquarters components during an era of budgetary constraints. OPEA conducts evaluations which assist executive management with the administrative, organizational, and operational aspects of the FBI. Depending on the nature of the evaluation, a variety of methodologies are used by OPEA, including qualitative strategies. This article presents the FBI’s use of focus groups as a tool of internal evaluation, in an environment where credibility is key to achieving meaningful cooperation, and identifies issues for consideration by other evaluators interested in its use.
DEFINITION
AND BRIEF
HISTORY
OF THE FOCUS
GROUP
A focus group is a small group interview where eight to twelve individuals are encouraged to interact and share information. Sharing is facilitated by group discussion among individuals with similar characteristics (Stewart & Shamdasani, 1990). Focus groups can be exploratory or confirmatory in nature; however, recent research on the application of focus groups in a needs assessment suggests that these groups may be “well suited for defining the status quo,” but not particularly beneficial in identifying potential solutions to problems (Buttram, 1990).
Barbara Poitras Dtiy l Management Washington, DC, 20535.
Analyst,
Federal
Bureau of Investigation,
Evaluation Practice, Vol. 14, No. 2, 1993, pp. 133-139.
9th Street & Pennsylvania
Copyright
ISSN: 0886-1633
@ 1993 by JAI Press, Inc.
All rights of reproduction
133
Ave., N.W.,
in any form reserved.
134
EVALUATION PRACTICE, 14(2), 1993
Focus groups produce qualitative data, most often provided as a transcript of participant comments (Merton et al., 1990). Unlike the Delphi or nominal group processes which are designed to produce consensus, focus groups facilitate an open exchange of participant perceptions, opinions, feelings, or reactions (Krueger, 1988).
THE INTERNAL
EVALUATION UNIT’S FOCUS GROUP INTERVIEW TECHNIQUE
During the past three years, OPEA staff members have conducted numerous focus group interviews with FBI employees assigned worldwide. These interviews have explored a wide range of issues such as reactions to changes in pay policies, identification of characteristics of successful FBI Special Agents, assessments of employee support for divisional reorganizations, needs assessments to establish policies to permit outside employment (second jobs), and exploration of alternative strategies to provide support (typing) services in FBI field offices. Although many divergent issues have been explored, a basic strategy for conducting the focus group interview is usually followed. Ensuring the success of this strategy is the credibility of OPEA evaluators, who, as internal evaluators and fellow members of the organization, understand the organizational policies and practices of the FBI. The focus group strategy of OPEA and considerations about its use are presented below. Factors to Consider Before Using Focus Groups Once an evaluation is initiated prepares a data collection strategy. if focus groups should be used. 1.
and its focus clearly identified, the evaluation team Two primary factors are considered when deciding
Resources: How much time is available before management and what resources are available for the evaluation?
needs the information
Focus groups are very efficient as several individuals can be quickly consulted in a short amount of time as compared to the time required to conduct multiple individual interviews or for the preparation, administration, collection, and analysis of traditional survey questionnaires. Additionally, conducting several focus groups within a field office helps to maximize the number of contacts, while minimizing OPEA travel costs and the disruption of regular field office operations. On average, each focus group session ranges between one and two hours and requires only a tape recorder and some tapes. In some instances, OPEA staff members are requested to assess the organizational climate with respect to a particular policy or issue, or to determine if a more comprehensive evaluation is warranted. In this context, the use of focus groups provides OPEA managers with information which can be used to allocate evaluation resources across multiple requests. 2.
What is the nature
of the research
question
to be addressed
by the evaluation?
The focus group questions of OPEA are typically exploratory in nature for identifying general issues of concern or employee reactions to agency policies. The most effective
Focus Groups: An lmportant Research Technique for lntemal Evaluation Units
135
questioning routes begin with an unbiased, open-ended question and works toward focusing on the specifics of the issue under examination. It is important that these questions be presented to the group in an objective manner, as participants will often try to please the facilitator by providing information which the group thinks is desired. When appropriate, OPEA also uses “confirmatory” focus group questions to validate data collected from other sources, such as written surveys, document reviews, or individual interviews. For example, upon review of surveys completed by FBI personnel assigned to the New York office, it appeared as though a large number of respondents would support the establishment of a new policy to assign a fixed tour of duty to the office in lieu of additional pay enhancements. Because this alternative would have a significant effect on employees assigned to New York and save the FBI a considerable amount of money, it was necessary to confirm this information. Due to tight time constraints-the Office of Personnel Management was waiting for a project status report-focus groups were conducted with some survey respondents. These focus groups revealed that employees were in fact trying to communicate their general concerns about the FBI’s transfer policy to executive management to encourage them to reevaluate this policy. Preparation for Focus Group Sessions
Once it has been determined that an evaluation lends itself to the focus group technique, the individual responsible for the evaluation typically drafts questions in consultation with other team members and the unit chief. A few carefully constructed openended questions are prepared in advance of each session. A funneling approach is used in the questions, beginning with a general question at the start of the session, followed by a succession of follow-up questions, eventually leading to the primary research question. As the most important information tends to surface near the end of each session, the sooner the group works towards the key issue, the greater the amount of information obtained. The first question should be one that all participants can answer to make them comfortable and begin contributing to the group. Once these questions have been refined, plans are made to conduct the sessions. Typically, the evaluator responsible for the study serves as the focus group facilitator and is assisted by another team member. Before each session, the facilitator prepares a brief statement to welcome the group, outlining the criteria for selection of group members, identifying an overall objective, and establishing the ground rules of the session, especially the need for individuals to speak one at a time, and indicating that there are no right or wrong answers to the interview questions. Participants are also assured that their names will not be linked to any comments. Typically, for each research question or issue examined, multiple focus group interviews are conducted. For example, if an issue is relevant to all FBI field divisions, then six to eight offices will be purposively selected to represent geographic regions or operational program areas, such as drugs or financial crimes. In each of these selected field divisions, three or four sessions with eight to twelve participants in each group will then be arranged. Familiarity with the organizational culture allows OPEA staff members to quickly determine selection strategies that will most efficiently yield qualified participants. The room in which the sessions are conducted is usually arranged in a U-shaped table configuration to encourage eye contact between the facilitator and focus group
136
EVALUATION PRACTICE, 14(2), 1993
participants. Each participant is seated along the outside perimeter of the table while the facilitator and assistant sit together in the front of the room. Although the facilitator leads the discussion, the assistant sits in front of the group as well because some participants are distracted by the presence of an individual, not actively involved in the process, sitting in the rear of the room. Selection Issues Depending on the nature of the inquiry, individuals selected for participation in focus groups may be purposively or randomly selected from the population they represent. For example, when examining the characteristics associated with successful FBI Special Agents, it was important to solicit information from individuals recognized as successful Special Agents. As a result, it was necessary to purposively select these participants with the assistance of management from each affected field office. However, when examining the possibility of modifications to the FBI’s pay policy, a random sample from a roster of all employees was appropriate to assess general reactions to proposed policy changes. In advance of each session, focus group participants are usually advised in writing of their selection and notified of the place, date, and time. Supervisors are also advised of the interview sessions and asked to release employees from their regular work assignments. As most focus groups are conducted during regular business hours in FBI offices, few attendance problems have been experienced. OPEA staff work closely with the managers of affected field offices or FBI Headquarters components to make administrative arrangements for these sessions. Key to the credibility of OPEA personnel as internal evaluators is the open communication with the focus group participants and their supervisors. OPEA personnel are candid about the nature of their studies and managers are well briefed in advance on the objectives of the focus group sessions. Although they are not provided the details of these sessions (i.e., who said what), they are afforded an exit conference to learn about the issue areas addressed, some general information which was discussed, and how this information confirms or refutes other data collected to date. Although it has been suggested that it may be preferable to use participants who do not know each other (Krueger, 1988), this is not always possible nor in fact desirable when conducting focus groups within the FBI. Despite the size of the FBI, (some 20,000 employees), the organization is a tightly knit community, due to rotational transfer policies, training programs, and career development opportunities. As a result, it is usually difficult to select a group of individuals within an office who do not know each other, especially if they have been assigned there for some period of time. Quite the contrary, the presence of an individual who is not known to others in these focus group sessions has been found to have an inhibiting effect on the other participants, as the desire for continued confidentiality once the group disbands cannot be assured. As a result, some individuals refrain from full participation. For example, some FBI employees are assigned to smaller offices or Resident Agency (RA) locations. Employees assigned to RAs which are geographically distant from a division’s headquarters city may not always be well known to each other. OPEA frequently conducts focus groups which include employees assigned to various RAs within a field division. Prior to these sessions, participants often query each other in an attempt to identify the names and assignments of the other participants. However, it is not always possible or desirable to exclude these
Focus Groups: An lmpoyfanf Research Technique joy lnfeynal Evaluation Units
137
individuals from focus groups. In an effort to address this problem when selecting or assigning individuals to groups, OPEA staff members try to “match up” some of the characteristics of participants to help ensure greater homogeneity within groups. For example, if Special Agents are selected from a variety of locations in the same investigative program area, such as drugs, there is a greater likelihood that they will be known to each other and more quickly accepted by the other participants. Facilitation
Issues
While it is important to establish an environment conducive to a frank and open discussion, it is important to remember that participants take important cues from the facilitator and assistant. As a result, the initial “small talk” among participants and the facilitator which tends to occur naturally prior to the beginning of a session can significantly affect the success of the group interview. For example, prior to a focus group conducted in a large field division in California, the facilitator made a general comment about the desirability of living in a warm climate. Unpredictably, one participant was offended by this comment, and accused the Washington D.C.-based facilitator of “having an East Coast bias” which would preclude an understanding of the problems associated with working in California, in particular, the high cost of living. As a result, this individual refrained from participation throughout much of the session, adversely affecting the other participants. Our experiences have demonstrated that once an individual begins to talk, his/her continued participation is virtually guaranteed. As a result, it is essential to encourage everyone’s participation from the beginning. We have found that if individuals are late for sessions, and do not hear the introductory remarks or participate in the opening question, they do not successfully bond with the rest of the group and usually make negligible contributions to the group. These individuals often disrupt those around them because they do not understand what is going on. As a result, we have informally adopted a policy that late arrivals will be met at the door by the assistant and politely advised that the group discussion has already begun and while we appreciate their interest, because they have arrived late, their attendance is no longer required. Documenting
the Focus Group Discussion
The use of audio recording equipment has been hotly debated within OPEA, and mixed results with its use have been obtained. When conducting sessions where classified information was discussed, audio recorders were not used to simplify data handling. During sessions which became emotionally charged due to the sensitivity of the issue being discussed, audio recorders were turned off and not used, once it was determined that the discussion was being inhibited. This inhibition is often demonstrated by respondent references to the equipment, by the provision of caveats before each comment, or gestures toward the equipment. In these instances, flip charts were placed in front of the group, and key points outlined as they were presented. This procedure assured participants that their comments were being accurately and anonymously recorded, thereby reducing the anxiety associated with the use of electronic recorders. In groups where more generic, less sensitive issues were discussed, such as the development of a career path for Special Agent attorneys or the role of the supervisor in FBI field offices, the use of audio recordings
138
EVALUATION PRACTICE, 14(2), 1993
provided a rich source of qualitative information for analysis and incorporation in OPEA reports. At the conclusion of each session, supplemental data collection instruments are disseminated to each participant. These are distributed at the end because by this time participants have developed some rapport and comfort level with the facilitator and assistant and are receptive to providing more personal information. Typically, these forms solicit basic demographic information, such as the year the participant entered the FBI, program area of assignment (such as Drugs or White Collar Crime), and the number of times transferred by the FBI. Additional questions are sometimes included to solicit more information on the subject addressed during the session. Data Analysis Immediately upon conclusion of the focus group sessions, tapes and notes are transcribed and demographic data from the supplemental forms are summarized. If many focus groups have been conducted resulting in a large amount of data to be processed, an off-the-shelf computer package such as R-Base (1990) is used to manage and analyze the data set. Once transcripts have been completed, the responses are reviewed, one question at a time. An overall statement is crafted to reflect the group’s response to each issue and representative quotations are selected for eventual inclusion in a report or briefing. The use of commercially available computer packages such as the ZyIndex (1988) to analyze the transcripts obtained during the focus groups, has been debated within OPEA. Although they are helpful in identifying the frequencies of words or phrases used by participants, there is a concern that the use of these packages may take comments out of their context. As a result, the intensity of comments, which can be a rich source of information, may be lost. Participant responses must be interpreted in terms of the discussion which proceeds and follows the response. Additionally, the internal consistency within a group is important, as participant perceptions are often changed as a result of the group’s interaction. It is important to identify if, when, and how these changes in perceptions occur subject to some of the same pitfalls associated with the traditional closedended questionnaire.
CONDUCTING FOCUS GROUPS IN THE LAW ENFORCEMENT COMMUNITY
The law enforcement environment is a tightly knit community where membership and credibility are essential to obtaining focus group participation. Additionally, security concerns are generated in some agencies, making it very difficult for outside evaluators to obtain unrestricted access to the necessary data. For example, at the outset of virtually every focus group session that OPEA has conducted with FBI Special Agents and the professional support staff, the facilitator and assistant are asked whether they are FBI employees or external consultants. The complete cooperation of these groups is almost never achieved until this question surfaces and is satisfactorily resolved. Sometimes focus group members take this screening a step further, querying the facilitator and assistant about their tenure within the FBI, the types of cases they have worked on, and the number
Focus Groups: An important Research Technique for internal Evaluation Units
139
of field divisions in which they have served. While cautious in revealing too much information about ourselves, we work to resolve the concerns of this group. Because OPEA performs an internal evaluation function, use of the focus group interview technique has been very successful. External evaluators seeking to use focus groups in law enforcement or similar environments are cautioned that they may not be fully accepted by group participants. As a result, the information which they obtain may be subject to the same pitfalls associated with the traditional closed-ended questionnaire. CONCLUSION Over the past several years, OPEA has conducted
numerous focus group sessions addressing a variety of issues. The use of this technique has yielded a tremendous amount of valuable data and has raised a number of implementation issues which have led to the refinement of this strategy, tailored to the needs of our specific operational environment. Some of the challenges to the success of focus groups confronted by OPEA are likely to be experienced by external evaluation consultants who seek to operate in a similar environment. REFERENCES Buttram, J. (1990).Focus groups: A starting point for needs assessment.Evaluation Practice, 11(3), 207-212.
Greenbaum, T.L. (1988). The practical handbook and guide to focus group research. Lexington, MA: Lexington Books. Krueger, R.A. (1990). FOCUSgroups. Washington, D. C.: American Evaluation Association Presession. Krueger, R.A. (1988). Focus groups: A practical guide for applied research. Newbury Park, CA: Sage Publications. Merton, R.K., Fiske, M., & Kendall, P.L. (1990). Zhefocused interview. N.Y.: The Free Press. Morgan, D.C. (1989). Focus groups as qualitative research. Newbury Park, CA: Sage Publications. Microrim, Inc. (1990). R-base users manual. Redmond, WA: Author. Stewart, D.W., & Shamdasani, P.N. (1990) Focus groups: Theory and practice. Newbury Park, CA: Sage Publications. Zylab Corporation. (1988). Zylndex. Arlington Heights, IL: Author.
Focus Groups: An Important Research Technique for Internal Evaluation Units BARBARA POITRAS DUFFY
INTRODUCTION
The FBI’s Office of Planning, Evaluation and Audits (OPEA) has had the primary responsibility for conducting internal program and policy reviews through management studies and evaluations for the past twenty years. These reviews are an important management tool as the FBI seeks to maximize the efficiency and effectiveness of the operations of 56 field divisions and headquarters components during an era of budgetary constraints. OPEA conducts evaluations which assist executive management with the administrative, organizational, and operational aspects of the FBI. Depending on the nature of the evaluation, a variety of methodologies are used by OPEA, including qualitative strategies. This article presents the FBI’s use of focus groups as a tool of internal evaluation, in an environment where credibility is key to achieving meaningful cooperation, and identifies issues for consideration by other evaluators interested in its use.
DEFINITION
AND BRIEF
HISTORY
OF THE FOCUS
GROUP
A focus group is a small group interview where eight to twelve individuals are encouraged to interact and share information. Sharing is facilitated by group discussion among individuals with similar characteristics (Stewart & Shamdasani, 1990). Focus groups can be exploratory or confirmatory in nature; however, recent research on the application of focus groups in a needs assessment suggests that these groups may be “well suited for defining the status quo,” but not particularly beneficial in identifying potential solutions to problems (Buttram, 1990).
Barbara Poitras Dtiy l Management Washington, DC, 20535.
Analyst,
Federal
Bureau of Investigation,
Evaluation Practice, Vol. 14, No. 2, 1993, pp. 133-139.
9th Street & Pennsylvania
Copyright
ISSN: 0886-1633
@ 1993 by JAI Press, Inc.
All rights of reproduction
133
Ave., N.W.,
in any form reserved.
134
EVALUATION PRACTICE, 14(2), 1993
Focus groups produce qualitative data, most often provided as a transcript of participant comments (Merton et al., 1990). Unlike the Delphi or nominal group processes which are designed to produce consensus, focus groups facilitate an open exchange of participant perceptions, opinions, feelings, or reactions (Krueger, 1988).
THE INTERNAL
EVALUATION UNIT’S FOCUS GROUP INTERVIEW TECHNIQUE
During the past three years, OPEA staff members have conducted numerous focus group interviews with FBI employees assigned worldwide. These interviews have explored a wide range of issues such as reactions to changes in pay policies, identification of characteristics of successful FBI Special Agents, assessments of employee support for divisional reorganizations, needs assessments to establish policies to permit outside employment (second jobs), and exploration of alternative strategies to provide support (typing) services in FBI field offices. Although many divergent issues have been explored, a basic strategy for conducting the focus group interview is usually followed. Ensuring the success of this strategy is the credibility of OPEA evaluators, who, as internal evaluators and fellow members of the organization, understand the organizational policies and practices of the FBI. The focus group strategy of OPEA and considerations about its use are presented below. Factors to Consider Before Using Focus Groups Once an evaluation is initiated prepares a data collection strategy. if focus groups should be used. 1.
and its focus clearly identified, the evaluation team Two primary factors are considered when deciding
Resources: How much time is available before management and what resources are available for the evaluation?
needs the information
Focus groups are very efficient as several individuals can be quickly consulted in a short amount of time as compared to the time required to conduct multiple individual interviews or for the preparation, administration, collection, and analysis of traditional survey questionnaires. Additionally, conducting several focus groups within a field office helps to maximize the number of contacts, while minimizing OPEA travel costs and the disruption of regular field office operations. On average, each focus group session ranges between one and two hours and requires only a tape recorder and some tapes. In some instances, OPEA staff members are requested to assess the organizational climate with respect to a particular policy or issue, or to determine if a more comprehensive evaluation is warranted. In this context, the use of focus groups provides OPEA managers with information which can be used to allocate evaluation resources across multiple requests. 2.
What is the nature
of the research
question
to be addressed
by the evaluation?
The focus group questions of OPEA are typically exploratory in nature for identifying general issues of concern or employee reactions to agency policies. The most effective
Focus Groups: An lmportant Research Technique for lntemal Evaluation Units
135
questioning routes begin with an unbiased, open-ended question and works toward focusing on the specifics of the issue under examination. It is important that these questions be presented to the group in an objective manner, as participants will often try to please the facilitator by providing information which the group thinks is desired. When appropriate, OPEA also uses “confirmatory” focus group questions to validate data collected from other sources, such as written surveys, document reviews, or individual interviews. For example, upon review of surveys completed by FBI personnel assigned to the New York office, it appeared as though a large number of respondents would support the establishment of a new policy to assign a fixed tour of duty to the office in lieu of additional pay enhancements. Because this alternative would have a significant effect on employees assigned to New York and save the FBI a considerable amount of money, it was necessary to confirm this information. Due to tight time constraints-the Office of Personnel Management was waiting for a project status report-focus groups were conducted with some survey respondents. These focus groups revealed that employees were in fact trying to communicate their general concerns about the FBI’s transfer policy to executive management to encourage them to reevaluate this policy. Preparation for Focus Group Sessions
Once it has been determined that an evaluation lends itself to the focus group technique, the individual responsible for the evaluation typically drafts questions in consultation with other team members and the unit chief. A few carefully constructed openended questions are prepared in advance of each session. A funneling approach is used in the questions, beginning with a general question at the start of the session, followed by a succession of follow-up questions, eventually leading to the primary research question. As the most important information tends to surface near the end of each session, the sooner the group works towards the key issue, the greater the amount of information obtained. The first question should be one that all participants can answer to make them comfortable and begin contributing to the group. Once these questions have been refined, plans are made to conduct the sessions. Typically, the evaluator responsible for the study serves as the focus group facilitator and is assisted by another team member. Before each session, the facilitator prepares a brief statement to welcome the group, outlining the criteria for selection of group members, identifying an overall objective, and establishing the ground rules of the session, especially the need for individuals to speak one at a time, and indicating that there are no right or wrong answers to the interview questions. Participants are also assured that their names will not be linked to any comments. Typically, for each research question or issue examined, multiple focus group interviews are conducted. For example, if an issue is relevant to all FBI field divisions, then six to eight offices will be purposively selected to represent geographic regions or operational program areas, such as drugs or financial crimes. In each of these selected field divisions, three or four sessions with eight to twelve participants in each group will then be arranged. Familiarity with the organizational culture allows OPEA staff members to quickly determine selection strategies that will most efficiently yield qualified participants. The room in which the sessions are conducted is usually arranged in a U-shaped table configuration to encourage eye contact between the facilitator and focus group
136
EVALUATION PRACTICE, 14(2), 1993
participants. Each participant is seated along the outside perimeter of the table while the facilitator and assistant sit together in the front of the room. Although the facilitator leads the discussion, the assistant sits in front of the group as well because some participants are distracted by the presence of an individual, not actively involved in the process, sitting in the rear of the room. Selection Issues Depending on the nature of the inquiry, individuals selected for participation in focus groups may be purposively or randomly selected from the population they represent. For example, when examining the characteristics associated with successful FBI Special Agents, it was important to solicit information from individuals recognized as successful Special Agents. As a result, it was necessary to purposively select these participants with the assistance of management from each affected field office. However, when examining the possibility of modifications to the FBI’s pay policy, a random sample from a roster of all employees was appropriate to assess general reactions to proposed policy changes. In advance of each session, focus group participants are usually advised in writing of their selection and notified of the place, date, and time. Supervisors are also advised of the interview sessions and asked to release employees from their regular work assignments. As most focus groups are conducted during regular business hours in FBI offices, few attendance problems have been experienced. OPEA staff work closely with the managers of affected field offices or FBI Headquarters components to make administrative arrangements for these sessions. Key to the credibility of OPEA personnel as internal evaluators is the open communication with the focus group participants and their supervisors. OPEA personnel are candid about the nature of their studies and managers are well briefed in advance on the objectives of the focus group sessions. Although they are not provided the details of these sessions (i.e., who said what), they are afforded an exit conference to learn about the issue areas addressed, some general information which was discussed, and how this information confirms or refutes other data collected to date. Although it has been suggested that it may be preferable to use participants who do not know each other (Krueger, 1988), this is not always possible nor in fact desirable when conducting focus groups within the FBI. Despite the size of the FBI, (some 20,000 employees), the organization is a tightly knit community, due to rotational transfer policies, training programs, and career development opportunities. As a result, it is usually difficult to select a group of individuals within an office who do not know each other, especially if they have been assigned there for some period of time. Quite the contrary, the presence of an individual who is not known to others in these focus group sessions has been found to have an inhibiting effect on the other participants, as the desire for continued confidentiality once the group disbands cannot be assured. As a result, some individuals refrain from full participation. For example, some FBI employees are assigned to smaller offices or Resident Agency (RA) locations. Employees assigned to RAs which are geographically distant from a division’s headquarters city may not always be well known to each other. OPEA frequently conducts focus groups which include employees assigned to various RAs within a field division. Prior to these sessions, participants often query each other in an attempt to identify the names and assignments of the other participants. However, it is not always possible or desirable to exclude these
Focus Groups: An lmpoyfanf Research Technique joy lnfeynal Evaluation Units
137
individuals from focus groups. In an effort to address this problem when selecting or assigning individuals to groups, OPEA staff members try to “match up” some of the characteristics of participants to help ensure greater homogeneity within groups. For example, if Special Agents are selected from a variety of locations in the same investigative program area, such as drugs, there is a greater likelihood that they will be known to each other and more quickly accepted by the other participants. Facilitation
Issues
While it is important to establish an environment conducive to a frank and open discussion, it is important to remember that participants take important cues from the facilitator and assistant. As a result, the initial “small talk” among participants and the facilitator which tends to occur naturally prior to the beginning of a session can significantly affect the success of the group interview. For example, prior to a focus group conducted in a large field division in California, the facilitator made a general comment about the desirability of living in a warm climate. Unpredictably, one participant was offended by this comment, and accused the Washington D.C.-based facilitator of “having an East Coast bias” which would preclude an understanding of the problems associated with working in California, in particular, the high cost of living. As a result, this individual refrained from participation throughout much of the session, adversely affecting the other participants. Our experiences have demonstrated that once an individual begins to talk, his/her continued participation is virtually guaranteed. As a result, it is essential to encourage everyone’s participation from the beginning. We have found that if individuals are late for sessions, and do not hear the introductory remarks or participate in the opening question, they do not successfully bond with the rest of the group and usually make negligible contributions to the group. These individuals often disrupt those around them because they do not understand what is going on. As a result, we have informally adopted a policy that late arrivals will be met at the door by the assistant and politely advised that the group discussion has already begun and while we appreciate their interest, because they have arrived late, their attendance is no longer required. Documenting
the Focus Group Discussion
The use of audio recording equipment has been hotly debated within OPEA, and mixed results with its use have been obtained. When conducting sessions where classified information was discussed, audio recorders were not used to simplify data handling. During sessions which became emotionally charged due to the sensitivity of the issue being discussed, audio recorders were turned off and not used, once it was determined that the discussion was being inhibited. This inhibition is often demonstrated by respondent references to the equipment, by the provision of caveats before each comment, or gestures toward the equipment. In these instances, flip charts were placed in front of the group, and key points outlined as they were presented. This procedure assured participants that their comments were being accurately and anonymously recorded, thereby reducing the anxiety associated with the use of electronic recorders. In groups where more generic, less sensitive issues were discussed, such as the development of a career path for Special Agent attorneys or the role of the supervisor in FBI field offices, the use of audio recordings
138
EVALUATION PRACTICE, 14(2), 1993
provided a rich source of qualitative information for analysis and incorporation in OPEA reports. At the conclusion of each session, supplemental data collection instruments are disseminated to each participant. These are distributed at the end because by this time participants have developed some rapport and comfort level with the facilitator and assistant and are receptive to providing more personal information. Typically, these forms solicit basic demographic information, such as the year the participant entered the FBI, program area of assignment (such as Drugs or White Collar Crime), and the number of times transferred by the FBI. Additional questions are sometimes included to solicit more information on the subject addressed during the session. Data Analysis Immediately upon conclusion of the focus group sessions, tapes and notes are transcribed and demographic data from the supplemental forms are summarized. If many focus groups have been conducted resulting in a large amount of data to be processed, an off-the-shelf computer package such as R-Base (1990) is used to manage and analyze the data set. Once transcripts have been completed, the responses are reviewed, one question at a time. An overall statement is crafted to reflect the group’s response to each issue and representative quotations are selected for eventual inclusion in a report or briefing. The use of commercially available computer packages such as the ZyIndex (1988) to analyze the transcripts obtained during the focus groups, has been debated within OPEA. Although they are helpful in identifying the frequencies of words or phrases used by participants, there is a concern that the use of these packages may take comments out of their context. As a result, the intensity of comments, which can be a rich source of information, may be lost. Participant responses must be interpreted in terms of the discussion which proceeds and follows the response. Additionally, the internal consistency within a group is important, as participant perceptions are often changed as a result of the group’s interaction. It is important to identify if, when, and how these changes in perceptions occur subject to some of the same pitfalls associated with the traditional closedended questionnaire.
CONDUCTING FOCUS GROUPS IN THE LAW ENFORCEMENT COMMUNITY
The law enforcement environment is a tightly knit community where membership and credibility are essential to obtaining focus group participation. Additionally, security concerns are generated in some agencies, making it very difficult for outside evaluators to obtain unrestricted access to the necessary data. For example, at the outset of virtually every focus group session that OPEA has conducted with FBI Special Agents and the professional support staff, the facilitator and assistant are asked whether they are FBI employees or external consultants. The complete cooperation of these groups is almost never achieved until this question surfaces and is satisfactorily resolved. Sometimes focus group members take this screening a step further, querying the facilitator and assistant about their tenure within the FBI, the types of cases they have worked on, and the number
Focus Groups: An important Research Technique for internal Evaluation Units
139
of field divisions in which they have served. While cautious in revealing too much information about ourselves, we work to resolve the concerns of this group. Because OPEA performs an internal evaluation function, use of the focus group interview technique has been very successful. External evaluators seeking to use focus groups in law enforcement or similar environments are cautioned that they may not be fully accepted by group participants. As a result, the information which they obtain may be subject to the same pitfalls associated with the traditional closed-ended questionnaire. CONCLUSION Over the past several years, OPEA has conducted
numerous focus group sessions addressing a variety of issues. The use of this technique has yielded a tremendous amount of valuable data and has raised a number of implementation issues which have led to the refinement of this strategy, tailored to the needs of our specific operational environment. Some of the challenges to the success of focus groups confronted by OPEA are likely to be experienced by external evaluation consultants who seek to operate in a similar environment. REFERENCES Buttram, J. (1990).Focus groups: A starting point for needs assessment.Evaluation Practice, 11(3), 207-212.
Greenbaum, T.L. (1988). The practical handbook and guide to focus group research. Lexington, MA: Lexington Books. Krueger, R.A. (1990). FOCUSgroups. Washington, D. C.: American Evaluation Association Presession. Krueger, R.A. (1988). Focus groups: A practical guide for applied research. Newbury Park, CA: Sage Publications. Merton, R.K., Fiske, M., & Kendall, P.L. (1990). Zhefocused interview. N.Y.: The Free Press. Morgan, D.C. (1989). Focus groups as qualitative research. Newbury Park, CA: Sage Publications. Microrim, Inc. (1990). R-base users manual. Redmond, WA: Author. Stewart, D.W., & Shamdasani, P.N. (1990) Focus groups: Theory and practice. Newbury Park, CA: Sage Publications. Zylab Corporation. (1988). Zylndex. Arlington Heights, IL: Author.