Implications of continuous quality improvement for program evaluation and evaluators

Articles

Implications of Continuous Quality Improvement for Program Evaluation and Evaluators MELVIN M. MARK AND EDWARD PINES

INTRODUCTION Imagine that, as an evaluator, you are about to work for the first time in an organization that has been operating a continuous quality improvement (CQI) program. How is a “CQI organization” likely to differ from other organizations you have worked with that have not been experienced in CQI? Image that, as a practicing evaluator, you are concerned with broadening your skill base. You have noticed that many organizations you have contracted with are hiring CQI Coordinators. What is the relationship between CQI and evaluation? Is training in evaluation relevant for such a position? Would you as an evaluator have something special to bring to a CQI program? Imagine that, as a member of a small evaluation shop in a state social service agency, you have conducted a variety of types of evaluation, including summative evaluations of specific programs. Your new agency head, however, is a believer in CQI. She also believes that, in an organization committed to CQI, no other form of evaluation is necessary. How reasonable is her position, and how do you argue to the contrary? In this paper, we focus on several implications that CQI programs are likely to have for evaluation and evaluators. In doing so, we address the preceding questions, which are formulated as hypothetical questions but may be increasingly common in reality. More generally, we hope to stimulate evaluators to consider explicitly the implications of the CQI movement for program evaluation (and vice versa). This paper is organized as follows: A brief description of CQI is presented followed by descriptions of what is different about a CQI organization and how such an organization might influence evaluation. Next, we address a key question: Does a CQI organization need evaluators or other evaluation activities? We end with some brief conclusions. Melvin M. Mark . Department of Psychology, 417 Moore Building, The Pennsylvania State University, University Park, PA 16802; and Edward Piies, Industrial Engineering Department, New Mexico State University. Las Cntces, NM 88003. Ewltution

Practice, Vol. 16, No. 2, 1995, pp. 131-139.

Copyright * 1995 by JAI Press, Inc.

ISSN: 08861633

Au tights of reproduction in any form reserved.

131

132

EVALUATION PRACTICE, 16(2), 1995

WHAT IS CONTINUOUS

QUALITY IMPROVEMENT

CQI, often known as Total Quality Management (TQM), is a team-based management strategy that emphasizes the improvement of the processes an organization uses to deliver its products and/ or services. It developed from activities initiated in Japanese manufacturing organizations by American consultants such as W. Edwards Deming and Joseph Juran in the 1950s. These consultants claimed that productivity would increase automatically if an organization focused on improving the quality of its production processes and on meeting customers’ requirements. CQI’ encompasses the following elements: 1) a focus on internal and external customers’ needs; 2) training in, and continual practice of, systems analysis and improvement techniques on the organization’s processes by all staff members; 3) the use of a team approach to research, analyze, and implement quality improvements; and 4) monitoring of process performance to aid in and validate quality improvements; which should result in 5) implementation of quality improvements that help products and/ or services meet customer requirements. In addition, CQI is based on the assumption that top-level management must commit to and participate in the program to achieve success. A consensuallyderived statement of objectives for the organization drives CQI efforts. A CQI approach directs staff to focus on the process of delivering a product or service, with emphasis on prevention of poor delivery before the fact, rather than on inspection after product or service delivery. In theory, by focusing on the improvement of theprocess of delivery, improvement should result in the outcome, that is, the product or service. Although CQI efforts can focus on the organization’s primary service/product, and therefore on overall organization effectiveness, CQI often focuses on more specific servicedelivery mechanisms or processes within an organization. For example, within a social service organization, a given CQI effort might address billing or client flow but not necessarily actual service delivery. CQI is based on the assumption that organizations must recognize staff expertise in providing a service or product. Staff rather than external consultants are the primary actors in this approach. They are often given time to work on a CQI project in addition to their normal assignments. CQI “teams” are typically interdisciplinary, consisting of staff from several areas and several levels within the organization. Staff members are trained to study organizational process using systems engineering and statistical techniques. Additionally, teams are trained in the procedures of group problem solving. Their objectives are to identify areas for improvement and to identify and assess potential improvement methods. The training is generally intended for all levels of staff, not just a specialized subset or volunteer group. One result of this is increased awareness of the process through which the organization’s service is provided. When an organization implements CQI, it attempts to identify areas that are causing problems in quality. The problems may be identified by individual program staff,

Continuous Qualifylmpnnwment

133

customers, or management. They are studied by a CQI team. The team conducts “customer-vendor analyses.” These analyses allow staff to view their work as a series of exchanges between vendors and customers. By analyzing these interfaces, staff attempt to improve service or product delivery. The CQI team makes recommendations to management for process improvements or, in some cases, implements the improvements without further management action. The use of problem-solving teams such as Quality Circles or other employee involvement programs-an early manifestation of the CQI movement-first became visible in the United States in the 1980s. Striven (1991) noted that quality circles are likely to have a large impact only if they are implemented with a balance of power that allows workers to participate actively in decisions, and that quality control processes often fall short by focusing on the production function without highquality field-testing. The same points can be applied to CQI and lead to predictions about the (implementation) conditions necessary for CQI efforts to be successful. Interestingly, there are reports that not all CQI implementations are successful (e.g., Capper & Jamison,l993; Shoop, 1993). For example, a recent Washington Post article on the subject was entitled, “Totaled Quality Management” (Mathews, 1993, emphasis added). Nevertheless, because of the perceived success of CQI efforts in manufacturing settings, many educational, government, and other non-manufacturing organizations are currently implementing CQI programs. The literature includes numerous descriptions of CQI being implemented in a variety of educational (e.g., Coate, 1990) and government settings (e.g., Brown, 1992; Carr & Littman, 1990; Jablonski, 1991; Shoop, 1993). Advocates of CQI suggest important benefits. For example, Carr and Littman (1990) claim that a government organization adopting CQI could expect to save 10 to 20 percent of its operating budget by reducing poorquality performance. They support this estimate with anecdotal evidence. The prospect of such cost reductions is of course a motivating factor for implementing CQI. WHAT IS DIFFERENT

ABOUT A CQI ORGANIZATION?

The evaluator is likely to see a variety of differences between a CQI environment and a traditional, non-CQI environment. Our contentions about the impact of CQI for evaluation should be seen as hypotheses or “best guesses”; we believe the intersection of evaluation and CQI is too young for confident, definitive statements on the impact of CQI on evaluation. How is a CQI organization different from a traditional organization? In traditional organizational management systems: 1) power to make change is centralized in management; 2) there are no specified procedures for problem identification and solution generation; and 3) quality improvement efforts, if any exist, are typically outcome-oriented, involving attempts to inspect at the end of the process and to remove or repair poor quality products or services.

EVALUATION PRACTICE, 16(2), 1995

134

In contrast, in well-implemented CQI efforts: 1) teams composed of workers from various levels are empowered to make changes; 2) problem identification and solution generation are based on specified techniques that involve data collection and statistical analysis; and 3) quality improvement efforts are fundamental and process-oriented, and are concerned with moving toward consistent (low variability), highquality performance. HOW MIGHT A CQI ORGANIZATION INFLUENCE EVALUATION TASKS AND APPROACHES? Staff

If the CQI process has worked well, staff will expect ownership of organizational change efforts-for they have been experiencing such ownership. They are more likely to question approaches to research and evaluation since they have used the same or similar techniques in their CQI work. They may ask why other tools are to be used, may desire explanations of the advantages of an unfamiliar method relative to methods they have successfully used, and may want translation of unfamiliar concepts into their CQI terminology. In general, they are likely to desire an active role in the design and implementation of the evaluation. If staff with CQI experience actively participate in evaluation, advantages may follow, such as those Lipps and Grants (1990) cited for a participatory approach to evaluation: 1) identification of practical constraints and program modifications not in program documents; 2) motivation of staff; and 3) improved feedback of results and utilization. On the other hand, staff members’ heightened expectations for involvement in research and organizational change can be a doubleedged sword. For example, substantially more time and effort may be required to negotiate plans for evaluation activities, to justify procedures that differ from familiar CQI procedures, and to discuss findings and their implications. Imagine now that an evaluator is about to enter an organization that has a history of a failed CQI program. A likely reason for such failure is that central management was unwilling to share decision making authority with CQI teams. If the organization has experienced failure in publicized CQI efforts, staff are likely to be cynical about any kind of information-based organizational change technique, including evaluation, and they may be cynical about the likelihood of utilization of results. Consequently, they may not wish to participate in evaluation efforts, which they may see as futile, and so data collection efforts involving staff may be hindered. Tasks and Approaches

Upfront planning of evaluation activities should be easier in an organization with a successful history of CQIs, as should the implementation of some types of evaluation.

Continuous Quality lmpmvement

135

Evaluability Assessment (Rutman, 1980; Smith, 1989; Wholey, 1987, 1989) is a prime example. The development of a program theory (Bickman, 1990) or model can be significantly easier in a CQI environment for the following reasons: 1.

2.

3.

The organization’s objectives, and the processes by which its objectives are to be met, will often have been documented-this may include documentation of the program, its activities and resources, and the casual chain through which the program is thought to operate. Personnel at all levels will be trained in systems engineering techniques, such as flow charting and the documentation of a program and its service delivery processes. Documentation and analysis of program delivery methods will presumably have led to the construction of some type of model of the program (or a program theory).

In addition, CQI projects may have been conducted to determine potential measures of program delivery and performance, so that the impact of CQI improvement projects could be assessed. Such work may be valuable in specifying measures of program performance, which is another task in evaluability assessment (Wholey, 1987). Indeed, in some instances, prior CQI work may even substitute for nearly all of the activities embodied in evaluability assessment or program theory development. While CQI employs a different set of techniques than evaluability assessment, it may accomplish many of the same objectives, and may resolve the organizational deficiencies that evaluability assessment was designed to rectify. An organization with a failed CQI effort, however, is unlikely to have adequate documentation of the program, its activities and resources, and the casual chain through which the program is thought to operate. Moreover, staff may not be very helpful in developing a model of the program and its processes, because they are skeptical of the value of contributing to such efforts. Summative Evaluation, when implemented in an organization with a successful history of CQI, is likely to be easier for the reasons just noted in the case of evaluability assessment/ program theory (e.g., the facilitation of planning and of utilization). However, summative evaluation may also be more challenging because of the potential of threats to internal and construct validity. In some instances, CQI can operate as an internal validity threat (Cook and Campbell, 1979), hindering casual attribution. This is most likely when CQI teams make changes in service delivery processes that influence the policy or procedural change that is being evaluated. For example, image a quasi-experimental evaluation of the effect of arrest in domestic assault cases, with two communities assigned to different methods for dealing with such cases (cf. Berk & Newton, 1989). If a CQI effort to reduce police response times in general had taken place in the experimental community, this could be a serious internal validity threat. In other instances, CQI activities may raise questions about construct validity. This is likely if the CQI efforts target the particular process (or general program) being evaluated. For example, imagine an evaluation of the effects of a substance abuse prevention program for junior high school students. If a CQI effort led to substantial modifications in the nature of the program midway through the evaluation, questions

EVALUATION PRACTICE, 16(2), 1995

would arise about the proper way to characterize the intervention in conceptual terms (e.g., the prevention program might shift from a self-esteem model to a social skiIIs model because of changes brought about by a CQI team). To avoid such problems, the researcher might specify clearly that the intervention is not subject to improvement during its evaluation. However, this creates a potentially serious external validity problem. The researcher would be evaluating the effect of an intervention that would not be the intervention actually used in the future (Cronbach, 1982). A preferable approach (as in cases that do not involve CQI) is to avoid premature summative evaluation, by waiting until the program has reached a reasonably steady (if not final) state. That is, during a developmental phase, CQI-based program changes could occur, with a summative evaluation subsequently performed during a period in which, by mutual consent, no additional program changes of note are anticipated. Of course, the very nature of a CQI organization reminds us that traditional summative evaluations are only capturing estimates of program impact at particular points in time, and that programs are to some extent always changing. Stakeholder analysis should be enhanced when an organization has a history of successful CQI. The CQI approach recognizes a series of “vendor-customer transactions” within the process. It is possible that information about stakeholders who are not customers is available from program staff as well. Thus, information about stakeholders may have already been generated by CQI teams. On the other hand, the focus on vendor-customer transactions may lead staff in CQI organizations to focus on those stakeholders who either provide inputs to the program or directly receive its services, to the exclusion of other interested parties; if so, this could lead to a narrower view of program stakeholders. Internal, formative evaluation that is continuous (&riven, 1991; Love, 1983) may be hard to distinguish from CQI. CQI provides an organizational philosophy and a set of techniques aimed at improving organizational functioning; and it includes a data-based assessment of the effects of changes. Almost inherently, CQI is a process that is carried out internal to the organization (though sometimes with external consultants to help CQI programs begin). Love (1983) describes two major characteristics of internal evaluation in practice: (I) providing relevant and timely information on which managers can base decisions and (2) influencing behavior, including decisionmaking behavior. From this perspective, CQI can be seen as a form of internal evaluation in that it also focuses on providing information for making actual changes in organizations. However, CQI’s focus is more on process improvements, with staff on CQI teams empowered to make changes. We expect that this focus on staff, rather than managers, as decision makers is one with which most internal evaluators would feel comfortable. DOES A CQI-ORGANIZATION

NEED EVALUATORS

OR EVALUATION?

Evaluators can improve CQI programs by educating CQI teams about the methodological and analysis techniques and conceptual frameworks used by evaluators. For example, the validity threats framework (e.g., Cook & Campbell, 1979) for experimental and quasiexperimental design; strategies such as randomization, stakeholder analysis, needs assessment, and other evaluation practices; and techniques for questionnaire construction, interviewing, and data analysis and display may be useful. Knowledge of these and other

ContinuousQuality lmmt

137

techniques familiar to evaluators should increase the ability of CQI teams to seek problems, document production processes, plan experiments, and analyze data. Some adherents of CQI believe that it removes the need for any other form of evaluation. We believe, to the contrary, that there is a compelling argument for summative evaluation of CQI organizations, at least of tax-supported social programs. Without summative evaluation, a program may continue even though it is not effective. (See Browne & Wildavsky, 1983, who anticipate this issue in the context of internal evaluation). CQI teams could exert considerable time and energy improving the quality and delivery of aspects of a program that at its core is ineffective or inefficient. “Building a better Edsel” is a potential slogan to remind us of such possibilities. CQI efforts may increase the likelihood that effective services are efficiently provided, but they certainly do not guarantee it. To estimate the effectiveness of a program, summative evaluation is called for. There may be less need for summative evaluation among for-profit CQI-using organizations. In these, monitoring of sales, market share, client satisfaction and, above all, return on investment, may suffice. However, unlike the private sector, social programs commonly have no market test to indicate effectiveness (or more properly, effectiveness as perceived by consumers). And, even if they did, acquiring a market share does not indicate an intervention’s true effectiveness, as evidenced by the scores of diet companies that make millions while not accomplishing long-term weight loss. Society can ill afford to improve the “quality,” that is, marketability, of ineffective social and educational programs. CQI’s focus on the customer can lead to CQI team decisions that improve “quality” only in terms of the customer’s perception of the program. For example, academics sometimes worry that good theatrics can overcome substandard content in student ratings of teacher effectiveness, and many critics of television argue that popularity is not a strong indicator of quality. The fear here is that in public organizations, dedication to customer assessed quality improvement could come at the expense of other aspects of program performance, such as cost-effectiveness. For tax-dependent social programs, then, wellconducted summative evaluation, particularly with a consideration of stakeholders’ perspectives, serves an important function that CQI does not. For both social programs and for-profit organizations, there may be value, at least in some instances, in subjecting the CQI program itself to an evaluation-rather than or in addition to evaluating the organization’s services/products. An evaluation of a CQI program might include, but not be limited to, the following questions: Are there obstacles to successful implementation of CQI in the organization? What is the rate of participation in CQI at different levels of the organization? What sort of organizational problems have been subjected to CQI work and what issues have been “off limit”! What has the CQI cost, including time spent on CQI teams? What are the apparent effects of the CQI program, including specific effects of CQI team decisions (e.g., the changes made by a CQI team on a particular production process), and more general effects of having a CQI program (e.g., on employee attitudes and behaviors)? By addressing such questions, evaluation could lead to improvements in the CQI program, and to better informed decisions about the merits of the CQI endeavor. It may be increasingly possible to evaluate CQI in this way as it continues to move from the private sector into educational and social programs and government agencies. And, if they are shown to be effective, CQI programs might be implemented and maintained in more organizations (Leonard & Miller, 1992).

138

EVALUATION PRACTICE, 16(2), 1995 CONCLUSIONS

CQI offers a structured approach to the analysis of an organization’s process and its improvement. As a result, evaluators should find that the successful implementation of CQI will generally provide advantages, as well as some challenges. In addition, we believe CQI programs could be strengthened if those implementing CQI received additional training in evaluation research, including such items as threats to validity and stakeholder analysis, and ifCQ1 projects and CQI organizations were evaluated. As CQI efforts become more important, it will be more common for evaluators to have knowledge and experience in CQI.

NOTE

1. This discussion of CQI is based in part on treatments of quality improvement strategies in Car-r and Littman (1990) and Jablonski (1991), to whom the reader is referred for further information.

REFERENCES

Berk, R. A., & Newton, P. J. (1985). Does arrest really deter wife battery? An effort to replicate the findings of the Minneapolis Police Experiment. American Sociological Review, SO(April), 253-262. Bickman, L. (1990). Advances in program theory. San Francisco: Jossey-Bass. Brown, J. H. (1992). Erie Excellence Council applies Deming’s principles. National Productivity Review, JJ(2), 181. Browne, A., & Wildavsky, A. (1983). Should evaluation become implementation? In A. J. Love, (Ed.), Developing effective internal evaluation. San Francisco: Jossey-Bass. Capper, C. A., & Jamison, M.T. (1993). Let the buyer beware: Total quality management and educational research and practice. Educational Researcher, 22, 25-30. Carr, D. K., & Littman, I. D. (1990). Excellence in government. Arlington, VA: Coopers & Lybrand. Coate, L. E. (1990). TQM on campus: Implementing total quality management in a university setting. Business officer, 24(5), 26-35.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation. Chicago: Rand McNally. Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass. Jablonski, J. R. (1991). Implementing total quality management. Albuquerque, NM: Technical Management Consortium. Leonard, E. C. Jr. & Miller, M. J. (1992). The role of evaluation in increasing the usefulness of employee involvement programs. Evaluation Practice, 13(l), 9-13. Lipps, G. & Grant, P. R. (1990). A participatory method of assessing program implementation. Evaluation

Review, J4(4), 427434.

Love, A. J. (1983). The organizational context and the development of internal evaluation. In A. J. Love (Ed.), Developing effectiveness internal evaluation. San Francisco: Jossey-Bass. Mathews, J. (1993). Total quality management [No. 183,6/6/93, Hl]. Washington Post. Rutman, L. (1980). Planning useful evaluations: Evaluability assessment. Beverly Hills: Sage. Striven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.

Continuous Quality lmpmvement

139

Smith, M. F. (1989). Evaluability assessment: A practical approach. Boston: Kluwer Academic Publishers. Shoop, T. (1993). Headlong into quality. Government Executive, 25(3), 19-23. Wholey, J. S. (1987). Evaluability assessment: Developing program theory. In L. Bickman (Ed.), Usingprogram theory in evaluation. San Francisco: Jossey-Bass. Wholey, J. S. (1989). Introduction: How evaluation can improve agency and program performance. In J. S. Wholey, K. E. Newcomber and Associates (Eds.), Improvinggovemmentperformance. San Francisco: Jossey-Bass.