Evaluating an Alternative Funding Plan

HEALTHCARE YANAGEYENT

FORUM -

GESTION D E S SOINS D E S A N T C

ORIGINAL ARTICLE

a&:‘

5 “@L‘

by M. Anderson and J. Cosby

Abstract There have been increasing ieveis of interest shown across Canada in the potential application of alternative funding arrangements for academic health science centres. This article outlines the approach, and some preliminaryresults, of the alternative funding plan evaluation at Queen’s University. Followinga discussionof the AFP itself, the

rlX?-;T

hysicians and other key stakeholders across the country have been showing increasing levels of interest in the potential application of alternative funding arrangements for academic health science centres. This has been fuelled, in part, by the fiscal realities of the past decade and a growing concern that the existing fee-for-service system for physicians may not be the most appropriate funding system for the academic environment. This article outlines the approach, and some preliminary results, of the alternative funding plan evaluation at Queen’s University.

The Queen’s AFP

R6sum6 s de s a d universitaires i n t h m n i de plus en

iminaires d’un pian de de rechangeadopt6 h

The Queen’s University Medical Faculty has had an AFP since July 1994. The AFP’s single-funding envelope replaced the separate funding that had come previously from a wide range of sources, such as the fee-for-service payment mechanism to the Queen’s clinicians and other sources such as the Ontario Ministry of Health, Ontario Ministry of Community and Social Services and specific budgets of the three participating hospitals. The central goal of the AFP was to secure a sustainable, predictable fixed funding foundation for the Medical Faculty. This was viewed as a positive development by all key stakeholders in the AFP agreement. At a general level, the AFP was expected to provide new opportunities for improving clinical care and the education and research missions of the academic health science centre. Changes in organizational relationships were also anticipated. The AFP is the first of its kind in North America, although several

departments within academic health science centres in Canada have adopted alternative payment plans. 1,2 The 1994 agreement with the provincial Ministry of Health signalled the completion of the AFP’s developmental phase. In April of that year, following months of discussions and meetings, the agreement was approved by 78 percent of those clinical faculty who voted (95 percent of the 207 eligible full-time faculty members voted). With the exceptions of Surgery, Anesthesia and Ophthalmology, all departments gave overwhelming support to the introduction of the AFP. In May, the proposed agreement to establish a new governance structure for the AFF was approved by 80 percent of the full-time clinical faculty voters. The new governance structure, the Southeastern Ontario Academic Medical Organization (SEAMO), includes representation from the Clinical Teachers Association at Queen’s University, Queen’s University, Kingston General Hospital, the Religious Hospitallers of St. Joseph (Hotel Dieu Hospital) and Providence Continuing Care Centre (St. Mary’s of the Lake Hospital). Two clinical departments, Diagnostic Radiology and Oncology, and the basic science departments chose not to participate in the AFP. In the agreement with the Ministry of Health, SEAMO is required to conduct an evaluation of the AFP, although the nature and extent of this evaluation has never been specified. Several research projects have been completed or are currently underway, and several more are in the planning stages. By June 1999, when the five-year agreement will come to an end, both SEAMO and the Ontario

d‘une telle Bvaluation.z S U Y Y E R 1 8 8 8 , VOL. 1 1 , NO. 2

@

a

$$- 3

CTe 1888, VOL. 11, NO. 2

HEALTHCARE YANAQEYENT

FORUM -

QESTION DES SOINS DE SANTa

Ministry of Health will have a number of studies at hand that will have evaluated various aspects of the AFP.

Evaluation of the AFP An evaluation of the AFP provides a number of benefits to many stakeholders: the funders (Ontario Ministry of Health); the providers (clinical faculty, Queen’s and the hospitals); the public; medical students; and other academic health science centres that may consider adopting alternative funding plan arrangements. The Queen’s Health Policy Research Unit, in association with a number of clinical and non-clinical Queen’s faculty members, has been conducting the evaluation research over the past two years. SEAM0 established an Evaluation Steering Committee in October 1994, and soon thereafter four working groups were formed (clinical, education, research and organization). In May of 1995, an external Scientific Advisory Committee comprising prominent health services researchers from across the country was established to provide critical input and guidance to the evaluative process. This group has since met twice with those involved with the evaluation at Queen’s. The first phase of the evaluation consisted of three elements: (1) the development of a documents model; (2) a clinical faculty survey; and (3) a round of interviews with department heads.3At the same time, three outcome-oriented studies were also initiated and are now nearing completion. The first phase of the evaluation, more commonly referred to in the program evaluation literature as the evaluability assessment,“-*cast the evaluation in terms of its “program theory.” ~n other words, what were the expectations of the AFP prior to its implementation? Consensus and a clear understanding of the expectations of the AFP are essential if the plan is to be rigorously evaluated, and if the evaluation is to remain focused. The program theory also lays the basis for examining implementation issues. If, for example, one of the intended effects of the AFP was to improve the research activity in the Medical Faculty, what mechanisms were put in place to ensure this occurred? If nothing was done in the organizational environment during the five years to facilitate improved research activity, then perhaps this is related to the implementation process rather than the AFP theory per se. Similarly, if a particular study was to show a negative outcome, that may have more to do with the process of implementation rather than with the AFP theory itself. Hence the importance of understanding how the AFT was implemented, which, in evaluation circles, is often referred to as process evaluation.

Documents Model The first element of stage one was a comprehensive review of the AFP doc~mentation.~ An extensive range of documents (for example, minutes of meetings, correspondence, information sheets, progress reports) was synthesized to five binders of critical background material for the evaluation. These were summarized further in a documents summary report and a “documents model.” The documents model (figure 1) identifies the changes that were expected to occur following the introduction of the AFP.

S U M M E R 1990, V O L . 11. NO. 2

Rcorpltntba

d

1

The two key components of the AFP,stabilized funding and the formation of a new governance structure, were to have two immediate effects: a change in physician incentives (i.e., a move away from fee-for-service payment to a new compensation structure) and the reorganization of resources. The most fundamental change that was expected to occur was a shift in workload for the clinical faculty. In addition to changing clinical practice patterns, hospital efficiency was expected to improve, a more effective governance and management structure was to be put in place and new organizational relationships were to be developed within the southeastern Ontario region. The “outputs” of the AFP were (1) to enhance the tripartite mission of the Medical Faculty; (2) improve research and education capacity; and (3) maintain and, ideally, improve clinical practice. If these goals are met, Queen’s will prove to be a more innovative effective academic health science centre because of the AFP.It will be able to maintain or improve the health status of the region’s population and provide the basis for a regionally based model of health services planning and communications. That then, is the AFP theory. It provides the basis for the evaluation research that is being conducted. No specific time horizons were placed on the expected activities identified in the documents model. However, it will probably take longer than two years to see all the changes anticipated with the AFP theory. This is an important point when considering the way upon which the AFP is to be evaluated. For example, CTC 1 8 9 0 , V O L . 11, N O . 2

HEALTHCARE YANAGEYENT

FORUM -

although physician alternatives may have been rarely used after two years, that is not to say they will not be used in the future. Moreover, the AFP, while perhaps not creating immediate changes, has created an organizational environment that can facilitate change, which in the past simply may not have been possible. It has been suggested, for example, that the current profound restructuring of the hospital system in Kingston would not have proceeded as smoothly had there not been an AFP in place.

QESTION D E S SOINS D E S A N T B

So what do the survey results mean? First, it is evident that the AFP has had an impact on the professional activities of the clinical faculty. Second, although there appears to be a strong interest in pursuing further academic work, the opportunities to do so are still relatively limited given the need to maintain the required level of clinical work. A general theme prevailing throughout the qualitative responses of the survey was that the AFP was good in theory, but the reality, after two years, had not been what the clinicians expected.

Survey of Clinical Faculty The second element of phase one was a survey of clinical faculty.lo Two-hundred-and-fourfull-time clinical faculty members responded to the survey (84 percent response rate), including 32 non-AFP clinical medical faculty. Response rates were at least 70 percent for each of the 15 clinical departments. The survey provided data on perceived changes to the clinician’s workplace. It helped to determine the extent to which changes in workload had occurred after two years of the AFP, and the capacity for such changes in the future. It also provided important information on education and research activities that has subsequently been used by those working groups as they develop evaluative research in those areas. In addition, several questions were asked regarding the clinicians’ ability (self-efficacy) and opportunity to shift their workload. Some of the key findings were as follows: Sixty-four percent of respondents stated that there had been “no change” or “marginal change” in their work. Seventy-two percent of respondents felt there was “small” or “no opportunity” to reduce their clinical workload in the future while at the same time become more involved in academic activities (i.e., education and research). Nineteen percent said they had experienced a “significant” to “very significant” change in their work as a result of the AFP. Forty-four percent stated that there was “good” to “great opportunities” for them to be more involved in academic work. Only four percent felt that there were “good” to “great opportunities” to reduce their clinical activity in the future and become more involved in academic activities. Almost a quarter of all faculty were “uncertain” as to what levels of opportunity would exist in the future to enable them to change their workload. Thirty percent of all respondents wanted to become more involved in externally funded research, while 25 percent wished to be more involved in resident training. These results reflect the strong desire of clinicians to be more involved in academic activities. When asked what policy or program had had the greatest impact on their education activities, 34 percent said the recent introduction of the new curriculum. Another 25 percent cited internal department changes, while only 18 percent attributed the greatest impact to the AFP. With regard to the greatest impact on their research activity, 29 percent of clinicians said internal department changes, and 18 percent cited external funding for research. Just 16 percent attributed the greatest impact to the AFP.

’

S U Y Y E R 1998, VOL. 11, NO. 2

Department Head Interviews The document review had identified the AFP theory, while the survey provided baseline information on the nature of the changes that had occurred and the perceived interests and opportunities for clinicians to change their mix of professional activities. The third element of phase one was a round of interviews with clinical department heads.’* The department head interviews built on the earlier elements and drew particular attention to the issues surrounding the AFP’s implementation. As could be expected, the department heads all had varying expectations for the AFP and varying opinions on the actual progress after two years. Much of the perceived impact depended on each department’s interpretation of the AFP,its pre-AFP history, the type of clinical services it provides, and each department head’s desired level of implementation. Department heads felt that the slow, but perhaps inevitable, pace of policy development has contributed to the AFP not yet realizing its full potential. Several department heads felt that careful consideration should be given when developing an AFP as to how to accommodate changes in workloads that affect service departments such as Radiology and Ane~thesiology.’~ Department heads felt that the resource sharing that now takes place among departments is a significant and challenging feature of the AFP when compared to pre-AFP days when there was little sharing of resources. They also felt that the AFP is contributing to the development of a more academically oriented environment. Most department heads felt that the many issues that had to be dealt with in the fmt two years have prevented the AFP from realizing its full potential. The quality of clinical work, they note however, has changed very little, although the approaches to clinical work have definitely changed.

Other Evaluation Research In addition to the work done in phase one, two other studies have sought to develop baseline data that can be used to track changes due to the AFP. Baseline data are being compiled from the curriculum vitanun of the clinical medical faculty in order to understand how the research background of physicians influences their involvement in research. (This is important because it would be naive to think that just because an opportunity exists to do more research that all clinicians will do so. Rather, the types of training they received before arriving at Queen’s may have a strong influence on the extent to which they embrace a new academically oriented environment). Baseline data are also being gathered in a survey of third- and fourth-year medical students, using Ottawa as a comparison site. Students are asked about their learning environment and their understanding of an AFP. The study will be repeated later in the evaluation process.

elk

1998, VOL. 11, NO. 2

HEALTHCARE YANABEYENT

FORUM -

Three outcome-oriented studies on clinical service are also nearing completion. These have examined changes in waiting times, changes to practice patterns as they relate to endoscopic procedures, and the perceptions of changes in clinical practice due to the AFP by referring physicians and clinical faculty. Preliminary results from these studies indicate that while some areas of practice need closer examination, other areas tend to suggest that the AFP has had a positive effect on clinical care. And when asked how the AFP should be considered in the context of other changes currently affecting health care, referring physicians and faculty clinicians considered other issues such as funding cuts, bed closures, staff cutbacks and program changes as being more significant to the provision of quality health care. Several other outcome studies are in the developmental stage. They focus on the following - areas: changes in the profile of clinical service provision, pre- and post-AFP implementation; changes in clinical practice content within the Queen’s Academic Health Science Centre, pre- and post-AFP implementation; monitoring health status (e.g., premature mortality) in terms of socioeconomic status and indicators of hospital care (utilization, morbidity data, etc.); exploring access to service using waiting times from the primary care physician to the specialist, followed by specialist to hospital care; further qualitative work on the perceptions of change in clinical faculty, medical students and key stakeholders in the Medical Faculty and hospitals; and an examination of the nature and extent, if any, of “downloading” clinical care from consultants to community health care. The nature and extent of changes occurring in research and education are, perhaps, the most difficult to determine. This is especially true over a five-year period. The Queen’s Health Policy (QHP) evaluation team is in the process of collecting data on research activity, but it will take longer than the five years of the agreement to fully determine the extent to which the AFP has contributed to increasing research activity. The conventional research indices, such as volume of research grants, publication citations, conference presentations and so on, will be collected, but it is both unwise and unlikely a firm causal connection between research activity and the AFP can be established in just five years.’ Similarly, in a five-year period it may be difficult to associate changes in conventional education indices, such as performance in national exams, directly to the AFP. What is more important in the short term for research and education is to evaluate the extent to which the AFP has helped to enhance a “spirit of enquiry” within the Medical Faculty. That requires an understanding of how the AFP was implemented, the commitment of the Dean’s office to encouraging greater levels of research and improved teaching practices, and the perceptions of the clinical faculty as to their own ability (self-efficacy) and opportunity to engage in more research and educational activity.’.’’

S U Y Y E R 1 9 9 8 , VOL. 11, NO. 2

BESTION DES SOINS OE SANTC

Challenges for Evaluation Research Evaluation research thus far has brought to light a number of issues that affect a comprehensive examination of the AFP. Practically, evaluation should be embedded within an AFP from the outset, with a clear focus established prior to its implementation. Financial and human resources must be secured well in advance of the implementation date, and a clear commitment from key stakeholders must be made for the evaluation to proceed effectively. Data sources must be identified and the limits of the data addressed in the methodologies of the various pieces of evaluative research. Equally important is the need to disseminate evaluation results in a timely manner to all stakeholders, and to ensure that stakeholders’ needs are addressed. Ultimately, however, the depth and breadth of the evaluation will be a function of the level of funding made available for the work to be ~ompleted.’~*’~ A number of methodological challenges also need to be addressed. These include the ability to generalize from the research being conducted on the Queen’s AFF’; in particular, being able to control for confounding variables (some of which include hospital restructuring, the effect of clinical practice guidelines and the introduction of new teaching curriculums). Moreover, there is difficulty in determining a ‘ h e ” comparison site for the research. (Which other health science centres would best serve as comparison sites? How many do you use? How will the uniqueness of those sites influence any interpretation of the changes occurring in the AFP environment? Will data at the provincial level be sufficient for the purposes of comparison?) There are also practical questions critical to the research design that are based on the resources available for the research. What, for example, will be the cost incurred for using various comparison sites? To what extent will electronic data be used as compared to more time-consuming and costly chart abstraction? Ultimately, the studies conducted are mediated by the financial resources available to undertake the evaluative research. There have been no other evaluations of alternative funding plans, so new territory is being charted. As such, there is an enormous learning curve for all involved in the evaluation process.

Summary The AFP has been in place for three years. There have been changes in the SEAM0 environment, and many clinicians would like to take the opportunities, should they exist, to become more involved in academic activity. Assuming that it will not lead to a decline in the quality of clinical care, such a change can only be seen as ultimately improving the health care of Canadians. Much of the concern voiced to the evaluators so far has focused on how the AFP has been implemented. As many observers will note, these concerns reflect the growing pains of an organization that is attempting to change significantly its culture and the way it delivers its research, education and clinical missions. That will not happen overnight. Much work remains to be done, but perhaps the most challenging aspects have already been dealt with: the huge investment in time for developing and promoting the AFP concept to all stakeholders, and the equally significant investment in time and resources required to implement and manage the AFP.

a

CTC 1 9 9 8 , V O L . 11, N O . 2

H EA L T H C A R E Y A N A O EY E NT

FORUM -

No one would have been able to predict the time and energy required to put an effective AFP into place prior to the introduction of the Queen’s AFP.The next two years will show whether the theory has worked to its fullest potential. If it has not, then the evaluation research work may be able to determine whether that was due to the theory itself, the way it was implemented, or some other exogenous variable as yet unknown. In any event, there are many lessons to be learned from the Queen’s experience. @

O E S T I O N D E S S O I N S D E SANTI!

14. Anderson MJ. Alternative Funding Plan Evaluation Workshop background paper. Prepared for the workshop on evaluating alternative funding plans in academic health centres. Queen’s University, Kingston, Ontario; March 1996. 15. Queen’s Health Policy Research Unit. Evaluating alternative funding plans in academic health centres. Report from a two-day workshop sponsored by Health Canada. Queen’s Health Policy Research Unit, Queen’s University, Kingston, Ontario;March 1996. Malcoh Anderson, PhD, is Associate Director, Research, at the Queen’s Health Policy Research Unit and,for two years, was project manager of the

References and Notes

AFP Evaluation.

1. Haslam RHA. Alternative Funding Plan, Department of Pediatrics, University of Toronto: is the AFP still alive? Annals RCPSC 1996;29(4):219-222.

Jarold Cosby, MA, is Research Coordinator at the Centrefor Health Economics and Policy Analysis (CHEPA).He conducted research on the AFP evaluation while a Research Associate at QHP.

2. Haslam RHA, Walker NE. Alternative funding plans: is there a place in academic medicine? Canadian Medical Association Journal 1993;147(7):1141-1146.

For more information on the evaluation of the AFP, contact Karen Parent, AFP Evaluation Project Manager, Queen’s Health Policy, Queen’s University, Kingston, Ontario, Canada K7L 3N6. Telephone (613) 5456387.

3. Cosby J. A proposal for an evaluability assessment of the AFP within SEAMO. Kingston (ON): Queen’s Health Policy Research Unit, Queen’s University; 1995. 4. Bickman L (4).Using program theory in evaluation. New directions for program evaluation. San Francisco: Jossey-Bass Inc. Publishers; 1987.

5. Chen HT. Theory-driven evaluations. Newbury Park (CA): SAGE; 1990. 6. Rutman L. Planning useful evaluations: evaluability assessment. Vol 96 of the SAGE Library of Social Research. Beverly Hills (CA): SAGE; 1980. 7. Smith MF. Evaluability assessment: a practical approach. Boston (MA): Kluwer Academic Publishers; 1989. 8. Wholey JS. Evaluability assessment: developing program theory. In: Bickman L, editor. Using program theory in evaluation. New directions for program evaluation. San Francisco: Jossey-Bass Inc. Publishers; 1987. 9. Anderson MJ, Schneider T, Thomas L, Cosby J. The Alternative Funding Plan: Documents Model. AFP Evaluation Technical Report Series. Kingston (ON): Queen’s Health Policy Research Unit, Queen’s University; 1996. 10. Cosby J, Middleton M. The perceptions of clinical medical faculty toward the Alternative Funding Plan. AFP Evaluation Technical Report Series. Kingston (ON): Queen’s Health Policy Research Unit, Queen’s University; 1996.

11 Cosby J, Middleton M, O’Connor M, Shulha L, Meyers K. Adjusting professional activities in an alternative funding environment: perceptions of clinical faculty after the first two years. Paper presented at the annual conference of the Association of Canadian Medical CollegesKanadian Association of Medical Educators1 Association of Canadian Teaching Hospitals, Halifax, Nova Scotia, May 1997. 12. Cosby J, Anderson MJ. Changing management to adapt to a new compensation culture: Implementing an alternative funding plan at Queen’s Faculty of Medicine. AFP Evaluation Technical Report Series. Kingston (ON): Queen’s Health Policy Research Unit, Queen’s University; 1997. 13. Duncan PG, Ballantyne M. Does the method of payment affect anaesthetic practice? An evaluation of an alternate payment plan. Canadian Journal of Anaesthesia 1997;44(5):503-510.

SUMMER 1 9 9 1 , VOL. 1 1 , NO. 2

@ ell!

1991, VOL. 1 1 , NO. 2