- Email: [email protected]
A multiobjective approach to evaluating and planning the allocation of inspection resources
European Journalof OperationalResearch52 (1991) 55-64 North-Holland
55
Theory and Methodology
A multiobjective approach to evaluating and planning the allocation of inspection resources Ronald Klimberg School of Management, Quantitative Methods Department, Boston University, Boston, MA 02215, USA
Charles ReVelle and Jared Cohon Department of Geography, and Environmental Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA Abstract: A two-objective, zero-one programming model of an inspection allocation problem faced by a federal agency is solved by a two-stage decision support system. The decision support system allows decision-makers to explore the entire range of outputs from alternative investments. A two-stage decision support system, called the national model for inspection selection (NMIS), was developed to solve an inspection allocation problem faced by the Food and Drug Administration (FDA). The FDA in its regulation of the pharmaceutical industry must determine the 'best' level of inspection resources to allocate nationally and among its districts, while satisfying at the same time mandated requirements for inspection frequency. The problem is initially formulated here as a large-scale, single-objective, zero-one programming model. We decompose this initial formulation into a sequence of smaller two-objective district-level subproblems and one two-objective national problem. Algorithms were developed to solve both the district-level subproblems and the national problem for the entire range of outputs from possible alternative investments. The NMIS decision support system is comprised of a set of computer programs which performs these algorithms. NMIS provides FDA management with a powerful tool to assist them in developing and evaluating their decisions at various levels of limited resources.
Keywords: Multiple objective integer programming, inspection, heuristics
1. Introduction: FDA's drug inspection selection problem The Food and Drug Administration (FDA) is the U.S. federal agency responsible for regulating the pharmaceutical industry. One of the FDA's major regulatory activities relative to the pharReceived June 1989; revisedOctober 1989
maceutical industry is the monitoring, the storage, and the distribution of all drugs to be used for human consumption. FDA's surveillance activities primarily consist of intermittently inspecting drug firms, and collecting and analyzing samples of drug products. Nationally, the pharmaceutical industry consists of about 4000 drug firms, producing thousands of different drug products. During any of these stages, from the manufacturing pro-
0377-2217/91/$03.50 © 1991 - ElsevierSciencePublishers B.V.(North-Holland)
56
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
cess until a pharmaceutical product reaches the consumer, a potential significant public health hazard may arise. Thousands of people could be seriously affected by just one small deviation in the production, storage, or distribution of a drug product. On average, approximately 10% of the drug inspections FDA annually performs results in violations classified as severe violations and about 35% are classified as minor violations (Klimberg, 1983). Whenever FDA detects a violation, they swiftly follow up with effective regulatory action until the violation is corrected. Historically, very few drug products have resulted in a significant public health problem and the overall quality of the drugs marketed is considered to be superior. However, one of these violations could represent a potential future catastrophe affecting thousands of people. Each year, FDA's headquarters management and district management determine the amount of resources to allocate nationally for inspecting drug firms and plan how to allocate these drug inspection resources among FDA's 24 districts across the country. Headquarters management seek to obtain an overall consistent strategy. Their objective is to achieve, via their allocation to the districts, a comprehensive coverage of all drug firms and of any particular national problem areas among all the districts. On the other hand, district management are given a certain amount of drug inspection resources to be used, and these district-level managers must decide on the actual drug firms and processes to inspect. District management decisions are further constrained by the legal requirement to inspect each drug firm at least once every two years. District management intentions are directed towards satisfying this biennial inspection requirement and rectifying any particular problem areas within their district. These allocation and selection decisions are made without coordination between headquarters and district management. A particular problem to a district may not appear to be a national problem and vice versa. This form of decision making may not be guaranteed to lead to superior choices and, in some cases, might not lead even to good choices. The problem is how to inform and coordinate these allocation/selection decisions among the districts and still satisfy the inspection frequency requirements. To achieve this coordination, we initially formulated a large single-objective zero-
one programming model. The large size of this formulation would have required a significant amount of computational effort to solve it. We, therefore, decomposed the formulation into a sequence of smaller two-objective district=level subproblems and one overarching two-objective national problem. After performing some empirical testing to observe the difficulty in obtaining an all-integer solution to the district-level subproblem, we were able to develop heuristics to solve the entire decomposed problem and corresponding national problem for the entire range of possible inspection investment. A two-stage decision support system (DSS), called the national model for inspection selection (NMIS), utilizing these heuristics was developed and applied to solve FDA's inspection selection problem. NMIS enables FDA management, both at headquarters and at the districts, to explore the tradeoffs between the inspection resources allocated and outputted. Inspection resources may in dollars or person-years and output may be measured in terms of the expected number of violations found. The initial large-scale, single-objective, zeroone programming model for FDA's drug inspection selection problem is formulated in the next section. This formulation was decomposed into 24 smaller district-level subproblems and one overarching national problem. This district-level subproblem was further restructured into a two-objective problem. We describe the first stage of NMIS, which is a heuristic to solve the district-level subproblem, in Section 3. The second stage of NMIS is an algorithm which utilizes the district solutions to solve the corresponding national problem and is presented in Section 4. We discuss the current and future application of NMIS at FDA in Section 5.
2. Formulation
An initial single-objective linear programming formulation of FDA's drug inspection selection problem is:
k = index of districts, 1 . . . . , K,
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
Decision variables
~_, xgk+ J~Qi
Xj.k
10 if process j in district k is to be inspected in the first year, otherwise, 1
Yjk =
0
E
Parameters
ajk = the amount of time required to inspect process j in district k, Pgk = the estimated probability of detecting a violation for process j in district k, B = the total amount of inspection time available nationwide in both years, Qi = the set of processes j in establishment i, I k = the set of establishments i in district k, D k = the set of all processes j in district k, F k = the set of establishments in district k not inspected last year, i.e., during 'year zero', K = the total number of districts. Problem I. K
E
E
k=l
j~Dk
(pjkXjk+PjkYjk)
(1)
subject to (2)
B 1 + B z <~B, K
~'~ b,k - B, = 0,
(3)
k=l K
E b2k - B2 = 0,
(4)
k=l
ajkxjk -- blk <~O,
k = 1,..., K,
(5)
ajkYjk--b2k <~O,
k= l ..... K,
(6)
b2k -- ( 1 -- o t ) b l k >~ O,
k = 1..... K,
(7)
b2k--(l+fllb,k~O,
k=l
(8)
jcDk
.j~Dk
..... K,
. . . . . K,
xjk>~l,
i~Fk,
k = l . . . . . K,
(10)
j~Q,
if process j in district k is to be inspected in the second year, otherwise,
Z=
i ~ I k, k = l
J~Qi
(9)
blk = the amount of inspection time available in the first year for district k, b2k = the amount of inspection time available in the second year for district k, B 1 = the total amount of inspection time available nationwide in the first year, B 2 = the total amount of inspection time available nationwide in the second year,
max.
~_, Ygk>~l,
57
where Xjk and Yjk are 0 or 1, j ~ D k, k = 1 . . . . . K; a, fl >/0 are prespecified parameters. The objective function, (1), maximizes the expected number of violative inspections generated (VIGs) in a two-year planning period. Equation (2) indicates the unknown total national resources available for the first and second year must be less than or equal to the known total amount of inspection time available for the two-year period. Equations (3) and (4) assure that total resources to be used in each district must sum over all districts to the national total resources for each year, respectively. Equations (5) and (6) are the individual district constraints on the amount of time available for each year. Equations (7) and (8) are resource boundary constraints for each district. These two equations force the second year resources for each district to be within a certain range, plus or minus, of the first year resources. The biennial inspection requirement which requires each drug firm to be inspected at least once every two years is expressed by equation (9). A consequence of this inspection frequency requirement is that all drug firms not inspected the year prior to the model's execution must have at least one inspection during the first year of planned inspections, and is expressed by equation (10). The probability of being out-of-compliance, Pjk, is one parameter required by this formulation which may be difficult to estimate. No empirical form exists to estimate the probability of a firm being out-of-compliance. Over the past several years, FDA has developed several different estimates to measure the overall level of compliance for the entire pharmaceutical industry. Each of these approaches, although politically useful, do not produce compliance rates nor probabilities at the level of detail required by this model. We generated probabilities of being out-of-compliance by applying a statistical technique called multiple logistic regression (MLR) (Klimberg, 1987). First, we classified the processes producing the thousands of drug products into a more manageable 12 general processes. All the drug products within a particular type of process were assumed to be to
58
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
some degree homogeneous in the way they are produced. This categorization enables us to compare the performances of all the drug products within a process type, as well as to compare them to other process types. Utilizing several years of historical surveillance data, each process type was analyzed separately using MLR, and current compliance probabilities were generated for each firm/process. A measure of the accuracy of the fitted logistic model is the value of the locus of a receiver operating characteristic (ROC) curve (Swets, 1988). The average value of the 12 ROC curves range from 0.59 to 0.81, with an average of 0.73.
3. District-level subproblem Problem I is a very large integer programming formulation, having on the order of about 3000 constraints and 12000 variables for the actual problem that FDA faces. If we relaxed the integer requirements and solved the resulting linear program, Problem I is still very unlikely to provide an all-integer solution. Some branch and bound, even a little, would be very expensive for such a large problem. We took two steps to improve the likelihood of producing an all-integer solution. Problem I has the necessary block angular structure required by decomposition methods. We decomposed Problem I into 24 district-level subproblems and one overarching national problem. Solutions of these smaller problems greatly reduce the overall amount of computational effort necessary to produce an all z e r o - o n e solution. However, the nonunit nature of the ajk's (the average time to inspect process j ) in the district-level subproblem will probably result in a non-all-integer solution to still be produced. We can further increase the likelihood of an integer termination by bringing the district total resource variables into the objective function. The usual method of dualizing constraints would have brought the entire constraint with known total year resource variables into the objective function. However, for our particular problem, the total yearly resource variables are unknown, ba and b 2. Only the total yearly resource variables were brought into the objective since these variables are unknown, and in any optimal solution the corresponding yearly resource constraints would be binding. The district-
level subproblem is now a two-objective, z e r o - o n e programming model, as shown below:
Decision variables for district k subproblem 1
if process j is to be inspected in the first year, otherwise,
xj = 0 1
if process j is to be inspected in the second year, otherwise,
yj = 0 b a = the the b 2 = the the
amount of inspection time available in first year for the district, amount of inspection time available in second year for the district,
Parameters aj = the amount of time required to inspect process j,
pj = the estimated probability of detecting a violation by inspection for process j, the set of processes j in establishment i, I = the set of establishments i in the district, D = the set of all processes j in the district, F = the set of establishments not inspected in the previous year (i.e., prior to the application of the two-year planning model).
Qi
=
Problem II.
max.
z,=
(pjxj+pjyj), j~D
(11)
z2 = - ( b , + b2) subject to
b 1 <~O,
(12)
~_, ajyj - b= <~O,
(13)
E a j x j -j~D
jED
E xj+ J~'Qi
Y'~ yj>~l,
i~1,
(14)
J~Qi
E xj>~l,
i~F,
(15)
b 2 - (1 - o~)b 1 ~ 0,
(16)
b 2 - (1 + •)61 .~<0,
(17)
j E Q,
where xj and Ya are 0 or 1, j ~ D; a, fl>~ 0 are prespecified parameters.
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
The first objective is to maximize the number of expected violations generated (VIGs), and the second objective is to minimize the total inspection time to be used. Equations (12) and (13) are the yearly resource constraints in the inspection time available. The biennial inspection requirements are expressed in eqs. (14) and (15). Equations (16) and (17) are the resource boundary constraints which require each year's resources to be used within the district to be relatively close. Equation (16) provides a lower bound for the amount of second year resource to be used, b 2, relative to the first year resources to be used, b 1. Correspondingly, eq. (17) provides an upper bound on b 2. The amount of resources to be used in the first and second year are closer together for smaller values of a and ft. (During our testing and application, we required the second year resources to be used to be close to the first year resources to be used, and also less than the first year resources to be used, i.e., we used a = 0.1 and fl = 0.0.) We did some empirical testing to evaluate the difficulty in producing all z e r o - o n e solutions to the district-level subproblem by using a small hypothetical data set. We relaxed the integer requirements and applied the weighting method of multiobjective linear programming for several different sets of weights (Cohon, 1978). Several patterns were observed. Different sets of weights had varying degrees of success in producing an all z e r o - o n e solution. Some weights generated all z e r o - o n e solutions immediately, while others required a significant amount of branch and bound. All the problems which produced fractions had at most one or two fractional variables. These noninteger variables always appeared to come from the firm/process with the smallest (or one of the smallest) pj/aj ratios of all the inspections in the solution. Further, in every case, the application of branch and bound always produced an all-integer solution exactly equivalent to the continuous solution except the fractional variables were now either zero or one. Overall, the all-integer solutions consisted predominantly of those inspections of firms/processes with the relatively larger pj/aj ratios. Finally, we observed that when we were given a particular all z e r o - o n e solution and if the relative weight on the resource objective was decreased, the next level of resources contained the same previous solution's inspections plus one more inspection. This added inspection was the inspect-
59
ion of the f i r m / p r o c e s s with the largest pJaj ratio of all the firms/processes not yet in the solution. Altogether, these observations are characteristic of the knapsack problem and the commonly applied heuristic to solve it. The relative benefit of inspecting a particular f i r m / p r o c e s s is measured by its marginal return, i.e., its pj/aj ratio. With one exception, we can predict the entire order in which new inspections will enter into the solution by the pJaj ratios. This exception occurs, generally, when the weight on the resource objective is relatively large in comparison to the number of violations objective. In such a case, we discovered what we call inefficient inspections in the solution, i.e., inspections of firms/processes with relatively small pJaj ratios. To understand why we sometimes find inefficient inspections in the solution, let us start with what we call the minimum time solution (we mean minimum time to be implied as being equivalent to minimum resources). The weight associated with the objective of maximizing the number of violations detected is equal to zero in the minimum time solution. In such a case, the district-level subproblem is now a single-objective problem of minimizing resources. The solution to the minim u m time solution is obvious. Each firm will have only one inspection during the two year period. This one inspection will be of the process within each firm having the smallest average time to inspect, i.e., i s = [qlaq = minj~Qaj]. (Remember each firm must have at least one inspection of one process in the two year planning period.) Nevertheless, it is highly unlikely that this particular process within the firm is also the process within the firm with the largest pj/aj ratio. As resources are increased from the minimum time solution or as more weight is attributed to maximizing VIGs, each inefficient inspection will be eventually driven out of solution and replaced by a more efficient inspection within the firm. We call this replacement of an inefficient inspection with a more efficient inspection an efficient shift. When an efficient shift occurs can be determined by the following ratio,
( pq-ps)/( a q - as) ,
(18)
where s = the index of the inefficient process within firm i and in the solution,
R. Klimberg et a L / .4 multiobjective approach to allocate inspection resources
60
q = the index of the process within firm i with the maximum pq//aq ratio. A solution generated by the weighting method from a multiobjective integer programming (MOIP) problem is called a supported noninferior point (Steuer and Choo, 1983). All the supported noninferior points can be generated by complete parameterization of the weights. Altogether, these supported noninferior points generate a convex hull in objective space. Every supported noninferior point for the district-level subproblem can be generated without the application of the weighting method by considering the pJai ratios and the above mentioned criterion of efficient shifts. The first stage of NMIS is a heuristic which applies these criteria to generate every supported noninferior point for the district-level subproblem. The heuristic starts by first finding the minimum time solution. Subsequent solutions are generated by adding new inspections, one at a time, based on their pj/aj ratio, or inspections are exchanged due
Reso,,rces
to an efficient shift, (18), until every firm/process is designated to be inspected once each year. The set of supported noninferior solutions produced by the first stage of NMIS demonstrates the relationship between the expected number of V1Gs and the amount of resources to be used for the entire range of possible values of time invested for a particular district. A concave curve is produced by joining adjacent supported noninferior points generated by this heuristic. We call this concave curve a district tradeoff curve, Figure 1. (The district tradeoff curve is a presentation device. Some points displayed on the tradeoff curve are not feasible.) Bowman (1976) demonstrated when solving MOIP problems that some of the noninferior points may not be generated when using the weighting method. These other solutions, not produced by the weighting method, nor by the first stage of NMIS, are called unsupported or 'weak' noninferior points. By definition, any unsupported noninferior point will always be below the line
Resources
~
,eso,.,rces I
-. National lW°
I
f
_ Resources
District
IGS
IGs
. . . . .
•
figs
~esources Figure 1. The national decomposedinspection selection )roblem
24
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
joining the two corresponding adjacent supported noninferior points. The incremental change in the amount of resources to be used and the expected number of VIGs between any pair of adjacent supported noninferior points is relatively small. Furthermore, w e found that each unsupported noninferior point in our problem tends to have several alternate optimal solutions associated with it. For these reasons, we felt it was not worth the price of significantly increasing solution costs, nor of any particular interest to decision-makers, to generate these unsupported noninferior points in this case. Given a particular supported noninferior point generated by the above heuristic, we still must decide-on how to assign these inspections between the two planning years. In general, F D A management preferred to maximize the expected number of violations in the first planning year. The first year of allocations were viewed as holding stronger implications for what inspections would actually accomplish. Furthermore, since the intent was to execute the model yearly, the second year of allocations from the current model's execution could be revised in the following year. An algorithm to determine yearly assignments of inspection must first consider the constraint (15) which requires all firms not inspected the previous year to have at least one inspection in the first planning year. We satisfy this requirement by first examining each of these firms which must have a first year inspection. One process is chosen from each of these firms and is assigned to be inspected in the first planning year. The one process chosen within each firm is the process with the largest p j / a j ratio of the processes within the firm which are in the current solution. After satisfying (15), the remaining unassigned inspections in the solution are assigned to be inspected in the first year according to their p j / a j ratio until the resource boundary constraints, (16) and (17), are satisfied. This procedure will maximize the number of VIGs in the first planning year.
inspections of drug firm/processes not yet in the solution. Solutions are generated for the entire range of possible alternative resource investments. As a result, a district tradeoff curve is produced for each district. By definition, each district tradeoff curve is a concave curve, Figure 1. The second stage of N M I S uses all the district tradeoff curves generated by N M I S ' s first stage and exploits their concavity to solve the national drug inspection selection problem and to generate a corresponding national tradeoff curve, Figure 1. The decomposed national inspection selection problem is a relatively small z e r o - o n e program and is formulated below: Problem IlL r -k max.
E E Gtxk, k= 1 t = 1
subject to /~ nk ~, E b k t X k t = B, ~= ~ t=l -~ ~ xkt = 1, k = 1 . . . . . K , t=l where
The order in which N M I S ' s first stage finds supported noninferior points to solve the districtlevel subproblem is based upon the efficiency criteria, p J a j ratio and efficient shifts, of the
(19)
(20) (21)
k, K = the index for all districts, k = 1 . . . . . K, nk = the total number of resource levels in district k, t = the index of resource levels, t = 1 . . . . . n~, B = the total amount of resources to be used nationwide in both years, bkt = the amount of resources used in district k with resource level t for both years, vk, = the amount of expected violations to be detected in district k with resource level t for both years (read from the district-level tradeoff curve from the solution to the district-level problem), 1 Xkt
4. Decomposed national problem
61
0
if resource level t is allocated to district k, otherwise.
The objective is to maximize the total expected number of V I G s in the two-year period with a given level of national drug inspection resources. Equation (20) requires the sum of the district drug inspection resources to be equal to some known
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
62
national level of inspection resources, B. The second constraint, (21), requires each district to have at least some minimum level of drug inspection resources allocated to it and only to have one level of resource allocation. Bringing the time budget into the objective and applying the weighting method to Problem III we get
Problem IV. g
max.
nk
~_, ~_, (hVkt--(1--X)bkt ) k=l
(22)
Since each district tradeoff curve is a concave function, we know that AVkl AVk2 Abkl >1 ~ >1 . . .
AVknk >1 Abknk '
k = l . . . . . K.
Suppose we are given some specific national solution at a particular resource level. The next increased level of resources would be determined by increasing X until the m a x i m u m value of )kVkt_ 1 - - ( 1 --h)bkt_ 1 shifts in one of the districts to a new resource level t. Let's say this occurs in district i: Then, the following is true for district i:
t=l
)ivit -- (1 - )~)b, >/Xvi,_ 1 -- (1 - X)b,t_x, subject to
hAvi, >1(1 - X) Ab,, ?l k
ZXkt=l,
k=l ..... K,
(23)
tffil
Abit >~
O
Another approach to solving Problem IV, which is more intuitive, exploits the concavity of the district tradeoff curves (Figure 1). We call this approach the 'greedy' algorithm. The algorithm apparently does not formally appear in any operations research texts, but seems to be known to numerous researchers in the operations research community. We define
AVkt
Avi,
the increase in the number of violations detected in district k in going from t - 1 resource level to resource level t, Abk, : the increase in the resource allocated to district k in going from t - 1 resource level to resource level t, :
That is, AVkt~Vkt--Vkt_l,
t=2,...,n
k,
Abkt= bkt-- bkt-1,
k = l ..... K.
(1 - )~) ~k
This result implies that as we vary X, as soon as the ratio ( 1 - h ) / ) ~ reaches a value less than AVkt/Abkt for any of the districts, there will be an increase in resources allocated, violations detected, and a new (supported) noninferior point found. If two or more districts have equivalent AOkt//Abkt values, we choose the district with the greatest Vk,/bk, value. Since each Vk(bk) is concave, as h increases, successively smaller values of AOkt/Abkt become larger than ( 1 - h ) / h and are eligible to enter the solution (or, in other words, less 'efficient' inspections enter the solution as h increases). The second stage of N M I S , then, employs the 'greedy' algorithm to generate the entire range of possible alternatives for the national problem. The algorithm starts by finding the national m i n i m u m resource level by selecting the minimum resource level from each district, i.e., each district's minim u m time solution. Subsequent solutions are produced by entering inspections in order of decreasing values of AOkt//Abkt until all the f i r m s / p r o c esses in each district are inspected twice. A concave curve, called the national tradeoff curve, is generated by joining adjacent solutions produced by the second stage of N M I S , Figure 1. The relationship between the expected number of violations detected and the amount of inspection resources to be allocated nationally is illustrated by the national tradeoff curve.
R. Klimberg et aL / A multiobjective approach to allocate inspection resources
5. Application
second stage of NMIS is executed again to determine the appropriate district inspection allocations for this specified national resource level. These district inspection levels are the allocation levels necessary to obtain the expected national number of VIGs specified on the national tradeoff curve. Inspection plans for each district, a list of firms/processes to inspect over the next two years while satisfying the inspection frequency requirements with the given level of inspection resources, are produced by re-executing the first stage of NMIS. Each year the model's parameters could be updated and current district and national tradeoff curves generated by executing the two stages of NMIS.
Interactive Fortran programs have been developed for each stage of the NMIS decision support system. A flowchart of NMIS is shown in Figure 2. Initially, the first stage of NMIS must be executed for each district. The decision support system will display each district tradeoff curve and save a file describing it. The second stage of NMIS uses these district tradeoff curves as input to generate and display the national tradeoff curve. Decision-makers would examine the national tradeoff curve and discuss the various tradeoffs of different resource levels. Once a consensus is obtained on a particular national resource level, the
.~ START3 INPUT ~ CERTAINLEVEL OFRESOURCES
DS I TRC I Tc fuRvETRADEOFF
NMIS Ist STAGE: DISTRICT LEVEL HEURISTIC
I
] PRINTAN )] I~ISPECTION l
~CREATE A DISTRICT/ TRADEOFFCURVE/ FILE /
N
l
PRINT
I OISTRICT
63
L N
NMI~ 2rid STAGE: NATIO~JAL"GREEDY" ALGORITH~
i
NATInNAL TRADEDFF CURVE
,,f. Figure 2. A flowchart of NMIS
DECISION-MAKERS DETERMINE ~ATIONALRESOURCE LEVEL
F
64
R. Klimberg et a L / A multiobjective approach to allocate inspection resources
The N M I S decision support system was first applied in 1987 and the results were presented to F D A management. Headquarters and district management closely examined N M I S ' outputs and the firm/process probability estimates of being out-of-compliance. These compliance probability estimates provide F D A management for the first time with a tangible estimate of the level of performance of each drug firm and their processes. Hence, listings of the probability estimates, sorted in several different ways, were produced for headquarters and district management and received particular interest. The managers seemed to be interested in examining the probabilities of lack of compliance of particular firms of personal interest and comparing the range of probabilities by process. The national and district tradeoff curves also received significant attention by F D A management. Along with each tradeoff curve, several previous years' actual resources used and actual number of violations detected were indicated so as to provide a frame of reference. District management initially examined their own tradeoff curve to see where they were on the curve and to compare their performance to the level of performance predicted by NMIS. Afterwards, district managers checked and compared how other districts performed. Headquarters management were interested in the varying levels of performance from district to district. A few districts actually displayed performances greater than the model's predictions. The national tradeoff curve drew particular attention from headquarters management. The Commissioner of Food and Drugs responded to recent trends illustrated by the national tradeoff curve to request for more drug inspection resources from Health and H u m a n Services (HHS) management. As a result, F D A ' s drug inspection program re-
cently received a substantial increase in drug inspection resources. Other top headquarters managements have also utilized the national tradeoff curve to assist them in planning and justifying future resource levels. As a result of this initial experience, F D A management wants NMIS, its probability estimates and tradeoff curves, to become an integral part of their strategic planning of the surveillance of the pharmaceutical industry. With further application, the outputs from N M I S will be used by district and headquarters managements as a tool to assist them in comparing drug industry performance. Future computer development plans are for a PC version of N M I S and the development of an interactive data base retrieval system to examine the regulatory surveillance information.
References Bowman, V.J. (1976), "On the relationship of the Tchebycheff norm and the efficient frontier of multiple-criteria objectives", in: H. Thiriez and S. Zionts (eds.), Multiple Criteria Decision Making, Springer, Berlin. Cohon, J.L. (1978), Multiobjective Programming and Planning Academic Press, New York. Klimberg, R.K. (1983), "An analysis of various disaggregations of the DQA inspection data: FY1978-1982", FDA Working Paper. Klimberg, R.K. (1987), "Inspectional scheduling: A two-stage decision support system using multiple objectives", Doctoral Thesis, The Johns Hopkins University. Klimberg, R.K., ReVelle, C., and Cohon, J.L. (1989), "NMIS: A national model for inspection selection with limited resources", in: A.G. Lockett and G. Islei (eds.), Improving Decision Making in Organisations, Springer-Verlag, Berlin. Steuer, R.E., and Choo, E.U. (1983), "An interactive weighted Tchebycheff procedure for multiple objective programruing", Mathematical Programming 26, August 1983. Swets, J.A. (1988), "Measuring the accuracy of diagnostic systems", Science June 1988.
55
Theory and Methodology
A multiobjective approach to evaluating and planning the allocation of inspection resources Ronald Klimberg School of Management, Quantitative Methods Department, Boston University, Boston, MA 02215, USA
Charles ReVelle and Jared Cohon Department of Geography, and Environmental Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA Abstract: A two-objective, zero-one programming model of an inspection allocation problem faced by a federal agency is solved by a two-stage decision support system. The decision support system allows decision-makers to explore the entire range of outputs from alternative investments. A two-stage decision support system, called the national model for inspection selection (NMIS), was developed to solve an inspection allocation problem faced by the Food and Drug Administration (FDA). The FDA in its regulation of the pharmaceutical industry must determine the 'best' level of inspection resources to allocate nationally and among its districts, while satisfying at the same time mandated requirements for inspection frequency. The problem is initially formulated here as a large-scale, single-objective, zero-one programming model. We decompose this initial formulation into a sequence of smaller two-objective district-level subproblems and one two-objective national problem. Algorithms were developed to solve both the district-level subproblems and the national problem for the entire range of outputs from possible alternative investments. The NMIS decision support system is comprised of a set of computer programs which performs these algorithms. NMIS provides FDA management with a powerful tool to assist them in developing and evaluating their decisions at various levels of limited resources.
Keywords: Multiple objective integer programming, inspection, heuristics
1. Introduction: FDA's drug inspection selection problem The Food and Drug Administration (FDA) is the U.S. federal agency responsible for regulating the pharmaceutical industry. One of the FDA's major regulatory activities relative to the pharReceived June 1989; revisedOctober 1989
maceutical industry is the monitoring, the storage, and the distribution of all drugs to be used for human consumption. FDA's surveillance activities primarily consist of intermittently inspecting drug firms, and collecting and analyzing samples of drug products. Nationally, the pharmaceutical industry consists of about 4000 drug firms, producing thousands of different drug products. During any of these stages, from the manufacturing pro-
0377-2217/91/$03.50 © 1991 - ElsevierSciencePublishers B.V.(North-Holland)
56
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
cess until a pharmaceutical product reaches the consumer, a potential significant public health hazard may arise. Thousands of people could be seriously affected by just one small deviation in the production, storage, or distribution of a drug product. On average, approximately 10% of the drug inspections FDA annually performs results in violations classified as severe violations and about 35% are classified as minor violations (Klimberg, 1983). Whenever FDA detects a violation, they swiftly follow up with effective regulatory action until the violation is corrected. Historically, very few drug products have resulted in a significant public health problem and the overall quality of the drugs marketed is considered to be superior. However, one of these violations could represent a potential future catastrophe affecting thousands of people. Each year, FDA's headquarters management and district management determine the amount of resources to allocate nationally for inspecting drug firms and plan how to allocate these drug inspection resources among FDA's 24 districts across the country. Headquarters management seek to obtain an overall consistent strategy. Their objective is to achieve, via their allocation to the districts, a comprehensive coverage of all drug firms and of any particular national problem areas among all the districts. On the other hand, district management are given a certain amount of drug inspection resources to be used, and these district-level managers must decide on the actual drug firms and processes to inspect. District management decisions are further constrained by the legal requirement to inspect each drug firm at least once every two years. District management intentions are directed towards satisfying this biennial inspection requirement and rectifying any particular problem areas within their district. These allocation and selection decisions are made without coordination between headquarters and district management. A particular problem to a district may not appear to be a national problem and vice versa. This form of decision making may not be guaranteed to lead to superior choices and, in some cases, might not lead even to good choices. The problem is how to inform and coordinate these allocation/selection decisions among the districts and still satisfy the inspection frequency requirements. To achieve this coordination, we initially formulated a large single-objective zero-
one programming model. The large size of this formulation would have required a significant amount of computational effort to solve it. We, therefore, decomposed the formulation into a sequence of smaller two-objective district=level subproblems and one overarching two-objective national problem. After performing some empirical testing to observe the difficulty in obtaining an all-integer solution to the district-level subproblem, we were able to develop heuristics to solve the entire decomposed problem and corresponding national problem for the entire range of possible inspection investment. A two-stage decision support system (DSS), called the national model for inspection selection (NMIS), utilizing these heuristics was developed and applied to solve FDA's inspection selection problem. NMIS enables FDA management, both at headquarters and at the districts, to explore the tradeoffs between the inspection resources allocated and outputted. Inspection resources may in dollars or person-years and output may be measured in terms of the expected number of violations found. The initial large-scale, single-objective, zeroone programming model for FDA's drug inspection selection problem is formulated in the next section. This formulation was decomposed into 24 smaller district-level subproblems and one overarching national problem. This district-level subproblem was further restructured into a two-objective problem. We describe the first stage of NMIS, which is a heuristic to solve the district-level subproblem, in Section 3. The second stage of NMIS is an algorithm which utilizes the district solutions to solve the corresponding national problem and is presented in Section 4. We discuss the current and future application of NMIS at FDA in Section 5.
2. Formulation
An initial single-objective linear programming formulation of FDA's drug inspection selection problem is:
k = index of districts, 1 . . . . , K,
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
Decision variables
~_, xgk+ J~Qi
Xj.k
10 if process j in district k is to be inspected in the first year, otherwise, 1
Yjk =
0
E
Parameters
ajk = the amount of time required to inspect process j in district k, Pgk = the estimated probability of detecting a violation for process j in district k, B = the total amount of inspection time available nationwide in both years, Qi = the set of processes j in establishment i, I k = the set of establishments i in district k, D k = the set of all processes j in district k, F k = the set of establishments in district k not inspected last year, i.e., during 'year zero', K = the total number of districts. Problem I. K
E
E
k=l
j~Dk
(pjkXjk+PjkYjk)
(1)
subject to (2)
B 1 + B z <~B, K
~'~ b,k - B, = 0,
(3)
k=l K
E b2k - B2 = 0,
(4)
k=l
ajkxjk -- blk <~O,
k = 1,..., K,
(5)
ajkYjk--b2k <~O,
k= l ..... K,
(6)
b2k -- ( 1 -- o t ) b l k >~ O,
k = 1..... K,
(7)
b2k--(l+fllb,k~O,
k=l
(8)
jcDk
.j~Dk
..... K,
. . . . . K,
xjk>~l,
i~Fk,
k = l . . . . . K,
(10)
j~Q,
if process j in district k is to be inspected in the second year, otherwise,
Z=
i ~ I k, k = l
J~Qi
(9)
blk = the amount of inspection time available in the first year for district k, b2k = the amount of inspection time available in the second year for district k, B 1 = the total amount of inspection time available nationwide in the first year, B 2 = the total amount of inspection time available nationwide in the second year,
max.
~_, Ygk>~l,
57
where Xjk and Yjk are 0 or 1, j ~ D k, k = 1 . . . . . K; a, fl >/0 are prespecified parameters. The objective function, (1), maximizes the expected number of violative inspections generated (VIGs) in a two-year planning period. Equation (2) indicates the unknown total national resources available for the first and second year must be less than or equal to the known total amount of inspection time available for the two-year period. Equations (3) and (4) assure that total resources to be used in each district must sum over all districts to the national total resources for each year, respectively. Equations (5) and (6) are the individual district constraints on the amount of time available for each year. Equations (7) and (8) are resource boundary constraints for each district. These two equations force the second year resources for each district to be within a certain range, plus or minus, of the first year resources. The biennial inspection requirement which requires each drug firm to be inspected at least once every two years is expressed by equation (9). A consequence of this inspection frequency requirement is that all drug firms not inspected the year prior to the model's execution must have at least one inspection during the first year of planned inspections, and is expressed by equation (10). The probability of being out-of-compliance, Pjk, is one parameter required by this formulation which may be difficult to estimate. No empirical form exists to estimate the probability of a firm being out-of-compliance. Over the past several years, FDA has developed several different estimates to measure the overall level of compliance for the entire pharmaceutical industry. Each of these approaches, although politically useful, do not produce compliance rates nor probabilities at the level of detail required by this model. We generated probabilities of being out-of-compliance by applying a statistical technique called multiple logistic regression (MLR) (Klimberg, 1987). First, we classified the processes producing the thousands of drug products into a more manageable 12 general processes. All the drug products within a particular type of process were assumed to be to
58
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
some degree homogeneous in the way they are produced. This categorization enables us to compare the performances of all the drug products within a process type, as well as to compare them to other process types. Utilizing several years of historical surveillance data, each process type was analyzed separately using MLR, and current compliance probabilities were generated for each firm/process. A measure of the accuracy of the fitted logistic model is the value of the locus of a receiver operating characteristic (ROC) curve (Swets, 1988). The average value of the 12 ROC curves range from 0.59 to 0.81, with an average of 0.73.
3. District-level subproblem Problem I is a very large integer programming formulation, having on the order of about 3000 constraints and 12000 variables for the actual problem that FDA faces. If we relaxed the integer requirements and solved the resulting linear program, Problem I is still very unlikely to provide an all-integer solution. Some branch and bound, even a little, would be very expensive for such a large problem. We took two steps to improve the likelihood of producing an all-integer solution. Problem I has the necessary block angular structure required by decomposition methods. We decomposed Problem I into 24 district-level subproblems and one overarching national problem. Solutions of these smaller problems greatly reduce the overall amount of computational effort necessary to produce an all z e r o - o n e solution. However, the nonunit nature of the ajk's (the average time to inspect process j ) in the district-level subproblem will probably result in a non-all-integer solution to still be produced. We can further increase the likelihood of an integer termination by bringing the district total resource variables into the objective function. The usual method of dualizing constraints would have brought the entire constraint with known total year resource variables into the objective function. However, for our particular problem, the total yearly resource variables are unknown, ba and b 2. Only the total yearly resource variables were brought into the objective since these variables are unknown, and in any optimal solution the corresponding yearly resource constraints would be binding. The district-
level subproblem is now a two-objective, z e r o - o n e programming model, as shown below:
Decision variables for district k subproblem 1
if process j is to be inspected in the first year, otherwise,
xj = 0 1
if process j is to be inspected in the second year, otherwise,
yj = 0 b a = the the b 2 = the the
amount of inspection time available in first year for the district, amount of inspection time available in second year for the district,
Parameters aj = the amount of time required to inspect process j,
pj = the estimated probability of detecting a violation by inspection for process j, the set of processes j in establishment i, I = the set of establishments i in the district, D = the set of all processes j in the district, F = the set of establishments not inspected in the previous year (i.e., prior to the application of the two-year planning model).
Qi
=
Problem II.
max.
z,=
(pjxj+pjyj), j~D
(11)
z2 = - ( b , + b2) subject to
b 1 <~O,
(12)
~_, ajyj - b= <~O,
(13)
E a j x j -j~D
jED
E xj+ J~'Qi
Y'~ yj>~l,
i~1,
(14)
J~Qi
E xj>~l,
i~F,
(15)
b 2 - (1 - o~)b 1 ~ 0,
(16)
b 2 - (1 + •)61 .~<0,
(17)
j E Q,
where xj and Ya are 0 or 1, j ~ D; a, fl>~ 0 are prespecified parameters.
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
The first objective is to maximize the number of expected violations generated (VIGs), and the second objective is to minimize the total inspection time to be used. Equations (12) and (13) are the yearly resource constraints in the inspection time available. The biennial inspection requirements are expressed in eqs. (14) and (15). Equations (16) and (17) are the resource boundary constraints which require each year's resources to be used within the district to be relatively close. Equation (16) provides a lower bound for the amount of second year resource to be used, b 2, relative to the first year resources to be used, b 1. Correspondingly, eq. (17) provides an upper bound on b 2. The amount of resources to be used in the first and second year are closer together for smaller values of a and ft. (During our testing and application, we required the second year resources to be used to be close to the first year resources to be used, and also less than the first year resources to be used, i.e., we used a = 0.1 and fl = 0.0.) We did some empirical testing to evaluate the difficulty in producing all z e r o - o n e solutions to the district-level subproblem by using a small hypothetical data set. We relaxed the integer requirements and applied the weighting method of multiobjective linear programming for several different sets of weights (Cohon, 1978). Several patterns were observed. Different sets of weights had varying degrees of success in producing an all z e r o - o n e solution. Some weights generated all z e r o - o n e solutions immediately, while others required a significant amount of branch and bound. All the problems which produced fractions had at most one or two fractional variables. These noninteger variables always appeared to come from the firm/process with the smallest (or one of the smallest) pj/aj ratios of all the inspections in the solution. Further, in every case, the application of branch and bound always produced an all-integer solution exactly equivalent to the continuous solution except the fractional variables were now either zero or one. Overall, the all-integer solutions consisted predominantly of those inspections of firms/processes with the relatively larger pj/aj ratios. Finally, we observed that when we were given a particular all z e r o - o n e solution and if the relative weight on the resource objective was decreased, the next level of resources contained the same previous solution's inspections plus one more inspection. This added inspection was the inspect-
59
ion of the f i r m / p r o c e s s with the largest pJaj ratio of all the firms/processes not yet in the solution. Altogether, these observations are characteristic of the knapsack problem and the commonly applied heuristic to solve it. The relative benefit of inspecting a particular f i r m / p r o c e s s is measured by its marginal return, i.e., its pj/aj ratio. With one exception, we can predict the entire order in which new inspections will enter into the solution by the pJaj ratios. This exception occurs, generally, when the weight on the resource objective is relatively large in comparison to the number of violations objective. In such a case, we discovered what we call inefficient inspections in the solution, i.e., inspections of firms/processes with relatively small pJaj ratios. To understand why we sometimes find inefficient inspections in the solution, let us start with what we call the minimum time solution (we mean minimum time to be implied as being equivalent to minimum resources). The weight associated with the objective of maximizing the number of violations detected is equal to zero in the minimum time solution. In such a case, the district-level subproblem is now a single-objective problem of minimizing resources. The solution to the minim u m time solution is obvious. Each firm will have only one inspection during the two year period. This one inspection will be of the process within each firm having the smallest average time to inspect, i.e., i s = [qlaq = minj~Qaj]. (Remember each firm must have at least one inspection of one process in the two year planning period.) Nevertheless, it is highly unlikely that this particular process within the firm is also the process within the firm with the largest pj/aj ratio. As resources are increased from the minimum time solution or as more weight is attributed to maximizing VIGs, each inefficient inspection will be eventually driven out of solution and replaced by a more efficient inspection within the firm. We call this replacement of an inefficient inspection with a more efficient inspection an efficient shift. When an efficient shift occurs can be determined by the following ratio,
( pq-ps)/( a q - as) ,
(18)
where s = the index of the inefficient process within firm i and in the solution,
R. Klimberg et a L / .4 multiobjective approach to allocate inspection resources
60
q = the index of the process within firm i with the maximum pq//aq ratio. A solution generated by the weighting method from a multiobjective integer programming (MOIP) problem is called a supported noninferior point (Steuer and Choo, 1983). All the supported noninferior points can be generated by complete parameterization of the weights. Altogether, these supported noninferior points generate a convex hull in objective space. Every supported noninferior point for the district-level subproblem can be generated without the application of the weighting method by considering the pJai ratios and the above mentioned criterion of efficient shifts. The first stage of NMIS is a heuristic which applies these criteria to generate every supported noninferior point for the district-level subproblem. The heuristic starts by first finding the minimum time solution. Subsequent solutions are generated by adding new inspections, one at a time, based on their pj/aj ratio, or inspections are exchanged due
Reso,,rces
to an efficient shift, (18), until every firm/process is designated to be inspected once each year. The set of supported noninferior solutions produced by the first stage of NMIS demonstrates the relationship between the expected number of V1Gs and the amount of resources to be used for the entire range of possible values of time invested for a particular district. A concave curve is produced by joining adjacent supported noninferior points generated by this heuristic. We call this concave curve a district tradeoff curve, Figure 1. (The district tradeoff curve is a presentation device. Some points displayed on the tradeoff curve are not feasible.) Bowman (1976) demonstrated when solving MOIP problems that some of the noninferior points may not be generated when using the weighting method. These other solutions, not produced by the weighting method, nor by the first stage of NMIS, are called unsupported or 'weak' noninferior points. By definition, any unsupported noninferior point will always be below the line
Resources
~
,eso,.,rces I
-. National lW°
I
f
_ Resources
District
IGS
IGs
. . . . .
•
figs
~esources Figure 1. The national decomposedinspection selection )roblem
24
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
joining the two corresponding adjacent supported noninferior points. The incremental change in the amount of resources to be used and the expected number of VIGs between any pair of adjacent supported noninferior points is relatively small. Furthermore, w e found that each unsupported noninferior point in our problem tends to have several alternate optimal solutions associated with it. For these reasons, we felt it was not worth the price of significantly increasing solution costs, nor of any particular interest to decision-makers, to generate these unsupported noninferior points in this case. Given a particular supported noninferior point generated by the above heuristic, we still must decide-on how to assign these inspections between the two planning years. In general, F D A management preferred to maximize the expected number of violations in the first planning year. The first year of allocations were viewed as holding stronger implications for what inspections would actually accomplish. Furthermore, since the intent was to execute the model yearly, the second year of allocations from the current model's execution could be revised in the following year. An algorithm to determine yearly assignments of inspection must first consider the constraint (15) which requires all firms not inspected the previous year to have at least one inspection in the first planning year. We satisfy this requirement by first examining each of these firms which must have a first year inspection. One process is chosen from each of these firms and is assigned to be inspected in the first planning year. The one process chosen within each firm is the process with the largest p j / a j ratio of the processes within the firm which are in the current solution. After satisfying (15), the remaining unassigned inspections in the solution are assigned to be inspected in the first year according to their p j / a j ratio until the resource boundary constraints, (16) and (17), are satisfied. This procedure will maximize the number of VIGs in the first planning year.
inspections of drug firm/processes not yet in the solution. Solutions are generated for the entire range of possible alternative resource investments. As a result, a district tradeoff curve is produced for each district. By definition, each district tradeoff curve is a concave curve, Figure 1. The second stage of N M I S uses all the district tradeoff curves generated by N M I S ' s first stage and exploits their concavity to solve the national drug inspection selection problem and to generate a corresponding national tradeoff curve, Figure 1. The decomposed national inspection selection problem is a relatively small z e r o - o n e program and is formulated below: Problem IlL r -k max.
E E Gtxk, k= 1 t = 1
subject to /~ nk ~, E b k t X k t = B, ~= ~ t=l -~ ~ xkt = 1, k = 1 . . . . . K , t=l where
The order in which N M I S ' s first stage finds supported noninferior points to solve the districtlevel subproblem is based upon the efficiency criteria, p J a j ratio and efficient shifts, of the
(19)
(20) (21)
k, K = the index for all districts, k = 1 . . . . . K, nk = the total number of resource levels in district k, t = the index of resource levels, t = 1 . . . . . n~, B = the total amount of resources to be used nationwide in both years, bkt = the amount of resources used in district k with resource level t for both years, vk, = the amount of expected violations to be detected in district k with resource level t for both years (read from the district-level tradeoff curve from the solution to the district-level problem), 1 Xkt
4. Decomposed national problem
61
0
if resource level t is allocated to district k, otherwise.
The objective is to maximize the total expected number of V I G s in the two-year period with a given level of national drug inspection resources. Equation (20) requires the sum of the district drug inspection resources to be equal to some known
R. Klimberg et al. / A multiobjective approach to allocate inspection resources
62
national level of inspection resources, B. The second constraint, (21), requires each district to have at least some minimum level of drug inspection resources allocated to it and only to have one level of resource allocation. Bringing the time budget into the objective and applying the weighting method to Problem III we get
Problem IV. g
max.
nk
~_, ~_, (hVkt--(1--X)bkt ) k=l
(22)
Since each district tradeoff curve is a concave function, we know that AVkl AVk2 Abkl >1 ~ >1 . . .
AVknk >1 Abknk '
k = l . . . . . K.
Suppose we are given some specific national solution at a particular resource level. The next increased level of resources would be determined by increasing X until the m a x i m u m value of )kVkt_ 1 - - ( 1 --h)bkt_ 1 shifts in one of the districts to a new resource level t. Let's say this occurs in district i: Then, the following is true for district i:
t=l
)ivit -- (1 - )~)b, >/Xvi,_ 1 -- (1 - X)b,t_x, subject to
hAvi, >1(1 - X) Ab,, ?l k
ZXkt=l,
k=l ..... K,
(23)
tffil
Abit >~
O
Another approach to solving Problem IV, which is more intuitive, exploits the concavity of the district tradeoff curves (Figure 1). We call this approach the 'greedy' algorithm. The algorithm apparently does not formally appear in any operations research texts, but seems to be known to numerous researchers in the operations research community. We define
AVkt
Avi,
the increase in the number of violations detected in district k in going from t - 1 resource level to resource level t, Abk, : the increase in the resource allocated to district k in going from t - 1 resource level to resource level t, :
That is, AVkt~Vkt--Vkt_l,
t=2,...,n
k,
Abkt= bkt-- bkt-1,
k = l ..... K.
(1 - )~) ~k
This result implies that as we vary X, as soon as the ratio ( 1 - h ) / ) ~ reaches a value less than AVkt/Abkt for any of the districts, there will be an increase in resources allocated, violations detected, and a new (supported) noninferior point found. If two or more districts have equivalent AOkt//Abkt values, we choose the district with the greatest Vk,/bk, value. Since each Vk(bk) is concave, as h increases, successively smaller values of AOkt/Abkt become larger than ( 1 - h ) / h and are eligible to enter the solution (or, in other words, less 'efficient' inspections enter the solution as h increases). The second stage of N M I S , then, employs the 'greedy' algorithm to generate the entire range of possible alternatives for the national problem. The algorithm starts by finding the national m i n i m u m resource level by selecting the minimum resource level from each district, i.e., each district's minim u m time solution. Subsequent solutions are produced by entering inspections in order of decreasing values of AOkt//Abkt until all the f i r m s / p r o c esses in each district are inspected twice. A concave curve, called the national tradeoff curve, is generated by joining adjacent solutions produced by the second stage of N M I S , Figure 1. The relationship between the expected number of violations detected and the amount of inspection resources to be allocated nationally is illustrated by the national tradeoff curve.
R. Klimberg et aL / A multiobjective approach to allocate inspection resources
5. Application
second stage of NMIS is executed again to determine the appropriate district inspection allocations for this specified national resource level. These district inspection levels are the allocation levels necessary to obtain the expected national number of VIGs specified on the national tradeoff curve. Inspection plans for each district, a list of firms/processes to inspect over the next two years while satisfying the inspection frequency requirements with the given level of inspection resources, are produced by re-executing the first stage of NMIS. Each year the model's parameters could be updated and current district and national tradeoff curves generated by executing the two stages of NMIS.
Interactive Fortran programs have been developed for each stage of the NMIS decision support system. A flowchart of NMIS is shown in Figure 2. Initially, the first stage of NMIS must be executed for each district. The decision support system will display each district tradeoff curve and save a file describing it. The second stage of NMIS uses these district tradeoff curves as input to generate and display the national tradeoff curve. Decision-makers would examine the national tradeoff curve and discuss the various tradeoffs of different resource levels. Once a consensus is obtained on a particular national resource level, the
.~ START3 INPUT ~ CERTAINLEVEL OFRESOURCES
DS I TRC I Tc fuRvETRADEOFF
NMIS Ist STAGE: DISTRICT LEVEL HEURISTIC
I
] PRINTAN )] I~ISPECTION l
~CREATE A DISTRICT/ TRADEOFFCURVE/ FILE /
N
l
I OISTRICT
63
L N
NMI~ 2rid STAGE: NATIO~JAL"GREEDY" ALGORITH~
i
NATInNAL TRADEDFF CURVE
,,f. Figure 2. A flowchart of NMIS
DECISION-MAKERS DETERMINE ~ATIONALRESOURCE LEVEL
F
64
R. Klimberg et a L / A multiobjective approach to allocate inspection resources
The N M I S decision support system was first applied in 1987 and the results were presented to F D A management. Headquarters and district management closely examined N M I S ' outputs and the firm/process probability estimates of being out-of-compliance. These compliance probability estimates provide F D A management for the first time with a tangible estimate of the level of performance of each drug firm and their processes. Hence, listings of the probability estimates, sorted in several different ways, were produced for headquarters and district management and received particular interest. The managers seemed to be interested in examining the probabilities of lack of compliance of particular firms of personal interest and comparing the range of probabilities by process. The national and district tradeoff curves also received significant attention by F D A management. Along with each tradeoff curve, several previous years' actual resources used and actual number of violations detected were indicated so as to provide a frame of reference. District management initially examined their own tradeoff curve to see where they were on the curve and to compare their performance to the level of performance predicted by NMIS. Afterwards, district managers checked and compared how other districts performed. Headquarters management were interested in the varying levels of performance from district to district. A few districts actually displayed performances greater than the model's predictions. The national tradeoff curve drew particular attention from headquarters management. The Commissioner of Food and Drugs responded to recent trends illustrated by the national tradeoff curve to request for more drug inspection resources from Health and H u m a n Services (HHS) management. As a result, F D A ' s drug inspection program re-
cently received a substantial increase in drug inspection resources. Other top headquarters managements have also utilized the national tradeoff curve to assist them in planning and justifying future resource levels. As a result of this initial experience, F D A management wants NMIS, its probability estimates and tradeoff curves, to become an integral part of their strategic planning of the surveillance of the pharmaceutical industry. With further application, the outputs from N M I S will be used by district and headquarters managements as a tool to assist them in comparing drug industry performance. Future computer development plans are for a PC version of N M I S and the development of an interactive data base retrieval system to examine the regulatory surveillance information.
References Bowman, V.J. (1976), "On the relationship of the Tchebycheff norm and the efficient frontier of multiple-criteria objectives", in: H. Thiriez and S. Zionts (eds.), Multiple Criteria Decision Making, Springer, Berlin. Cohon, J.L. (1978), Multiobjective Programming and Planning Academic Press, New York. Klimberg, R.K. (1983), "An analysis of various disaggregations of the DQA inspection data: FY1978-1982", FDA Working Paper. Klimberg, R.K. (1987), "Inspectional scheduling: A two-stage decision support system using multiple objectives", Doctoral Thesis, The Johns Hopkins University. Klimberg, R.K., ReVelle, C., and Cohon, J.L. (1989), "NMIS: A national model for inspection selection with limited resources", in: A.G. Lockett and G. Islei (eds.), Improving Decision Making in Organisations, Springer-Verlag, Berlin. Steuer, R.E., and Choo, E.U. (1983), "An interactive weighted Tchebycheff procedure for multiple objective programruing", Mathematical Programming 26, August 1983. Swets, J.A. (1988), "Measuring the accuracy of diagnostic systems", Science June 1988.