- Email: [email protected]
A model for the evaluation of computer software packages
Comput. & Indus. Engng Yol. 7, No. 4, pp. 309-315, 1983
0360--8352/83 $3.00+ .00 Pergamon Press Ltd.
Printed in Great Britain.
A MODEL FOR THE EVALUATION OF COMPUTER SOFTWARE PACKAGES G. LARRYSANDERSt College of Business Administration, Texas Tech University, U.S.A. PARVIZ GHANDFOROUSH:~ College of Business, Virginia Polytechnic Institute and State University, U.S.A.
and LARRYM. AUSTIN§ College of Business Administration, Texas Tech University, Lubbock, TX 79409, U.S.A.
(Received in revised form June 1982) Abstract--The tremendous increase in recent years in the demand for, and availability of, computer software packages, has led researchers to investigate better ways of evaluating them for selection. The selection of software packages is confounded because criteria for evaluation are dependent on the system type. For example, decision support systems should be judged largely on the basis of subjective criteria, while transaction processing systems can be evaluated primarily on quantifiable factors. We discuss the use of a well-known model used in site selection, as a means to evaluate software packages according to subjective or objective factors. We illustrate application of the model with a small example. 1. INTRODUCTION
In the past few years, there has been a tremendous increase in the demand for computer software packages, and software firms have produced a variety of such packages in response to this demand. These packages range from simple audit sampling programs to complex, highly sophisticated financial planningsystems. With this burgeoning demand has come an increased interest in methods of evaluation. Obviously, the software packages are standardized, rather than being tailored to the individual needs of companies and organizations. With a dozen financial packages available from different vendors, for example, how does a manager select the one with the best features--and at the right price--that will be suited to the information and processing needs of the organization? This problem is further complicated by the function and orientation of three different types of information systems. The first type, transaction processing systems (TPS), are designed for handling routine repetitive functions within an organization. An example of a TPS would be a payroll system. The second type, management information systems (MIS), are designed around and within the confines of transaction processing systems, and are typically oriented towards monitoring and controlling the utilization of organizational resources. The third type, decision support systems (DSS), are targeted towards assisting managers in resolving semistructured (and sometimes unstructured) decision processes [4]. Different evaluation criteria need to be applied to the three types of systems. TPS are concerned with improving the efficiency of routine organizational processes, and their goal is to process data quickly and at a reduced cost. The criteria used in evaluating TPS are largely objective and can be quantified. MIS are also intended to increase organizational efficiency; however, they also provide non-quantifiable benefits in the form of better insight for managerial decision making. However, the benefits of DSS are mostly non-quantifiable using the traditional cost/benefit approach[5], so that their orientation demands that they be evaluated primarily on their perceived contribution to the decision making process [8]. As Kleijnen observed: "... the cost/benefit approach does not seem to contribute much to the analysis of computerized information systems."[6]. Thus, a model for evaluating software packages must be able to handle both subjective factors, critical to DSS success, and objective factors. tLecturer in Management Information Systems. ~:Assistant Professor of Quantitative Sciences. §Associate Dean and Professor of Quantitative Sciences. 309
310
G . L . SANDERS et al. 2. AN E V A L U A T I O N
MODEL
In order to structure the process of selecting a software package given the range of criteria, a model was sought that had the following properties. (1) The ability to evaluate transaction processing systems, management information systems, and decision support systems. (2) The ability to transform qualitative considerations into numerical units; (3) The ability to express both quantitative and qualitative ratings in the same numerical units: (4) The ability to allow the decision maker to explicitly express his or her judgment as to the relative importance of quantitative and qualitative factors; and (5) The capacity for performing sensitivity analysis on the results of the evaluation. As is often the case in the decision sciences, models developed specifically to address a problem in one area, are easily adaptable to other, seemingly unrelated, problem areas. The "key" is to recognize the broad similarities between problems rather than their operational dissimilarities. Such is the case in the present situation.
The Brown-Gibson multidimensional location model One key strategic problem in operations management is the optimal location of physical facilities. These decisions usually represent the commitment of significant financial resources, and are among the most critical decisions that managers must make. Consequently, a great deal of research has been done in developing models to assist decision makers in this regard. One such model is the Brown-Gibson Multidimensional Location Model[l], that first appeared in print in 1972. The model expresses quantitative considerations as monetary costs, and transforms the total cost for each feasible site into a "score" between zero and one (the scores sum to one). For each qualitative factor, the model translates nominal ratings into numerical scores, and allows the decision maker to express his view of the relative importance of the subjective criteria by weighting them (both the scores for each site on each criterion, and the associated weighted scores, sum to one). The maker then selects a weighting that represents his subjective view of the relative importance of the aggregate objective score, in relation to the aggregate subjective score. Finally, certain critical factors are considered; these are "knockout" criteria that may eliminate some sites at the outset. The Brown-Gibson model is essentially an enhanced scoring model. One advantage of scoring models is that their use encourages decision makers to become more cognizant of decision-related criteria and attributes, the net effect being a decision process that reflects the complexities of the decision environment [6]. In mathematical notation, the Brown-Gibson Multidimensional Location Model is stated as follows.
LMi = CFM~ - [X • OFMi + (1 - X ) . SFM~],
i = 1. . . . . n,
(l)
where LM~ = the location measure for site i; CFM~ = 0 or 1, the critical factor measure for location i; OFM~ = the objective factor measure for location i, 0 ~
k OFMi
=
1"
i-I
SFMi = the subjective factor measure for location i, 0 ~
~
SFM~ =
1;
i=1
X = the weight given to the objective factor measure, 0 ~
A modelfor the evaluationof computersoftwarepackages
311
for each site is then computed as the harmonic mean of the n factors. That is: 1
OFCI • i=l
Note in (2) that the larger the OFC~ (cost), the smaller the OFMi (score). To compute SFM~, the decision maker first rates each site on each subjective factor using a nominal scale. For example, availability of skilled labor for the sites may be characterized as: poor, marginal, below average, good, very good or excellent. These nominal ratings are then transformed to numerical scores by assigning ordinal values, in increasing order of desirability, to the nominal categories. The subjective weights (SW~k = factor) are calculated as follows: SWik = SWik ~SWik
k = 1. . . . . n.
(3)
i=1
Obviously, the subjective scores for each of the k factors sum to one. The decision maker then determines subjectively the relative importance of the K subjective factors. Brown and Gibson suggested using a binary preference approach to rank the factors from 1 to K, with the most preferred receiving a score of K, etc. However, any scoring system that assigns numerical ratings in increasing order of preference may be used. The subjective factor weights SFWk are then computed as follows: SFWk SFWk = K SFWk
k = 1. . . . . K.
(4)
k=l
Again, the subjective factor scores sum to one. The subjective factor measures for each site are then computed as follows. K
SFMi = ~ (SFWk • SWik),
i = 1. . . . . n.
(5)
k=l
Because both the subjective factor scores and the subjective scores sum to one, it is obvious that the subjective factor measures sum to one as well. Thus, we have expressed both objective factor measures and subjective factor measures for each site as numerical socres on the same scale. All that remains for the decision maker to do is to select a value for X--the relative weight he wishes to place on the objective factor measure.
Sensitivity analysis Rather than explicitly choosing a value for X, we recommend plotting the n linear relationships (X is the independent variable, and the LMi are the n dependent variables), and examining the resulting relationships among sites. Many times, several of the sites are dominated by one or more sites (i.e. no value of X would indicate that these sites should be selected). These sites can be discarded, and the analysis focused on fewer candidates. This process will be illustrated by an example problem in the next section. 3. EVALUATION OF COMPUTER SOFTWARE The similarity between the site selection problem and the software selection problem is obvious. Both have quantitative and qualitative considerations that must be addressed, and both require the decision maker to select one of n competing alternatives. The Brown-Gibson Multidimensional Location Model has been used to select landing sites for helicopters employed in logging operations, to choose plant sites for the construction of mobile homes and for pre-fabricated houses [3] and for other purposes. The model has also gained acceptance by its
312
G, L. SANDERSet al.
inclusion in several textbooks (e.g. [2], [7]). Thus, we propose the use of this model in the evaluation and selection of computer software packages. In fact, the only notational change needed is to define: CSMi = Computer Software Measure for the ith package, so that, in the model discussed above, CSM~ = LM~. The application of the model to computer software is best illustrated by an example.
Software evaluation criteria There are many dimensions to the software selection process. In terms of critical, objective, and subjective factors, the following dimensions have been selected for this illustration:
Critical factors (1) Hardware compatibility If the package won't run on our computer, it is obviously useless to consider it. (2) Operational compatibility If we need a package for financial planning, an auditing package is inappropriate.
Objective factors (1) Initial purchase cost (2) Costs of operation, maintenance, and modification (3) Costs of training users and operators
Subjective factors (1) (2) (3) (4) (5) (6) (7)
Quality of program documentation. Ease of use by managers and analysts. Information quality. Provision of audit trails. Expandability. "Track record" of vendor. Quality of security system.
An example Our example involves the decision to acquire a financial accounting package from among six available software packages. Preliminary investigation reveals that two of the packages will not run on the company's minicomputer, and a third will not support enough accounts. These three packages are not considered further, since they are assigned CFMi = 0. The other packages are labeled A, B and C and the objective factor costs are exhibited in Table 1.
Table 1. Objective factor costs Packages Costs
A
B
C
Purchase Cost
$ 9,000
$13,000
$11,000
Cost of Operation, Maintenance, Modification
$ 2,000
$ 1,500
$ 1,000
Cost of Training Users, Operators
$ 1,000
$ 1,500
$ 1,000
Total Cost
$12,000
$16,000
$13,000
313
A model for the evaluation of computer software packages
The objective factor measures are computed as follows. Let S = 1/12000+ 1/16000+ 1/13000. Then OFMA = (1/2000)S = 0.374 OFMB = (1/16000)S = 0.281 OFMc = (1/13000)S = 0.345. Turning to the subjective factors, the seven subjective factors have been characterized using nominal scales, and the results are exhibited in Table 2. Noting that we have chosen to use a simple ordinal ranking scheme for the seven subjective factors on each package, the resulting subjective weights are shown in Table 3. Observe that subjective factor (4), "Audit Trails," received the same rating for all three packages, and technically may be discarded. However, we retain it for clarity in discussing the example. The subjective factors are ordered in descending order of importance, and we use a simple ordering scheme to represent the subjective factor scores; that is, the jth score = 8 - j, where j is the index number of the factor. Therefore SFWj=(8-j)/y~.=
i=(8-j)/28.
The subjective factor measure can now be computed, as in (3). For example, SFMA = (2/6). (7/28) + (2/5). (6/28) + (1/6). (5/28) + (1/3). (4/28) + (3/6). (3/28) + (1/6). (2/28) + (1/6) • (1/28) = 0.318. Likewise SFMB = 0.377;
SFMc = 0.305.
Table 2. Subjective factor evfluations (Rank) Packages Subjective Factors
A
B
C
1. Quality of Documentation
Good (2)
Complete (3)
Sketchy (1)
2. Ease of Use
Simple (2)
Complicated {1)
Simple (2)
3. Information Quality
Good (i)
Excellent (3)
Very Good (2
4. Audit Trails
Yes (i)
Yes (I)
Yes (I)
5. Expandabillty
Yes (3)
No (i)
Possibly (2)
Questionable(l]
Excellent (3)
Good (2)
Poor (i)
Acceptable (3)
Harginal (2)
6. Vendor "Track Record" 7. Quality of Security
314
G, L. SANDERSet al. Table 3. Subjective weights Packages Subjective Factors
A
B
C
i. Quality of Documentation
2/6
3/6
i/6
2. Ease of Use
2/5
1/5
2/5
3. Information Quality
1/6
3/6
2/6
4. Audit Trails
i/3
I/3
I/3
5. Expandability
3/6
I/6
2/6
6. Vendor "Track Record"
1/6
3/6
2/6
7. Quality of Security
I/6
3/6
2/6
We are now prepared to perform the sensitivity analysis on our evaluation model. We note that CSMA = 0.374(X) + 0.318(1 - X) -- 0.056(X) + 0.318 CSMB = 0.281(X) + 0.377(1 - X) = 0.377 - 0.096(X) CSMc = 0.345(X) + 0.305(1 - X) = 0.040(X) + 0.305. The three linear equations are plotted in Fig. 1. Note in the example that package C is dominated by package A and can be discarded as a viable choice. We also see that package B is preferred for 0 ~
In summary, we have proposed a slightly altered version of the Brown-Gibson Multidimensional Location Model as an alternative to current models used to evaluate computer software packages. One particularly appealing aspect of this model is that a decision support system can be easily built to accommodate the decision maker. Brown and Gibson[3] developed a FORTRAN program to operationalize their model, and we constructed an interactive program in BASIC that requires little technical knowledge of the user. However, the model is not without shortcomings, and some caveats are in order. First, use of the model does not remove subjectivity from the evaluation process--it merely quantifies and scales it. Therefore, the care with which the subjective factors are ranked, and the nominal evaluation of each factor for each package, are critical. Second, the use of the harmonic mean to compute the objective factor weights can present difficulties, in that the model is sensitive to differences in cost only in a multiplicative sense. For example, two packages with costs of $10,000 and $12,000, respectively, would yield objective factor measures identical with those of packages costing $100,000 and $120,000, respectively-even though the cost differentials are $2,000 and $20,000, respectively. Decision makers should be sensitive to this feature of the model, and take it into account--perhaps in the determination of X--when making the final decision.
A model for the evaluation of computer software packages
315
CSMi
0.6
0.5
0.4
CSMA CSMc 0.3
CSMB
0.2
0.1
0
0.2
0.4
0.6
0.8
1.0
X
Fig. 1. Sensitivity analysis: an example. REFERENCES 1. P. A. Brown & D. F. Gibson, A quantified model for site selection--Application to a multiplant location problem. AIIE Trans. 4, 1-10 (March 1972). 2. Elwood S. Buffa, Modern Production/Operations Management, 6th Edn. Wiley, New York (1980). 3. D. F. Gibson, Private communication, 1982. 4. Peter G. Keen & Michael Scott Morton, Decision Support Systems: An Organizational Perspective. Addison-Wesley, Reading, Mass. (1978). 5. Peter G. Keen, "Value Analysis: Justifying Decision Support Systems," MIS Quarterly, Vol. 5, No. 1 (March 1981), pp. 1-15. 6. Jack P. C. Kleijnen, Computers and Profits: Quantifying Financial Benefits of Information, Addison-Wesley, Reading, Mass. (1980). 7. James L. Riggs, Production Systems: Planning, Analysis, and Control. Wiley, New York (1981). 8. Gemma M. Welsch, A multidimensional measure of perceived decision support system implementation success. Working paper, School of Accounting, DePaul University, Chicago, Illinois (1981).
0360--8352/83 $3.00+ .00 Pergamon Press Ltd.
Printed in Great Britain.
A MODEL FOR THE EVALUATION OF COMPUTER SOFTWARE PACKAGES G. LARRYSANDERSt College of Business Administration, Texas Tech University, U.S.A. PARVIZ GHANDFOROUSH:~ College of Business, Virginia Polytechnic Institute and State University, U.S.A.
and LARRYM. AUSTIN§ College of Business Administration, Texas Tech University, Lubbock, TX 79409, U.S.A.
(Received in revised form June 1982) Abstract--The tremendous increase in recent years in the demand for, and availability of, computer software packages, has led researchers to investigate better ways of evaluating them for selection. The selection of software packages is confounded because criteria for evaluation are dependent on the system type. For example, decision support systems should be judged largely on the basis of subjective criteria, while transaction processing systems can be evaluated primarily on quantifiable factors. We discuss the use of a well-known model used in site selection, as a means to evaluate software packages according to subjective or objective factors. We illustrate application of the model with a small example. 1. INTRODUCTION
In the past few years, there has been a tremendous increase in the demand for computer software packages, and software firms have produced a variety of such packages in response to this demand. These packages range from simple audit sampling programs to complex, highly sophisticated financial planningsystems. With this burgeoning demand has come an increased interest in methods of evaluation. Obviously, the software packages are standardized, rather than being tailored to the individual needs of companies and organizations. With a dozen financial packages available from different vendors, for example, how does a manager select the one with the best features--and at the right price--that will be suited to the information and processing needs of the organization? This problem is further complicated by the function and orientation of three different types of information systems. The first type, transaction processing systems (TPS), are designed for handling routine repetitive functions within an organization. An example of a TPS would be a payroll system. The second type, management information systems (MIS), are designed around and within the confines of transaction processing systems, and are typically oriented towards monitoring and controlling the utilization of organizational resources. The third type, decision support systems (DSS), are targeted towards assisting managers in resolving semistructured (and sometimes unstructured) decision processes [4]. Different evaluation criteria need to be applied to the three types of systems. TPS are concerned with improving the efficiency of routine organizational processes, and their goal is to process data quickly and at a reduced cost. The criteria used in evaluating TPS are largely objective and can be quantified. MIS are also intended to increase organizational efficiency; however, they also provide non-quantifiable benefits in the form of better insight for managerial decision making. However, the benefits of DSS are mostly non-quantifiable using the traditional cost/benefit approach[5], so that their orientation demands that they be evaluated primarily on their perceived contribution to the decision making process [8]. As Kleijnen observed: "... the cost/benefit approach does not seem to contribute much to the analysis of computerized information systems."[6]. Thus, a model for evaluating software packages must be able to handle both subjective factors, critical to DSS success, and objective factors. tLecturer in Management Information Systems. ~:Assistant Professor of Quantitative Sciences. §Associate Dean and Professor of Quantitative Sciences. 309
310
G . L . SANDERS et al. 2. AN E V A L U A T I O N
MODEL
In order to structure the process of selecting a software package given the range of criteria, a model was sought that had the following properties. (1) The ability to evaluate transaction processing systems, management information systems, and decision support systems. (2) The ability to transform qualitative considerations into numerical units; (3) The ability to express both quantitative and qualitative ratings in the same numerical units: (4) The ability to allow the decision maker to explicitly express his or her judgment as to the relative importance of quantitative and qualitative factors; and (5) The capacity for performing sensitivity analysis on the results of the evaluation. As is often the case in the decision sciences, models developed specifically to address a problem in one area, are easily adaptable to other, seemingly unrelated, problem areas. The "key" is to recognize the broad similarities between problems rather than their operational dissimilarities. Such is the case in the present situation.
The Brown-Gibson multidimensional location model One key strategic problem in operations management is the optimal location of physical facilities. These decisions usually represent the commitment of significant financial resources, and are among the most critical decisions that managers must make. Consequently, a great deal of research has been done in developing models to assist decision makers in this regard. One such model is the Brown-Gibson Multidimensional Location Model[l], that first appeared in print in 1972. The model expresses quantitative considerations as monetary costs, and transforms the total cost for each feasible site into a "score" between zero and one (the scores sum to one). For each qualitative factor, the model translates nominal ratings into numerical scores, and allows the decision maker to express his view of the relative importance of the subjective criteria by weighting them (both the scores for each site on each criterion, and the associated weighted scores, sum to one). The maker then selects a weighting that represents his subjective view of the relative importance of the aggregate objective score, in relation to the aggregate subjective score. Finally, certain critical factors are considered; these are "knockout" criteria that may eliminate some sites at the outset. The Brown-Gibson model is essentially an enhanced scoring model. One advantage of scoring models is that their use encourages decision makers to become more cognizant of decision-related criteria and attributes, the net effect being a decision process that reflects the complexities of the decision environment [6]. In mathematical notation, the Brown-Gibson Multidimensional Location Model is stated as follows.
LMi = CFM~ - [X • OFMi + (1 - X ) . SFM~],
i = 1. . . . . n,
(l)
where LM~ = the location measure for site i; CFM~ = 0 or 1, the critical factor measure for location i; OFM~ = the objective factor measure for location i, 0 ~
k OFMi
=
1"
i-I
SFMi = the subjective factor measure for location i, 0 ~
~
SFM~ =
1;
i=1
X = the weight given to the objective factor measure, 0 ~
A modelfor the evaluationof computersoftwarepackages
311
for each site is then computed as the harmonic mean of the n factors. That is: 1
OFCI • i=l
Note in (2) that the larger the OFC~ (cost), the smaller the OFMi (score). To compute SFM~, the decision maker first rates each site on each subjective factor using a nominal scale. For example, availability of skilled labor for the sites may be characterized as: poor, marginal, below average, good, very good or excellent. These nominal ratings are then transformed to numerical scores by assigning ordinal values, in increasing order of desirability, to the nominal categories. The subjective weights (SW~k = factor) are calculated as follows: SWik = SWik ~SWik
k = 1. . . . . n.
(3)
i=1
Obviously, the subjective scores for each of the k factors sum to one. The decision maker then determines subjectively the relative importance of the K subjective factors. Brown and Gibson suggested using a binary preference approach to rank the factors from 1 to K, with the most preferred receiving a score of K, etc. However, any scoring system that assigns numerical ratings in increasing order of preference may be used. The subjective factor weights SFWk are then computed as follows: SFWk SFWk = K SFWk
k = 1. . . . . K.
(4)
k=l
Again, the subjective factor scores sum to one. The subjective factor measures for each site are then computed as follows. K
SFMi = ~ (SFWk • SWik),
i = 1. . . . . n.
(5)
k=l
Because both the subjective factor scores and the subjective scores sum to one, it is obvious that the subjective factor measures sum to one as well. Thus, we have expressed both objective factor measures and subjective factor measures for each site as numerical socres on the same scale. All that remains for the decision maker to do is to select a value for X--the relative weight he wishes to place on the objective factor measure.
Sensitivity analysis Rather than explicitly choosing a value for X, we recommend plotting the n linear relationships (X is the independent variable, and the LMi are the n dependent variables), and examining the resulting relationships among sites. Many times, several of the sites are dominated by one or more sites (i.e. no value of X would indicate that these sites should be selected). These sites can be discarded, and the analysis focused on fewer candidates. This process will be illustrated by an example problem in the next section. 3. EVALUATION OF COMPUTER SOFTWARE The similarity between the site selection problem and the software selection problem is obvious. Both have quantitative and qualitative considerations that must be addressed, and both require the decision maker to select one of n competing alternatives. The Brown-Gibson Multidimensional Location Model has been used to select landing sites for helicopters employed in logging operations, to choose plant sites for the construction of mobile homes and for pre-fabricated houses [3] and for other purposes. The model has also gained acceptance by its
312
G, L. SANDERSet al.
inclusion in several textbooks (e.g. [2], [7]). Thus, we propose the use of this model in the evaluation and selection of computer software packages. In fact, the only notational change needed is to define: CSMi = Computer Software Measure for the ith package, so that, in the model discussed above, CSM~ = LM~. The application of the model to computer software is best illustrated by an example.
Software evaluation criteria There are many dimensions to the software selection process. In terms of critical, objective, and subjective factors, the following dimensions have been selected for this illustration:
Critical factors (1) Hardware compatibility If the package won't run on our computer, it is obviously useless to consider it. (2) Operational compatibility If we need a package for financial planning, an auditing package is inappropriate.
Objective factors (1) Initial purchase cost (2) Costs of operation, maintenance, and modification (3) Costs of training users and operators
Subjective factors (1) (2) (3) (4) (5) (6) (7)
Quality of program documentation. Ease of use by managers and analysts. Information quality. Provision of audit trails. Expandability. "Track record" of vendor. Quality of security system.
An example Our example involves the decision to acquire a financial accounting package from among six available software packages. Preliminary investigation reveals that two of the packages will not run on the company's minicomputer, and a third will not support enough accounts. These three packages are not considered further, since they are assigned CFMi = 0. The other packages are labeled A, B and C and the objective factor costs are exhibited in Table 1.
Table 1. Objective factor costs Packages Costs
A
B
C
Purchase Cost
$ 9,000
$13,000
$11,000
Cost of Operation, Maintenance, Modification
$ 2,000
$ 1,500
$ 1,000
Cost of Training Users, Operators
$ 1,000
$ 1,500
$ 1,000
Total Cost
$12,000
$16,000
$13,000
313
A model for the evaluation of computer software packages
The objective factor measures are computed as follows. Let S = 1/12000+ 1/16000+ 1/13000. Then OFMA = (1/2000)S = 0.374 OFMB = (1/16000)S = 0.281 OFMc = (1/13000)S = 0.345. Turning to the subjective factors, the seven subjective factors have been characterized using nominal scales, and the results are exhibited in Table 2. Noting that we have chosen to use a simple ordinal ranking scheme for the seven subjective factors on each package, the resulting subjective weights are shown in Table 3. Observe that subjective factor (4), "Audit Trails," received the same rating for all three packages, and technically may be discarded. However, we retain it for clarity in discussing the example. The subjective factors are ordered in descending order of importance, and we use a simple ordering scheme to represent the subjective factor scores; that is, the jth score = 8 - j, where j is the index number of the factor. Therefore SFWj=(8-j)/y~.=
i=(8-j)/28.
The subjective factor measure can now be computed, as in (3). For example, SFMA = (2/6). (7/28) + (2/5). (6/28) + (1/6). (5/28) + (1/3). (4/28) + (3/6). (3/28) + (1/6). (2/28) + (1/6) • (1/28) = 0.318. Likewise SFMB = 0.377;
SFMc = 0.305.
Table 2. Subjective factor evfluations (Rank) Packages Subjective Factors
A
B
C
1. Quality of Documentation
Good (2)
Complete (3)
Sketchy (1)
2. Ease of Use
Simple (2)
Complicated {1)
Simple (2)
3. Information Quality
Good (i)
Excellent (3)
Very Good (2
4. Audit Trails
Yes (i)
Yes (I)
Yes (I)
5. Expandabillty
Yes (3)
No (i)
Possibly (2)
Questionable(l]
Excellent (3)
Good (2)
Poor (i)
Acceptable (3)
Harginal (2)
6. Vendor "Track Record" 7. Quality of Security
314
G, L. SANDERSet al. Table 3. Subjective weights Packages Subjective Factors
A
B
C
i. Quality of Documentation
2/6
3/6
i/6
2. Ease of Use
2/5
1/5
2/5
3. Information Quality
1/6
3/6
2/6
4. Audit Trails
i/3
I/3
I/3
5. Expandability
3/6
I/6
2/6
6. Vendor "Track Record"
1/6
3/6
2/6
7. Quality of Security
I/6
3/6
2/6
We are now prepared to perform the sensitivity analysis on our evaluation model. We note that CSMA = 0.374(X) + 0.318(1 - X) -- 0.056(X) + 0.318 CSMB = 0.281(X) + 0.377(1 - X) = 0.377 - 0.096(X) CSMc = 0.345(X) + 0.305(1 - X) = 0.040(X) + 0.305. The three linear equations are plotted in Fig. 1. Note in the example that package C is dominated by package A and can be discarded as a viable choice. We also see that package B is preferred for 0 ~
In summary, we have proposed a slightly altered version of the Brown-Gibson Multidimensional Location Model as an alternative to current models used to evaluate computer software packages. One particularly appealing aspect of this model is that a decision support system can be easily built to accommodate the decision maker. Brown and Gibson[3] developed a FORTRAN program to operationalize their model, and we constructed an interactive program in BASIC that requires little technical knowledge of the user. However, the model is not without shortcomings, and some caveats are in order. First, use of the model does not remove subjectivity from the evaluation process--it merely quantifies and scales it. Therefore, the care with which the subjective factors are ranked, and the nominal evaluation of each factor for each package, are critical. Second, the use of the harmonic mean to compute the objective factor weights can present difficulties, in that the model is sensitive to differences in cost only in a multiplicative sense. For example, two packages with costs of $10,000 and $12,000, respectively, would yield objective factor measures identical with those of packages costing $100,000 and $120,000, respectively-even though the cost differentials are $2,000 and $20,000, respectively. Decision makers should be sensitive to this feature of the model, and take it into account--perhaps in the determination of X--when making the final decision.
A model for the evaluation of computer software packages
315
CSMi
0.6
0.5
0.4
CSMA CSMc 0.3
CSMB
0.2
0.1
0
0.2
0.4
0.6
0.8
1.0
X
Fig. 1. Sensitivity analysis: an example. REFERENCES 1. P. A. Brown & D. F. Gibson, A quantified model for site selection--Application to a multiplant location problem. AIIE Trans. 4, 1-10 (March 1972). 2. Elwood S. Buffa, Modern Production/Operations Management, 6th Edn. Wiley, New York (1980). 3. D. F. Gibson, Private communication, 1982. 4. Peter G. Keen & Michael Scott Morton, Decision Support Systems: An Organizational Perspective. Addison-Wesley, Reading, Mass. (1978). 5. Peter G. Keen, "Value Analysis: Justifying Decision Support Systems," MIS Quarterly, Vol. 5, No. 1 (March 1981), pp. 1-15. 6. Jack P. C. Kleijnen, Computers and Profits: Quantifying Financial Benefits of Information, Addison-Wesley, Reading, Mass. (1980). 7. James L. Riggs, Production Systems: Planning, Analysis, and Control. Wiley, New York (1981). 8. Gemma M. Welsch, A multidimensional measure of perceived decision support system implementation success. Working paper, School of Accounting, DePaul University, Chicago, Illinois (1981).