- Email: [email protected]
A Multi-Criteria Economic Evaluation Framework for Control System Configuration – Framework and Case Study
2nd IFAC Workshop on Dependable Control of Discrete Systems DCDS’09 Bari, Italy, June 10-12, 2009
A Multi-Criteria Economic Evaluation Framework for Control System Configuration – Framework and Case Study Peng Zhao*, Yan Lu **, Mohsen A. Jafari*, Davood Golmohammadi* * Dept. of Industrial & Systems Engineering, Rutgers University, Piscataway, NJ 08854, USA ** Siemens Corporate Research, Princeton, NJ 08540, USA (email: [email protected]) Abstract: The underlying methodology includes three main components: A digital factory/simulation model, a knowledge base/expert system and a multi-criteria evaluation model to compute the scores of different control designs and configurations on economic terms. For the economic evaluation, an existing methodology, Non-Traditional Capital Investment Criteria (NCIC) is used which allows us to incorporate into the analysis both traditional criteria, readily measurable in financial benefits, and non-traditional criteria that are not easily measurable based on their financial benefits. These non-financial benefits could be quantitative (measurable, but not necessarily in dollars) or qualitative (not measurable at all). An example is used to demonstrate this method by comparing the economic value of two control design alternatives for a singulator-- a centralized control where one motion controller controls all the axes, and a distributed configuration where the control of the axes are taken by two controllers, working autonomously, and interacting whenever necessary. Keywords: Multi-criteria Economic Evaluation, Distributed Control, Analytical Hierarchy Process, Digital Factory.
1. INTRODUCTION In a recent article, Lu and Jafari (2007) echoed the concerns raised by Hall (2005) and others about the lack of widespread interest on the use of Distributed Artificial Intelligent Control, or “DAIC” in industrial applications. In particular, these authors stressed the lack of any appropriate commercially available tool to compute (i) the cost of design and cost of ownership of DAIC systems, and (ii) the value of these systems in terms of potential benefits that they can offer compared to more traditional technologies. In this article a framework is proposed for the economic evaluation of distributed control designs against centralized ones. The underlying methodology has three main components: (i) A digital factory/simulation model to estimate system performance and some measurable factors under different configurations, (ii) A knowledge base/expert system which provides guidelines for establishing decision criteria, and interacts with the user to estimate other quantifiable factors whose values cannot be obtained from simulations and weigh the factors for alternative design and configurations. (iii) A multi-criteria evaluation model to compute the scores of different control designs and configurations on economic terms. For the economic evaluation, an existing methodology, Non-Traditional Capital Investment Criteria (NCIC), developed by Boucher and MacStravic (1991) and Gogus and Boucher (1998) is used. This approach allows us to incorporate into the analysis both traditional criteria, readily measurable in financial benefits, and non-traditional criteria that are not easily measurable based on their financial benefits. These non-financial benefits could be quantitative
978-3-902661-44-9/09/$20.00 © 2009 IFAC
(measurable, but not necessarily in dollars) or qualitative (not measurable at all). Furthermore, the approach proposed here also deals with the condition when the outcome or impact of some evaluation criteria within alternatives falls into a fuzzy domain. While the underlying techniques proposed are general and can be applied to different applications, in this article we will focus our analysis on a case study for easy illustration. The example used in the case study is the control design for a simplified version of the parcel “singulator” of Siemens. We want to compare the economic value of the singulator with two control designs: (i) A centralized control where one motion controller controls all the modules of the singulator, (ii) A distributed configuration where the control of the singulator modules are divided into two controllers, working autonomously, and interacting whenever necessary. 2. ECONOMIC EVALUATION FRAMEWORK Figure 1 below illustrates the proposed framework for the economic evaluation of alternative control configurations. The kernel of the framework is a Multi-Criteria Economic Evaluation Engine, which calculates a score for each specific control design. By comparing the scores of the alternative control configurations, the design with the highest value is considered as the best solution. The inputs to the economic evaluation engine come from: (i) Knowledge Base (KB) module which provides decision hierarchy consisting of a set of alternatives, their criteria categories, and the criteria, as well as some application domain specific data – for instance, how much can be saved for maintenance and training (ii)
140
10.3182/20090610-3-IT-4004.0027
Digital Factory (DF) module providing quantifiable measures to the evaluation engine for each configuration. Some typical quantifiable measures are system average throughput and utilization of various resources in the system. (iii) Domain Expertise (DE) module determining weights and rankings of various evaluation criteria used in the model. In this section, we will describe the details of the modules and the steps to follow for the economic evaluation of distributed control designs against centralized ones.
Knowledge Base
User’s control requirements & alternative configurations
Digital Factory
Decision hierarchy & domain data
Quantifiable measures Pairwise comparison
Multi-criteria Economic Evaluation Engine
Economically optimal configuration
Domain Expertise
Figure 1 Framework architecture 2.1 Multi-Criteria Economic Evaluation Engine Many multi-criteria economic models (Chandra et al. 1988, Wabalickis et al. 1988 and Falkner et al. 1990) have been proposed in the last decade for evaluation and justification of advanced technology investments. Saaty’s Analytic Hierarchy Process (or “AHP”) (1980) and Boucher and MacStravicis NCIC model (1991) are two of the most popular models. The NCIC decision modeling approach is similar in some respects to AHP. Both NCIC and AHP start with subjective pairwise comparison of process to rate criteria, and then calculate the importance of each criterion using the eigenvector approach. Maximum eigenvector is used in both models to check for logical consistency. However, unlike AHP, NCIC interprets the relative importance of each criterion in monetary terms. Furthermore, NCIC provides additional checks on economic consistency and ranks alternatives with respect to their worth. NCIC has been used in different applications. Boucher (1998) examined the use of multi-criteria modeling at machine level automation investment. The study involved the evaluation of generic types of filling equipment used in packaged food manufacturing. In another article, Boucher (1996) examined an investment decision problem at production line automation level. Gogus and Boucher (1998) extended NCIC to fuzzy domain for the cases where experts’ opinion or level of preference is not crisp and contains varying degrees of impression. Imprecision is also possible in outcomes of events that influence an alternative. In this article we adopt the NCIC decision model for our control design economic evaluation framework, which involves construction of a decision hierarchy, pairwise comparison of criteria within each category included in the hierarchy, consistency analysis, computation of criteria weights for each alternative, and computation of the net worth for each alternative. We will show how these steps can be applied to our case study.
2.2 Constructing the Decision Hierarchy and Collecting Domain Data Page margins The construction of a decision hierarchy is the first and the most important step in our analysis. It includes definition of hierarchy levels, breakdown of evaluation criteria categories, and association of cost or benefit to each of these categories. At the same time, many of the unquantifiable criteria require elicitation of expert judgment and unbiased estimations. While some of these requirements are for the end-users, others involve the technology owners and system integrators. A decision hierarchy consists of a set of alternatives, their criteria categories, and their criteria. In the first step the investment alternatives and their evaluation criteria are identified. The evaluation criteria are the expected impacts or outcomes of the investment alternatives. These outcomes may be quantitative (measurable) or qualitative. In addition, these outcomes may be benefits which increase the value of the alternative or costs which decrease the value of the alternative. All significant outcomes of an investment alternative should be reflected in the choice of evaluation criteria. Evaluation criteria are specific to an alternative. A single criterion may be assigned to only one alternative or to all alternatives, if applicable. Also, a particular criterion may be a benefit for one alternative and a cost for another alternative. This information should be reflected in the decision hierarchy. Choosing the decision criteria is the most crucial step of the analysis. Great care should be taken to select evaluation criteria that will represent all expected impacts of an investment alternative. Criteria should be quantified whenever possible. Furthermore, in order to avoid excessive comparison, criteria can be classified into categories. Figure 2 shows a schematic of a decision hierarchy.
Best Over all Alternative
Level 1
Level 2
Level 3 Level 4
Alternative 1
Category 1
Criteria
1
... Criteria L 1
...
. . ..
Category 2
Criteria
Alternative n
Category K
2
... Criteria L 2
...
Figure 2 A decision hierarchy
141
Criteria Ik
... Criteria Lk
...
. . ..
Criteria should be independent of each other. Consider, for example, quality improvement. Quality improvement can result in a cost saving due to less rework and scrap. Quality improvement can also result in less defective product reaching the customer, thus improving customer satisfaction. If the cost savings associated with less rework and scrap has been computed and included as part of the expected annual benefit, then the additional merit of the criterion labeled "Quality Improvement" is limited to that which is expected from improving customer satisfaction alone. If the value of less rework and scrap has not been computed as part of an alternative's annual benefit, then the quality improvement criterion represents the effect of both factors.The decision hierarchy used in this article reflects the views of the authors and by no means is being proposed as a general paradigm. We believe that for all practical purposes, a KB system will be required to address this major issue. The architecture of such KB system is outside the scope of this article. 2.3 Collecting Quantifiable Data For the non-quantifiable data, they can be obtained from some domain experts. But to collect the quantifiable data of the system before a system is built, digital factory may be the easiest way to be used in design phase. A digital factory offers an integrated approach to virtually run a process without having a real one. Simulation is the key technology within this concept. Event-based, time-based or hybrid simulation models can be built to any degree of details for conceptual system designs required by the analysis. By using simulation, some issues of the proposed system can be estimated. Almost in all cases and applications, materials flow and system layout must be included in these models. To have a fair comparison of various control configurations, information flow and communication networking also need to be part of the simulations. One can even go further and include business processes for design and implementation in the simulations. From these simulations, different performance measures associated with the criteria defined in the decision hierarchy for the evaluation can be computed for each control design and configuration. 2.4 Pair wise Comparison The next step of the analysis is to derive weights for the individual criteria, from which the implied annual benefit of criteria can be computed. These weights are based on the judgments of relative importance of criteria when compared in a pairwise fashion. For each alternative, pairwise comparison matrices are formed for each of the non-financial criteria categories (i.e., material conversion, information conversion, strategic).Comparison matrices for a criteria category of an alternative consist of all criteria within the group plus an additional criterion of annual Dollars. This annual dollars criterion represents the total dollar value of financial criteria in the Annual Dollars category. This criterion is measured in dollars and exists in the comparison matrix to provide a link between relative weights and dollar values. For each alternative, the relative importance of criteria is determined by making pairwise comparisons within the
benefit and cost hierarchies. We form pairwise comparison matrices for the non-financial criteria category by deriving weights for its individual evaluation criteria. From these weights the annual benefit or cost of a criterion is computed. These weights are based on expert judgment of relative importance of criteria when compared in pairwise fashion. To form a comparison matrix for a given criteria category, we include all the criteria in that category and a criterion representing the total dollar value of the financial criteria in the “Annual Dollars” category. The purpose of this criterion is to provide a reference in financial terms for calculating the relative importance of the non-financial criteria. We will assume that for some criteria the judgments will be fuzzy while for others it will be crisp. For fuzzy judgment either linguistic variables or fuzzy numbers can be used. We will use linguistic variables suggested by Gogus and Boucher: AS= “About the Same as”; SG=”Slightly Greater than”; MoG=“Moderately Greater than”; MuG=“Much Greater than”, or MMG=”Much Much Greater than”. An important aspect of linguistic fuzzy variables is that they are specific to individuals, but the membership function for the same variables for different individuals is different. Hence, this function needs to be determined on an individual basis. In the case of fuzzy numbers, individuals are asked to provide their responses with a range defined by lowest possible value, most likely value, and highest possible value. These are taken as the extreme points and mid point of a Triangular Fuzzy Number (or “TFN”). In case that linguistic fuzzy variables are used, one can still build TFNs from the membership functions (see Turksen 1988, Turksen 1991 and Gogus 1998 for details). The details of how to apply these modules and the economic evaluation engine into practice will be elaborated in the next section with the Singulator case study. 3. CASE STUDY – A SIMPLIFIED VERSION OF SIEMENS SINGULATOR Parcel singulator is widely used to convert an input stream of disorganized parcels into one or more single file output streams. The Siemens Singulator design integrates advanced real time vision with distributed mechanical design and motion control technology (Reznik 2004). The system kinematics is modularized into distributed (matrix-like) conveyor architecture, allowing parcels to be rotated and translated individually. As a product, Siemens Singulator uses a central controller to control all 7 by 12 actuators and the motion cycle is 33msec. If there are multiple CPUs to share the computation load of the motion control, both the system performance and robustness are expected to be improved. Our case study is motivated by comparing the distributed control design with the central design for such a singulator considering the economics values. For easier illustration of the concept of the economic analysis method, the case study is based on a simplified version of the singulator with only 4 by 8 actuators, which wouldn’t change the analysis result. The schematic of the simplified singulator is shown in Figure 3, with four main modules: the bulk conveyor, the singulator bed with a matrix like arrangement of actuated tiles, the steering wheel diverter (SWD), and a takeaway. Parcels with different shapes and orientations arrive on the bulk conveyor. The induction of parcels from the bulk conveyor to the
142
singulator bed is conditioned on keeping a specified minimum gap between the parcels. To realize this, the first two rows of tiles on the singulator bed move slower than the rest so as to separate the parcels. The following tiles move with a higher speed, where the parcels are already distanced apart with at least the minimum gap. Parcels move sequentially to the SWD and from there they are discharged through the takeaway conveyor. The induction speed is 0.5 m/sec while the discharge speed can be as high as 1.5 m/sec. Theoretically, the motion control configuration for the above system could span from a fully centralized system to a fully distributed system where each actuated part has its own controller. In this article, we only consider two configurations – a fully “centralized control” where all the modules and tiles are controlled by a single controller; and a “distributed configuration” where the control of the actuated parts are distributed to two controllers. In either configuration, the motion controller is responsible for computing the speeds and mechanical settings of the various actuators in the singulator. Settings computed in every cycle include speed of the bulk conveyor, speed of every individual tile or belt on the singulator bed, and orientation of every column in SWD. For a centralized configuration, the speeds for all the tiles on the 4x8 singulator matrix are set by a single motion controller. Thus, the maximum speeds for the singulator is conditioned by this controller’s cycle time. For the distributed configuration, the speed control is divided between the two controllers, each in charge of a 2x8 matrix of tiles. In the case of distributed configuration, we will assume that the each motion control cycle also includes a communication cycle when the two controllers must communicate to position those parcels sitting on tiles within both the matrices. Bulk conveyor
Singulator bed
SWD takeaway
We note that the impact of stoppage time on the system throughput rate is different for the two alternatives that we consider here.
Level 3
Operational/ safety
Annual dollars Annualized first cost
Level 4
Design & implementation
% of change in System throughput % of change in operation stoppage
System Dynamical Response
Change in design complexity
Ease of expansion
Increase in testing time Increase in training requirements
Improvement in system safety & fault tolerance Improvement in maintainability Cost more in system maintenance/supplies
Figure 4 Proposed breakdown of Level 3 & 4 In case of fuzzy outcomes, Level 3 and 4 need to be structured according to cost and benefit classes. A hierarchy is proposed below assuming a reasonable level of maturity for any technology used. Again, this hierarchy can be easily modified from one application to another. It is also noted that it is possible for a Level 3 category to be in both cost and benefit class, with different criteria at Level 4. According to the model of Figure 5, we assume reduction in time for the implementation of distributed control configuration. At the same time, it is also assumed that for the distributed configuration, the design complexity, testing time, and training requirements will increase compared to the centralized one, thus they are categorized as cost. Our rationale is that the implementation is often done by a third party, i.e., system integrator, which will supposedly have a mature level of technology for the implementation. But the design process, testing and training greatly involve in-house resources, which most probably will be less mature in terms of new technologies. Distributed Control
Figure 3 Simplified Singulator Benefits
3.1 Constructing Decision Hierarchy
Operational/ safety
We start NCIC by building a decision hierarchy model – it starts with the identification of design alternatives followed by setting up evaluation criteria. These criteria are the outcomes or the expected impacts of the design alternatives. These outcomes may be costs (decrease the value of the alternative) or benefit (increase the value of the alternative). For evaluation of an alternative distributed control design, the expected cost or benefit structure is considered in comparison to a base centralized configuration. Thus, all the costs and benefits for the distributed configuration will be calculated in an incremental basis of switching from a centralized configuration to a distributed one. Figure 4 illustrates our proposed list of four categories in Level 3, and evaluation criteria in each category in Level 4. Under Operational/Safety category, we have both % of change in throughput rate and % of change in stoppage time.
Costs
System Dynamical Response
% of change in System throughput % of change in operation stoppage
Ease of expansion
Annual dollars Annualized first cost
Design & implementation Increase in design complexity
Operational/ safety Cost more in system maintenance/supplies
Increase in testing time Increase in training requirements
Improvement in system safety & fault tolerance Improvement in system maintainability
Figure 5 Proposed costs and benefits hierarchy for distributed control configuration 3.2 Evaluation Prior to any further economic evaluation, the most likely or range of values have to be estimated for all the quantifiable measures that are mapped out in the above decision hierarchy model. While some of these measures can be obtained from a
143
knowledge base or a historical database, others require simulation of the alternative configurations. Table 1 lists the comparing criteria for the singuator case. It contains quantifiable and unquantifiable criteria. Row 1 is the monetary difference between the investments. This estimation was obtained from domain experts. Row 2 and 3 are quantifiable criteria with values computed from simulations. Our simulations indicate that, on the average, the distributed configuration provides 5% increase in throughput and 5% reduction in overall stoppage compared to the centralized system. The other criteria in the table are unquantifiable and were obtained through interviews with experts and also through investigation of the literature. Table 1 Quantifiable criteria and their data source Criteria Change in annualized cost in the investment % of change in throughput % of change in stoppage Change in safety & fault tolerance Change in maintainability Change in ease of expansion Change in design complexity Change in testing time Change in training requirements Change in system maintenance/supplies
Data Source KB & expert view Simulation Simulation KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view
Amount $100,000 5% -5% ---
Table 2 Pairwise comparison matrix for design/implementation under costs Design/implementation A B C A – Cost=100,000 ----B – Change in design complexity ----C – Increase in testing time ----D – Increase in training requirements
D
----
Table 3 Pairwise comparison matrix for design/implementation under costs Design/implementation A - Cost=100,000 B – Change in design complexity C – Increase in testing time D – Increase in training requirements
A -----MoG
B 1/MoG ------
C SG MoG
D AS SG
1/SG
1/MoG
------
1/SG
AS
1/SG
SG
------
3.4 Consistency Check & Irrational Weight Factors
---
Since each comparison is made independently, after the tables are filled, consistency of the decision maker’s judgment has to be checked. NCIC computes a consistency ratio that measures how far a decision-maker's judgments are from perfect consistency. The value of 0 means perfect consistency. It is recommended that pairwise comparison matrices having a consistency ratio greater than 0.20 should be reviewed. Highly inconsistent matrices can distort the final result. See Turksen 1988, Turksen 1991 and Gogus 1998 for details of consistency ratio.
-----------
3.3 Pair wise Comparison of Evaluation Criteria In this subsection, the pair wise comparison tables are presented for both cost and benefit categories. Table 2 shows an example of such a table to be filled. Entry (i, j) in tables represents the strength of preference of criterion A over criterion B. For instance, to consider entry (A, C) of Table 2, the question to be asked here must be: “With respect to distributed control configuration with two PLCs, what is the relative value or importance of $100,000 in annualized cost to increase in testing time?” Every table is considered as a matrix, which is reciprocal. If an entry of the matrices is crisp, say 1, it is shown as a triple (1, 1, 1) indicating that the lowest likely, most likely and highest likely values are all the same. In the case of fuzzy responses, linguistic (AS, MoG, MuG, MMG) or fuzzy numbers between 1 and 9 are used. For singulator case study, three domain experts/decision makers were interviewed for judgments of the comparison independently. Each one filled out the four comparison tables. As an example, Table 3 shows a result from Expert 1 on pairwise comparison matrix for design/implementation under costs.
In the case of the judgment 1, the consistency is shown below in Table 4. CRm was the consistency ratio for the mean values; CRg was calculated by taking the geometric means of lower and upper bounds. For more detail, please refer to Boucher, T. O., Gogus, O. M., Bruins, R., Descovich, T., and Litman, N., (1996). Table 4 shows that both CRm and CRg for all the tables are within the range. The comparison of decision maker 1 is considered consistent. Table 4 Consistency of decision maker 1’s judgment Consistency Ratio CRm CRg operational/safety under benefits 0.0712 0.1891 dynamical response under benefits 0 0 design/implementation under costs 0.0326 0.079 operational/safety under costs 0 0 3.5 Implied annual costs and benefits of criteria After testing the pair wise comparisons matrices for judgmental consistency, the implied annual benefits of criteria can be computed. The implied annual benefits/costs of criteria and the total value are also in fuzzy set, which is represented ~ a , a , a , where as a triangle a l m n
144
al : smallest possible value of the fuzzy strength of preference
scales and structure may need to be incorporated into the analysis. These issues require attention from research community before such a framework can be effectively used in practice.
am : most likely value of the fuzzy strength of preference an : highest possible value of the fuzzy strength of preference Table 5 shows the calculated results. Table 5 Implied annual benefits and costs of criteria (all numbers are dollar amounts in 1000s)
REFERENCES Boucher, T. O. and MacStravic, E. L., Multi-attribute evaluation within a present worth framework and its relation to analytic hierarchy process, The Engineering Economist. Vol.37. No.1, pp. 1-32, 1991.
From the above table we conclude that the evaluations by all the interviewees are consistent and that they all favour distributed configuration to a centralized one as measured by implied annual dollars. 5. CONCLUSION AND FUTURE WORK Here we demonstrated a methodological framework that a decision maker can systematically use to choose an automation design alternative by combining different evaluation criteria according to both quantifiable and nonquantifiable measures. While we used NCIC for the demonstration, there are similar methods in the literature that one can use both in crisp and fuzzy domains for the analysis. What makes the results of this paper novel is the integration of this economic evaluation methodology to the overall design process through a simulation engine and knowledge base, as demonstrated in Figure 1. To build a commercially viable software product based on the ideas presented in this article, there are issues that need further investigation, such as development of decision hierarchy and the way pairwise comparisons must be developed and conducted. Decision hierarchy and the selection of criteria are application dependent and their extent of accuracy depends on the domain expertise of the analyst(s) in charge. While existing commercial simulation engines can be used for evaluation of some quantifiable measures, we are not aware of any existing methodology which can be used to assist decision makers for properly framing these comparisons and building the decision hierarchy. There are other issues that such a framework must take into account. For instance, it may be necessary to evaluate a design alternative from both design and operational perspectives. Thus the evaluation must incorporate interviewees from all these interested parties and their evaluation criteria (which are often contradictory and of different scales) must be integrated into one cohesive decision hierarchy. Furthermore, if the system under the design is intended for an external client, then different cost and benefit
Boucher, T. O. and MacStravic, E. L., (1991), Multi-attribute evaluation within a present worth framework and its relation to analytic hierarchy process, The Engineering Economist, volume(37),No.1, page1-32. Boucher, T. O., Luxhoj, J.T., Descovich, T. and Litman, N., (1993). Multicriteria evaluation of automated filling systems: a case study, Journal of Manufact Systems, volume(12),No.5, page357-378. Boucher, T. O., Gogus, O. M., Bruins, R., Descovich, T., and Litman, N., (1996). Multicriteria Evaluation of a Centralized Control Station for a Production Line: A Case Study, Journal of Engineering Valuation and Cost Analysis, volume (1), No.1, page17-32. Chandra, J., Schall, S. O., (1988). Economic Justification of Flexible Manufacturing Systems Using the Leontief Input-Output Model, The Engineering Economist, volume(34),No.1, page27-50. Falkner, C. H., Benhajla, S., (1990). Multi-attribute decision models in the justification of CIM systems, The Engineering Economist, volume(35),No.2, page91-114. Gogus, O. M. and Boucher, T. O., (1998). Fuzzy NCIC, The Engineering Economist, volume(43),No.3, page203-246. Hall, K. H., Staron, R. J. and Vrba, P., (2005). Experience with Holonic and Agent Based Control Systems and their Adoption by Industry. Holonic and Multi-Agent Systems for Manufacturing, page1-10. Springer Berlin / Heidelberg. Lu, Y. and Jafari, A. M., (2007). Distributed Intelligent Industrial Automation – Issues and Challenges, Automation Technology in Practice. Reznik, D., Genc, Y., Schiesser, R. and Wynblatt, M. (2004). A New-Generation Parcel Singulator: Smaller, Quieter and Programmable, Automation Technology in practice international, Oldenbourg Indutrieverlag volume (2), No.2. Saaty, T.L., (1980). The Analytic Hierarchy Process, McGraw-Hill, Inc. Turksen, I. B., (1988). Stochastic fuzzy sets: a survey, Combining Fuzzy Imprecision with Probabilistic Uncertainty in Decision Making, SpringerVerlag, New York, page168-183. Turksen, I. B., (1991). Measurement of membership functions and their assessment, Fuzzy Sets and Systems 40, page538
145
A Multi-Criteria Economic Evaluation Framework for Control System Configuration – Framework and Case Study Peng Zhao*, Yan Lu **, Mohsen A. Jafari*, Davood Golmohammadi* * Dept. of Industrial & Systems Engineering, Rutgers University, Piscataway, NJ 08854, USA ** Siemens Corporate Research, Princeton, NJ 08540, USA (email: [email protected]) Abstract: The underlying methodology includes three main components: A digital factory/simulation model, a knowledge base/expert system and a multi-criteria evaluation model to compute the scores of different control designs and configurations on economic terms. For the economic evaluation, an existing methodology, Non-Traditional Capital Investment Criteria (NCIC) is used which allows us to incorporate into the analysis both traditional criteria, readily measurable in financial benefits, and non-traditional criteria that are not easily measurable based on their financial benefits. These non-financial benefits could be quantitative (measurable, but not necessarily in dollars) or qualitative (not measurable at all). An example is used to demonstrate this method by comparing the economic value of two control design alternatives for a singulator-- a centralized control where one motion controller controls all the axes, and a distributed configuration where the control of the axes are taken by two controllers, working autonomously, and interacting whenever necessary. Keywords: Multi-criteria Economic Evaluation, Distributed Control, Analytical Hierarchy Process, Digital Factory.
1. INTRODUCTION In a recent article, Lu and Jafari (2007) echoed the concerns raised by Hall (2005) and others about the lack of widespread interest on the use of Distributed Artificial Intelligent Control, or “DAIC” in industrial applications. In particular, these authors stressed the lack of any appropriate commercially available tool to compute (i) the cost of design and cost of ownership of DAIC systems, and (ii) the value of these systems in terms of potential benefits that they can offer compared to more traditional technologies. In this article a framework is proposed for the economic evaluation of distributed control designs against centralized ones. The underlying methodology has three main components: (i) A digital factory/simulation model to estimate system performance and some measurable factors under different configurations, (ii) A knowledge base/expert system which provides guidelines for establishing decision criteria, and interacts with the user to estimate other quantifiable factors whose values cannot be obtained from simulations and weigh the factors for alternative design and configurations. (iii) A multi-criteria evaluation model to compute the scores of different control designs and configurations on economic terms. For the economic evaluation, an existing methodology, Non-Traditional Capital Investment Criteria (NCIC), developed by Boucher and MacStravic (1991) and Gogus and Boucher (1998) is used. This approach allows us to incorporate into the analysis both traditional criteria, readily measurable in financial benefits, and non-traditional criteria that are not easily measurable based on their financial benefits. These non-financial benefits could be quantitative
978-3-902661-44-9/09/$20.00 © 2009 IFAC
(measurable, but not necessarily in dollars) or qualitative (not measurable at all). Furthermore, the approach proposed here also deals with the condition when the outcome or impact of some evaluation criteria within alternatives falls into a fuzzy domain. While the underlying techniques proposed are general and can be applied to different applications, in this article we will focus our analysis on a case study for easy illustration. The example used in the case study is the control design for a simplified version of the parcel “singulator” of Siemens. We want to compare the economic value of the singulator with two control designs: (i) A centralized control where one motion controller controls all the modules of the singulator, (ii) A distributed configuration where the control of the singulator modules are divided into two controllers, working autonomously, and interacting whenever necessary. 2. ECONOMIC EVALUATION FRAMEWORK Figure 1 below illustrates the proposed framework for the economic evaluation of alternative control configurations. The kernel of the framework is a Multi-Criteria Economic Evaluation Engine, which calculates a score for each specific control design. By comparing the scores of the alternative control configurations, the design with the highest value is considered as the best solution. The inputs to the economic evaluation engine come from: (i) Knowledge Base (KB) module which provides decision hierarchy consisting of a set of alternatives, their criteria categories, and the criteria, as well as some application domain specific data – for instance, how much can be saved for maintenance and training (ii)
140
10.3182/20090610-3-IT-4004.0027
Digital Factory (DF) module providing quantifiable measures to the evaluation engine for each configuration. Some typical quantifiable measures are system average throughput and utilization of various resources in the system. (iii) Domain Expertise (DE) module determining weights and rankings of various evaluation criteria used in the model. In this section, we will describe the details of the modules and the steps to follow for the economic evaluation of distributed control designs against centralized ones.
Knowledge Base
User’s control requirements & alternative configurations
Digital Factory
Decision hierarchy & domain data
Quantifiable measures Pairwise comparison
Multi-criteria Economic Evaluation Engine
Economically optimal configuration
Domain Expertise
Figure 1 Framework architecture 2.1 Multi-Criteria Economic Evaluation Engine Many multi-criteria economic models (Chandra et al. 1988, Wabalickis et al. 1988 and Falkner et al. 1990) have been proposed in the last decade for evaluation and justification of advanced technology investments. Saaty’s Analytic Hierarchy Process (or “AHP”) (1980) and Boucher and MacStravicis NCIC model (1991) are two of the most popular models. The NCIC decision modeling approach is similar in some respects to AHP. Both NCIC and AHP start with subjective pairwise comparison of process to rate criteria, and then calculate the importance of each criterion using the eigenvector approach. Maximum eigenvector is used in both models to check for logical consistency. However, unlike AHP, NCIC interprets the relative importance of each criterion in monetary terms. Furthermore, NCIC provides additional checks on economic consistency and ranks alternatives with respect to their worth. NCIC has been used in different applications. Boucher (1998) examined the use of multi-criteria modeling at machine level automation investment. The study involved the evaluation of generic types of filling equipment used in packaged food manufacturing. In another article, Boucher (1996) examined an investment decision problem at production line automation level. Gogus and Boucher (1998) extended NCIC to fuzzy domain for the cases where experts’ opinion or level of preference is not crisp and contains varying degrees of impression. Imprecision is also possible in outcomes of events that influence an alternative. In this article we adopt the NCIC decision model for our control design economic evaluation framework, which involves construction of a decision hierarchy, pairwise comparison of criteria within each category included in the hierarchy, consistency analysis, computation of criteria weights for each alternative, and computation of the net worth for each alternative. We will show how these steps can be applied to our case study.
2.2 Constructing the Decision Hierarchy and Collecting Domain Data Page margins The construction of a decision hierarchy is the first and the most important step in our analysis. It includes definition of hierarchy levels, breakdown of evaluation criteria categories, and association of cost or benefit to each of these categories. At the same time, many of the unquantifiable criteria require elicitation of expert judgment and unbiased estimations. While some of these requirements are for the end-users, others involve the technology owners and system integrators. A decision hierarchy consists of a set of alternatives, their criteria categories, and their criteria. In the first step the investment alternatives and their evaluation criteria are identified. The evaluation criteria are the expected impacts or outcomes of the investment alternatives. These outcomes may be quantitative (measurable) or qualitative. In addition, these outcomes may be benefits which increase the value of the alternative or costs which decrease the value of the alternative. All significant outcomes of an investment alternative should be reflected in the choice of evaluation criteria. Evaluation criteria are specific to an alternative. A single criterion may be assigned to only one alternative or to all alternatives, if applicable. Also, a particular criterion may be a benefit for one alternative and a cost for another alternative. This information should be reflected in the decision hierarchy. Choosing the decision criteria is the most crucial step of the analysis. Great care should be taken to select evaluation criteria that will represent all expected impacts of an investment alternative. Criteria should be quantified whenever possible. Furthermore, in order to avoid excessive comparison, criteria can be classified into categories. Figure 2 shows a schematic of a decision hierarchy.
Best Over all Alternative
Level 1
Level 2
Level 3 Level 4
Alternative 1
Category 1
Criteria
1
... Criteria L 1
...
. . ..
Category 2
Criteria
Alternative n
Category K
2
... Criteria L 2
...
Figure 2 A decision hierarchy
141
Criteria Ik
... Criteria Lk
...
. . ..
Criteria should be independent of each other. Consider, for example, quality improvement. Quality improvement can result in a cost saving due to less rework and scrap. Quality improvement can also result in less defective product reaching the customer, thus improving customer satisfaction. If the cost savings associated with less rework and scrap has been computed and included as part of the expected annual benefit, then the additional merit of the criterion labeled "Quality Improvement" is limited to that which is expected from improving customer satisfaction alone. If the value of less rework and scrap has not been computed as part of an alternative's annual benefit, then the quality improvement criterion represents the effect of both factors.The decision hierarchy used in this article reflects the views of the authors and by no means is being proposed as a general paradigm. We believe that for all practical purposes, a KB system will be required to address this major issue. The architecture of such KB system is outside the scope of this article. 2.3 Collecting Quantifiable Data For the non-quantifiable data, they can be obtained from some domain experts. But to collect the quantifiable data of the system before a system is built, digital factory may be the easiest way to be used in design phase. A digital factory offers an integrated approach to virtually run a process without having a real one. Simulation is the key technology within this concept. Event-based, time-based or hybrid simulation models can be built to any degree of details for conceptual system designs required by the analysis. By using simulation, some issues of the proposed system can be estimated. Almost in all cases and applications, materials flow and system layout must be included in these models. To have a fair comparison of various control configurations, information flow and communication networking also need to be part of the simulations. One can even go further and include business processes for design and implementation in the simulations. From these simulations, different performance measures associated with the criteria defined in the decision hierarchy for the evaluation can be computed for each control design and configuration. 2.4 Pair wise Comparison The next step of the analysis is to derive weights for the individual criteria, from which the implied annual benefit of criteria can be computed. These weights are based on the judgments of relative importance of criteria when compared in a pairwise fashion. For each alternative, pairwise comparison matrices are formed for each of the non-financial criteria categories (i.e., material conversion, information conversion, strategic).Comparison matrices for a criteria category of an alternative consist of all criteria within the group plus an additional criterion of annual Dollars. This annual dollars criterion represents the total dollar value of financial criteria in the Annual Dollars category. This criterion is measured in dollars and exists in the comparison matrix to provide a link between relative weights and dollar values. For each alternative, the relative importance of criteria is determined by making pairwise comparisons within the
benefit and cost hierarchies. We form pairwise comparison matrices for the non-financial criteria category by deriving weights for its individual evaluation criteria. From these weights the annual benefit or cost of a criterion is computed. These weights are based on expert judgment of relative importance of criteria when compared in pairwise fashion. To form a comparison matrix for a given criteria category, we include all the criteria in that category and a criterion representing the total dollar value of the financial criteria in the “Annual Dollars” category. The purpose of this criterion is to provide a reference in financial terms for calculating the relative importance of the non-financial criteria. We will assume that for some criteria the judgments will be fuzzy while for others it will be crisp. For fuzzy judgment either linguistic variables or fuzzy numbers can be used. We will use linguistic variables suggested by Gogus and Boucher: AS= “About the Same as”; SG=”Slightly Greater than”; MoG=“Moderately Greater than”; MuG=“Much Greater than”, or MMG=”Much Much Greater than”. An important aspect of linguistic fuzzy variables is that they are specific to individuals, but the membership function for the same variables for different individuals is different. Hence, this function needs to be determined on an individual basis. In the case of fuzzy numbers, individuals are asked to provide their responses with a range defined by lowest possible value, most likely value, and highest possible value. These are taken as the extreme points and mid point of a Triangular Fuzzy Number (or “TFN”). In case that linguistic fuzzy variables are used, one can still build TFNs from the membership functions (see Turksen 1988, Turksen 1991 and Gogus 1998 for details). The details of how to apply these modules and the economic evaluation engine into practice will be elaborated in the next section with the Singulator case study. 3. CASE STUDY – A SIMPLIFIED VERSION OF SIEMENS SINGULATOR Parcel singulator is widely used to convert an input stream of disorganized parcels into one or more single file output streams. The Siemens Singulator design integrates advanced real time vision with distributed mechanical design and motion control technology (Reznik 2004). The system kinematics is modularized into distributed (matrix-like) conveyor architecture, allowing parcels to be rotated and translated individually. As a product, Siemens Singulator uses a central controller to control all 7 by 12 actuators and the motion cycle is 33msec. If there are multiple CPUs to share the computation load of the motion control, both the system performance and robustness are expected to be improved. Our case study is motivated by comparing the distributed control design with the central design for such a singulator considering the economics values. For easier illustration of the concept of the economic analysis method, the case study is based on a simplified version of the singulator with only 4 by 8 actuators, which wouldn’t change the analysis result. The schematic of the simplified singulator is shown in Figure 3, with four main modules: the bulk conveyor, the singulator bed with a matrix like arrangement of actuated tiles, the steering wheel diverter (SWD), and a takeaway. Parcels with different shapes and orientations arrive on the bulk conveyor. The induction of parcels from the bulk conveyor to the
142
singulator bed is conditioned on keeping a specified minimum gap between the parcels. To realize this, the first two rows of tiles on the singulator bed move slower than the rest so as to separate the parcels. The following tiles move with a higher speed, where the parcels are already distanced apart with at least the minimum gap. Parcels move sequentially to the SWD and from there they are discharged through the takeaway conveyor. The induction speed is 0.5 m/sec while the discharge speed can be as high as 1.5 m/sec. Theoretically, the motion control configuration for the above system could span from a fully centralized system to a fully distributed system where each actuated part has its own controller. In this article, we only consider two configurations – a fully “centralized control” where all the modules and tiles are controlled by a single controller; and a “distributed configuration” where the control of the actuated parts are distributed to two controllers. In either configuration, the motion controller is responsible for computing the speeds and mechanical settings of the various actuators in the singulator. Settings computed in every cycle include speed of the bulk conveyor, speed of every individual tile or belt on the singulator bed, and orientation of every column in SWD. For a centralized configuration, the speeds for all the tiles on the 4x8 singulator matrix are set by a single motion controller. Thus, the maximum speeds for the singulator is conditioned by this controller’s cycle time. For the distributed configuration, the speed control is divided between the two controllers, each in charge of a 2x8 matrix of tiles. In the case of distributed configuration, we will assume that the each motion control cycle also includes a communication cycle when the two controllers must communicate to position those parcels sitting on tiles within both the matrices. Bulk conveyor
Singulator bed
SWD takeaway
We note that the impact of stoppage time on the system throughput rate is different for the two alternatives that we consider here.
Level 3
Operational/ safety
Annual dollars Annualized first cost
Level 4
Design & implementation
% of change in System throughput % of change in operation stoppage
System Dynamical Response
Change in design complexity
Ease of expansion
Increase in testing time Increase in training requirements
Improvement in system safety & fault tolerance Improvement in maintainability Cost more in system maintenance/supplies
Figure 4 Proposed breakdown of Level 3 & 4 In case of fuzzy outcomes, Level 3 and 4 need to be structured according to cost and benefit classes. A hierarchy is proposed below assuming a reasonable level of maturity for any technology used. Again, this hierarchy can be easily modified from one application to another. It is also noted that it is possible for a Level 3 category to be in both cost and benefit class, with different criteria at Level 4. According to the model of Figure 5, we assume reduction in time for the implementation of distributed control configuration. At the same time, it is also assumed that for the distributed configuration, the design complexity, testing time, and training requirements will increase compared to the centralized one, thus they are categorized as cost. Our rationale is that the implementation is often done by a third party, i.e., system integrator, which will supposedly have a mature level of technology for the implementation. But the design process, testing and training greatly involve in-house resources, which most probably will be less mature in terms of new technologies. Distributed Control
Figure 3 Simplified Singulator Benefits
3.1 Constructing Decision Hierarchy
Operational/ safety
We start NCIC by building a decision hierarchy model – it starts with the identification of design alternatives followed by setting up evaluation criteria. These criteria are the outcomes or the expected impacts of the design alternatives. These outcomes may be costs (decrease the value of the alternative) or benefit (increase the value of the alternative). For evaluation of an alternative distributed control design, the expected cost or benefit structure is considered in comparison to a base centralized configuration. Thus, all the costs and benefits for the distributed configuration will be calculated in an incremental basis of switching from a centralized configuration to a distributed one. Figure 4 illustrates our proposed list of four categories in Level 3, and evaluation criteria in each category in Level 4. Under Operational/Safety category, we have both % of change in throughput rate and % of change in stoppage time.
Costs
System Dynamical Response
% of change in System throughput % of change in operation stoppage
Ease of expansion
Annual dollars Annualized first cost
Design & implementation Increase in design complexity
Operational/ safety Cost more in system maintenance/supplies
Increase in testing time Increase in training requirements
Improvement in system safety & fault tolerance Improvement in system maintainability
Figure 5 Proposed costs and benefits hierarchy for distributed control configuration 3.2 Evaluation Prior to any further economic evaluation, the most likely or range of values have to be estimated for all the quantifiable measures that are mapped out in the above decision hierarchy model. While some of these measures can be obtained from a
143
knowledge base or a historical database, others require simulation of the alternative configurations. Table 1 lists the comparing criteria for the singuator case. It contains quantifiable and unquantifiable criteria. Row 1 is the monetary difference between the investments. This estimation was obtained from domain experts. Row 2 and 3 are quantifiable criteria with values computed from simulations. Our simulations indicate that, on the average, the distributed configuration provides 5% increase in throughput and 5% reduction in overall stoppage compared to the centralized system. The other criteria in the table are unquantifiable and were obtained through interviews with experts and also through investigation of the literature. Table 1 Quantifiable criteria and their data source Criteria Change in annualized cost in the investment % of change in throughput % of change in stoppage Change in safety & fault tolerance Change in maintainability Change in ease of expansion Change in design complexity Change in testing time Change in training requirements Change in system maintenance/supplies
Data Source KB & expert view Simulation Simulation KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view KB & expert view
Amount $100,000 5% -5% ---
Table 2 Pairwise comparison matrix for design/implementation under costs Design/implementation A B C A – Cost=100,000 ----B – Change in design complexity ----C – Increase in testing time ----D – Increase in training requirements
D
----
Table 3 Pairwise comparison matrix for design/implementation under costs Design/implementation A - Cost=100,000 B – Change in design complexity C – Increase in testing time D – Increase in training requirements
A -----MoG
B 1/MoG ------
C SG MoG
D AS SG
1/SG
1/MoG
------
1/SG
AS
1/SG
SG
------
3.4 Consistency Check & Irrational Weight Factors
---
Since each comparison is made independently, after the tables are filled, consistency of the decision maker’s judgment has to be checked. NCIC computes a consistency ratio that measures how far a decision-maker's judgments are from perfect consistency. The value of 0 means perfect consistency. It is recommended that pairwise comparison matrices having a consistency ratio greater than 0.20 should be reviewed. Highly inconsistent matrices can distort the final result. See Turksen 1988, Turksen 1991 and Gogus 1998 for details of consistency ratio.
-----------
3.3 Pair wise Comparison of Evaluation Criteria In this subsection, the pair wise comparison tables are presented for both cost and benefit categories. Table 2 shows an example of such a table to be filled. Entry (i, j) in tables represents the strength of preference of criterion A over criterion B. For instance, to consider entry (A, C) of Table 2, the question to be asked here must be: “With respect to distributed control configuration with two PLCs, what is the relative value or importance of $100,000 in annualized cost to increase in testing time?” Every table is considered as a matrix, which is reciprocal. If an entry of the matrices is crisp, say 1, it is shown as a triple (1, 1, 1) indicating that the lowest likely, most likely and highest likely values are all the same. In the case of fuzzy responses, linguistic (AS, MoG, MuG, MMG) or fuzzy numbers between 1 and 9 are used. For singulator case study, three domain experts/decision makers were interviewed for judgments of the comparison independently. Each one filled out the four comparison tables. As an example, Table 3 shows a result from Expert 1 on pairwise comparison matrix for design/implementation under costs.
In the case of the judgment 1, the consistency is shown below in Table 4. CRm was the consistency ratio for the mean values; CRg was calculated by taking the geometric means of lower and upper bounds. For more detail, please refer to Boucher, T. O., Gogus, O. M., Bruins, R., Descovich, T., and Litman, N., (1996). Table 4 shows that both CRm and CRg for all the tables are within the range. The comparison of decision maker 1 is considered consistent. Table 4 Consistency of decision maker 1’s judgment Consistency Ratio CRm CRg operational/safety under benefits 0.0712 0.1891 dynamical response under benefits 0 0 design/implementation under costs 0.0326 0.079 operational/safety under costs 0 0 3.5 Implied annual costs and benefits of criteria After testing the pair wise comparisons matrices for judgmental consistency, the implied annual benefits of criteria can be computed. The implied annual benefits/costs of criteria and the total value are also in fuzzy set, which is represented ~ a , a , a , where as a triangle a l m n
144
al : smallest possible value of the fuzzy strength of preference
scales and structure may need to be incorporated into the analysis. These issues require attention from research community before such a framework can be effectively used in practice.
am : most likely value of the fuzzy strength of preference an : highest possible value of the fuzzy strength of preference Table 5 shows the calculated results. Table 5 Implied annual benefits and costs of criteria (all numbers are dollar amounts in 1000s)
REFERENCES Boucher, T. O. and MacStravic, E. L., Multi-attribute evaluation within a present worth framework and its relation to analytic hierarchy process, The Engineering Economist. Vol.37. No.1, pp. 1-32, 1991.
From the above table we conclude that the evaluations by all the interviewees are consistent and that they all favour distributed configuration to a centralized one as measured by implied annual dollars. 5. CONCLUSION AND FUTURE WORK Here we demonstrated a methodological framework that a decision maker can systematically use to choose an automation design alternative by combining different evaluation criteria according to both quantifiable and nonquantifiable measures. While we used NCIC for the demonstration, there are similar methods in the literature that one can use both in crisp and fuzzy domains for the analysis. What makes the results of this paper novel is the integration of this economic evaluation methodology to the overall design process through a simulation engine and knowledge base, as demonstrated in Figure 1. To build a commercially viable software product based on the ideas presented in this article, there are issues that need further investigation, such as development of decision hierarchy and the way pairwise comparisons must be developed and conducted. Decision hierarchy and the selection of criteria are application dependent and their extent of accuracy depends on the domain expertise of the analyst(s) in charge. While existing commercial simulation engines can be used for evaluation of some quantifiable measures, we are not aware of any existing methodology which can be used to assist decision makers for properly framing these comparisons and building the decision hierarchy. There are other issues that such a framework must take into account. For instance, it may be necessary to evaluate a design alternative from both design and operational perspectives. Thus the evaluation must incorporate interviewees from all these interested parties and their evaluation criteria (which are often contradictory and of different scales) must be integrated into one cohesive decision hierarchy. Furthermore, if the system under the design is intended for an external client, then different cost and benefit
Boucher, T. O. and MacStravic, E. L., (1991), Multi-attribute evaluation within a present worth framework and its relation to analytic hierarchy process, The Engineering Economist, volume(37),No.1, page1-32. Boucher, T. O., Luxhoj, J.T., Descovich, T. and Litman, N., (1993). Multicriteria evaluation of automated filling systems: a case study, Journal of Manufact Systems, volume(12),No.5, page357-378. Boucher, T. O., Gogus, O. M., Bruins, R., Descovich, T., and Litman, N., (1996). Multicriteria Evaluation of a Centralized Control Station for a Production Line: A Case Study, Journal of Engineering Valuation and Cost Analysis, volume (1), No.1, page17-32. Chandra, J., Schall, S. O., (1988). Economic Justification of Flexible Manufacturing Systems Using the Leontief Input-Output Model, The Engineering Economist, volume(34),No.1, page27-50. Falkner, C. H., Benhajla, S., (1990). Multi-attribute decision models in the justification of CIM systems, The Engineering Economist, volume(35),No.2, page91-114. Gogus, O. M. and Boucher, T. O., (1998). Fuzzy NCIC, The Engineering Economist, volume(43),No.3, page203-246. Hall, K. H., Staron, R. J. and Vrba, P., (2005). Experience with Holonic and Agent Based Control Systems and their Adoption by Industry. Holonic and Multi-Agent Systems for Manufacturing, page1-10. Springer Berlin / Heidelberg. Lu, Y. and Jafari, A. M., (2007). Distributed Intelligent Industrial Automation – Issues and Challenges, Automation Technology in Practice. Reznik, D., Genc, Y., Schiesser, R. and Wynblatt, M. (2004). A New-Generation Parcel Singulator: Smaller, Quieter and Programmable, Automation Technology in practice international, Oldenbourg Indutrieverlag volume (2), No.2. Saaty, T.L., (1980). The Analytic Hierarchy Process, McGraw-Hill, Inc. Turksen, I. B., (1988). Stochastic fuzzy sets: a survey, Combining Fuzzy Imprecision with Probabilistic Uncertainty in Decision Making, SpringerVerlag, New York, page168-183. Turksen, I. B., (1991). Measurement of membership functions and their assessment, Fuzzy Sets and Systems 40, page538
145