- Email: [email protected]
The use of system dynamics as a tool for intermediate level technology evaluation: three case studies
J. Eng. Technol. Manage. 20 (2003) 193–204
The use of system dynamics as a tool for intermediate level technology evaluation: three case studies E.F. Wolstenholme∗,1 Leeds Business School, Leeds, UK
Abstract This set of three case studies suggest that traditional approaches to technological evaluation are static and either high level and simplistic, or low level and complex. Further, that they tend to evaluate technology in terms of itself, rather than the domain it is intended to support. A holistic and dynamic method, based upon system dynamics simulation modelling, is described for the early evaluation of technology at an intermediate and balanced level. The method is demonstrated by applying it to the evaluation of management information systems (MIS) in the defence industry and new drugs in the pharmaceuticals industry. The approach involves the creation of dynamic simulation models of the anticipated domain of application of the technology and their use as a test bed to evaluate the systemic impact of the technology over time. Very importantly, the method allows the contribution of each aspect of the technology to be assessed and facilitates investigation of alternative structural and operating changes to the domain itself, to make best use of the technology. © 2003 Elsevier B.V. All rights reserved. Keywords: System dynamics; Technology; Technological evaluation; Simulation; Systems
1. Introduction Traditional approaches to evaluating new technologies have tended to focus either, at a high and simplifying level or at a very detailed, complex and specific level. High level assessments usually take the form of balancing the estimated costs of the technology with the anticipated benefits. Cost estimates might be based on previous applications of the technology in other domains or from trials or from statistical sources. A characteristic of high level assessments is that they are essentially static and focus on evaluating the ∗ Present address: Cognitus Ltd., 1 Park View, Harrogate, North Hampshire HG1 5LY, UK. E-mail address: [email protected] (E.F. Wolstenholme). 1 Professor of Business Learning (Leeds Business School); Director (Cognitus Ltd.).
0923-4748/$ – see front matter © 2003 Elsevier B.V. All rights reserved. doi:10.1016/S0923-4748(03)00018-3
194
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
technology in terms of itself, rather than in terms of benefits to the overall performance of the domain of application. Low level assessments tend to take a more domain-orientated approach. However, they are often characterised by complex and detailed assessments of the impact of the technology, often using tools designed for other purposes and usually with a particular emphasis on singular performance elements of the domain. There is a growing need to develop an intermediate level of technology evaluation, which can capture the domain-centred orientation of low level approaches, but also provide a balanced assessment of impact across the whole domain of application. To the present, one of the problems in developing such an intermediate approach has been the lack of suitable tools. This case study reports on a methodology based on system dynamics (Sterman, 2000; Wolstenholme, 1990; Morecroft and Sterman, 1994; Richardson and Pugh, 1981) for intermediate level technological assessment, providing an entirely original framework, which is showing considerable promise. The approach involves the creation (with the active participation of domain users and managers) of maps and dynamic simulation models of the anticipated domain of application of the technology, at an appropriate level of resolution, as a test bed to evaluate its global, rather than local, impact. The emphasis being on developing insights and learning about the systemic consequences of adapting a complex technology (Senge, 1990; De Geus, 1988). A three-stage methodology has been developed which will be described prior to presentation and analysis of existing applications. Three mature and two embryonic applications are described. 2. The three-stage methodology 2.1. Stage I: model the domain of application The first stage of the approach is to create, calibrate and validate an appropriate system dynamics model of the intended domain of application of the technology. Technology is defined here in its widest sense as any science-based application, ranging from hardware, though management information systems, to drugs. The type of model needed will depend, therefore, both on the technology and the domain. This base model must assume the present operation of the domain, which may make use of low or out-dated technologies. It must also incorporate appropriate scenarios and global performance measures by which to assess performance in the domain. Note here that the modelling process itself will usually help with the definition of performance measures. An important part of the method is that the process of model building itself should improve understanding of the domain of application. It is vital, therefore, that the base model should capture knowledge of the domain in the widest sense. It follows that the model should be constructed with a variety of people from the domain and that the model should form an acceptable and credible representation of the domain to them. It should represent an extension of their shared mental models of the structure, organisation and current operating policies of the domain.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
195
2.2. Stage II: technology assessment The second stage of the method is to use the model as a test bed on which to evaluate the impact of the new technology. This stage involves superimposing on the model the anticipated effects of the new technology throughout the whole domain of application. The model then assumes the role of a test “flight simulator” for the technology and forms a dynamic experiential learning environment. Here, potential developers and users of the technology can explore the interactions of assumptions and parameter values they have defined, and learn holistically about the potential impact of the new technology. 2.3. Stage III: technology accommodation in the domain The third stage of the method is perhaps the most significant. It builds on the fact that to make the best use of any significant new technology it is often necessary to change procedures and policies to accommodate it. The modelling process described facilitates experimentation to redesign the domain of application to achieve this. It could involve modifying any aspect of the domain and testing the effects of the modifications. For example: • • • • •
re-engineering processes; changing information paths and policies; changing organisational boundaries and responsibilities for particular activities; eliminating delays or increasing or reducing capacities; identifying high leverage intervention points where changes might produce the greatest outcome for the smallest input.
The detailed use of the method in two domains will now be presented. These concern the assessment of management information systems in a defence context (two applications) and drug evaluation in the pharmaceuticals industry (one application). A summary of applications in three other domains, which are at an embryonic stage of development will also be given. 3. Case studies in the use of the methodology 3.1. MIS evaluation in defence 3.1.1. The technology In the two case studies reported here, the technology was a complex management information system (MIS), although work was also carried out to evaluate various defence hardware items (Wolstenholme, 1993). The investment being made in MIS in defence organisations is enormous and, although much effort has been devoted to creating structured methods to aid the design of MIS, the area of MIS evaluation remains less developed. The traditional high level technological assessment approach used in the defence domain, prior to technology design and/or purchase, is that of an overall concept requirement analysis and feasibility study. The traditional low
196
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
level approach is that of using very detailed discrete entity simulators, designed for complex battle gaming. The emphasis to date has been, at worst, on simply assessing MIS in terms of its ability to process data and, at best, on evaluation of the impact of the MIS on sub-functions of the organisation at the detailed operational level. Attempts to introduce an intermediate level of technological assessment have centred on the use of prototypes. Whilst this approach is useful, it still suffers from a focus on real life experiments in only one part of the domain. Two examples will now be given of applying the systems-based intermediate level methodology in the defence MIS field. In both cases, a key element of the method was not to model the information system per se, but rather in terms of its anticipated effects on the physical and information processes of the organisation and the operational strategies which convert information into action. The procedure was to identify the information attributes (timeliness, accessibility, accuracy, relevance and comprehensibility) affected by each MIS feature and to incorporate their effect on parameters in the model. Of particular importance was the use of the method to assess the value of individual parts of the MIS as a basis for setting priorities for detail design and development. This work provided a direct assistance to incremental software engineering. 3.2. MIS evaluation in defence supply chains 3.2.1. Stage I model There are numerous complex interactions in military supply chains and many replications of each stage. However, at an intermediate level of analysis the chain can be considered as a four-stock system, consisting of Corps, Division, Brigade and Battle Groups, with supplies passing down the chain, subject to transport availability, and orders for supply passing back up the chain. Such a typical chain was modelled for a number of commodities. The work reported here will deal with ammunition as an example. The stage I model was based on the procedures and policies used under a manual information system for ammunition ordering and supply. The main complication of military supply chains over civilian supply chains is that the front line stocks are subject to significant loss, both due to attrition and due to being left behind as the tempo of an encounter unfolds. Losses of this nature were explicitly modelled and proved to have an important bearing on the behaviour of the supply chain. The main scenario used for experimentation with the model was an encounter of the battle groups with an enemy force, requiring the maintenance of a given rate of fire and hence use of ammunition to survive. The stage I model in the case study was validated against the expectations and views of defence management and professionals and against other more detailed simulation models. 3.2.2. Stage II model The features of the MIS technology incorporated into the stage II model were route management data, transport management data, monitoring of ground dump stocks, an on-line communications and a database. The information attributes affected by these features were identified and these, in turn, were related to parameters in the model.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
197
For example, on-line communications and data base features of the MIS relate to the information attributes of timeliness, accuracy and accessibility. Timeliness changes were represented by alterations in the values of the delays associated with information transmission and by reductions in the periods between reports. Accessibility was represented by the inclusion of information links between new sources and destinations of the information enabled by the MIS. For example, the availability of information at Corps and Division concerning the state of forward dumps and the stock in transit. Accuracy was quantified in terms of the difference between the perceived and true values of a variable and modelled by a random number function representing a distribution of values about the true value of the variable in question. 3.2.3. Stage II model results and the contribution of individual features of the MIS The stage II model results showed the rather obvious base result that it was possible for the battle groups to survive for longer for the same ammunition expenditure when using the proposed MIS, compared to using the existing and largely manual information and communications system. However, and more importantly, the model also provided a means of carrying out sensitivity analysis to investigate under what changes in assumptions and policies this proved not to be the case. The base model also provided some totally unexpected and surprising results, particularly with regard to the value of some parts of the proposed MIS. Three examples will be given here. (a) Firstly, the greatest ammunition losses occurred if only the route and/or transport management functions of the MIS were implemented. The majority of these losses occurred at the Brigade and Battle Group locations of the supply chain. (b) Secondly, the use of the improved information on the state of the forward dumps led, on its own, to the greatest reduction in ammunition losses. However, the savings provide small compensation for the overall losses incurred from improving the transport management. (c) Thirdly, although the installation of the database and the associated on-line communications did contribute to improving the overall performance of the supply chain, their impact was not as great as expected. This result emphasises that, although the availability and quality of information is increased by the MIS, significant policy changes must be implemented to really take advantage of the facilities which it provides. In summary, the main effect of implementing the MIS as shown by the model was one of allowing greater quantities of ammunition to reach the front lines. This has the effect of increasing the ability of the Battle Groups to maintain the desired firing rate. Although this effect has the potential to give greater operational capability to the front line gun units commanders, it also increases ammunition losses and reduces the flexibility at Division on Corps to reallocate ammunition between Battle Groups as the pattern of military operation changes. The results opened up an interesting debate about the whole purpose of the MIS, the objectives of the logistics function, the performance measures by which it might be judged and the need to change operating procedures and policies to make best use of the MIS. This thinking formed the basis of stage III of the methodology.
198
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
3.2.4. The stage III model The task of stage III model of the methodology was to examine the limitations of the prevailing policies and structure within the host system of the MIS and to investigate changes which may be necessitated or enabled by the introduction of MIS. The addition of this stage to the assessment process is consistent with the view that a proposed MIS must be evaluated in terms not only of its immediate impact, but also in terms of its potential to support managerial action. The use of the basic model and, in particular, the comparison of its results with those representing the effects of implementing the MIS, led to an explicit statement of logistics objectives as follows: (a) (b) (c) (d)
to enable the military operation to be achieved; to minimise ammunition losses and ammunition usage; to balance operational flexibility and operational capability; to minimise the resources/capacity required to achieve a military operation, particularly that of transport.
Of these, objective (a) was accepted as the prime objective in that it represents the requirement for logistics to support operational objectives. However, of particular importance here is objective (c), which provides an example of how the method helps the definition of opportunities for policy change. One unintended consequence of the simulated implementation of the MIS was to create a distorted distribution of ammunition stocks across the supply chain, with too much being ‘sucked’ through to the Battle Group end. This result quickly led thinking towards the possibility of using this distribution as a very visual performance measure, defined as the ‘ammunition profile’. The thinking also led to the possibility of having commanders negotiate the desired size of stocks at each point in the supply chain. In other words, the definition of a ‘desired ammunition profile’ across the chain and a set of policies to achieve it. The concept here was to balance front line stocks for immediate use but with high loss rates, with stocks further back, which could be reassigned as a battle unfolded. The sequence of action emanating from this chain of events led to a greater degree of openness and communication between managers in different parts of the supply chain. Other policy changes were defined in the context of regulating transport capacity and contingency stocks (less were required of both when using the MIS for the same level of risk). 3.3. MIS evaluation on the battlefield 3.3.1. Background The second application of the methodology in defence MIS concerned an MIS at a very early stage of its development life cycle and one to be implemented in a very unstructured area of military operation—the area of the tactical battlefield. The MIS studied was a Battlefield Management Information System and its purpose was to improve the command and control process, thereby giving commanders and staff more time for decision making based on accurate, relevant and timely information.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
199
There were three problems which made the evaluation of the Battlefield Management Information System difficult. The first was the sheer size and complexity of the battlefield operation and the proposed MIS. The second was that the MIS was only a concept and thus no data actually existed upon which to base an evaluation. The third problem was that many different perceptions existed of battlefield operations and this meant that the conceptualisation of a relevant base model of the host domain depended, in turn, on the anticipated scope of the MIS. 3.3.2. Stage I model The base model of the battlefield was conceived in terms of the following activities: • • • • • • • •
resource deployment (i.e. a commander allocating his available troops); reinforcement; logistics; target location; target acquisition and engagement; target tracking and co-ordination; weapon fire; movement (in particular the ‘shoot and scoot’ process, whereby a unit which has fired a given number of rounds moves to conceal its position, in case it has been identified by its flash or infrared signature); • vehicle repair; • secure communications (digital encryption). It should be noted that these functions were identified in an intense knowledge acquisition phase involving many defence personnel and were expressed in high level terms. They in fact subsume many lower level functions. The identification of these functions was paramount to identifying resources on the battlefield and further iterative discussions led to defining relevant states for these resources for inclusion in the model. Eventually, four states were conceptualised for enemy forces: unobserved, observed, acquired-as-targets and dead, through which a force progresses sequentially. However, later on in the modelling process it became clear that a backwards transition process was also required for some circumstances. For example, if an enemy unit withdraws out of range before it is killed then it must be allowed to revert to the unobserved state. Friendly units were also conceptualised as being in one of the above four states, plus two others. In order to provide a representation of the shoot and scoot cycle, a ‘hidden’ state was introduced whereby enemy units were unable to inflict damage on transient friendly units. A repair cycle was also introduced in order to represent those friendly units which had been damaged by enemy activity, but were recoverable; and also those which had broken down. All weapon types for both sides were combined into a single resource state of ‘units’. The base model was validated against the results from detailed battlefield simulators and run under a scenario of enemy forces advancing on a friendly position and using performance measured in terms of the loss rate of units on each side and the advance or retreat achieved.
200
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
3.3.3. Construction of the stage II model The features of the MIS technology incorporated into the model were: • digital maps; • improved communication of enemy locations between friendly units to improve fire control; • monitoring of targets to prevent duplication of attack effort and hence overkill; • secure communications; • type of threat alarm; • built-in test equipment. As in the previous case, the features of the MIS were related to a set of basic information attributes, which were in turn related to parameters in the model. 3.3.4. Stages II and III models’ results The results from the stage II model showed clearly that without the MIS the friendly force is defeated faster than with the MIS. With a Battlefield MIS in place, it would appear that friendly forces are able to inflict higher attrition on enemy units and are thus able to survive longer. Again, these base observations were rather obvious and the main value of the model was its ability to support intensive ‘what-if’ analysis. In each experiment carried out it was the process of interpreting how the results were obtained by superimposing the MIS, which gives rise to improved understanding. Again, some surprising results emerged from the stage II model which related both to identifying inadequacies in the design of the technology and/or to the need for operational procedures to be changed. Experimentation with the stage II model revealed the need for changed procedures in a number of areas to make best use of the MIS. For example, procedures and policies were required to: • reallocate targets assigned to friendly units which had been destroyed or hidden and hence unable to fire; • remove targets when they had moved out of range; • handle the increase in targets brought about by using the MIS. A further interesting aspect of insight generation arising in this case study was that the structures that gave rise to most of the insights were generic. For example, the problems associated with the reallocation of targets could easily be applied to the reallocation of logistics. If the MIS automatically sends ammunition reports to a commander or a logistics co-ordination point, there needs to be procedures in place to reallocated reports if the unit generating the report is eliminated. Whilst over-ordering is unlikely to be affected seriously by the attrition of one unit, the problem becomes more significant when aggregated across the entire Battle Group. Consideration of generic structures and procedures is a way of generating further, non-obvious insights. 3.3.5. Contribution of the individual MIS features The battlefield model also proved to be an excellent way of assessing the individual contribution of each part of the MIS. The model was run with each individual component of the MIS switched on in turn and with all parts switched on together. No one component of the
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
201
MIS alone made an outstanding difference to any model performance measures. However, the total set of components made a very significant difference to the model performance. This case provided a classical example of the fact that the sum of the technology was much greater than its individual parts and that this synergy arose through the way in which the parts interacted to support one another. 3.4. Drug evaluation in the pharmaceutical industry 3.4.1. The technology In the pharmaceutical industry, the technology of interest was that of new drugs. These are normally evaluated at a very high level in terms of their price and clinical benefits. Low level detailed assessments involve clinical trials (statistical sampling of individual groups of patients in different domains of application). The predominant methodology for pharmacoeconomic assessment is that of decision analysis. This involves the creation of decision trees which are used to combine data on the success (probabilities and outcomes) of individual products collected from clinical trials, using an expected value criterion. In practice, attempts are made to fit this approach at all levels of evaluation. A broadening of drug assessment is emerging in the development of the new subject of pharmacoeconomics. This shift of thinking is largely driven by drug companies, but in a world of cost conscious heath services there is a growing need to evaluate the global resource savings associated with new products within the end-user (i.e. purchaser) domain, as well as clinical efficacy. In this industry, work with the dynamic simulation approach has involved evaluating both clinical and disease inhibiting drugs. The main examples of the former are the new developments in anaesthetics and examples of the latter would be asthma treatments or ulcer eradicators. The types of domain models used in drug assessment applications depend on the product. For example, in the case of a disease control product the base model may be a model of the disease progression. In the case of a clinical product, the model may be a specific or typical representation of some of the many settings and procedures in which the product will be used. These might vary considerably, say between and within types of institutions or countries. The work reported here involves the evaluation of anaesthetics in hospitals. 3.4.2. Stage I model A number of models were developed from advice and knowledge provided by clinicians, anaesthetists and managers from a range of different hospital settings. These models incorporated the whole patient flow process from arrival to discharge since it was postulated that the potential benefits resulting from new anaesthetics could occur not only in the operating theatres, but also the recovery or day wards. The states of patient flows incorporated into the model were arrival and preparation, theatre, recovery, day ward and home. The model referenced here was based around in-patient surgery over a working day for one type of operational procedure. Such a model was considered to be sufficiently aggregate to connect the whole patient flow process to staffing and discharge policies, but sufficiently detailed to relate to individual patient movements. It could also be easily related to available cost data. The idea of creating generic models which reflect a broad
202
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
spectrum of local practices in hospital care organisations is becoming an established part of pharmacoeconomic practice (Kenigsberg, 1996). The performance measures used in the model were those which were considered to be meaningful and important to the decision-makers in hospitals. The focus here was on the trend towards more day care surgery and on exploiting ways to create capacity in day care to allow greater patient throughput, possibly involving higher costs, but associated higher revenues. All resource calculations (staff and materials) in the model were based on the patient flows. The stage I model was validated against cost data collected from a studies aimed at quantifying and valuing in monetary terms resource use at each stage of selected surgical procedures, with particular relevance to anaesthesia. The model was run under a number of existing hospital policies and practices relating to time spent by anaesthetists with patients in recovery and the time at which patients were discharged to home. 3.4.3. The stage II model The stage II model incorporated the claimed benefits of new anaesthetic drugs by modifying a whole range of model parameters. For example, the average time taken at each stage of the patient flow process and the numbers of staff and materials used. The stage II model demonstrated that the composite effect of the claimed benefits of the anaesthetic would be significant. Certainly, in terms of the capability of the hospital to complete a list of patients quicker. However, that the savings in cost and time were not necessarily realisable, since staff were full time, or sufficiently great to enable more patients to be treated in a day. 3.4.4. The stage III model The stage III model was used to evaluate the effects of changing a number of managerial policies in the hospital, which were facilitated by the increased safety levels of the new drug. These policies included transferring responsibility for recovering patients from anaesthetists to nursing staff at an earlier point in the patient flow process and allowing patients to be discharged earlier. Superimposing these new policies on the model indicated that very substantial savings were possible in the patient cycle time, sufficient to accommodate unto two extra patients per day in to the existing daily operating schedule with their associated revenue implications. The message from this work was again that a new technology on its own made some, but not excessive, impact on the economics of its domain of application. However, the effects of new managerial policies, which were facilitated by introducing the new drugs, could make a significant economic impact. Further, that the policy savings were not associated with the point of application of the technology, but in areas of the domain remote from this (for example, the recovery and ward areas of the hospital). 3.5. Other embryonic applications In addition to the above case studies, other applications of the method described in this report are underway, but at an early stage of development.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
203
3.5.1. Oil industry fluid tracking Here the technology was high-specification water infusion to enhance oil recovery. Such technology is traditionally evaluated in detail by using oil reservoir simulators designed for other purposes and related only to the reservoir and production functions of the oil business. The objective therefore was to introduce a much broader evaluation approach to assess the broad impact of water infusion technology across all operating functions. That is, at the intermediate level to assess the technology in terms of its impact on not just the production, but also the marketing and financial domains of the business. A real-time fluid tracking model was developed for this purpose to provide semi-quantitative answers to what-if questions concerning the impact of the technology at an early stage in the assessment process. 3.5.2. Manufacturing and project management Similar work is currently being undertaken in manufacturing and project management, where again high and low level assessment methods prevail. The low level methods usually centre on very detailed simulation and planning software. In the case of manufacturing there are detailed production-line simulators for materials resource planning and in the case of project management detailed project scheduling software. The approach in both cases has been to develop more holistic higher level models to give a wider view on the overall potential of the technology, which helps to focus priorities for large investments. An important stage III model effect was he use of the model to help validate the parameters used in the detailed material planning system. 4. Conclusions This case study has introduced the concept of a three stage procedure for broad and intermediate level assessment of technology and has demonstrated its application in a number of areas in practice. The main advantages of the approach are that it provides: • An assessment of the technology in terms of its effect on the dynamic behaviour of its domain of application, rather than an assessment in terms of itself. • A framework to indicate the way in which the technology interacts with its domain of application. The indicated benefits of new technology from this type of assessment can be surprising and counter-intuitive. This contrasts strongly with static cost benefit analysis, which often assumes that the benefits of each part of the technology are independent and that the combined effect of the technology is a linear summation of its parts. • A way of sharing thinking about the technology between managers in different functional areas of the organisation at an early enough time for all to be involved in the analysis. • Experiential learning about the technology and the domain of application and their interaction by providing a quantitative basis for ‘what-if’ analysis. • A way of determining the overall merits of a technology and, in particular, its possible side effects, prior to a full and costly commitment.
204
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
In general, the method is showing a high level of promise. It will become more and more relevant and important as the sophistication and cost of technology increases and the ease of use of modelling software improves. References De Geus, A.P., 1988. Planning as learning. Harvard Business Review 66 (2) March–April, 1988. Kenigsberg, P.A., 1996. Economic evaluation in anaesthesia a production process approach, EPH Feb. 1996, vol. 2 (1). Morecroft, J.D.W., Sterman, D.S. editors. Modelling for Learning Organisations. Pegasus. Richardson, G.P., Pugh, A.L., 1981. Introduction to System Dynamics Modelling with DYNAMO. Pegasus. Senge, P., 1990. The Fifth Discipline. Doubleday. Sterman, J.D., 2000. Business Dynamics, Systems Thinking and Modelling for a Complex World. Irwin, McGraw-Hill. Wolstenholme, E.F., 1990. System Enquiry—A System Dynamics Approach. Wiley, New York. Wolstenholme, E.F., 1993. The Evaluation of Management Information Systems—A Dynamic and Holistic Approach. Wiley, New York.
The use of system dynamics as a tool for intermediate level technology evaluation: three case studies E.F. Wolstenholme∗,1 Leeds Business School, Leeds, UK
Abstract This set of three case studies suggest that traditional approaches to technological evaluation are static and either high level and simplistic, or low level and complex. Further, that they tend to evaluate technology in terms of itself, rather than the domain it is intended to support. A holistic and dynamic method, based upon system dynamics simulation modelling, is described for the early evaluation of technology at an intermediate and balanced level. The method is demonstrated by applying it to the evaluation of management information systems (MIS) in the defence industry and new drugs in the pharmaceuticals industry. The approach involves the creation of dynamic simulation models of the anticipated domain of application of the technology and their use as a test bed to evaluate the systemic impact of the technology over time. Very importantly, the method allows the contribution of each aspect of the technology to be assessed and facilitates investigation of alternative structural and operating changes to the domain itself, to make best use of the technology. © 2003 Elsevier B.V. All rights reserved. Keywords: System dynamics; Technology; Technological evaluation; Simulation; Systems
1. Introduction Traditional approaches to evaluating new technologies have tended to focus either, at a high and simplifying level or at a very detailed, complex and specific level. High level assessments usually take the form of balancing the estimated costs of the technology with the anticipated benefits. Cost estimates might be based on previous applications of the technology in other domains or from trials or from statistical sources. A characteristic of high level assessments is that they are essentially static and focus on evaluating the ∗ Present address: Cognitus Ltd., 1 Park View, Harrogate, North Hampshire HG1 5LY, UK. E-mail address: [email protected] (E.F. Wolstenholme). 1 Professor of Business Learning (Leeds Business School); Director (Cognitus Ltd.).
0923-4748/$ – see front matter © 2003 Elsevier B.V. All rights reserved. doi:10.1016/S0923-4748(03)00018-3
194
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
technology in terms of itself, rather than in terms of benefits to the overall performance of the domain of application. Low level assessments tend to take a more domain-orientated approach. However, they are often characterised by complex and detailed assessments of the impact of the technology, often using tools designed for other purposes and usually with a particular emphasis on singular performance elements of the domain. There is a growing need to develop an intermediate level of technology evaluation, which can capture the domain-centred orientation of low level approaches, but also provide a balanced assessment of impact across the whole domain of application. To the present, one of the problems in developing such an intermediate approach has been the lack of suitable tools. This case study reports on a methodology based on system dynamics (Sterman, 2000; Wolstenholme, 1990; Morecroft and Sterman, 1994; Richardson and Pugh, 1981) for intermediate level technological assessment, providing an entirely original framework, which is showing considerable promise. The approach involves the creation (with the active participation of domain users and managers) of maps and dynamic simulation models of the anticipated domain of application of the technology, at an appropriate level of resolution, as a test bed to evaluate its global, rather than local, impact. The emphasis being on developing insights and learning about the systemic consequences of adapting a complex technology (Senge, 1990; De Geus, 1988). A three-stage methodology has been developed which will be described prior to presentation and analysis of existing applications. Three mature and two embryonic applications are described. 2. The three-stage methodology 2.1. Stage I: model the domain of application The first stage of the approach is to create, calibrate and validate an appropriate system dynamics model of the intended domain of application of the technology. Technology is defined here in its widest sense as any science-based application, ranging from hardware, though management information systems, to drugs. The type of model needed will depend, therefore, both on the technology and the domain. This base model must assume the present operation of the domain, which may make use of low or out-dated technologies. It must also incorporate appropriate scenarios and global performance measures by which to assess performance in the domain. Note here that the modelling process itself will usually help with the definition of performance measures. An important part of the method is that the process of model building itself should improve understanding of the domain of application. It is vital, therefore, that the base model should capture knowledge of the domain in the widest sense. It follows that the model should be constructed with a variety of people from the domain and that the model should form an acceptable and credible representation of the domain to them. It should represent an extension of their shared mental models of the structure, organisation and current operating policies of the domain.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
195
2.2. Stage II: technology assessment The second stage of the method is to use the model as a test bed on which to evaluate the impact of the new technology. This stage involves superimposing on the model the anticipated effects of the new technology throughout the whole domain of application. The model then assumes the role of a test “flight simulator” for the technology and forms a dynamic experiential learning environment. Here, potential developers and users of the technology can explore the interactions of assumptions and parameter values they have defined, and learn holistically about the potential impact of the new technology. 2.3. Stage III: technology accommodation in the domain The third stage of the method is perhaps the most significant. It builds on the fact that to make the best use of any significant new technology it is often necessary to change procedures and policies to accommodate it. The modelling process described facilitates experimentation to redesign the domain of application to achieve this. It could involve modifying any aspect of the domain and testing the effects of the modifications. For example: • • • • •
re-engineering processes; changing information paths and policies; changing organisational boundaries and responsibilities for particular activities; eliminating delays or increasing or reducing capacities; identifying high leverage intervention points where changes might produce the greatest outcome for the smallest input.
The detailed use of the method in two domains will now be presented. These concern the assessment of management information systems in a defence context (two applications) and drug evaluation in the pharmaceuticals industry (one application). A summary of applications in three other domains, which are at an embryonic stage of development will also be given. 3. Case studies in the use of the methodology 3.1. MIS evaluation in defence 3.1.1. The technology In the two case studies reported here, the technology was a complex management information system (MIS), although work was also carried out to evaluate various defence hardware items (Wolstenholme, 1993). The investment being made in MIS in defence organisations is enormous and, although much effort has been devoted to creating structured methods to aid the design of MIS, the area of MIS evaluation remains less developed. The traditional high level technological assessment approach used in the defence domain, prior to technology design and/or purchase, is that of an overall concept requirement analysis and feasibility study. The traditional low
196
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
level approach is that of using very detailed discrete entity simulators, designed for complex battle gaming. The emphasis to date has been, at worst, on simply assessing MIS in terms of its ability to process data and, at best, on evaluation of the impact of the MIS on sub-functions of the organisation at the detailed operational level. Attempts to introduce an intermediate level of technological assessment have centred on the use of prototypes. Whilst this approach is useful, it still suffers from a focus on real life experiments in only one part of the domain. Two examples will now be given of applying the systems-based intermediate level methodology in the defence MIS field. In both cases, a key element of the method was not to model the information system per se, but rather in terms of its anticipated effects on the physical and information processes of the organisation and the operational strategies which convert information into action. The procedure was to identify the information attributes (timeliness, accessibility, accuracy, relevance and comprehensibility) affected by each MIS feature and to incorporate their effect on parameters in the model. Of particular importance was the use of the method to assess the value of individual parts of the MIS as a basis for setting priorities for detail design and development. This work provided a direct assistance to incremental software engineering. 3.2. MIS evaluation in defence supply chains 3.2.1. Stage I model There are numerous complex interactions in military supply chains and many replications of each stage. However, at an intermediate level of analysis the chain can be considered as a four-stock system, consisting of Corps, Division, Brigade and Battle Groups, with supplies passing down the chain, subject to transport availability, and orders for supply passing back up the chain. Such a typical chain was modelled for a number of commodities. The work reported here will deal with ammunition as an example. The stage I model was based on the procedures and policies used under a manual information system for ammunition ordering and supply. The main complication of military supply chains over civilian supply chains is that the front line stocks are subject to significant loss, both due to attrition and due to being left behind as the tempo of an encounter unfolds. Losses of this nature were explicitly modelled and proved to have an important bearing on the behaviour of the supply chain. The main scenario used for experimentation with the model was an encounter of the battle groups with an enemy force, requiring the maintenance of a given rate of fire and hence use of ammunition to survive. The stage I model in the case study was validated against the expectations and views of defence management and professionals and against other more detailed simulation models. 3.2.2. Stage II model The features of the MIS technology incorporated into the stage II model were route management data, transport management data, monitoring of ground dump stocks, an on-line communications and a database. The information attributes affected by these features were identified and these, in turn, were related to parameters in the model.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
197
For example, on-line communications and data base features of the MIS relate to the information attributes of timeliness, accuracy and accessibility. Timeliness changes were represented by alterations in the values of the delays associated with information transmission and by reductions in the periods between reports. Accessibility was represented by the inclusion of information links between new sources and destinations of the information enabled by the MIS. For example, the availability of information at Corps and Division concerning the state of forward dumps and the stock in transit. Accuracy was quantified in terms of the difference between the perceived and true values of a variable and modelled by a random number function representing a distribution of values about the true value of the variable in question. 3.2.3. Stage II model results and the contribution of individual features of the MIS The stage II model results showed the rather obvious base result that it was possible for the battle groups to survive for longer for the same ammunition expenditure when using the proposed MIS, compared to using the existing and largely manual information and communications system. However, and more importantly, the model also provided a means of carrying out sensitivity analysis to investigate under what changes in assumptions and policies this proved not to be the case. The base model also provided some totally unexpected and surprising results, particularly with regard to the value of some parts of the proposed MIS. Three examples will be given here. (a) Firstly, the greatest ammunition losses occurred if only the route and/or transport management functions of the MIS were implemented. The majority of these losses occurred at the Brigade and Battle Group locations of the supply chain. (b) Secondly, the use of the improved information on the state of the forward dumps led, on its own, to the greatest reduction in ammunition losses. However, the savings provide small compensation for the overall losses incurred from improving the transport management. (c) Thirdly, although the installation of the database and the associated on-line communications did contribute to improving the overall performance of the supply chain, their impact was not as great as expected. This result emphasises that, although the availability and quality of information is increased by the MIS, significant policy changes must be implemented to really take advantage of the facilities which it provides. In summary, the main effect of implementing the MIS as shown by the model was one of allowing greater quantities of ammunition to reach the front lines. This has the effect of increasing the ability of the Battle Groups to maintain the desired firing rate. Although this effect has the potential to give greater operational capability to the front line gun units commanders, it also increases ammunition losses and reduces the flexibility at Division on Corps to reallocate ammunition between Battle Groups as the pattern of military operation changes. The results opened up an interesting debate about the whole purpose of the MIS, the objectives of the logistics function, the performance measures by which it might be judged and the need to change operating procedures and policies to make best use of the MIS. This thinking formed the basis of stage III of the methodology.
198
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
3.2.4. The stage III model The task of stage III model of the methodology was to examine the limitations of the prevailing policies and structure within the host system of the MIS and to investigate changes which may be necessitated or enabled by the introduction of MIS. The addition of this stage to the assessment process is consistent with the view that a proposed MIS must be evaluated in terms not only of its immediate impact, but also in terms of its potential to support managerial action. The use of the basic model and, in particular, the comparison of its results with those representing the effects of implementing the MIS, led to an explicit statement of logistics objectives as follows: (a) (b) (c) (d)
to enable the military operation to be achieved; to minimise ammunition losses and ammunition usage; to balance operational flexibility and operational capability; to minimise the resources/capacity required to achieve a military operation, particularly that of transport.
Of these, objective (a) was accepted as the prime objective in that it represents the requirement for logistics to support operational objectives. However, of particular importance here is objective (c), which provides an example of how the method helps the definition of opportunities for policy change. One unintended consequence of the simulated implementation of the MIS was to create a distorted distribution of ammunition stocks across the supply chain, with too much being ‘sucked’ through to the Battle Group end. This result quickly led thinking towards the possibility of using this distribution as a very visual performance measure, defined as the ‘ammunition profile’. The thinking also led to the possibility of having commanders negotiate the desired size of stocks at each point in the supply chain. In other words, the definition of a ‘desired ammunition profile’ across the chain and a set of policies to achieve it. The concept here was to balance front line stocks for immediate use but with high loss rates, with stocks further back, which could be reassigned as a battle unfolded. The sequence of action emanating from this chain of events led to a greater degree of openness and communication between managers in different parts of the supply chain. Other policy changes were defined in the context of regulating transport capacity and contingency stocks (less were required of both when using the MIS for the same level of risk). 3.3. MIS evaluation on the battlefield 3.3.1. Background The second application of the methodology in defence MIS concerned an MIS at a very early stage of its development life cycle and one to be implemented in a very unstructured area of military operation—the area of the tactical battlefield. The MIS studied was a Battlefield Management Information System and its purpose was to improve the command and control process, thereby giving commanders and staff more time for decision making based on accurate, relevant and timely information.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
199
There were three problems which made the evaluation of the Battlefield Management Information System difficult. The first was the sheer size and complexity of the battlefield operation and the proposed MIS. The second was that the MIS was only a concept and thus no data actually existed upon which to base an evaluation. The third problem was that many different perceptions existed of battlefield operations and this meant that the conceptualisation of a relevant base model of the host domain depended, in turn, on the anticipated scope of the MIS. 3.3.2. Stage I model The base model of the battlefield was conceived in terms of the following activities: • • • • • • • •
resource deployment (i.e. a commander allocating his available troops); reinforcement; logistics; target location; target acquisition and engagement; target tracking and co-ordination; weapon fire; movement (in particular the ‘shoot and scoot’ process, whereby a unit which has fired a given number of rounds moves to conceal its position, in case it has been identified by its flash or infrared signature); • vehicle repair; • secure communications (digital encryption). It should be noted that these functions were identified in an intense knowledge acquisition phase involving many defence personnel and were expressed in high level terms. They in fact subsume many lower level functions. The identification of these functions was paramount to identifying resources on the battlefield and further iterative discussions led to defining relevant states for these resources for inclusion in the model. Eventually, four states were conceptualised for enemy forces: unobserved, observed, acquired-as-targets and dead, through which a force progresses sequentially. However, later on in the modelling process it became clear that a backwards transition process was also required for some circumstances. For example, if an enemy unit withdraws out of range before it is killed then it must be allowed to revert to the unobserved state. Friendly units were also conceptualised as being in one of the above four states, plus two others. In order to provide a representation of the shoot and scoot cycle, a ‘hidden’ state was introduced whereby enemy units were unable to inflict damage on transient friendly units. A repair cycle was also introduced in order to represent those friendly units which had been damaged by enemy activity, but were recoverable; and also those which had broken down. All weapon types for both sides were combined into a single resource state of ‘units’. The base model was validated against the results from detailed battlefield simulators and run under a scenario of enemy forces advancing on a friendly position and using performance measured in terms of the loss rate of units on each side and the advance or retreat achieved.
200
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
3.3.3. Construction of the stage II model The features of the MIS technology incorporated into the model were: • digital maps; • improved communication of enemy locations between friendly units to improve fire control; • monitoring of targets to prevent duplication of attack effort and hence overkill; • secure communications; • type of threat alarm; • built-in test equipment. As in the previous case, the features of the MIS were related to a set of basic information attributes, which were in turn related to parameters in the model. 3.3.4. Stages II and III models’ results The results from the stage II model showed clearly that without the MIS the friendly force is defeated faster than with the MIS. With a Battlefield MIS in place, it would appear that friendly forces are able to inflict higher attrition on enemy units and are thus able to survive longer. Again, these base observations were rather obvious and the main value of the model was its ability to support intensive ‘what-if’ analysis. In each experiment carried out it was the process of interpreting how the results were obtained by superimposing the MIS, which gives rise to improved understanding. Again, some surprising results emerged from the stage II model which related both to identifying inadequacies in the design of the technology and/or to the need for operational procedures to be changed. Experimentation with the stage II model revealed the need for changed procedures in a number of areas to make best use of the MIS. For example, procedures and policies were required to: • reallocate targets assigned to friendly units which had been destroyed or hidden and hence unable to fire; • remove targets when they had moved out of range; • handle the increase in targets brought about by using the MIS. A further interesting aspect of insight generation arising in this case study was that the structures that gave rise to most of the insights were generic. For example, the problems associated with the reallocation of targets could easily be applied to the reallocation of logistics. If the MIS automatically sends ammunition reports to a commander or a logistics co-ordination point, there needs to be procedures in place to reallocated reports if the unit generating the report is eliminated. Whilst over-ordering is unlikely to be affected seriously by the attrition of one unit, the problem becomes more significant when aggregated across the entire Battle Group. Consideration of generic structures and procedures is a way of generating further, non-obvious insights. 3.3.5. Contribution of the individual MIS features The battlefield model also proved to be an excellent way of assessing the individual contribution of each part of the MIS. The model was run with each individual component of the MIS switched on in turn and with all parts switched on together. No one component of the
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
201
MIS alone made an outstanding difference to any model performance measures. However, the total set of components made a very significant difference to the model performance. This case provided a classical example of the fact that the sum of the technology was much greater than its individual parts and that this synergy arose through the way in which the parts interacted to support one another. 3.4. Drug evaluation in the pharmaceutical industry 3.4.1. The technology In the pharmaceutical industry, the technology of interest was that of new drugs. These are normally evaluated at a very high level in terms of their price and clinical benefits. Low level detailed assessments involve clinical trials (statistical sampling of individual groups of patients in different domains of application). The predominant methodology for pharmacoeconomic assessment is that of decision analysis. This involves the creation of decision trees which are used to combine data on the success (probabilities and outcomes) of individual products collected from clinical trials, using an expected value criterion. In practice, attempts are made to fit this approach at all levels of evaluation. A broadening of drug assessment is emerging in the development of the new subject of pharmacoeconomics. This shift of thinking is largely driven by drug companies, but in a world of cost conscious heath services there is a growing need to evaluate the global resource savings associated with new products within the end-user (i.e. purchaser) domain, as well as clinical efficacy. In this industry, work with the dynamic simulation approach has involved evaluating both clinical and disease inhibiting drugs. The main examples of the former are the new developments in anaesthetics and examples of the latter would be asthma treatments or ulcer eradicators. The types of domain models used in drug assessment applications depend on the product. For example, in the case of a disease control product the base model may be a model of the disease progression. In the case of a clinical product, the model may be a specific or typical representation of some of the many settings and procedures in which the product will be used. These might vary considerably, say between and within types of institutions or countries. The work reported here involves the evaluation of anaesthetics in hospitals. 3.4.2. Stage I model A number of models were developed from advice and knowledge provided by clinicians, anaesthetists and managers from a range of different hospital settings. These models incorporated the whole patient flow process from arrival to discharge since it was postulated that the potential benefits resulting from new anaesthetics could occur not only in the operating theatres, but also the recovery or day wards. The states of patient flows incorporated into the model were arrival and preparation, theatre, recovery, day ward and home. The model referenced here was based around in-patient surgery over a working day for one type of operational procedure. Such a model was considered to be sufficiently aggregate to connect the whole patient flow process to staffing and discharge policies, but sufficiently detailed to relate to individual patient movements. It could also be easily related to available cost data. The idea of creating generic models which reflect a broad
202
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
spectrum of local practices in hospital care organisations is becoming an established part of pharmacoeconomic practice (Kenigsberg, 1996). The performance measures used in the model were those which were considered to be meaningful and important to the decision-makers in hospitals. The focus here was on the trend towards more day care surgery and on exploiting ways to create capacity in day care to allow greater patient throughput, possibly involving higher costs, but associated higher revenues. All resource calculations (staff and materials) in the model were based on the patient flows. The stage I model was validated against cost data collected from a studies aimed at quantifying and valuing in monetary terms resource use at each stage of selected surgical procedures, with particular relevance to anaesthesia. The model was run under a number of existing hospital policies and practices relating to time spent by anaesthetists with patients in recovery and the time at which patients were discharged to home. 3.4.3. The stage II model The stage II model incorporated the claimed benefits of new anaesthetic drugs by modifying a whole range of model parameters. For example, the average time taken at each stage of the patient flow process and the numbers of staff and materials used. The stage II model demonstrated that the composite effect of the claimed benefits of the anaesthetic would be significant. Certainly, in terms of the capability of the hospital to complete a list of patients quicker. However, that the savings in cost and time were not necessarily realisable, since staff were full time, or sufficiently great to enable more patients to be treated in a day. 3.4.4. The stage III model The stage III model was used to evaluate the effects of changing a number of managerial policies in the hospital, which were facilitated by the increased safety levels of the new drug. These policies included transferring responsibility for recovering patients from anaesthetists to nursing staff at an earlier point in the patient flow process and allowing patients to be discharged earlier. Superimposing these new policies on the model indicated that very substantial savings were possible in the patient cycle time, sufficient to accommodate unto two extra patients per day in to the existing daily operating schedule with their associated revenue implications. The message from this work was again that a new technology on its own made some, but not excessive, impact on the economics of its domain of application. However, the effects of new managerial policies, which were facilitated by introducing the new drugs, could make a significant economic impact. Further, that the policy savings were not associated with the point of application of the technology, but in areas of the domain remote from this (for example, the recovery and ward areas of the hospital). 3.5. Other embryonic applications In addition to the above case studies, other applications of the method described in this report are underway, but at an early stage of development.
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
203
3.5.1. Oil industry fluid tracking Here the technology was high-specification water infusion to enhance oil recovery. Such technology is traditionally evaluated in detail by using oil reservoir simulators designed for other purposes and related only to the reservoir and production functions of the oil business. The objective therefore was to introduce a much broader evaluation approach to assess the broad impact of water infusion technology across all operating functions. That is, at the intermediate level to assess the technology in terms of its impact on not just the production, but also the marketing and financial domains of the business. A real-time fluid tracking model was developed for this purpose to provide semi-quantitative answers to what-if questions concerning the impact of the technology at an early stage in the assessment process. 3.5.2. Manufacturing and project management Similar work is currently being undertaken in manufacturing and project management, where again high and low level assessment methods prevail. The low level methods usually centre on very detailed simulation and planning software. In the case of manufacturing there are detailed production-line simulators for materials resource planning and in the case of project management detailed project scheduling software. The approach in both cases has been to develop more holistic higher level models to give a wider view on the overall potential of the technology, which helps to focus priorities for large investments. An important stage III model effect was he use of the model to help validate the parameters used in the detailed material planning system. 4. Conclusions This case study has introduced the concept of a three stage procedure for broad and intermediate level assessment of technology and has demonstrated its application in a number of areas in practice. The main advantages of the approach are that it provides: • An assessment of the technology in terms of its effect on the dynamic behaviour of its domain of application, rather than an assessment in terms of itself. • A framework to indicate the way in which the technology interacts with its domain of application. The indicated benefits of new technology from this type of assessment can be surprising and counter-intuitive. This contrasts strongly with static cost benefit analysis, which often assumes that the benefits of each part of the technology are independent and that the combined effect of the technology is a linear summation of its parts. • A way of sharing thinking about the technology between managers in different functional areas of the organisation at an early enough time for all to be involved in the analysis. • Experiential learning about the technology and the domain of application and their interaction by providing a quantitative basis for ‘what-if’ analysis. • A way of determining the overall merits of a technology and, in particular, its possible side effects, prior to a full and costly commitment.
204
E.F. Wolstenholme / J. Eng. Technol. Manage. 20 (2003) 193–204
In general, the method is showing a high level of promise. It will become more and more relevant and important as the sophistication and cost of technology increases and the ease of use of modelling software improves. References De Geus, A.P., 1988. Planning as learning. Harvard Business Review 66 (2) March–April, 1988. Kenigsberg, P.A., 1996. Economic evaluation in anaesthesia a production process approach, EPH Feb. 1996, vol. 2 (1). Morecroft, J.D.W., Sterman, D.S. editors. Modelling for Learning Organisations. Pegasus. Richardson, G.P., Pugh, A.L., 1981. Introduction to System Dynamics Modelling with DYNAMO. Pegasus. Senge, P., 1990. The Fifth Discipline. Doubleday. Sterman, J.D., 2000. Business Dynamics, Systems Thinking and Modelling for a Complex World. Irwin, McGraw-Hill. Wolstenholme, E.F., 1990. System Enquiry—A System Dynamics Approach. Wiley, New York. Wolstenholme, E.F., 1993. The Evaluation of Management Information Systems—A Dynamic and Holistic Approach. Wiley, New York.