Evaluating the costs and benefits of laboratory information management systems

?? Tutorial

15

Chemometrics and Intelligent Laboratoty Systems: Laboratory Information Management, 13 (1991) 15-36 Elsevier Science Publishers B.V., Amsterdam

Evaluating the costs and benefits of laboratory information management systems * Randall R. Stein Chesapeake

Software,

Inc., Brandywine

Five Building, PA 19317

(Received

U.S. Routes

I and 202, Chadds

Ford,

(USA)

31 July 1990; accepted

18 September

1990)

Abstract

Stein, R.R., 1991. Evaluating the costs and benefits of laboratory information L.aboratoT systems: Laboratory Information Management, 13: 15-36.

management

systems.

Chemometrics

and Intelligent

A cost-benefit analysis is an effective tool for a manager using or considering a laboratory information management system (LIMS). Before implementation, the analysis will indicate the financial impact of a LIMS on the laboratory, and provide a sound basis for making a purchase decision. After a purchase, the analysis will provide a yardstick for measuring the success of the LIMS implementation. This tutorial explains how to perform an effective cost-benefit analysis. Considerations unique to the laboratory environment are discussed. A pragmatic approach to economics is taken, with the intention of achieving a sufficient level of accuracy for making managerial decisions without an undue expenditure of time. To assist in the analysis, detailed enumerations of specific LIMS cost and benefit factors, as well as common analysis errors, are included. Techniques for using the cost-benefit analysis to speed up and capitalize upon the benefits of a LIMS implementation

are discussed.

CONTENTS Abstract

. .. . .

.......................................

1 Introduction

..................................

................................ 2.1 First steps .................................

2 Getting

started

2.2 Laboratories

are different

from other business

..

operations

.................... 3.1 Exploratory studies .......................... 3.2 Fixed budget studies ......................... 4 LIMS costs ................................... 4.1 Direct vs. indirect costs ....................... 4.2 Initial vs. ongoing costs ....................... 4.3 Tangible vs. intangible costs ....................

in part at the Fourth

0925-5281/91/$03.50

International

LIMS conference,

0 1991 - Elsevier Science

.. .

... ... ..

15 17 17 17

3 How much analysis is necessary?

* Presented

. .. .. .

Publishers

. . . .

Pittsburgh,

.. . . . .

PA, June 1990.

B.V. All rights reserved

. . . .

. . . .

.. . . ..

..

.. .. . .. . .. . . .

...

18 18 19

. . .. . .

19 19 19 20 20

Laboratory Information Management

16

4.4

Specific cost factors ........................... 4.4.1 Initial costs .............................

20

.

5.5 Analyzing

intangible

5.5.1 Extrema 5.5.2 Breakeven

or unpredictable

22

.

23

.

23 23 23

.

........

benefits

20 22

.

........................... 5 LIMS benefits .................................. 5.1 Tangible benefits ............................. 5.2 Intangible benefits ............................ 5.3 Unpredictable benefits ......................... 5.4 Medium-term vs. long-term benefits ................ 4.4.2 Ongoing costs

25

......................... analysis ........................

. .. . .

analysis

in a LIMS purchase ........ ........................ 5.6.2 Laboratory throughput ..................... 5.6.3 Quality of data .......................... ................... 5.6.4 Laboratory management ...................... 5.6.5 Quality of operations 5.6.6 Regulatory compliance ..................... ..................... 6 Industry-specific considerations ...................... 6.1 Public-sector organizations ............ 6.2 Research and development organizations 6.3 Regulated industries ........................... ..................... 6.4 Manufacturing organizations ..................... 6.5 Defense-sector organizations ............ 7 Applying LIMS cost-benefit analysis results 5.6 Benefit factors to consider

25 25 25

5.6.1 Data management

7.1 Economics,

accounting,

and other unpleasant

25

..

..

or off-the-shelf

26 26

.

26 26

.

.. ... . . ...

.

27 27

.

27 27

..

. ...

27

..

27 28

..

.

28

..

28

... ... . .

. .

.

...................... ........................ Initial drop in productivity .................... Return to pre-LIMS productivity ................... Plateau at pre-LIMS productivity .................... Rapid productivity improvement .............. Plateau followed by slower improvements

29 29 29

8 Time course of LIMS productivity

8.1 8.2 8.3 8.4 8.5

9 Summary

of cost-benefit

analysis steps

30 30

..................

..................................... ................................ 11 Acknowledgements

10 Conclusions

Appendix: Common errors in laboratory cost-benefit analyses ..................... A.1 Common errors in expectations A.2 Common

errors associated

with initial LIMS cost factors

A.3 Common

errors associated

with ongoing

LIMS costs

A.4 Common errors in analysis ........................ References .........................................

26 26

.

subjects

systems?

..

. ..

7.3.1 What size system should be implemented? custom,

26 26

... . 7.1.2 Cost of money and scheduling ...... . . 7.1.3 Net return on investment . . .... 7.2 Valid measurements for project approval . ... . 7.3 Drawing conclusions from an analysis . . .. 7.3.2 In-house,

.

. .

... . ..

7.1.1 Cash flow analysis

.

. .

......

...

. . . . .

. . . .

.. . . .

31 31

.. .. ..

.... . . . . . . . . . .

31 31 31 32 32 32 33

..

33

.. .. ..

34 35 36

??

?? Tutorial

1 INTRODUCTION

A cost-benefit analysis is an important technique for a manager who is considering or using laboratory information management systems (LIMS). Prior to the purchase of a LIMS, a costbenefit analysis will indicate the revenues and expenses that may be expected after the LIMS is in use. This may provide justification for the LIMS or warn that economic benefits alone may not justify the system. After the purchase, the costbenefit analysis may be used to provide a yardstick for the LIMS implementation effort, indicating whether the expected benefits have been achieved, whether expenses are being properly contained, when further customization of the system is warranted, and what laboratory operations should be examined to improve laboratory productivity. This tutorial explains how to perform an effective, pragmatic cost-benefit analysis of a LIMS. A pragmatic cost-benefit analysis is an exercise in balance: obtaining sufficient accuracy without investing an excess of time or money in the analysis itself. This article will try to simplify the process by indicating the most important considerations in the analysis and how to extract good management information without wasting time agonizing over every penny. It is far more common for a LIMS cost-benefit estimate to err on the side of being falsely optimistic than falsely pessimistic. It is not uncommon for a laboratory to appear to be running more efficiently after a LIMS implementation, but for no cost savings to show up on the bottom line. To combat this tendency, many common mistakes in the performance of cost-benefit analyses have been summarized in the Appendix. A few things need to be remembered when performing a LIMS cost-benefit analysis. First, cost-benefit analysis is an important part of the decision whether to implement a LIMS, but does not usually constitute the entire decision. Organizations frequently choose to implement procedures for reasons other than direct cost effectiveness. The cost-benefit analysis should assist the decision-making process, not supplant it. Second, performing a cost-benefit analysis is

17

not difficult, but it can be deceptive. Implementation of a LIMS requires attention to many details of scientific procedure and laboratory organization, sometimes making it difficult to switch back to a managerial perspective and consider expense and benefit centers rather than scientific details. The take-home message from this article should be that a LIMS cost-benefit analysis of sufficient accuracy to provide sound management decisionmaking information can be performed. A final admonition before starting: cost-benefit analyses always represent alternatives. To make the analysis complete, consider all reasonable alternatives, and always include the costs and benefits of not doing anything at all!

2 GETTING

STARTED

This section benefit analysis

will discuss project.

how to begin

a cost-

2.1 First steps The very first step in starting an analysis should be to establish a reasonable accuracy goal, e.g. 10%. Keep this continuously in mind. Do not waste days on an item that is 0.1% of the project, but do not miss those small items that add up to a large figure. Bear in mind the need to look for items where the difference between costs and benefits are the greatest. This may be quite different from the items where the magnitude of the costs or benefits are the greatest, because many items that represent sizable benefits are accompanied by equivalent and canceling costs. The most important requirement for performing an accurate analysis is to be sure not to miss any important cost or benefit factors. Early in the analysis, circulate to one’s peers a list of the items thought to be important (not the analysis of those items), and specifically request that it be reviewed for completeness. Often a person with a different perspective will discover other important factors. Do not circulate the analysis itself: the analysis is too distracting from the task of searching for cost and benefit factors.

Laboratory

18

Try to avoid running out of time in the middle of the analysis. As soon as there is a preliminary list of cost and benefit factors, prioritize the items in the order that they will bring net benefits (or costs) to the organization. Then work down the list. This way, the estimate will continually be refined and one can see when it is stabilizing. Common errors in expectations for LIMS are enumerated in the Appendix.

2.2 Laboratories

are different from

other business

operations

If one is experienced at cost analysis, but not experienced in laboratory operations, it is important to realize that the laboratory differs from other types of business operations, and the approach to the cost-benefit analysis must be correspondingly different. Both laboratory and conventional systems are dependent upon the timing of how items move through the organization. However, in the laboratory, there tend to be more constraints on that timing, while operating costs and quality are more dramatically dependent upon it. For example, in many non-laboratory departments, delays in providing information may cause someone to sit idle. However, in the laboratory, a similar delay may invalidate a procedure, making it necessary to restart an analysis from the beginning, or worse, such a mistake can ruin an entire sample or experiment. The timing requirements of analytical methods make laboratory procedures less flexible than other types of business procedures. Achieving one benefit may mean sacrificing another. For example, an automated laboratory system may achieve an important improvement in efficiency and quality by avoiding the error-prone transcription of data. However, avoiding transcription requires analysts to enter data into a LIMS the moment they become available, even if it is not completely convenient to do so. Such temporal constraints tend to make a LIMS less flexible than manual procedures, and increases the probability that automated procedures may cause new bottlenecks in the laboratory.

Information

Management

??

It could be concluded that a LIMS tends to alter the flow of data and materials through the laboratory. One should examine each change, looking for potential bottlenecks as well as benefits. In general, it is necessary to consider the effect of automation on the movement of materials as well as movement of data; resist the temptation to treat these two areas as somehow common: they are not the same. Understanding the differences between the movement of data and materials reveals potential efficiency improvements that are difficult to quantify when considered together. Analyzing time-dependent (real-time) laboratory operations may require special analysis skills. There are a number of inter-disciplinary techniques that can help ameliorate these difficulties, including real-time analysis techniques developed for the real-time software and process control environments and time-motion techniques. The discussion of these techniques goes beyond the scope of this article, but consulting an expert in this area is recommended. Do not infer from these warnings that LIMS is not a powerful productivity tool: it certainly is. However, to produce realistic expectations, enthusiasm must be tempered with the understanding that almost every savings is partially offset by an associated cost. If a benefit is found without an associated cost, take it as a warning that some aspect of the analysis may have been neglected.

3 HOW MUCH

ANALYSIS

IS NECESSARY?

Cost-benefit analyses are performed for two primary reasons: establishing optimal expenditures and justifying projects. Ideally, a cost-benefit analysis is performed during the planning stages of a project. The benefits achieved by various expenditure levels are examined and the level of expenditures that are optimal is selected. Realistically, this approach is not always possible. Often, budgets are set far in advance, by managers far removed from the laboratories, or the budget is too small. In these cases, the cost-benefit analysis is performed to justify the use of a portion of a fixed budget. In the ideal case, the question is

?? Tutorial

19

“How much should I spend for the best return?” while in the pragmatic case, “Will an expenditure of $200000 be beneficial?” should be asked. If there is a fixed budget, plan to spend a specific fraction of this budget on initial work, including the cost analysis. If one is looking for an optimal expenditure, base the decision upon the resources that are involved. These options are elaborated upon below. Finally, trust instinct; if one is uncomfortable with the analysis, it is useless, and work must continue. However when it is felt all the important points have been covered, then stop. Avoid the expensive temptation to refine your analysis to perfection. 3. I Exploratory

studies

When looking for optimal expenditures, the amount of company resources that will be handled by the potential system should be used to decide how long to spend on the analysis. LIMS may be classified as small, medium or large, with commensurate cost-benefit analysis expenditures as shown in Table 1. 3.2 Fixed budget studies When on a fixed LIMS budget, use a specific fraction of it for initial tasks prior to the LIMS purchase. These total initial costs should amount to lo-15% of the project. The initial tasks may be broken down as shown in Table 2. The largest fraction of the initial time should be spent specifying your needs. This includes discussing the pro-

TABLE

1

LIMS expenditures Laboratory

based

feature

Project cost, in dollars Reports People Laboratories Samples per year Analyses per year Analysis time, weeks Expenditure, in dollars

upon project

size

Small

Medium

Large

60000 20 5 2 6000 30000 1 3000

3OOOcO 70 50 8 40000 250000 3 9000

1500000 300 500 30 300000 2000000 15 45000

TABLE Initial

2 analysis

tasks

Task

Percent of initial tasks

Specify LIMS requirements Education about LIMS Analyze available products Cost-benefit analysis

40-50 20 10 20-30

posed system and studying the laboratories. Educating oneself about LIMS and analyzing available products are usually combined efforts that amount to the second largest fraction of this time. Performing the cost-benefit analysis comes third. The time that will be required to configure the LIMS is not included as part of these initial costs. These fractions will shift depending upon whether the selection is made by a committee or a single person, whether travel is involved, and so on. The larger the project and the more important an accurate prediction of the ultimate costs, the larger a percentage should be devoted to the cost analysis. This division should give a budget to use for the cost-benefit analysis effort that will translate to an estimate of the time that can be devoted to the project.

4 LIMS COSTS

LIMS costs are usefully divided in several ways: 1. direct vs. indirect costs; 2. initial vs. ongoing costs; 3. tangible vs. intangible costs. Each of these categories will be discussed below, followed by a list of specific cost factors that should be considered. 4.1 Direct vs. indirect costs Direct costs are expenditures most closely related to a task. Indirect costs are related expenditures: the overhead of doing things. Examples of direct and related indirect costs are shown in Table 3.

Laboratory

20

TABLE

3

Examples

Equipment

Management

4

4.2 Initial vs. ongoing costs of direct and indirect

costs

Related

Direct cost Employee

Information

salary

purchases

Maintenance fees Labor to install cables

indirect

costs

Vacation time Insurance Office and laboratory space Electricity Supplies Accounting and personnel costs Time to select equipment Space for equipment Equipment installation time Administration of contract Disruption to the laboratory Loss of time Potential power loss

The most important indirect cost is for personnel. It is usually easy to get this figure: most company controllers can give it immediately. Unfortunately, many cost analyses - especially those where increased staffing is required ignore indirect costs. Typical indirect costs for personnel range from 70 to 120% of salary, with 80% being a conservative figure for a manufacturing organization and 90% being a conservative figure for a research and development one. A chemist earning $70000 per year really costs a company about $126000 per year; an entry-level technician earning $25 000 per year really costs $45 000. Discussions of changes in staffing are generally viewed by LIMS vendors with slightly less enthusiasm than swimming in shark-infested waters. As an analyst, you should not beg this question. Avoid making managerial decisions and treat all potential changes in staffing as if they were going to actually take place. This way, the costs and benefits of the automated system are clearly presented. If it is desirable to present the costs of retraining personnel to another job, do so as a separate line item from personnel costs, so it is clear that this is a policy cost, not a system cost. If hiring and firing personnel are required, do not forget the considerable costs associated with those activities.

It is necessary to separate the initial costs of a LIMS from the ongoing costs. Initial costs include [l] items such as consulting, actual equipment purchase or lease, modifications to the site, staff to manage and oversee the project, training, the cost of disruptions, equipment installations, preparing internal documentation, and so on. These are termed “initial” because they occur only once, although they may not all happen at the very beginning of the project. Ongoing costs include software, hardware and new facilities maintenance costs, depreciation, and operations overhead. Frequently in the LIMS arena, ongoing costs are much more important than initial capital expenses. 4.3 Tangible vs. intangible

costs

Most costs that are identified are tangible. Intangible costs include considerations such as a drop in morale caused by the disruption of a new system. Most intangible factors turn out to be benefits rather than costs, so these items are discussed among the benefits, below. The treatment of intangible costs and intangible benefits is the same. 4.4 Specific cost factors Following are lists of the common cost factors in a LIMS purchase. Each industry has a different set of factors that are more prominent and others that are less important. First the initial costs will be discussed, then the ongoing costs. 4.4. I Initial costs The major initial cost factors in a LIMS purchase are summarized in Table 4. Common errors made when specifying initial LIMS costs are enumerated in the Appendix. Notice the large number of initial costs in Table 4 associated with the time required for personnel to figure out what to do, develop new functions, or manage a portion of the LIMS effort. Failure to anticipate these types of costs are the major reason that managers look back on their

?? Tutorial

TABLE Initial

21

4 LIMS costs

Category

Cost factor

Hardware

Computer storage devices (disks, backup devices, archive devices) Wiring, power conditioning Climate control Furniture near the computer Networking: interfaces, servers Networking: cabling, wiring closets User interaction: terminals, workstations Display devices: printers, plotters Sample ID: bar code readers, printers and label applicators Cabling and wiring for devices above Repair documentation, spares Operating system System development tools, including communications and compilers Networking and connectivity software components LIMS core components LIMS optional components LIMS development tools Database systems and development tools Generic instrument interface software Special instrument interface software, e.g. chromatography, process control Graphics and printing software Documentation: users, managers, development Furniture for storing documentation Personnel to manage acquisition and installation Disruption due to installation Loss of incompletely depreciated equipment that is replaced Lost space for computers, terminals, printers Decide where to place computers, terminals, printers, and cables Perform customization and configuration Develop and reproduce computer usage procedures Develop and reproduce LIMS usage procedures Develop and reproduce new laboratory procedures Determine off-line vs. on-line data needs Develop archiving and backup procedures Computer training, for users, managers, hardware engineers, support staff LIMS training, for users, managers, support personnel Instrument interface training, for users, managers, hardware engineers, support staff Accessory software training, for users, managers, support personnel Lost work during training Decrease in productivity during training, installation and adaptation Time to identify items that must be present in the database Time to configure database to include the necessary items Time to create necessary database forms Time to set up system security. desired audit trails, and enter valid users Time to design and create necessary reports Time to design and create necessary charts and alarms Time to configure laboratory for tests. samples, instruments, projects, labs, etc. Enter analysis procedures Set up accounting system, if used Develop interfaces to other corporate systems Hook up instruments, reconfigure automated instruments

Software

Installation

Configuration

and conversion

Laboratory

22

LIMS implementation efforts and wonder why they didn’t achieve their forecasted goals. It is not unusual for the personnel costs involved in meetings to exceed the initial purchase cost of a small LIMS. This is as it should be: there is no substitute for planning! Work closely with your intended computer vendors to establish hardware costs. Large computer vendors are highly skilled at determining these needs in areas such as power, power condi-

TABLE Ongoing

5 LIMS costs

Category

Cost factor

Supplies

Printer paper, forms, ribbons Laser printer paper, forms, toner, drums Plotter paper, pens Bar code labels, ribbons Fuses, keys, cables, spare parts Media for backups and archives Cleaning supplies Computer hardware Networking hardware Operating system LIMS core software LIMS option software Database software Other software packages Administration of maintenance contracts Installation of updates Performing in-house preventative maintenance Supervising in-house preventative maintenance Performing non-preventative maintenance Supervising non-preventative maintenance Performing backups and archiving Supervising backups and archiving Archival costs Internal support personnel time Training of new personnel due to normal promotions and turnover Power Computer lease and rental costs Communication rental (telephone, satellite, etc.) Depreciation costs

Maintenance

Personnel

Miscellaneous

fees

Information

Management

W

tioning, climate control, media storage and wiring. The recommendation is to seek expert advice if multiple sites or networked computers will be involved. Do not neglect the cost of wiring buildings and running cables to desks and instruments. Let the LIMS software vendor specify how much computer power will be required, and avoid the temptation to purchase the computer system before you have careful discussions of the performance requirements with the software vendor. If pressured to do so, force the LIMS vendor and system vendor to sit down together. Calculating upon storage requirements requires determining the needs for off-line vs. on-line data, and will also depend critically upon how the LIMS handles this aspect of the database. This varies significantly from vendor to vendor, so recommendations are not easily generalized. 4.4.2 Ongoing costs The major ongoing cost factors in a LIMS purchase are listed in Table 5. It is important to remember that ongoing costs begin to accumulate as soon as the project starts. Only a few of them are polite enough to wait a year. Ongoing costs that are associated with retraining and upgrades of the software and hardware are frequently underestimated. Depreciation costs may be a large fraction of the total computer system costs. When estimating ongoing costs, be sure to include estimates of the future direction of the organization into the calculations. Are increasing numbers of samples or tests anticipated? Integration with new external systems? New regulatory requirements? Changes in quality control procedures? Shifts in clientele or the basic mission of your organization?

5 LIMS BENEFITS

It is also useful to divide several categories: - Tangible benefits _ Intangible benefits _ Unpredictable benefits - Medium-term vs. long-term

LIMS

benefits

benefits

into

?? Tutorial

23

Benefits are nearly always ongoing throughout the life of a project, although they frequently change over a period of years. In rare cases, benefits will occur only at a specific point in time, e.g. a bonus from a supplier because the new system meets a quality specification. This article will assume that benefits are ongoing. Many personnel have the tendency to overestimate the benefits of automation. This especially applies to the proponents of a new system, even to those who can very accurately estimate costs. As was pointed out previously, there is almost always a cost associated with every benefit, so if an unbalanced benefit is discovered, it is likely that a cost factor has been missed. Each of the above categories will be discussed, followed by a list of specific benefit factors that should be considered.

5. I Tangible benefits Tangible benefits are the benefits that are easily assigned monetary figures. These include features such as reductions in the costs of calculating and reporting, improved capacity and security of record keeping, faster and more thorough searching, faster and more complex analysis of data, less time spent responding to customer inquiries, reduced or shifted manpower and improved feedback on resource usage.

5.2 Intangible

benefits

Strictly speaking, there may not really be such a thing as an intangible benefit. Given enough time and enough information, one can assign a monetary amount to any benefit. Pragmatically, however, there are no resources to analyze certain benefits, so these are called intangible. Intangible benefits typically include ‘improved image because of a state-of-the-art system’ and ‘improved reputation in the industry’. Benefits may be welldefined in one industry, but difficult to enumerate (and therefore intangible) in another industry, such as the ubiquitous ‘better service to our clients’ and the ever-popular ‘better laboratory management’.

The ideal situation may be when tangible benefits justify a system and intangible benefits are a bonus. However, many LIMS are purchased precisely because of their intangible benefits, so due consideration must be given to them. It is interesting to note that intangibles most frequently show on the benefits side of the balance sheet. This is not surprising: although there are many intangible costs associated with businesses, they are more frequently associated with new business opportunities than changes to an existing operation. 5.3 Unpredictable

benefits

Unpredictable benefits are benefits that do not happen frequently enough to predict statistically when they will occur, or that occur too irregularly to assign a monetary benefit. There are frequently reductions in the incidence of calamities caused by the improved quality offered by the LIMS. When LIMS benefits sound like a presentation from a life insurance salesman, unpredictable benefits are probably being discussed. What is considered unpredictable is often dependent upon the industry and business sector involved. For example, a mistake in a state environmental laboratory may allow an environmental calamity to be perpetrated. Estimating the monetary benefit to the state of improving the quality in that laboratory may require a risk analysis that is well beyond the scope of a LIMS feasibility study. On the other hand, an industrial environmental laboratory performing exactly the same analyses may be able to place very reliable numbers on the benefits to the corporation. 5.4 Medium-term

vs. long-term

benefits

Establishing long-term benefits has a difficulty common to automation systems: the technology is changing too quickly to make accurate predictions. Benefits that accrue over periods of 10 years or more should be treated very cautiously and be suspected of being inaccurate. These include long-term operations costs and some public goodwill and system flexibility issues. Medium-term

Laboratory

24

TABLE

Information

Management

6

LIMS benefits Benefit factor

Category Data management

Laboratory

Quality

[l]

throughput

of data

Laboratory

management

Data may be stored more systematically More data conveniently stored per event Track a greater number of relationships Track more complex relationships Change data more quickly Data may be more secure Security can be more highly tailored Security may require less managerial oversight Audit trails can be automated Transactions are more traceable Managerial information on the amount and type of data are improved Data may be more complete Data can be recalled more quickly and flexibly Reliability of data storage and retrieval is improved Data may be handled more flexibly Reporting is faster Unit printing costs may be reduced Cost of transferring data may be reduced New relationships among data are more easily examined Data are available when needed by requiring its entry Entry of sample information may be faster Sample identification may be faster Reduced transcription (offset by possible increased data entry time) Faster QA/QC or statistical process control (SPC) Speed of analysis Automation of entire analysis Faster response to queries Faster and automated reporting Automatic validation of results Inspection only of discrepancies Assistance in remembering procedures; prompt for required data More accurate data because of automated acquisition More accurate data because of range checking More accurate sample identification due to bar codes Reduced transcription errors Improved instrumentation reliability from QC procedures Improved instrumentation reliability from automated calibration procedures More involved QC procedures SPC on laboratory analyses Automatic enforcement of validation procedures Automatic enforcement of laboratory analysis procedures Automatic enforcement of instrument standardization procedures Automatic prompt for missing data Improved accuracy of laboratory management information Increased quantity of laboratory management information Reduced labor to produce laboratory management information More instruments, analysts, results handled per manager Reduction in lost opportunities due to inadequate knowledge Reduction in unsolved backlog problems Improved validation procedures

W

?? Tutorial

TABLE

25

6 (continued) Benefit

Category Quality

of operations

Regulatory

compliance

factor

SPC on manufactured products Statistical quality control on laboratory operations Communicate with corporate CIM system Improved QA/QC on products Reduced testing costs Correlate laboratory analyses and process measurements Faster solutions to production problems tested in laboratory Faster notification of backlog problems Improved electronic data interchange capabilities Automated communication with inventory, ordering and materials Automated regulatory report generation Regulation-compliant audit trail Enforce regulation-compliant laboratory procedures Enforce regulation-compliant result validation Improved response to changing regulations Fast compliance with regulatory audits

benefits, on the other hand, can be considered with confidence. 5.5 Analyzing

intangible

or unpredictable

benefits

Intangible benefits are frequently treated as simply being different from tangible benefits. Tangible benefits are measured on a monetary scale, and intangible benefits are measured on another scale. This makes firm decision making on the basis of intangible benefits very shaky. The following techniques are particularly useful for assigning monetary costs to intangible benefits [l-3]: 5.5.1 Extrema analysis To analyze by extrema, assign best-case and worst-case costs to the intangible item. If there is more information, use ‘pessimistic’, ‘realistic’ and ‘optimistic’ categories. 5.5.2 Breakeven analysis To analyze by breakeven values, work backwards to determine what benefit is required to make the system break even. For example, if the company needs to save $50000 by better handling of customer inquiries and there are 1000 inquiries per year, $50 per inquiry must be saved to break even.

planning

systems

When unable to analyze intangible benefits by the aforementioned techniques, the benefits should be clearly itemized so they can be carefully considered. Sometimes, intangible benefits are restatements of political goals and have nothing to do with costs, and vigilance is needed to filter these out. Statistical techniques exist for the analysis of benefits that are difficult to predict. These techniques, referred to as risk analysis, are beyond the scope of this article and are described elsewhere WI. 5.6 Benefit factors

to consider in a LIMS

purchase

As mentioned previously, each industry has different items that come to prominence and others that are less important, and any specific factor can be a cost or a benefit, depending upon whether it increases or decreases as a result of the LIMS. Specific benefit factors to consider, listed in Table 6, have been loosely grouped into the following general areas. 5.6.1 Data management Computerized systems provide a number of generic data management advantages [l] that are not unique to LIMS, but apply to most computerized systems.

Laboratory

26

5.62 Laboratory throughput Benefits from improvements in throughput arise from several distinct areas, including improvements in handling samples, completing the necessary paperwork or data entry for handling samples, and generating the end product, be it a certificate of analysis, report or signal to a process control system. Benefits are incurred because of time reductions that translate directly to reductions in personnel or increases in sample load per person. In the author’s opinion, the ability to provide practical productivity improvements for a specific environment is the area of the greatest difference among LIMS vendors’ products. 5.6.3 Quality of data Data quality benefits curacy in data entry and niques. This may involve tion of data transcription analysis of the data that

arise from improved acnew quality control techthe elimination or reducor improved statistical are entered.

5.6.4 Laboratory management Laboratory management benefits arise from improved control of the laboratory operation. They show up as decreased labor costs to perform analyses or manage laboratory personnel. The ability to provide meaningful managerial information differs greatly between vendors and implementations. 5.6.5 Quality of operations Operations quality is important in environments where the laboratory is not an end unto itself, but is a service for another organization. These benefits apply to that other organization rather than the laboratory. This is common in the manufacturing quality assurance and quality control laboratory. The potential for LIMS to improve quality in the manufacturing environment is discussed in detail elsewhere [4]. 5.6.6 Regulatory compliance Although regulatory compliance may be placed in several of the previous categories, it is so important to the involved industries that it is used separately here.

Information

Management

??

Bear in mind that these categories are highly interrelated. When estimating benefits, just like ongoing costs, make sure that the analysis includes an estimate of the future of the business, as described in Part 4.4.2. Allowances should be made for new business opportunities and changes in the level of benefits as the goals of the organization change.

6 INDUSTRY-SPECIFIC

CONSIDERATIONS

Several types of industries or organizations that require special consideration in performing a cost-benefit analysis are discussed in this section. 6. I Public-sector

organizations

Public-sector organizations have classic problems in performing cost-benefit analyses. Rather than existing to make a profit, they serve some form of public good, and instead of having paying customers, they serve an entire community. Efficiency or productivity may take a second seat to political motivations, and decisions in the publicsector organization are likely to consider factors other than the cost-benefit analysis more heavily than other organizations. In spite of these differences, an analysis can be completed in a similar manner. The biggest difference is that the number of intangibles and unpredictables increase because the ‘public good’ encompasses more than a corporate benefit. Among other considerations, note that tax writeoffs that may be very important in the corporate environment may not exist in the public-sector agency. 6.2 Research

and development

organizations

Research facilities are characterized by relatively high personnel costs and an indirect relationship between automation expenditures and improved research. Modestly higher salaries for research personnel are usually accompanied by significantly higher indirect costs, so personnel tend to be more limited than in other environments. As a result, the tangi-

?? Tutorial

ble justification for LIMS in this environment is frequently founded in saving time or reducing the need for personnel. The development laboratory may find tangible benefits (and costs) through improvements offered to manufacturing facilities. Facilities that are service-oriented (performing studies based upon requests from manufacturing sites or customers) will find the traditional LIMS benefits to be applicable. Less tangible benefits include providing an environment where scientists spend more time on scientific issues, emotional or marketing issues such as company prestige and the ability to attract desired personnel with state-of-the-art facilities, and fulfillment of corporate research philosophy. Some LIMS features assume greater importance in the research and development environment. LIMS capabilities related to personnel overhead, such as project definitions and scheduling flexibility, have a greater impact on system benefits than in most other environments. LIMS that can enforce analytical protocols may allow work to be shifted from scientists to lower-level personnel. As assistance in the interpretation and analysis of data, are also more important than in other environments. Improved data retrieval provided by a LIMS may or may not produce a quantifiable benefit, depending upon the type of research being performed and the capabilities of the LIMS. LIMS are currently primitive in their integration of electronic notebook concepts.

6.3 Regulated industries

27

6.4 Manufacturing

The goals of LIMS in the manufacturing QA/QC environment are quite different from the more widely discussed research or analytical laboratory [4]. The benefits of the LIMS in the manufacturing environment lie in improvements of process quality and speed of identifying errors; improving the efficiency of the laboratory itself is frequently a secondary benefit. Other areas that are emphasized in this environment compared with other industries include the cost of exchanging timely information with other systems (ordering, shipping, quality, corporate, process) and the cost of handling product changes or other manufacturing line changes. 6.5 Defense-sector

organizations

Defense-sector organizations have unique specification, procurement and support requirements that can dramatically affect the cost of a system, regardless of the use of the resulting system. Special care must be taken to incorporate these expenses into an analysis.

7 APPLYING SULTS

LIMS

COST-BENEFIT

ANALYSIS

RE-

This section will discuss how to combine the cost and benefit factors and make use of the results. 7.1 Economics,

In regulated industries, LIMS are seen as a means of coping with regulations, especially reporting and tracking of information. In this environment, it is important that a clear picture is drawn of how the company meets these obligations, as it is frequently desirable to evaluate benefits not only on the basis of meeting existing regulations, but also on the potential to meet proposed regulations and cope with future changing regulations. The cost of responding to regulatory audits may be included. Unpredictable benefits are frequently important in regulated industries.

organizations

accounting,

and

other

unpleasant

subjects

Before we discuss how to draw conclusions from an analysis, a brief discussion of economics is necessary. 7.1.1 Cash jlow analysis Although there are many types of economic analyses, most laboratory automation systems are analyzed on a cash flow basis. Cash flow economics have the benefit of being accessible without a lifetime of study, but have the disadvantage that today’s money is different from tomorrow’s mon-

28

ey. Unless all of the expenses and all of the benefits arrive at the same moment in time, it is necessary to compensate for this change. The greatest difficulty in performing this analysis is not performing the calculations or deciding what figures to use, but is the rapid rate at which the technology is changing; e.g. state-of-the-art computers just a few years ago are doorstops today. Initial LIMS costs are typically 50% hardware, and the costs of software tend to track the costs of hardware fairly closely. Also, the amount of most ongoing costs and benefits tend to drift over several years. As a result, the costs side of these calculations is enormously difficult to predict more than five years into the future, and the benefits side becomes untrustworthy without a very detailed analysis. Focus initially on one-year and three-year analyses (starting from the time the system is put into use). These will give a good idea what the initial capital risk is and where the project is headed. A venture that is being justified solely on the basis of cost savings that takes more than three years to break even should only be considered if the project is being supported strongly at the highest management levels. Projects that are worthy of longer paybacks should also be worthy of a more extensive cost-benefit analysis! 7.1.2 Cost of money and scheduling Since today’s and tomorrow’s money cannot be compared directly, it is necessary to adjust costs and benefits to a specific point in time. This is done by adjusting costs or benefits by a factor that accounts for the appreciating value of money, called the cost of money. The easiest ways to compensate for the changing value of money are either to add the cost of money to each year’s total and accumulate the totals to find the net cost at the end of the period (future value approach), or to adjust all costs and benefits to their equivalent in today’s money (present value approach). The cost of money is expressed as a discount rate. Most corporate accountants can give a sufficiently accurate value for this number which may be the cost to the organization of borrowing money.

Laboratory

Information

Management

W

To apply the cost of money, it is necessary to schedule the times at which one expects to receive benefits or incur expenses. Pick a reasonable starting day, such as the installation date or the date of the first non-planning expenditure, to be ‘day 0’, and work out a schedule. #Using a spreadsheet, calculate the future value of every expense. Sufficient accuracy can often be obtained by treating expenses the first 6 months as occurring at day 1, months 6-18 as one year, and so on. Annually compounded interest calculations will usually be sufficiently accurate, as will the assumption that benefits can be immediately invested and costs immediately subtracted. 7.1.3 Net return on investment Net return on investment (NROI) is a commonly used term. In its simplest form, it is the interest rate needed in order to accumulate the same money benefits as the project, including adjusting for the cost of money. When the money comes in and goes out at regular time periods (or sufficiently often relative to the length of the project) this is simple. However, this is rarely the case. There are many technical variations on this calculation, so find out how it is normally done at your company and use that technique. Also be aware that NROI figures are sometimes very sensitive to the selection of the period of time over which they are calculated, and do not represent the project well if large aperiodic expenses or revenues are incurred. 7.2 Valid measurements

for project approval

What numbers do we look at to determine if a LIMS is cost-effective? Any cost analysis textbook will point out that the concept of ‘maximum benefit for minimum cost’ is worthless. Minimum costs are obtained by spending nothing, while maximum benefits are usually obtained by spending far too much. In some situations, it is desirable to calculate maximum benefit for a given cost. This is useful for a project that was funded prior to the costbenefit analysis! This has the problem of ignoring huge benefits obtained at a small additional cost.

??

Tutorial

Some people try to determine the ‘maximum benefit for minimum cost’ by analyzing the ratio of benefits to costs. This technique is overly sensitive to small changes in costs, but is appropriate when comparing alternatives with widely varying benefits [5]. The most common measurements are maximum net benefit (the present value of benefits less the present value of costs) and internal rate of return. Internal rate of return is most appropriate [5] where payoff comes many years after investment, an area that is very difficult to analyze in a LIMS environment. Maximum net benefit is, therefore, the recommended approach and has the benefit of being easy to calculate and apply. If there is a need to look at the absolute cost or benefit figures, be exceedingly careful. It is very easy to treat costs as benefits, and vice versa. For example, if it is decided a system can be maintained by a vendor less expensively than maintaining it in-house, is this a savings in manpower (a benefit) or a decrease in maintenance (a cost)? 7.3 Drawing conclusions from an analysis As mentioned earlier, this article addresses performing a cost-benefit analysis without counting pennies - the 5 to 20% error range (depending upon the industry). With this attitude, the results of a cost-benefit analysis will fall into three rather clear areas: 1. the LIMS is clearly justified, 2. the LIMS is clearly a cost disaster, or 3. the decision cannot be made on the basis of cost. This is important information. The biggest business catastrophes occur in cases that are expected to be profitable but turn out to be marginal, not those where the profit is less than expected. If managers cannot provide a rule of thumb, assume that a justifiable system must have a profit margin of 20% or a net return on investment (NROI) of no more than 4 years. The cost-benefit analysis can do much more than simply present a bottom-line go vs. no go figure. Some additional common questions that can be asked and assigned costs and benefits are discussed below. Common errors made when com-

29

pleting a cost-benefit the Appendix.

analysis

are enumerated

in

7.3.1 What size system should be implemented? The cost-benefit analysis frequently needs to consider alternative system sizes. Should a single centralized or distributed system serve the corporation? Are departmental systems justified? Should single-laboratory systems be linked? Each of these will have a different, sometimes surprising, distribution of costs and benefits. Larger systems tend to have lower costs per head, but larger initial costs. However, this depends critically on the number of potential users: a system that is least expensive when there are 500 users may be the most expensive if there are only 20 potential users of the system. Three general principles must be remembered when systems of different sizes are compared. Present value calculations are imperative. Eyeballing the figures may be very misleading. Consider all available methods of financing. A system that is overly costly when purchased may be justifiable when leased. Compare only systems of the same net capacity. For the first approximation, it is accurate to compare a large system against two smaller systems of half the capacity, but not a large system directly against a smaller system. If forced to compare systems of different sizes, the ratio of benefits to costs is a more appropriate measure of the project than net benefits [1,2]. 7.3.2 In-house, custom, or off-the shelf system? Many laboratory automation cost-benefit analyses seek to decide whether to develop a system in-house, use custom development, purchase offthe-shelf packages or a combination of these. Furthermore, many laboratory automation systems that begin with commercial packages require significant customization for effective use in a particular environment. This poses a potential dilemma: accurately estimating the cost of development projects requires a careful specification of the work to be performed, but the point of the study may be to decide whether it is worth the expense of producing such a specification!

Laboratory

30

The analysis of the costs of custom software development are outside the scope of this article, and can be found elsewhere (cf. ref. 6). Estimation of software development costs should only be performed by a person that is experienced at this task, has a track record of success, and preferably does this as a full-time job. If your organization does not have personnel with the aforementioned qualifications, or if they are considering only a partially customized system, the choices are limited to (i) hiring a consultant or finding a person within the organization that can make this assessment, or (ii) obtaining firm, fixed-price quotations for the work to be analyzed. However, be forewarned that commercial vendors will be reluctant to spend the large amount of time required to provide precise pricing figures if one is still trying to decide whether to do the work in-house. There are several approaches to the dilemma of how to determine whether it is worth developing a detailed specification for a custom project. A surprisingly common solution - pretending that the cost of custom software does not exist - is obviously foolish. A more reasonable approach is to obtain ball park estimates for the software, then treat them as firm numbers. This is as successful as the estimates are accurate, and may be a useful approach if the custom development does not comprise a large fraction of the total project. For example, a vendor providing an estimate for software development closely related to their system should be able to estimate the cost of the system with a 30% accuracy. If the custom software is only 20% of the total system, this error only biases the final results by 6%. Another approach is to separate the custom development from the basic LIMS installation, and perform the custom work as a separate phase or a separate project. This is not unlike ignoring the costs altogether, but is effective when the custom work can be eliminated if necessary or is not needed for the basic use of the LIMS. This strategy is commonly seen in political situations where adding new systems is difficult but obtaining support for installed systems is easy. Finally, some companies develop the specifica-

Information

Management

??

tions for the custom work, and solicit non-binding bids. Some vendors will go halfway in this area, using partial specifications to create a less accurate estimate.

8 TIME COURSE

OF LIMS PRODUCTIVITY

Changes in productivity following the installation of a LIMS typically follow a course like that shown in Fig. 1. The following productivity phases may be observed: 8.1 Initial drop in productivity Immediately following installation, productivity may drop because the new system impacts many aspects of laboratory operations. This drop commonly ranges from 10 to 50% of pre-LIMS productivity. The drop in productivity begins when training is started, which is usually just before the LIMS is installed. The extent of the drop can be minimized by thorough pre-installation training and careful con-

100

I

Time

Fig. 1. Expected LIMS benefits vs. time. The figure shows the percentage of the total expected benefits achieved as a function of time in a typical LIMS installation. The time course over which each fall, rise or plateau in achieving LIM benefits varies greatly among installations. Refer to the text for a more complete description of this figure.

?? Tutorial

31

sideration of potential bottlenecks, rarely be avoided altogether. 8.2 Return to pre-LIMS

although

it can

productivity

There is usually substantial motivation to return quickly to the efficiency the laboratory achieved prior to the LIMS installation. As the drop in productivity may directly impact cash flow in the organization, it is important for the cash flow analysis to estimate how deep the drop will be and how long it will last. This should be estimated from past experience with organizational changes. Periods ranging from a few days to six months are typical, depending upon the complexity of the system, the extent of training, and how well the transition is organized. The time to return to the pre-LIMS productivity can frequently be shortened by as much as 50% by concentrating carefully on achieving the specific benefit factors that are expected early in the implementation process. Sometimes it is desirable to plan a longer period of modestly reduced productivity to prevent a large drop in productivity that cannot be tolerated. However, avoid phasing in the LIMS too slowly, especially if the LIMS enforces a stepwise approach to sample and test result handling. In such cases, a slow implementation can prevent employees from using the LIMS because other employees have not completed prerequisite tasks. This can lead to a vicious cycle of disillusionment with the LIMS and reduced use of the system. 8.3 Plateau at pre-LIMS

productivity

In some installations, the motivation to improve productivity above the initial level is lacking, resulting in a plateau close to the productivity level existing before the LIMS installation. This is particularly true when much of the benefit of the system is in increased sample throughput, or changes in procedures are required to take advantage of the productivity improvements offered by the LIMS. This plateau is rarely a necessary limitation, and can usually be overcome with proper management if it is noticed in time.

8.4 Rapid productivity

improvement

The initial rise in productivity after the system is in full use is frequently achieved quickly, typically over a three to six month period, although the period varies tremendously from industry to industry. 8.5 Plateau followed

by slower improvements

Most systems plateau at 50 to 75% of their expected productivity gain, after the largest benefit factors are achieved. At this point, laboratory management must switch their emphasis from seeking the major benefits to refining their system and pursuing smaller benefits. These improvements necessarily occur more slowly than the initial gains. When the plateau is reached, the laboratory reporting functions of the LIMS itself may assist the manager in identifying areas in which further improvements may be made. It generally takes an additional 6 to 24 months to realize the full benefit of the system. To summarize with an example, if a forecast of $lOOOOO/year gain in productivity is made, expect no gain the first half year (due to an initial decline followed by offsetting improvements), a gain of about $33000 the next half year (two thirds of the expected $66000/year rate), and a slow increase to the full $lOOOOO/year over the next year or two.

9 SUMMARY

OF COST-BENEFIT

ANALYSIS

STEPS

The techniques described in this and the preceding sections may be summarized in the following steps that can be followed to perform a costbenefit analysis: 1. Determine the budget for the analysis. 2. Given the budget, establish a reasonable accuracy goal. 3. Create a list of cost and benefit factors, consulting the tables in this article. 4. Circulate the list of cost and benefit factors, looking for forgotten expenses and benefits. 5. Prioritize the factors (as far as possible) in order of net benefit.

Laboratory

32

6. Note whether factors are tangible, intangible or unpredictable, and whether tangible factors are initial or ongoing. 7. Assign monetary values to tangible costs and benefits that are not based on time-motion studies. 8. Study the laboratory and assign monetary values to costs and benefits that are based on time-motion studies. Be careful to look for potential automation bottlenecks as well as benefits. 9. Assign values to intangible and predictable factors using the extrema or breakeven methods (where you are able). 10. Check for common mistakes, using the tables and examples in this article. 11. Assign dates to when each cost and benefit is incurred. 12. Use the schedule and cost of money to correct the cost and benefit figures to a specific date. the sensitivity of your analysis to 13. Consider your assumptions, predictions of the company’s future, and the values of intangibles. 14. Summarize the results over 1 and 3 years by combining the appropriate initial and ongoing costs and benefits. If warranted, use longer analysis times. Describe all intangible or unpredictable benefits that are not translated into money.

10 CONCLUSIONS

Does the cost-benefit analysis work? For the type of analysis discussed in this article, which attempts to balance the cost of performing the analysis against the accuracy of the resulting figures, it may be concluded that this type of costbenefit analysis does work, and is an effective management tool. Just how well does it work? The overall accuracy achieved by consultants from our company ranges from 80 to 95% (depending upon the industry) for projects ranging from $50000 to $2 000 000. Manufacturing industries, where benefits are more precisely defined, tend to have the most accurate analyses.

Information

Management

??

Also, more accurate analyses are obtained when the analyst is working under a manager that controls the strategic direction of the organization. This leads to better estimates of the scope and application of the system, especially the personnel that will be affected by the system. When the analysis is conducted at a lower level, misperceptions in the scope of the system tend to introduce errors. This level of accuracy will certainly avoid the disastrous 200% and larger overruns that plague many large software projects in general, and large LIMS implementations in particular. However, a 5% accuracy is likely to be achieved only by an experienced analyst with a thorough knowledge of the laboratory environment, an unbiased approach to the project, and free access to management. The net accuracy of 80-95% cited above must be balanced against the fact that most analyses tend to have compensating errors. The 95% accuracy does not apply line-by-line. This is expected in a survey analysis such as this: the key is to not miss any important cost or benefit factors. Also, the organizations on which these figures are based are those that hire consultants for this type of work. It is possible that organizations that hire consultants to perform independent studies have better internal cost controls than organizations that do not expose themselves to independent evaluations. In conclusion, the LIMS cost-benefit analysis is an effective tool that can be used as an accurate aid to managerial planning.

11 ACKNOWLEDGEMENTS The author would like to acknowledge the valuable suggestions made by Mr. Stephen A. Allen, Dr. Melody S. Kasprzak, Ms. Elizabeth E.H. Schreppler and Mr. James R. Bowen.

APPENDIX

Common errors in laboratory

cost-benefit

analyses

This appendix describes some common errors the author has observed in cost-benefit analyses.

?? Tutorial

33

Some of these pertain to any automation system, while others are specific to LIMS. The errors may be divided into the following categories: 1. errors in expectations; 2. errors associated with initial LIMS costs; 3. errors associated with ongoing LIMS costs; 4. errors in analysis. A.1 Common

errors in expectations

1 Expecting a paperless office. A common mistake in any form of laboratory automation is to write off all paper costs in the expectation of a paperless office. In fact, an automated system will eliminate the exchange of paper. However, it is our repeated experience that some individuals prefer printed records and, given the ability to easily print well-organized reports, enormously increase the amount of paper they use personally. Typically the total expenditures on paper, ribbons, and so on increase about 30% with an automated system. 2 Expecting immediate increase in productivity. LIMS-induced productivity increases are not gained in full immediately. Productivity undergoes several distinct phases, as shown in Fig. 1 and discussed above. It is 3 Expecting lowered base maintenance costs. frequently expected that the LIMS’ ability to track instrumentation service and calibration will reduce maintenance costs. This is an area of misunderstanding. Such tracking can predict failure, avoid having equipment entirely out of commission, avoid the use of faulty equipment, and eliminate major repairs. A significant net savings in maintenance costs may result. However, basic routine maintenance must still be performed and routine maintenance costs are not eliminated. Furthermore, the automated instruments that are desirable for use with a LIMS may be significantly more expensive and more complex than manual instrumentation. 4 Expecting turnkey products. Automation systems are rarely desirable as turnkey products. This is not to say they are not available as such - if

one is lucky enough to be in one of the industries that has been heavily pursued by the LIMS vendors. However, in each organization there appears to be a certain amount of customization (not configuration) that is worth far more than its cost in terms of improved productivity or compatibility with existing systems. It is not unrealistic to look for a system that will not be customized - it is often simply not in your best interest to do so. Reliability is5 Reliability of automated systems. sues are often neglected. The impact of failures in automated data acquisition should be carefully calculated into both the cost analysis and the design of all automated systems. Can the laboratory tolerate a failure of the interfaces to the instruments? What are the alternatives? A.2 Common cost factors

errors associated with initial LIMS

Make sure that cables can run where 1 Cabling. they are required, and there are no structural obstacles in the building. An unnoticed fire wall can ruin an estimate! If terminal or other communication servers are used, make sure that power will be available where they are installed. 2 Telephone lines. If modem communications (e.g. telephone) are to be used, make sure they are compatible with the organization’s telephone system. Some electronic phone systems will not support standard modems. 3 Instrument interfaces. Consider the lengths of wires that must be run to all instruments. There may be a need for special equipment to establish reliable links to these instruments communicating over a long distance. Instrument interfaces using analog signals should usually be as close as possible to the location of the measurement. It will be nec4 Terminal and printer placement. essary for you to determine where terminals and printers must be placed, and furniture may have to be purchased for these devices.

Laboratory

34

5 Graphics devices and compatibility. The difference in price between display devices with different graphics capability is large and should be considered carefully. Make sure compatibility is considered in addition to cost. 6 In-house maintenance. If the organization maintains any of the system in-house, be sure to include appropriate engineering documentation. This can be quite costly and difficult to obtain if not ordered with the system. The initial purchase of spare parts should be included. 7 Sample identify printers, included. tions if processed.

identification equipment. Equipment to samples such as bar code readers and label applicators, and so on should be Do not scrimp on ergonomic consideraa large number of samples are to be

8 Database and LIMS tools. This is a particularly dangerous cost area at the current time, as many LIMS require a large purchase of database system and database interface tools. In the effort to standardize on database tools, many vendors use popular third-party databases, making these purchases more complicated. Make sure that one has all of the software components that are required for using, maintaining, and modifying the LIMS. Do not forget to 9 Instrument interface software. include the costs of instrument interfaces as this is frequently more expensive than expected. Include both generic interfaces as well as special interfaces for equipment such as chromatography or process control systems. This IO Graphics, analysis and printing software. area is often insufficiently analyzed. Be sure to consider the usability and flexibility of the software in addition to its bottom-line cost. I I Development tools. This will include database interfaces, communication interfaces, compilers and LIMS options. Development tools add significant additional costs to a system and are easy to miss.

Information

Management

??

12 Documentation: users, managers, and system development. Make sure there is adequate documentation for each category of personnel using the system. Do not underestimate the costs of purchasing dozens or hundreds of copies of documentation. If the documentation set is large, fumiture may also be required. Although most purchasers 13 Configuration time. and vendors pay appreciable attention to the necessary system configuration, the cost of sorting out the configuration needs is often overlooked. In some cases, this can be substantial. Carefully examine which 14 Changes in personnel. personnel are performing each procedure, before and after the automation. A classic mistake, frequently repeated in the LIMS literature, is to note the important LIMS benefit of reducing time (e.g. by automating calculations) without balancing this against differences in the pay scales of the persons that enter the data. A.3 Common LIMS costs

errors

associated

with

ongoing

Do not as1 Efficiency of automated procedures. sume that computerized procedures are more efficient than manual procedures. Some manual procedures (e.g., checking off desired tests on a form) may be much faster than their computerized counterparts. The automation of calculations falls into this category. The benefit of automated calculations often depends critically upon how much data are transferred and how completely a procedure is automated. The classic problem in this regard is when readings from an automated instrument must be corrected by data that are entered manually. For example, the time savings from an instrument that automatically transfers test results to a LIMS appear to be obvious. However, suppose that those results must be normalized to the weight or volume of the sample. If the entry of the weight or volume value is also automated, the process will be highly efficient. However, if the calculation cannot proceed until the weight is manually entered, the

?? Tutorial

benefit will be much smaller or possibly even worse than the manual entry of both values. The use of bar code labels also falls into this category. Make sure that improvements in the use of bar codes are not obliterated by the time required to print the labels. Generally, if labels are not scanned several times or used in automated equipment, their productivity must be carefully examined. 2 Costs of quality control. Do not forget that the quality improvements arising from improved statistical analysis of data may be associated with increased managerial time to perform and monitor the analysis. The laboratory is generally 3 Timing constraints. very sensitive to changes in timing of procedures, as discussed above. The changes due to the LIMS should be examined carefully. While examining procedures, look for other common procedural pitfalls, as many manual procedures in analytical chemistry rely on technical staff to make intelligent decisions concerning the samples they are analyzing. This may be as simple as diluting a concentrated sample, or as complex as interpreting the validity of a complex NMR spectrum. Frequent mistakes in estimating LIMS savings are: _ forgetting that an analyst still needs time to perform decision-making steps in a procedure, ~ underestimating the difficulty of forcing decision-making flexibility into a LIMS that does not provide for them, and - failing to account for new activities required by the automated system. A.4 Common errors in analysis [I] 1 Double counting. The classic mistake in costbenefit analysis is to include the same cost or benefit in two ways [1,2]. This is a particular hazard when the cost analyst is trying hard to include all possible benefits, particularly when considering secondary effects [2]. For example, if you transfer a person from the laboratory to serve as a LIMS manager and count the cost of that person as part of the cost of the project, do not

35

also count the reduction in efficiency of that other department due to the loss of that person. 2 Irresponsible analysts. There are usually strong incentives for an analyst to tell his client or management what they want to hear. It is easier to manipulate an analysis to a desired conclusion than to be completely unbiased, especially in an inter-disciplinary area such as this. This problem applies both to consultants and in-house staff. Selection of an analyst should carefully consider this problem. Any person with something to gain from the success or failure of the system should not be allowed to perform the analysis. Use a consultant that does not have a commitment to the project, or accountants without a personal interest in the project. 3 Omitting significant costs. This is simply overlooking costs. The most common costs to overlook are overhead - electricity, space, company services - and the downtime while performing routine maintenance and upgrades of both computer and laboratory equipment. 4 Hidden costs. These are costs that are incurred elsewhere in an organization. A classic example is training for other groups that will use your system. More difficult is assigning overhead. For example, if using existing computer facilities, fair share of the costs of computer power, storage, etc. must be allowed. 5 Spillovers. A new major system may alter salary, promotion or hiring patterns. These are usually not considered, but they may be appreciable. 6 Past events. Cost-benefit analyses should only consider the future; the past should be constant for all alternatives. For example, never add in the cost or benefit of an existing system. 7 Pecuniary benefits. Benefits distantly related to the system should not usually be included in the analysis. For example, a LIMS purchase may cause a vendor to open a branch office in your area,

Laboratory

36

stimulating the local economy. Don’t get carried away in adding this to the benefits of the system. 8 Non-market costs. Many costs are set by competitive market values, such as the cost of a computer. Some are not, and one must be careful in converting these into money. For example, suppose a backup power for the computer system is installed. There may be a temptation to use the difference in cost of recovering from power failures before and after the backup power system was installed. However, if the costs of the incidents were insured, a more appropriate measure would be the change in the insurance rates. Make sure to com9 Varying analysis periods. pare like with like. A common simple mistake is to compare unit costs over different periods of time. For example, this could mean comparing monthly costs to 28-day costs. Similarly, if undertaking a cash-flow analysis, be sure that any alternatives operate over the same periods of time. The cost of a two-year project must not be compared to the cost of a 2.5 year project. In this case, the proper treatment is to take the cost of the 2.5 year project and subtract its remaining worth at the end of 2 years. IO Changes in pay scale. Make sure to compare the pay scale of the person performing each task before and after the task is automated. Frequently, automated systems reduce the time to perform tasks, but shift the responsibility to more highly paid personnel.

Information

Management

fl

I1 Inadequate decision-making. When a procedure is automated, it is important to check the extent to which the automated procedure provides the flexibility of the manual system. Never forget the training and intelligence of the analyst. Any time an analyst makes a decision, it is likely that it will be necessary to enter this information into the automated system. Do not forget to account for this time and do not underestimate the difficulty of forcing information into an automated system that does not readily accept it. For example, if an analyst dilutes turbid samples, but a LIMS does not allow the dilution information to be entered easily, the time required to trick the system into accepting the information may be longer than the time gained by using the LIMS.

REFERENCES

J.L. Ring and E.L. Schrems, Cost-benefit analysis in information systems development and operation, Computing Suruqvs, 10 (1975) 19-34. P.G. Sassone and W.A. Schaffer, Cost- Benefit Analysis, A Handbook, Academic Press, New York, 1978. H.G. Thuesen, W.J. Fabrycky and G.J. Thuesen, Engineering Economy, Prentice-Hall, EngIewood Cliffs, NJ, 5th ed., 1977. R.R. Stein, Improving efficiency and quality by coupling quality assurance/quality control testing and process control systems with a laboratory information management system, Process Control and Quality, 1 (1990) 3-14. E.J. Mishan, Economics for Social Decisions, Praeger Publishers, New York, 1973. B.W. Boehm, Software Engineering Economics, Prentice-Hall, Englewood Cliffs, NJ, 1981.