Utilization of laboratory resources: developments in knowledge-based ordering systems

mternathndJournalof

IMedical International

ELSEVIER

Utilization

Journal of Bio-Medical

Computing 40 (1995) 17-30

of laboratory resources: developments in knowledge-based ordering systems

L. Boon-Fallem-““, E. Sokal”, P.G. Nightingaleb, M. Petersb, J.B. Otte”: J.M. Ketelslegers”, E. De Wilde”, A. Hasmanc aCliniques b Wolfson “University

Universitaires St. Luc, Brussels, Belgium Computer Laboratory, Birmingham, UK of Limburg, Maastricht, The Netherlands

Received 27 March 1995; accepted 2 May 1995

Abstract This paper describes two rule based decision support systems. The first system is used to screen incoming test requests for adequacy on the basis of signs and symptoms volunteered by the requesting GPs. The system was tested using a database of 794 requests for a TSH test. About 17% of the test requests were correctly identified as unnecessary. In total, 0.5% of the tests were incorrectly labelled as unnecessary. This concerned 4% of the patients that appeared to have hyperthyroidism and 23% of the patients that appeared to have hypothyroidism on the basis of TSH and FT4 results. The other system is a rule-based clinical decision support system for the requesting of laboratory investigations, originally designed for use at a hospital within the UK, that was implemented in a predominantly French-speaking hospital in Belgium. This involved the modification of the system to allow multilingual operation, and also the implementation of a completely new set of investigation protocols. The purpose of this study was to assess the transferability, both of the system itself, and of its benefits. The system was introduced gradually and has only recently been in full operation. However, the findings from the first months of routine use of the system indicate that the transfer of the system to a different clinical environment has been successful. Although it is too early to assess fully the impact on laboratory utilization, the clinicians believe that it is improving the appropriateness of investigations. Keywords:

haviour;

Clinical Feedback

* Corresponding

protocols;

Knowledge

bases (computer);

Liver

transplantation;

author.

0020-7101/95/$09.50 0 1995 Elsevier Science Ireland Ltd. All rights reserved SSDZOO20-7101(95)01118-X

TSH

test; Test requesting

be-

18

L. Boon-Falleur

et al. / International

Journal

1. Introduction The potential of knowledge-based computer systems to influence physician use of laboratories is being examined within the Advanced Informatits in Medicine programme of the European Commission of the European Union as part of the OpenLabs project (A2028) [l]. This is a 3-year project which started in January 1992 and is concerned with the application of advanced informatics and telematics in the optimization of clinical laboratory services. OpenLabs includes activities relating to pre-analytical, analytical and post-analytical stages of the laboratory testing process. Most activities involve knowledge-based systems and/or the telematic exchange of information, and the project seeks to define an open architecture and communication standard within which the numerous established and emerging functionalities of a laboratory information system may be built from different components, from different suppliers, into a unified functional system. The pre-analytical activity reflects a growing international concern that rising costs of laboratory medicine are due, in part, to an increase in unnecessary testing, and that by improving the appropriateness of tests selected by both general practitioners and hospital doctors, this trend can be reversed. 2. Protocol

management

Clinical protocols (guidelines) defining panels of biological parameters to be assayed and monitored are, theoretically, important tools for the clinician in the daily process of managing diagnostic and investigational procedures. Protocols defining the management of patients in a variety of conditions are based on the clinical and scientific experience of specialized senior medical and laboratory staff, and their routine implementation is critical for the appropriate, efficient and consistent care of patients in a variety of circumstances. However, protocols, particularly in less specialized areas of medicine, are frequently informal and their implementation by the junior, less experienced, medical staff, in whose hands the routine

of Biomedical

Computing

40 (1995)

17-30

management of patients largely resides, is impaired by problems of communication and divergence in interpretation. In specialist areas such as liver and renal medicine, protocols are often more formalised, but their consistent implementation by junior medical staff is again impaired this time by the sheer impossibility of computing appropriate patient-specific testing schedules reflecting complex pathology and differing severities of illness. The net result of these practical difficulties in implementing investigation protocols is, most frequently, the ‘defensive’ over-requesting of more common ‘monitoring’ tests and the under-requesting of the less common, more esoteric tests, often required to confirm diagnosis. Both these practices can lead to the inappropriate use of resources, either by unnecessary testing, or by prolonging hospitalisation. A similar situation can be observed in primary care. Here also, GPs may order more tests than is necessary. It was shown that providing feedback to GPs with respect to their investigation ordering behaviour reduced the number of tests requested [2]. In this situation, the feedback was based on guidelines and work agreements that had been proposed by representatives of the GPs and specialists. The feedback was given by a specialist who screened the requests. Given the theoretical importance of protocols and their obvious potential to improve laboratory utilization, it is important to investigate ways of supporting their practical implementation in routine patient management. Knowledge-based systems expressing locally agreed guidelines in the form of IF-THEN-ELSE rules provide a possible approach, and their use to reinforce guidelines automatically in specialist units is now being evaluated in a number of centres. In this contribution, the results of the OpenLabs-related study are reported. The first part of the study concerns the translation of guidelines and work agreements written in natural language into rules of a rule-based system. The question to be answered here was whether the ensuing automatic feedback system could replace or even improve the manual feedback system. The second part of the study reports the porting of a protocol management system that was implemented in one country to another country.

L. Boon-Falleur

3. Feedback

et al. 1 International

Journal

to GPs

The guidelines and work agreements, as available and used in Maastricht, the Netherlands, concern a large number of different investigations. On the basis of these guidelines, the adequacy of investigations requested by GPs was half yearly assessed by an independent specialist. All requests for investigations are sent to the so-called DCC (Diagnostic Coordinating Centre) of the Academic Hospital, Maastricht, where they are: (a) posted in a local database; and (b) copied on hospital order forms. After the investigations are performed, the results are sent to the DCC. Here the results are added to the database and then sent to the GPs. After half a year, all requests ordered by all GPs during a randomly selected month are reviewed. All GPs receive feedback about the adequacy of the request investigations. In order to make assessment of the adequacy of the investigations possible, the GPs agreed to add medical data on the order form, including the reason for request. This way of providing feedback reduced the number of inadequately requested investigations considerably [2]. Since the manual system is time consuming, it was decided to design a computerized feedback system that would determine for each request whether the investigation was necessary.

3.1. The feedback system The feedback system was installed at the DCC. A test requesting system was installed at each GP site. Test requests together with additional medical data are sent electronically to the DCC. Here the data are stored in a database. The feedback system retrieves newly requested investigations from the database and analyses them. The core of the feedback system is a rule-based expert system. The system automatically screens the test requests coming from GPs. This expert system is built using the in-house developed shell SIMPLEXYS [3,4]. It uses a three-valued logic (the rule’s conclusion can be either true, false or unknown). The system is programmed .in PASCAL, and the rule base together with the supporting software is compiled. The system allows the use of different types of rules. Although in expert system’s jargon, a rule is commonly considered to be an ‘if-then’ directive

of Biomedical

Computing

40 (1995)

17-30

19

that is itself always a true proposition, in our case, when we speak of a rule, we refer to the rule’s conclusion. So the rule COMPLAINTS tells us whether the patient has any complaints. The value of the rule (or of the consequence of the rule in the expert system’s jargon) can be true, false or unknown. One type of rule that is most equivalent to ‘if-then’ rules is the so-called evaluation rule: COMPLAINTS: ‘Does the patient have any complaints?’ WEIGHTINCREASE OR WEIGHTLOSS OR.. . The above rule says: if the patient has increased in weight (WEIGHTINCREASE is true), or the patient has lost weight (WEIGHTLOSS is true) or... then the patient has complaints (COMPLAINTS becomes true). Before the rule is used, COMPLAINTS has the unknown value. We refer to the above rule as rule COMPLAINTS but COMPLAINTS is actually the consequence of the rule and WEIGHTLOSS and WEIGHTINCREASE antecedents. are the WEIGHTLOSS and WEIGHTINCREASE themselves can again be the consequents of other rules. ASK rules ask questions from the user, who can either answer with ‘y’, ‘n’ or ‘?’ (these answers are internally translated into true, false or unknown). An example of an ASK rule is the following: WEIGHTINCREASE: ‘weight increase’ ASK THEN FA: WEIGHTLOSS. In ASK rules, the value of the consequent is determined by the answer of the user. In this example also, another feature of SIMPLEXYS is presented. If according to the answer, WEIGHTINCREASE is assigned the value true, then at the same time the value of WEIGHTLOSS, that in principle can also be determined by another ASK rule, can directly be assigned the value false. This means that in this case the latter ASK rule will not be executed (the rule’s consequent has a definite value). Another type of rule are the TEST rules. These rules perform tests on available data: HYPERSCORE: ‘score hyperthyroidism’ TEST score: = 0; if t(PALPITATIONS) then score: = score + 2; if....

20

L. Boon-Falleur

et al. 1 International

Journal

.. if score > 15 then TEST: = TR else if (score 2 7) and (score < = 15) then TEST: = PO else if score < 7 then TEST: = FA ENDTEST In this case, the TEST rule HYPERSCORE tests, through the use of a scoring system, whether hyperthyroidism is present, possibly present, or absent. In this example, PALPITATIONS is another ASK rule that inquires about the presence of palpitations. If palpitations are present (the function t is then assignedthe value true), the score will be raised by 2 points. The rule HYPERSCORE is given the value true when the variable TEST is assignedthe value true, etc. In our system, TEST rules are also used instead of ASK rules for retrieving the relevant data from the database since the data are already available in a file. Another feature of SIMPLEXYS is its control structure. Since this expert system is used in signal processing environments, it has one more control structure than the usual expert systems. In SIMPLEXYS, the notion of run is used. A run is the process that derives all necessaryconclusions given a set of inputs. This notion is used in our application: in each run a separate test request is analyzed. SIMPLEXYS also provides constructs to handle context. The context (e.g. a certain test request) determines which rules will be evaluated. The work agreements and guidelines concerning thyroid disease, mononucleosis infectiosa and allergy tests, that are used in the manual feedback system, have been translated into rules used by the expert system. Although the work agreements and guidelines cannot always be directly translated into rules, our experience is that, with some additional information, it is possible to implement the work agreements in the form of rules. In some cases(as in the case of thyroid disease);scoring systemsare used to determine the adequacy of a test. 3.2. Results

The feedback system was tested using requests for TSH tests. The feedback system contained 40 rules, with which the adequacy of the test could be determined. In the case of TSH test, a scoring system was used. Each relevant sign or symptom is

of Biomedical

Computing

40 (1995)

17-30

awarded a score. Positive scoresare given to signs and symptoms that are indicative for hyperthyroidism, whereas negative scores are assigned to signs and symptoms that are indicative for hypothyroidism. The scores are based on values provided by Crooks and Wayne [5,6]. On the basis of the signs and symptoms, a total score is calculated. Since the GPs volunteer the medical data (they are not presented with a computerized questionnaire, via which, all relevant data are asked; a situation which also existed in the manual system), the amount of data varies from patient to patient. A test was considered unnecessary when the total score was equal to zero. A test set of 794 requests was used to determine the quality of the feedback system. Only requests were included for those patients suspectedof having thyroid disease.This set contained 754 euthyroid cases,13 hypothyroid casesand 27 hyperthyroid cases(the TSH result was taken as the golden standard). In total, 138 caseshad a zero total score. Of these cases,134 appeared to be euthyroid, three were hypothyroid and one hyperthyroid. On the basisof the scores,therefore, about 17% of the tests were correctly identified as unnecessary. The four incorrectly identified cases(0.5% of the tests, 4% of the hyperthyroid casesand 23% of the hypothyroid cases)contained, on examination, no evidence in favour of hypo- or hyperthyroidism. 4. Feedback in secondary care A protocol system for secondary care was evaluated in the Paediatric Liver Transplantation Unit of the Cliniques Universitaires Saint-Luc, Universite Catholique de Louvain (UCL), Brussels. The systemunder evaluation is a multilingual version of the Liver Unit Management Protocol System (LUMPS) developed by the Wolfson Computer Laboratory (WCL), University of Birmingham, UK in collaboration with the Liver Unit, Queen Elizabeth Hospital, Birmingham. LUMPS is written in MUMPS and runs on a 486 IBM compatible PC, which is linked to local laboratory information systems. LUMPS [7] automatically proposes, at terminals in ward areas, laboratory investigations appropriate to individual patients’ clinical condition and recent pathol-

L. Boon-Falleur

et al. / International

Journal

ogy. The doctors review proposed tests and add to, delete from, or simply accept the tests proposed for their patients as required. The system produces request documentation, provides a convenient way of reviewing test results and warns of potentially ‘dangerous’ results. Importantly, the system provides feedback of laboratory usage, facilitating refinement of guidelines. Test proposals are based on locally agreed investigation guidelines and relate to defined clinical classifications which may be symptom, disease or sub-disease-based. Laboratory data are acquired automatically from laboratory computer systems via the hospital IT infrastructure and test results care available .both to the system’s rulebase, and in the Liver Unit as soon as results are authorised by the laboratory. Investigation guidelines can be supported in either or both of the following ways: proactive support proposes appropriate investigation (i.e. at ward level, patient-specific investigation schedules, reflecting current pathology and clinical condition and based on locally agreed guidelines); reactive support denies inappropriate investigation by pointing out deviations or possible deviations from locally agreed practice. These techniques can be used together or separately, the net effect being that, only those tests which the clinician and the laboratory agree are necessary for the management of the patient, are requested routinely. Routine operation of LUMPS at the Queen Elizabeth Hospital, Birmingham has resulted in: a significant reduction in the number of Clinical Chemistry tests requested (audit data are not available for Haematology); a significant saving in medical staff time; improved appropriateness and continuity of management, notwithstanding frequent changes in junior medical staff [8]. These effects have been maintained for almost 3 years [91. 4.1. Implementing LUMPS at UCL Setting. The Cliniques Universitaires

Saint-Luc is the University Hospital of UCL, which is located in Brussels, Belgium. It is a 900-bed hospital with 48 Intensive Care Unit (ICU) beds plus several intensive hospitalisation units (e.g. neona-

of Biomedical

Computing

40 (1995)

17-30

21

tology; sterile unit; transplantation units - liver, kidney, bone marrow, cardiac, etc.). Approximately 30 000 inpatients (with an average length of stay of 9 days) and 270 000 outpatients are treated at the hospital per year. The hospital’s Department of Clinical Biology provides laboratory services in: Clinical Chemistry, including Endocrinology; Haematology including Coagulation and’ Immuno-Haematology; Microbiology including Bacteriology, Serology and Virology. Most of the laboratories are connected to the Laboratory Information System (LIS), the main exceptions being Virology and Immuno-Haematology. In total, the LIS-connected laboratories receive an average of around 2500 requests per day, involving some 13 500 individual tests. The LIS is entirely an in-house development written in PLI1, with the exception of the analyzer interfaces, for which the PGP System is used. Responsibility for the system lies with the Laboratory Information Team, a division of the Pathology Department, staffed by three programmer analysts, one Department of Clinical Chemistry liaison officer and four operators. There is a uni-directional interface between the LIS and the Hospital Information System (HIS), which is also an in-house development: the LIS sends laboratory results to the HIS in a standard ASCII file format at 2-min intervals, at which point the results can be viewed by users of the HIS (and in particular by the clinicians in the Paediatric Liver Unit). Removal of language restrictions. Evaluation within the EEC of this approach to managing demand for laboratory resource requires the introduction of a suitable system into routine service within a European hospital. A first requirement therefore was to modify the system to remove English language restrictions. All screen displays, user prompts, etc., must be in the natural language of the host country and shorthand keyboard entries must also reflect that language. Where several languages are in common use within a hospital, the language used at any terminal should be the user’s preferred language. All language-dependent text, including literal strings (fixed text within static or dynamic pseudo

22

L. Boon-Falleur

et al. 1 International

Journal

‘window’ displays), code lists, data entry validation and hypertext help, has been isolated therefore within the original LUMPS framework and is now held in indexed data files which have, as a primary key, a unique code indicating the language of the record content. Programs which use text now reference the appropriate language file using the language flag associated with the user code. Given sufficient processor power, the system can now support simultaneously any number of languages, the only constraints being: (i) each language must use left to right sequencing of discrete alphanumeric characters; (ii) overall text layout of screens is based on the English language version, with limited allowance for differing lengths of prompts; (iii) a computer literate person with reasonable knowledge of the system is needed to set up and/or modify a new language version; and (iv) where more than one language is used, one must be designated as the primary language for codes and expansions. The functionality of the multilingual system is identical to the original, and implementation in a new language can be accomplished without further programming. Modification of the system required some 5 man-months effort. Changes within the program code involved some 387 separate modules, which together contain some 21 000 lines of code. Much of this work was automated by creating routines which scanned the source programs, identifying language-dependent text strings, and performing operations necessary for language independence. Unfortunately, literal strings occur frequently in situations where text is used for some internal operation rather than for communication with the user (e.g. as data field delimiters, global field keys, character translation masks, etc.), and therefore some 2500 instances proved equivocal and required human intervention. De$nition of local guidelines. Guidelines to be expressed as IF-THEN-ELSE rules were developed by senior clinical and laboratory staff at UCL, based on their Transplant Unit’s existing written protocols. The 31 guidelines included static elements (fixed tests on fixed days for spe-

of Biomedical

Computing

40 (1995)

17-30

cified clinical condition) and, for the LIS-connetted laboratories, dynamic elements (tests reflect current pathology as well as clinical condition). These guidelines were then encapsulated within the rulebase by WCL (as 206 rules). Communications. LUMPS sits on the UCL laboratory Ethernet and was originally connected to the Paediatric Liver Unit via the LIS and the hospital token ring network, to which the laboratory network is bridged. This connection was something of a challenge and although a reliable working system was achieved, complicated logging-in procedures and substantial time delays were unpopular and a dedicated telephone line has been substituted. System tailoring. This included the translation of LIS definitions into the LUMPS system, involving the definition of some 1350 tests and 70 test ‘batteries’. The bulk of this was effected by automatic programmed transfer between the two systems, although some refinement of resulting definitions was needed. Locally required screen formats and request documentation were also defined. Some 38 different request forms for all laboratories connected to the LIS and 59 different tube label formats were defined to mimic existing UCL forms. For unconnected laboratories, request forms have been replaced by a simple list of requested investigations, from which multi-part request documents (required for billing purposes) are reconstituted by laboratory staff. Laboratory data transfer. Data are transmitted to LUMPS from the LIS in an analogous way to that used to transmit LIS data to the HIS. Translation. Time constraints restricted the translation into French to clinical user screens. System maintenance screens, accessed only by the system manager, have not yet been translated. Training. LUMPS is managed by the Informatits Group of the Department of Clinical Biology. Transfer of expertise at a systems, operations and application level was necessary to sustain routine operation; however, the high level of in-house expertise available resulted in the rapid completion of a fairly detailed educational/training programme. Members of the group quickly became fully conversant with LUMPS, and can effect guideline and application tailoring/set-up changes

L. Boon-Falleur

et al. 1 International

Journal

as well as supporting routine operation. They also provided training for all the users, and to this end, they produced a user manual illustrating the main features of the system. Introduction into routine service. The following three-stage strategy was adopted for the gradual implementation and testing of the software: (i) system installation and configuration, training of support staff and registration of user definitions; (ii) identification of patients to the system, training of ward staff, validation of basic screen design and translation, request form definitions and refinement of static protocols by monitoring of system proposals; (iii) the gradual introduction of the system, including acceptance of test proposals and printing of request documents for static protocols, refinement of dynamic protocols by monitoring proposals by the system, and finally, acceptance and printing for dynamic protocols. Implementation of the system started in late 1993, and it was introduced into routine use on 21st March 1994. A detailed review of the dynamic component of the guidelines led to significant modifications which were implemented in September 1994. The system is used by administrative, nursing, and junior and senior medical staff. Patient demographic data are input by ward clerks, often some days in advance for forecast admissions. Classification of patients, results review and acceptance and/or modification of test proposals is done by junior medical staff, with some involvement of senior clinicians, Printing of request documentation for newly admitted patients is administered by nursing staff, provided test proposals have been vetted by junior doctors. Printing for other patients is undertaken by junior doctors. Software support for the system is provided by WCL from Birmingham. Details of any problems with the system are supplied by fax and any necessary software modifications implemented via a modem. Since the gradual introduction of the system was completed, both the hardware and software reliability have been high, and consequently the support required has been minimal.

of Biomedical

Computing

40 (1995)

17-30

23

4.2. Audit data An integral part of the system is the audit program, which analyses all test results received by the system from the LIS and categorises each result as ‘protocol’, ‘inserted’ or ‘unsolicited’: protocol tests are those proposed by the rulebase and accepted by the user; inserted tests are those added by the user via the system; unsolicited tests are those requested manually rather than via the system. As demonstrated by Fig. 1, during the period from the 21st March 1994 to 15th November 1994 the system was used to propose tests and print request documents for patients belonging to three major classifications: ‘BPRT’ (for pre-transplant assessment), ‘BPOT’ (for post-transplant assessment) and ‘TRH’ (governing transplantation and immediate post-transplant follow-up). The protocols associated with the classifications BPOT and BPRT contain only static rules. Figs. 2 and 3 show the number of tests requested on each patient in these categories. In each graph, the patients are arranged in chronological order based on the date of classification. Use of the system was spasmodic initially and only stabilized around 1st May 1994 (from patient 19 onwards in Fig. 2 and patient seven onwards in Fig. 3), when the percentage of tests proposed by the protocols for the categories BPOT and BPRT has been 49% and 33%, respectively. From Figs. 2 and 3 it can be seen that a high proportion of the unsolicited tests for these classifications was due to a very small number of patients. These patients were classified as BPOT or BPRT for an unusually long time, suggesting either that these patients were extremely atypical or that the classifications should have been ended sooner. If the effect of these patients on the overall percentages is reduced by considering just the tests requested on the day of classification (when the vast majority of tests is requested for most of the other patients), then since 1st May 1994 the percentage of tests proposed by the protocols for the categories BPOT and BPRT has been 79% and 77%, respectively. Thus, for most patients governed by static protocols, the system is performing well.

24

L. Boon-Falleur

et al. / International

Journal

of Biomedical

Computing

40 (1995)

17-30

ABIL

I II I

&DEN0 ASCITE BIOPSIE+TRHSUIU BPOT BPRT

PROTOCOL

CICLO UNSOLICITED

DIARR+INSLl+UILS HDIG INSU INSU+WILS INSUFFHED REJET+CICLO SEP

TRH+TRHS+ASCITE+POST_OP

TRH+TRHS+POST-OP

TRH+TRHS+POST-.OP+CICLO

TRHSUIU

TRHSUIU+SEP

0

2000

6000

4000

NUNBER

OF

6000

10000

TESTS

Fig. 1. The total number of tests requested on patients in each classification (21 March 1994- 15 November 1994).

L. Boon-Falleur

et al. / International

Journal

of Biomedical

Computing

40 (1995)

17-30

25

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

n

49 51 53 55 57

PROTOCOL

UNSOLICITED

59 61 63

INSERTED

65 67 69 71 73 75 77 79 61 63 65 67 69

0

100

200

NUfiBER

Fig. 2. The total

number

of tests requested

on each patient

with

OF

300

400

TESTS

classification

‘BPOT’

(21 March

1994-

15 November

1994).

L. Boon-Falleur

et al. / International

Journal

of Biomedical

Computing

40 (1995)

17-30

2 3 4

6 7

9 10 1.1

I Irn

12

PROTOCOL 17 UNSOLICITED

16 19

INSERTED

20 21 22 23 24 26 26 27 26

Cl I 0

I 100

300

200

NUMBER

Fig.

3. The total

number

of tests requested

on each patient

with

OF

400

500

TESTS

classification

‘BPRT’

(21 March

1994-

15 November

1994).

L. Boon-Falleur

et al. / International

Journal

of Biomedical

Computing

40 (1995)

17-30

21

Table 1 Laboratory usage for patients classified as ‘BPOT’ or ‘BPRT’ Aug-Ott Number of patients

Bacteriology Chemistry Enzymology Blood Gasometry :Haematology Coagulation Nuclear Medicine Auto-immune Laboratory Serology Toxicology Urine Chemistry Urgent Virology Total

1993

Aug Ott 1994

Aug-Ott

1993

Aug-Ott

1994

36

35

Number of tests

Number of tests

Tests per patients

Tests per patient

296 953 15 33

329 1051

8.2 26.5 0.4 0.9

9.4 30.0 0.0 0.6

721

139

141

124

48

12

20.2 3.9 1.3

21.1 3.5 2.1

22

5

0.6

0.1

56

9 15

15

7

0.5 1.6 0.4

0.3 2.1 0.2

18.7 0.7 83.9

15.7 0.9 86.0

17

672 25 3020

0 22

548 30 3011

The effect of the introduction of the system on laboratory usage was assessedby comparing data from patients belonging to these two classesin two 3-month periods before and after the introduction of the system. The results are summarised in Table 1, which shows the total laboratory usage to be remarkably stable. The rules associated with the TRH classification are mostly dynamic, and are still being refined. However, the large number of unsolicited tests for this classification (see Fig. 1) is partly explained by classification errors on the ward. One of the difficulties is that patients stay in the ICU for a variable number of days after transplantation. The ICU is not connected to the LUMPS system, which consequently was tailored to disable automatic proposals during the ICU period. Thus the process of classification is more complex in this case. Since the system has only recently been used to print request documentation for dynamic protocols, there is little information available to enable

the impact on requesting to be assessed.However, a comparison of the numbers of tests requested for transplanted patients in two 3-month periods before and after the introduction of the system is given in Table 2. There was an overall reduction in laboratory usage from 979 to 949 tests per patient (a 3.0% decrease), largely due to the number of general chemistry tests (chemistry, enzymology, urine chemistry and urgent) falling from 369 to 339 per patient ( - 8.2%). It should be noted that bacteriology testing increased by 9.9%; since there are virtually no rules applicable to bacteriology testing included in the system, an increase (or decrease) in the number of requests for these tests should be independent of the use of the system. Fig. 1 indicates that the number of tests requested for patients classified as TRH was a large proportion of the total number. Therefore, it is likely that the full benefit of the system will only be realized once it is used to manage these patients throughout their stay.

L. Boon-Falleur et al. / International Journal of Biomedical Computing 40 (1995) 17-30

28 Table 2 Laboratory

usage for patients Aug-Ott

Number patients

of

1993

10

Number Bacteriology Chemistry Enzymology Blood Gasometry Haematology Coagulation Nuclear Medicine Serology Toxicology Urine Chemistry Urgent Virology Total

classified

as ‘TRH’ Aug-Ott

1994

Aug-Ott

1993

Aug-Ott

1994

10

of tests

Number

of tests

Tests per patient

Tests per patient

1346 687 29 902

1479 616 3 712

134.6 68.7 2.9 90.2

147.9 61.6 0.3 71.2

3170 285 5

3232 208 0

317.0 28.5 0.5

323.2 20.8 0.0

20 368 10

4 469 0

2.0 36.8 1.0

0.4 46.9 0.0

2965 4 9791

2771 0 9494

296.5 0.4 979.1

277.1 0.0 949.4

5. Conclusions 5.1. Evaluation of feedback to GPs The described feedback system provides feedback about the adequateness of the requests of GPs on the basis of volunteered data. In the manual system, the GPs added medical data to the request form that would (in their eyes sufficiently) support their requests. This means that the amount of medical data available can vary from case to case. However, if not enough data are provided, it is difficult to assessthe adequacy of the test request. Even with this limitation, the feedback system was able to correctly identify that about 17% of the test requests were unnecessary. The number of errors made by the system appeared to be rather low for hyperthyroidism, but relatively large for hypothyroidism. However, in these cases the medical data provided were equivocal and did not support thyroid disease sufficiently. A better result could probably have been obtained when the system.!was instructed to only indicate that too little data were provided, although in this case, the number of correctly identified unnecessary tests would decrease for the same reason.

It is considered to present GPs computerized forms for each test they request, so that they can indicate for all relevant signs and symptoms whether they are present or not. This would assure that the medical data were complete so that the expert system probably can better assessthe adequatenessof the test. Such a feature, however, will take more of the busy GP’s time. The amount of data entered by each GP can also be monitored over time. If a GP consistently presents too little data, this can be determined and an appropriate feedback can then be given. The reason for request has, of course, also to be taken into account. In this study, all requests for monitoring the state of a patient having thyroid disease were not included. Only the requests for ruling in or ruling out thyroid diseasewere analyzed. The number of requests for which it is stated that the investigation is requested, because the patient urged the GP to do so, was very small. In the future, the distribution of the various reasons for request provided during a certain period will also be analyzed. Since the system has not been used on a large scale yet, it was not possible to determine the

L. Boon-Falleur

et al. 1 International

Journal

impact of providing ‘instantaneous’ feedback on the requesting behaviour of the GPs. The results, however, indicate that the number of test requests can be reduced by up to 17%. 5.2. Evaluation of feedback in secondary care The flexibility afforded by the high degree of parameterisation within LUMPS made the introduction of the system into the clinical environment at UCL relatively straightforward at the technical level; the adaptability of the program proved to be excellent regarding LIS interfacing for result transmission, and the terminal emulation used to access the system was easily tailored to the local constraints. Some problems in performance were experienced; however, these have now been largely solved by some rationalisation of internal data gathering and storing techniques. The original LUMPS system was designed for use in a UK clinical environment, where the requesting behaviour differs appreciably from that in Belgium. Consequently, the protocols implemented at UCL needed to be entirely different from those used in the UK, and also needed to include more laboratories (e.g. microbiology). Static rules for laboratory investigation have been easy to define and validate. However, dynamic rules have presented more difficulties, and UCL believe the formulation of an improved set of guidelines will be an important consequence of the project. The UCL team’s realisation, that the dynamic rules they initially defined did not really reflect their actual requirements, is not uncommon in WCL’s experience. Few clinicians have actually achieved that which the system allows them to do, i.e. to effect reliably a strategy for laboratory investigation [lo]. Establishing the optimum pattern usually requires a little time! When the original LUMPS system was introduced onto the Liver Unit in Birmingham, it gained almost immediate acceptance by medical staff because it provided easy and fast access to the results of laboratory investigations (which was not otherwise available). Thus, the system was providing its users with a significant and perceptible benefit from the outset and so little further encouragement was needed to promote the use of the system. This was not the case at UCL because the

of Biomedical

Computing

40 (1995)

17-30

29

results of laboratory investigations were already available on the HIS and consequently, it has taken rather longer to establish routine use of the system. In its first year of operation, the original LUMPS system produced significant savings in medical staff time [8] (through the automatic generation of request documentation and the ready availability of results) and also significantly reduced overall and out-of-hours laboratory usage [9]. In addition, the appropriateness of laboratory testing improved; compliance with protocols increased, indicating that necessary tests were more likely to be requested, but overall usage decreased, suggesting that the number of unnecessary tests had fallen. The system has been used consistently since its introduction, and at present approximately 83% of all laboratory investigations requested are proposed by the system. The indication from UCL is that any reduction in laboratory usage since the introduction of the system has so far been small. However, the relatively large number of investigations requested in addition to those proposed by the system suggests that the users may not yet have learned to ‘trust’ the system to propose the appropriate investigations. Once this does happen, the number of extra investigations requested may fall, thus producing a significant reduction in laboratory usage. The users at UCL would prefer the same user interface as their established HIS, which also transmits laboratory results to ward areas. Whilst a common interface should be an eventual goal, customisation of user-interface was not possible within the context of this study. The principal objective at UCL and in the several other UK sites where the system is running, is the development and evaluation of the concept of automatic reinforcement of laboratory investigation guidelines. Ultimately, should the concept prove generally applicable, order communication and/or clinical management systems will provide comprehensive knowledge-based functionality, including guidelines for many other aspects of clinical management. Although the clinicians would like some modifications to be made to the system, they are already convinced about its clinical usefulness: they believe it helps to avoid omissions in requesting (a particular example is its use with research or drug study

30

L: Boon-Falleur

et al. / International

Journal

protocols, where registering a patient in the appropriate category causes the required investigations to be proposed automatically); it also prevents the prescription of unnecessary tests often requested by junior doctors or students with insufficient clinical experience or pathological knowledge (e.g. the unnecessary requesting of a caeruloplasmin level in neonatal cholestasis) and, since the ward doctor can view previous results and outstanding requests for tests simultaneously on the same screen, helps to reduce the number of tests repeated unnecessarily. In addition, the automatic printing of requests and labels produces a considerable time saving, which is greatly appreciated by the nurses and junior doctors. Although the system has been running for a comparatively short time, the users’ involvement and enthusiasm is encouraging, as is the interest from other disciplines and specialties. The system has gained acceptance from the users, and the laboratories are particularly eager that it continues in routine use to allow time for the potential benefits to be realized. Acknowledgements We acknowledge the financial support European Union for the AIM OpenLabs (A2028).

of the project

of Biomedical

Computing

40 (1995)

17-30

References 111O’Moore

R et al: OpenLabs: the application of advanced informatics and telematics for optimization of clinical laboratory services, Comp Meth Prog Biomed, 45 (1994) 137-140. PI Pop P and Winkens RAG: A diagnostic center for general practitioners: results of individual feedback on diagnostic actions, JR Coil Gen Prac, 39 (1989) 507-508. experiment: Real time expert 131 Blom JA: The SIMPLEXYS systems in patient monitoring, PhD thesis, Technical University Eindhoven, 1990. RAG and Blom JL: To test [41 Hasman A, Pop P, Winkens or not to test, that is the question, Clin Chim Acta, 222 (1993) 49-56. IPC and Wayne EJ: Statistical methods PI Crooks J, Murray applied to the clinical diagnosis of thyrotoxicosis, Q J Med, (1959) 21 l-234. studies in thyroid 161Wayne EJ: Clinical and metabolic disease (The second of two Lumleian lectures delivered before the Royal College of Physicians of London on April 14 and 16, 1959), Br Med J, (1960) 78-90. BA, Ryan MF, 171 Peters M, Clark IR, Parekh J, McCauley Neuberger JM and Mutimer D: Automatic application of rule-based decision support; a specialist unit investigation manager. In: Current Perspectives in Healthcare Computing (Ed: B Richard), The British Journal of Healthcare Computing, Surrey, UK, 1991, pp. 1299136. D, McCauley BA, Nightingale PG, Ryan MF, 181Mutimer Peters M and Neuberger JM: Computerised protocols for laboratory investigation and their effect on use of medical time and resources, J Clin Pathol, 45 (1992) 5722574. PG, Peters M, Mutimer D and Neuberger [91 Nightingale JM: Effects of a computerised protocol management system on ordering of clinical tests, Qua1 Health Care, 3 (1994) 23-28. test demand by clinicians ~ com1101Peters M: Managing puter assisted guidelines, J Clin Pathol, (in press).