- Email: [email protected]
Knowledge acquisition for medical diagnosis systems
K135 letter
Knowledge acquisition for medical diagnosis systems Adi Armoni
The methodology of the development of knowledge bases for expert systems is based on questioning of human experts. Until now, very little attention has been paid to the question of the consistency and accuracy of the expert's estimations. This is a subject of great importance, especially in areas requiring the estimation of probability events. In the research partially presented in the paper, a multistage methodology was developed to elicit knowledge requiring subjective judgement. The paper discusses an empirical application in the field of medical diagnosis, and includes the results and conclusions drawn from it.
Keywor~ k n o ~
~
~ ,
.m~y,
The classic model of the development of knowledge bases is based on the location of THE expert, the identification of his/her expertise, and the elicitation of his/her knowledge.
DEFINITIONS
•
Probability groups evaluated: a priori probability: the prevalence of event K, P(K), the evidence E, P(E), or positive results of testing X, P(X), o posterior diagnostic probability: the probability of the existence of event K when given evidence E, P(K/E), o conditional probabilities of evidence: the probability of the existence of evidence E given the occurrence of event K, P(E/K), o posterior diagnostic probabilities of test," the probability of the occurrence of event K when given positive results of test X, P(K/X), o conditional probability of test: the probability of positive results of test X when given occurrence of event K, P(X/K). o
medi~l
In the light of the many studies dealing with the representation of knowledge in the knowledge bases of expert systems and different methods of operation, not enough attention has been paid to the difficulties involved in the elicitation of the knowledge required from human experts. This applies particularly to knowledge that is not strictly factual, but is mainly based on the experience and judgement of a human expert. From studies of this subject, it has become clear that the limitations of the decision maker in dealing with uncertain information can impair the accuracy, quality, reliability, and consistency of his/her decisions [1-3]. Until now, expert systems were based entirely on the evaluations of human experts. There were no further opportunities to test them by changing the acquisition direction and crossing the results received in contrast with the direct evaluations of the human expert [4].
•
Research Unit, School of Business Administration, College of Manal~nent, Tel-Aviv,Israel Paper received28 March 1994. Revisedpaper received9 August 1994. Accepted22 September 1994
In conclusion, the concepts of consistency and accuracy in an expert's estimations can be identified using the
•
•
Objective probability: The values of the probability on which there is agreement in the professional literature [5-13]. Consistency ofestimatedprobabilities: The difference between probabilities elicited directly and the same probabilities calculated normatively according to Bayesian theory. Accuracy of estimated probabilities: The difference between a group of probabilities estimated by an expert (directly or calculated normatively) and their objective values.
0950-7051/95/$09.50 O 1995 Elsevier Science B.V. All rights reserved Knowledge-Based Systems Volume 8 Number 4 August 1995 223
Knowledge acquisition for medical diagnosis systems: A Armoni
EMPIRICAL APPLICATION FROM MEDICAL DIAGNOSIS FIELD
Xe
General
R(Xe, Xo)
Xo /
/
R(Xe,Xn)
\ Xn
Figure 1 Concepts of consistency and accuracy [X~:values of subjective probabilities as elicited from expert; X.: values of probabilities as calculated normatively; Xo: objective values of probabilities.]
triangle shown in Figure 1. The level of the consistency and accuracy of the expert in his/her estimation of probabilities is expressed through the following correlation. R(Xe, Xn) is the degree of consistency of the expert, and R(Xe, Xo) is the degree of accuracy of the expert. R is estimated by measures that will be described later in the study.
Many expert systems have been developed in the field of medical diagnosis. As stated above, there is a fear in those carrying out the questioning as to whether the probability values found in the knowledge bases of these systems express accurately and consistently the estimations of the human expert who constituted the basis for the development of the system [14-16]. The disease that was chosen to represent the assumptions of the research is type 1 diabetes (insulin dependent diabetes mellitus (IDDM)). This disease was chosen because of its complexity and complications on one hand, and its relatively high prevalence on the other hand.
Stages of empirical experiment The experiment had four stages: •
ASSESSMENT OF ACCURACY AND CONSISTENCY OF EXPERT'S ESTIMATIONS •
•
•
Assessment of level of consistency of estimations: © a realization of the anomaly caused by lack of
•
consistency of the expert (i.e. the proportion of calculated probabilities larger than 1), © the degree of fitness (if any), between values of probability directly elicited, and those calculated normatively (i.e. the correlation coefficient), © a quantitative concept for the difference between the direct probabilities and the normatively calculated probabilities via an average of the absolute differences; in order to allow for the accurate estimation of differences between direct probabilities and normatively calculated ones, a confidence interval is determined for the value of Xdj (the direct evaluation of the expert).
•
Presentation and analysis of research results
Assessment of level of accuracy of estimations." © a quantitative measure of the accuracy of the expert's estimations; an assessment of the difference between the expert's estimation and the objective values through the average of absolute differences, o an assessment of the fitness between the expert's estimations and the objective values, as calculated by the correlation coefficient, © a sensitivity analysis of trends in the expert's estimates, i.e. whether he/she tends to overestimate, underestimate, or be accurate.
224
The components of knowledge were elicited, and a questionnaire was constructed for the estimation of probabilities. The questionnaire included 110 question based on a numeric scale of 0-100 with which the estimating doctor was requested to indicate his estimation of the probability of events. The initial sample included 13 internists and diabetologists. In response to their comments, the questionnaire was reduced to 75 questions and the wording was changed for part of the questions. The population of 86 doctors participating in the final version of the questionnaire included 22 endocrinologists/diabetologists, 19 internists, 18 family doctors, 16 pediatricians, three oncologists, and eight other doctors (orthopedists, surgeons, emergency doctors, dermatologists). Each of the doctors completed the questionnaire with a surveyor from the research staff. The research results and the recommendations for methods of eliciting information were analysed.
Knowledge-Based Systems Volume 8 Number 4 August 1995
According to the three measures of consistency defined in this study, there is a distinct preference (alfa=O.O1) for endocrinologists in conditional probability estimation and especially for rare symptoms. Estimates of high levels of consistency were provided by pediatricians, especially in relation to the diagnostic probabilities related to common symptoms. The internists and the doctors belonging to the group from various specialties provided the estimates with the lowest level of consistency. An explanation for the findings described above can be drawn from the fact that pediatricians encounter the disease first (in juvenile diabetes), and, as a result, they have an advantage in terms of the diagnosis of the disease and in the estimation of the diagnostic probabilities. The endocrinologists, on the other hand, specifically treat a population of diabetics,
Knowledge acquisition for medical diagnosis systems: A Armoni
•
•
•
and are more exposed to cases with rare symptoms and complications related to the disease. Thus they have an advantage in the estimation of the conditional probabilities. A distinct preference in the consistency of the estimations was discovered for the group of doctors with the maximal number of years of professional experience (20 years and more). It is clear from the results that, as the number of years of professional experience of the estimating doctor increases, the consistency in his/her estimations increases, in accordance with the three measures of consistency described above. The most accurate probability estimations for conditional probability groups with common symptoms related to type 1 diabetes were provided by pediatricians. The most accurate estimates for probability groups related to rare symptoms of type 1 diabetes were provided by endocrinologists. Note that the results mentioned above are distinct (alfa=0.01). In contrast to the results for the consistency of estimations, the most accurate estimations were not necessarily provided by the doctors with the maximal number of years of professional experience. The most accurate estimations of a priori probabilities, and of conditional probabilities, especially those related to rare symptoms, were provided by doctors with professional experience of 11-20 years. The most accurate estimations for probabilities related to common symptoms were provided by doctors with the maximal number o f years of professional experience (20 years and more). An explanation for the findings may be the 'passion for research' of a group of doctors with professional experience of 10-15 years. They thus have a relative advantage in estimating probabilities of rare symptoms. On the other hand, the group of experienced doctors exhibits a clear preference in their estimation of probabilities related to common symptoms, and this is of course due to their rich professional
experience. •
A positive correlation was found between the experts
who provided the most accurate estimates and the groups who provided the most consistent estimates (alfa = 0.01). It is significantly clear that the experts who provided the most accurate estimates were the same ones who provided the most consistent ones.
CONTRIBUTION OF RESEARCH TO DEVELOPMENT OF KNOWLEDGE BASES FOR EXPERT SYSTEMS The definition of measures and the development of techniques for the testing of the internal consistency of the estimated probabilities. This technique is based on a numeric test o f the estimated probabilities. The experiments that have been carried out up to now to test the consistency of the expert were based on the repeated questioning of the expert (with similar questions and different periods of time).
•
•
•
•
The definition of measures and the development of techniques for the testing of the accuracy o f the expert's estimates. In this case, up until now, a renowned expert has been chosen, and the degree of accuracy in his/her estimates has not been examined in any way. Regarding the field of medical diagnosis, the study proves that there is a positive correlation between the degree of consistency and the degree o f accuracy o f the expert. The experts who are more accurate in their estimates are also the ones who show the most consistency. This finding is important, especially in the light of the fact that, in some o f the probabilities, it is difficult (or even impossible) to locate the objective probabilities. The study proves that the differences in the degree o f the consistency and accuracy o f the knowledge elicited from the experts are dependent on professional characteristics and on professional expertise and the number o f years of experience. This contradicts the approach accepted today, according to which, differences stem from personal characteristics only and are caused by difficulties which the individual encounters in his/her estimation of probabilities [17,18]. The findings from the empirical application of the methodology to the field of diagnostic medicine show that, in contrast to the norm accepted today, basing the development of knowledge bases o f expert systems on the knowledge of a single expert (who is renowned), it is necessary to build the knowledge base from the knowledge elicited from a number of experts with different types and levels of professional expertise, and with different numbers of years of professional experience. This recommendation comes from the 'relative strength' (the high degree of consistency and accuracy) o f each one of the experts in a different field o f probabilities.
REFERENCES 1 Fischhoff, B 'Debiasing in judgment under uncertainty: heuristics and biases' in Kahnman, D, Slovic, P and Tversky, A (Eds.) Cambridge University Press, USA (1982) 2 Healy, A F and Kubovy, M 'The effects of payoffs and prior probabilities on indices of performance and cutoff location in recognition memory' Memory and Cognition Vol 6 (1978) pp 544-553 3 Healy, A F and Kubovy, M 'Probability matching and the formation of conservative decision rules in memory analog of signal detection' Journal of Experimental Psychology: Human Learning and Memory Vol 7 (1981) pp 344-354 4 Fischhoff, B and Beyth-Marom, R 'Hypothesis evaluation from a Bayesian perspective' Psychological Review Vol 90 (1983) pp 239-260 5 Melin, K et al. 'Diabetes mellitus some complications till porotitis epidemica' Nord. Med. Vol 60 (1958) pp 1715-1717 6 Delbridge, L, Ctercteko, C, Reeve, T and Le Quesne, P 'The atiology of diabetic neuropatic ulceration of the foot' British Journal of Surgery (1985) Vol 72 No 1 pp 1-6 7 Thomas, P, Watkins, P and Ward, J 'Diabetic neuropathy' in Kenn, H and Jarrett, J (Eds.) Complications of Diabetes (2nd Ed.) Edward Arnold, UK (1982) pp 109-136 8 Martin, F and Hopper, L 'The relationship of acute insulin sensitivity to the progression of vascular disease in long term type 1 diabetes mellitus' Diabetologia Vol 30 (1987) pp 149-153 9 Jensen, T et al. 'Coronary heart disease in young type 1 diabetic patients with and without diabetic nephropathy: incidence and risk factors' Diabetologia Vol 30 No 3 (1987) pp 144-148
Knowledge-Based Systems Volume 8 Number 4 August 1995 225
Knowledge acquisition for medical diagnosis systems: A Armoni 10 Jeffrey, F and Lisa, H 'Epidemiologic approach to the etiology of type 1 diabetes mellitus and its complications' New England Journal of Medicine Vol 317 No 22 (1987) pp 1390-1398 11 Ginsberg-Fellner, M e t al. 'Diabetes mellitus and autoimmunity in patients with the congenital rubella syndrome' Reviews of Infectious Diseases Vol 7 Supp 1 (1985) pp 170--176 12 Delbridge, L e t al. 'Factors associated with development of foot lesions in the diabetic' Surgery Vol 93 (1983) pp 78-82 13 Gamble, D 'The epidemiology of insulin dependent diabetes, with particular reference to the relationship of virus infection to its etiology' Epidemiology Reviews Vol 2 (1980) pp 49-70 14 Shortliffe, E H Computer Based Medical Consultation: M Y C I N North-Holland, USA 15 Fryback, G 'Baye's theorem and conditional nonindependence of
226
data in medical diagnosis' in Computer Assisted Medical Decision Making Vol 1 Springer-Vedag (1985) pp 183-195 16 Ben-Bassat, et al. 'Sensitivity analysis in Bayesian classification models: muitiplicative deviations' IEEE Transactions on Pattern Analysis and Machine Intelligence Vol PAMI-2 No 3 (1980) pp 261-266 17 Mark, D and Cookson, J 'The role of clinical judgment analysis in the development of medical expert systems' Proceedings of the Second European Conference on Artificial Intelligence in Medicine (1989) 18 Shortliffe, E H and Buchanan, B G 'A model of inexact reasoning' Medicine & Mathematical Biasciences Vol 23 (1975) pp 351-379
Knowledge-Based Systems Volume 8 Number 4 August 1995
-
Knowledge acquisition for medical diagnosis systems Adi Armoni
The methodology of the development of knowledge bases for expert systems is based on questioning of human experts. Until now, very little attention has been paid to the question of the consistency and accuracy of the expert's estimations. This is a subject of great importance, especially in areas requiring the estimation of probability events. In the research partially presented in the paper, a multistage methodology was developed to elicit knowledge requiring subjective judgement. The paper discusses an empirical application in the field of medical diagnosis, and includes the results and conclusions drawn from it.
Keywor~ k n o ~
~
~ ,
.m~y,
The classic model of the development of knowledge bases is based on the location of THE expert, the identification of his/her expertise, and the elicitation of his/her knowledge.
DEFINITIONS
•
Probability groups evaluated: a priori probability: the prevalence of event K, P(K), the evidence E, P(E), or positive results of testing X, P(X), o posterior diagnostic probability: the probability of the existence of event K when given evidence E, P(K/E), o conditional probabilities of evidence: the probability of the existence of evidence E given the occurrence of event K, P(E/K), o posterior diagnostic probabilities of test," the probability of the occurrence of event K when given positive results of test X, P(K/X), o conditional probability of test: the probability of positive results of test X when given occurrence of event K, P(X/K). o
medi~l
In the light of the many studies dealing with the representation of knowledge in the knowledge bases of expert systems and different methods of operation, not enough attention has been paid to the difficulties involved in the elicitation of the knowledge required from human experts. This applies particularly to knowledge that is not strictly factual, but is mainly based on the experience and judgement of a human expert. From studies of this subject, it has become clear that the limitations of the decision maker in dealing with uncertain information can impair the accuracy, quality, reliability, and consistency of his/her decisions [1-3]. Until now, expert systems were based entirely on the evaluations of human experts. There were no further opportunities to test them by changing the acquisition direction and crossing the results received in contrast with the direct evaluations of the human expert [4].
•
Research Unit, School of Business Administration, College of Manal~nent, Tel-Aviv,Israel Paper received28 March 1994. Revisedpaper received9 August 1994. Accepted22 September 1994
In conclusion, the concepts of consistency and accuracy in an expert's estimations can be identified using the
•
•
Objective probability: The values of the probability on which there is agreement in the professional literature [5-13]. Consistency ofestimatedprobabilities: The difference between probabilities elicited directly and the same probabilities calculated normatively according to Bayesian theory. Accuracy of estimated probabilities: The difference between a group of probabilities estimated by an expert (directly or calculated normatively) and their objective values.
0950-7051/95/$09.50 O 1995 Elsevier Science B.V. All rights reserved Knowledge-Based Systems Volume 8 Number 4 August 1995 223
Knowledge acquisition for medical diagnosis systems: A Armoni
EMPIRICAL APPLICATION FROM MEDICAL DIAGNOSIS FIELD
Xe
General
R(Xe, Xo)
Xo /
/
R(Xe,Xn)
\ Xn
Figure 1 Concepts of consistency and accuracy [X~:values of subjective probabilities as elicited from expert; X.: values of probabilities as calculated normatively; Xo: objective values of probabilities.]
triangle shown in Figure 1. The level of the consistency and accuracy of the expert in his/her estimation of probabilities is expressed through the following correlation. R(Xe, Xn) is the degree of consistency of the expert, and R(Xe, Xo) is the degree of accuracy of the expert. R is estimated by measures that will be described later in the study.
Many expert systems have been developed in the field of medical diagnosis. As stated above, there is a fear in those carrying out the questioning as to whether the probability values found in the knowledge bases of these systems express accurately and consistently the estimations of the human expert who constituted the basis for the development of the system [14-16]. The disease that was chosen to represent the assumptions of the research is type 1 diabetes (insulin dependent diabetes mellitus (IDDM)). This disease was chosen because of its complexity and complications on one hand, and its relatively high prevalence on the other hand.
Stages of empirical experiment The experiment had four stages: •
ASSESSMENT OF ACCURACY AND CONSISTENCY OF EXPERT'S ESTIMATIONS •
•
•
Assessment of level of consistency of estimations: © a realization of the anomaly caused by lack of
•
consistency of the expert (i.e. the proportion of calculated probabilities larger than 1), © the degree of fitness (if any), between values of probability directly elicited, and those calculated normatively (i.e. the correlation coefficient), © a quantitative concept for the difference between the direct probabilities and the normatively calculated probabilities via an average of the absolute differences; in order to allow for the accurate estimation of differences between direct probabilities and normatively calculated ones, a confidence interval is determined for the value of Xdj (the direct evaluation of the expert).
•
Presentation and analysis of research results
Assessment of level of accuracy of estimations." © a quantitative measure of the accuracy of the expert's estimations; an assessment of the difference between the expert's estimation and the objective values through the average of absolute differences, o an assessment of the fitness between the expert's estimations and the objective values, as calculated by the correlation coefficient, © a sensitivity analysis of trends in the expert's estimates, i.e. whether he/she tends to overestimate, underestimate, or be accurate.
224
The components of knowledge were elicited, and a questionnaire was constructed for the estimation of probabilities. The questionnaire included 110 question based on a numeric scale of 0-100 with which the estimating doctor was requested to indicate his estimation of the probability of events. The initial sample included 13 internists and diabetologists. In response to their comments, the questionnaire was reduced to 75 questions and the wording was changed for part of the questions. The population of 86 doctors participating in the final version of the questionnaire included 22 endocrinologists/diabetologists, 19 internists, 18 family doctors, 16 pediatricians, three oncologists, and eight other doctors (orthopedists, surgeons, emergency doctors, dermatologists). Each of the doctors completed the questionnaire with a surveyor from the research staff. The research results and the recommendations for methods of eliciting information were analysed.
Knowledge-Based Systems Volume 8 Number 4 August 1995
According to the three measures of consistency defined in this study, there is a distinct preference (alfa=O.O1) for endocrinologists in conditional probability estimation and especially for rare symptoms. Estimates of high levels of consistency were provided by pediatricians, especially in relation to the diagnostic probabilities related to common symptoms. The internists and the doctors belonging to the group from various specialties provided the estimates with the lowest level of consistency. An explanation for the findings described above can be drawn from the fact that pediatricians encounter the disease first (in juvenile diabetes), and, as a result, they have an advantage in terms of the diagnosis of the disease and in the estimation of the diagnostic probabilities. The endocrinologists, on the other hand, specifically treat a population of diabetics,
Knowledge acquisition for medical diagnosis systems: A Armoni
•
•
•
and are more exposed to cases with rare symptoms and complications related to the disease. Thus they have an advantage in the estimation of the conditional probabilities. A distinct preference in the consistency of the estimations was discovered for the group of doctors with the maximal number of years of professional experience (20 years and more). It is clear from the results that, as the number of years of professional experience of the estimating doctor increases, the consistency in his/her estimations increases, in accordance with the three measures of consistency described above. The most accurate probability estimations for conditional probability groups with common symptoms related to type 1 diabetes were provided by pediatricians. The most accurate estimates for probability groups related to rare symptoms of type 1 diabetes were provided by endocrinologists. Note that the results mentioned above are distinct (alfa=0.01). In contrast to the results for the consistency of estimations, the most accurate estimations were not necessarily provided by the doctors with the maximal number of years of professional experience. The most accurate estimations of a priori probabilities, and of conditional probabilities, especially those related to rare symptoms, were provided by doctors with professional experience of 11-20 years. The most accurate estimations for probabilities related to common symptoms were provided by doctors with the maximal number o f years of professional experience (20 years and more). An explanation for the findings may be the 'passion for research' of a group of doctors with professional experience of 10-15 years. They thus have a relative advantage in estimating probabilities of rare symptoms. On the other hand, the group of experienced doctors exhibits a clear preference in their estimation of probabilities related to common symptoms, and this is of course due to their rich professional
experience. •
A positive correlation was found between the experts
who provided the most accurate estimates and the groups who provided the most consistent estimates (alfa = 0.01). It is significantly clear that the experts who provided the most accurate estimates were the same ones who provided the most consistent ones.
CONTRIBUTION OF RESEARCH TO DEVELOPMENT OF KNOWLEDGE BASES FOR EXPERT SYSTEMS The definition of measures and the development of techniques for the testing of the internal consistency of the estimated probabilities. This technique is based on a numeric test o f the estimated probabilities. The experiments that have been carried out up to now to test the consistency of the expert were based on the repeated questioning of the expert (with similar questions and different periods of time).
•
•
•
•
The definition of measures and the development of techniques for the testing of the accuracy o f the expert's estimates. In this case, up until now, a renowned expert has been chosen, and the degree of accuracy in his/her estimates has not been examined in any way. Regarding the field of medical diagnosis, the study proves that there is a positive correlation between the degree of consistency and the degree o f accuracy o f the expert. The experts who are more accurate in their estimates are also the ones who show the most consistency. This finding is important, especially in the light of the fact that, in some o f the probabilities, it is difficult (or even impossible) to locate the objective probabilities. The study proves that the differences in the degree o f the consistency and accuracy o f the knowledge elicited from the experts are dependent on professional characteristics and on professional expertise and the number o f years of experience. This contradicts the approach accepted today, according to which, differences stem from personal characteristics only and are caused by difficulties which the individual encounters in his/her estimation of probabilities [17,18]. The findings from the empirical application of the methodology to the field of diagnostic medicine show that, in contrast to the norm accepted today, basing the development of knowledge bases o f expert systems on the knowledge of a single expert (who is renowned), it is necessary to build the knowledge base from the knowledge elicited from a number of experts with different types and levels of professional expertise, and with different numbers of years of professional experience. This recommendation comes from the 'relative strength' (the high degree of consistency and accuracy) o f each one of the experts in a different field o f probabilities.
REFERENCES 1 Fischhoff, B 'Debiasing in judgment under uncertainty: heuristics and biases' in Kahnman, D, Slovic, P and Tversky, A (Eds.) Cambridge University Press, USA (1982) 2 Healy, A F and Kubovy, M 'The effects of payoffs and prior probabilities on indices of performance and cutoff location in recognition memory' Memory and Cognition Vol 6 (1978) pp 544-553 3 Healy, A F and Kubovy, M 'Probability matching and the formation of conservative decision rules in memory analog of signal detection' Journal of Experimental Psychology: Human Learning and Memory Vol 7 (1981) pp 344-354 4 Fischhoff, B and Beyth-Marom, R 'Hypothesis evaluation from a Bayesian perspective' Psychological Review Vol 90 (1983) pp 239-260 5 Melin, K et al. 'Diabetes mellitus some complications till porotitis epidemica' Nord. Med. Vol 60 (1958) pp 1715-1717 6 Delbridge, L, Ctercteko, C, Reeve, T and Le Quesne, P 'The atiology of diabetic neuropatic ulceration of the foot' British Journal of Surgery (1985) Vol 72 No 1 pp 1-6 7 Thomas, P, Watkins, P and Ward, J 'Diabetic neuropathy' in Kenn, H and Jarrett, J (Eds.) Complications of Diabetes (2nd Ed.) Edward Arnold, UK (1982) pp 109-136 8 Martin, F and Hopper, L 'The relationship of acute insulin sensitivity to the progression of vascular disease in long term type 1 diabetes mellitus' Diabetologia Vol 30 (1987) pp 149-153 9 Jensen, T et al. 'Coronary heart disease in young type 1 diabetic patients with and without diabetic nephropathy: incidence and risk factors' Diabetologia Vol 30 No 3 (1987) pp 144-148
Knowledge-Based Systems Volume 8 Number 4 August 1995 225
Knowledge acquisition for medical diagnosis systems: A Armoni 10 Jeffrey, F and Lisa, H 'Epidemiologic approach to the etiology of type 1 diabetes mellitus and its complications' New England Journal of Medicine Vol 317 No 22 (1987) pp 1390-1398 11 Ginsberg-Fellner, M e t al. 'Diabetes mellitus and autoimmunity in patients with the congenital rubella syndrome' Reviews of Infectious Diseases Vol 7 Supp 1 (1985) pp 170--176 12 Delbridge, L e t al. 'Factors associated with development of foot lesions in the diabetic' Surgery Vol 93 (1983) pp 78-82 13 Gamble, D 'The epidemiology of insulin dependent diabetes, with particular reference to the relationship of virus infection to its etiology' Epidemiology Reviews Vol 2 (1980) pp 49-70 14 Shortliffe, E H Computer Based Medical Consultation: M Y C I N North-Holland, USA 15 Fryback, G 'Baye's theorem and conditional nonindependence of
226
data in medical diagnosis' in Computer Assisted Medical Decision Making Vol 1 Springer-Vedag (1985) pp 183-195 16 Ben-Bassat, et al. 'Sensitivity analysis in Bayesian classification models: muitiplicative deviations' IEEE Transactions on Pattern Analysis and Machine Intelligence Vol PAMI-2 No 3 (1980) pp 261-266 17 Mark, D and Cookson, J 'The role of clinical judgment analysis in the development of medical expert systems' Proceedings of the Second European Conference on Artificial Intelligence in Medicine (1989) 18 Shortliffe, E H and Buchanan, B G 'A model of inexact reasoning' Medicine & Mathematical Biasciences Vol 23 (1975) pp 351-379
Knowledge-Based Systems Volume 8 Number 4 August 1995
-