- Email: [email protected]
Determination of Class II and Class III skeletal patterns: Receiver operating characteristic (ROC) analysis on various cephalometric measurements
CONTINUING EDUCATION ARTICLE
Determination of Class II and Class III skeletal patterns: Receiver operating characteristic (ROC) analysis on various cephalometric measurements Unae Kim Han, DMD, MPH, MS, and Young H. Kim, DMD, MS Lexington and Weston, Mass. Receiver operating characteristic analysis is an excellent method for evaluating and comparing the performance of diagnostic tests. The purpose of this study was to use the receiver operating characteristic analysis to evaluate the diagnostic ability of several cephalometric measurements in determining the presence of Class II and Class III skeletal patterns. Receiver operating characteristic analysis was performed on 976 cases. Fifteen cephalometric measurements were evaluated. A computer software program ROC ANALYZER was used to tabulate the areas under the curves and to perform the statistical comparison between the curves. The results of this study indicated that the Anteroposterior Dysplasia Indicator had the best diagnostic ability in identifying cases with Class II and Class III skeletal patterns. WITS Appraisal and Overjet were highly effective in diagnosing cases with Class II skeletal pattern. WITS Appraisal, Convexity, AB Plane Angle and Overjet also performed well in diagnosing cases with Class III skeletal pattern. (Am J Ortho Dentofacial Orthop 1998;113:538-45.)
S
ince its introduction, lateral cephalometric radiographs have been widely used in the field of orthodontics to diagnose craniofacial anomalies, to determine a diagnosis and treatment plan of an individual patient’s malocclusion, to monitor and predict facial growth, and to evaluate treatment effects. The anteroposterior relationship between the maxilla and the mandible is considered the most important diagnostic criterion in orthodontics. Successful treatment planning and analysis of treatment results depend on reliable diagnostic information. Numerous cephalometric measurements and analyses have been proposed to determine the skeletal pattern of individual patients causing much confusion and controversy. It seems appropriate to ask the following question: How can we measure the quality of diagnostic information provided by cephalometric measurements in determining the skeletal pattern of an individual patient? Efficacy of a cephalometric measurement, e.g., ANB angle, in diagnosing Class II or Class III skeletal pattern can be characterized by its sensitivity, specificity, predictive value positive, predictive value negative, accuracy, and likelihood ratio.1 Methods to calculate these measures can be illustrated with the 2 3 2 decision matrix (Table I). In private practice. Reprint requests to: Unae Kim Han, 789 Massachusetts Ave., Lexington, MA 02173 Copyright © 1998 by the American Association of Orthodontists. 0889-5406/98/$5.00 1 0 8/1/84775
538
The sensitivity is the true-positive ratio and is calculated as (TP/TP1FN). This measures the proportion of diseased patients correctly identified as positive. The specificity is the true-negative ratio and is calculated as (TN/FP1TN). This measures the proportion of disease-free patients correctly identified as negative. The predictive value positive is the probability that a patient with a positive result actually has the disease and is calculated as (TP/TP1FP). The predictive value negative is the probability that a patient with a negative test result actually does not have the disease and is calculated as (TN/TN1FN). The accuracy of a diagnostic measure is the sum of the true-positive and true-negative outcomes divided by all outcomes. This measures the fraction of patients for which the diagnostic test is correct. The likelihood ratio is the true-positive ratio (TP/ TP1FN) divided by the false-positive ratio (FP/ FP1TN). Tests with high likelihood ratios are better than those with low likelihood ratios.2 By observing the decision matrix that is used to calculate the sensitivity, specificity, predictive value positive, predictive value negative, accuracy, and likelihood ratio, it is obvious that the data changes simply by arbitrarily selecting different cut-off value to discriminate the test positive versus the test negative. Therefore the overall assessment of the test can be better made with a receiver operating characteristic (ROC) analysis.3 By changing the cut-off value for a positive (or negative) test several times, several different pairs of false-positive ratio
Han and Kim 539
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table I. Decision matrix Presence of Disease or Condition (e.g., Class II Skeletal Pattern) Present Test Result (e.g., ANB angle)
Absent
Total
Abnormal
TP
FP
TP 1 FP
Normal
FN
TN
FN 1 TN
Total
TP 1 FN
FP 1 TN
Table II. Sample cases N Normal Occlusion Class I Malocclusion Class II Malocclusion Class III Malocclusion Total
102 214 624 36 976
(1-specificity) and true-positive ratio (sensitivity) can be obtained. These pairs are plotted as the X and Y coordinate values on the ROC curve (Fig. 1). Both axes of the graph range in value from 0 to 1. The further upward and to the left this curve lies in the diagram, the better and more diagnostic is the test. This curve is called the ROC curve of a diagnostic test because it describes the inherent detection characteristics of the test and because the receiver of the test information can operate at any point of the curve by using an appropriate cut-off value. Determination of the area under the ROC curve provides a quantitative assessment of a test’s diagnostic performance and allows the comparison of diagnostic ability of different tests.4,5 Since Lee Lusted introduced the concept of ROC curves to medicine in 1971, the use of ROC curves in medical research has become prominent.6-8 They have been used extensively in the field of diagnostic radiology.9.10 Researchers in psychology have used the ROC analysis to study the discrimination behavior by isolating the effects of the observer response bias.11 In dentistry, the ROC analysis has been proposed as the method to evaluate the efficacy of various types of dental radiographs in detecting dental caries and periodontal diseases.12,13 In orthodontics, Wardlaw et al.14 used ROC analysis to evaluate the relationship between several cephalometric measurements and the presence of anterior openbite. They found that the Overbite Depth Indicator15 had the highest diagnostic value in discriminating between patients with and without openbite.15 Choi and Chang16 used the ROC analysis to evaluate the ability of various cephalometric measurements to identify patients with Class III malocclusion. They found Wits Ap-
Fig. 1. Typical conventional ROC curve. Area under curve (gray area) is estimated to evaluate diagnostic performance.
praisal to be highly effective in identifying patients with Class III malocclusion. The ROC analysis is an excellent method for evaluating and comparing the performance of diagnostic tests. The purpose of this study was to use the ROC analysis to evaluate the diagnostic ability of several cephalometric measurements in determining the presence of Class II and Class III skeletal patterns. MATERIAL AND METHODS Data collection
The data for this study were obtained from 976 cases selected from the office of one orthodontist. The age of the sample group ranged from 8 years to 17 years with the average of 11 years and 10 months. The sample included 102 cases with normal occlusion, 214 cases with Class I malocclusion, 624 cases with Class II malocclusion, and 36 cases with Class III malocclusion (Table II). The cases were selected after a careful screening of the study models to exclude any cases with early loss of deciduous molars, with an obvious mesial shift of the posterior teeth, or with a significant functional shift of the mandible. Study models were used to determine the molar displacement of each case. The cuspal relationships of the maxillary and mandibular first molars were obtained by direct measurements made on the study models. The molar displacement was read as zero when the mesiobuccal groove of the mandibular first molar occluded with the tip of the mesiobuccal cusp of the maxillary first molar. The molar displacement
540 Han and Kim
was read as negative when the mandibular first molar deviated in the distal direction. The molar displacement was read as positive when the mandibular first molar deviated mesially. The measurement was read to 0.1 mm with Cenco gauge calipers. An average figure of each case was obtained from the right and left side measurements. Fifteen cephalometric measurements pertinent to the anteroposterior skeletal dysplasia were made on the lateral cephalograms. The measurements were read to 0.5 mm for linear measurements and 0.5° for angular measurements. The following cephalometric measurements were obtained for the ROC Analysis: 1. Facial angle: The angle formed by the Frankfort horizontal plane and the facial plane17 2. Angle of convexity17 3. AB plane angle17 4. SNA angle 5. SNB angle 6. ANB angle 7. SN-Pog angle 8. PP-MP (palatal plane to mandibular plane) angle 9. MP-SN (mandibular plane to sella-nasion) angle 10. MP-FH (mandibular plane to Frankfort Horizontal) angle 11. Anteroposterior dysplasia indicator (APDI): The combined reading of the facial angle 6 AB plane angle 6 palatal plane angle. The palatal plane angle is formed by the palatal plane and the FH plane. This angle is positive when the palatal plane slopes downward anteriorly relative to the FH plane and is negative when the palatal plane slopes upward anteriorly relative to the FH plane.18 12. Gonial angle 13. Y-axis 14. Wits appraisal 15. Overjet ROC analysis
ROC curve plots the true-positive ratio versus false-positive ratio while the cut-off value for a positive (or negative) result is varied (Fig. 1). Determination of the area under the ROC curve provides a quantitative assessment of a test’s diagnostic performance. Evaluating the areas under ROC curves of different diagnostic tests allows comparison of their diagnostic ability.
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
The conventional least-squares curve fitting procedures are inappropriate for ROC curves because both axes represent dependent variables and both are subject to error. Several investigators proposed a parametric curve-fitting technique for ROC curves using maximum-likelihood estimation. This estimation is based on the distributional assumption of binormality in the normal and abnormal populations. This method can estimate the area under a curve and the standard error of that area.19-21 More recently, a nonparametric method has been proposed to estimate the area under ROC curves. This method calculates the area with the trapezoidal rule and is equivalent to Mann-Whitney U-Statistic.22,23 This method also allows calculation of the area and its standard error. It has been known that graphically the better test (of the two) appears closer to the upper left hand corner of the ROC graph. If the area under the curve and its standard error can be measured, a statistical comparison of the two curves can be made. Hanley and McNeil24,25 and McClish26 proposed that given noncorrelated cases, one can simply divide the differences in areas under two ROC curves by the standard error of the area difference to obtain the statistical comparison. The task of measuring the areas under ROC curves and statistically comparing different curves requires computational assistance. Several computer software programs have recently become available facilitating the calculations. The availability of these programs has contributed to the broader acceptance and wider usage of ROC analysis in medical research. For this study, ROC ANALYZER developed by Centor et al.27,28 was used. The ROC ANALYZER allows computation of the area using the parametric maximum-likelihood estimation method or the nonparametric method. The program also performs the statistical comparison of two curves with the methods proposed by Hanley and McNeil.24 The cephalometric measurements obtained for this study represent a set of continuous data. In order to perform ROC analysis with the ROC ANALYZER, these continuous data were converted to categorical data. Ogilvie and Creelman19 said that there is usually little advantage in having many points on the ROC curve. Five or six wellseparated points with suitable sample size should give adequate precision for estimating the area under ROC curves. Six categories were selected using the mean and the standard deviation of each cephalometric measurement.
Han and Kim 541
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table III. Sample rating category data for cephalometric measurement: ANB angle and condition: Class II malocclusion Condition
ANB Angle
Absent (Normals)
Table IV. Sample rating category data for cephalometric measurement: ANB angle and condition: Class III malocclusion Condition
Present (Abnormals) ANB Angle
3 # 20.5 20.4 # 3 2.1 # 3 4.6 # 3 7.1 # 3 9.6 # 3 Total
# # # #
2.0 4.5 7.0 9.5
17 108 139 84 4 0 352
0 56 211 257 92 8 624
The cases were grouped initially on the basis of the presence of Class II malocclusion. Of the total 976 cases, 624 cases had Class II malocclusion and 352 cases did not. These groups were used to perform ROC analysis to identify cases with Class II malocclusion. The cases were then regrouped on the basis of the presence of Class III malocclusion. Thirty-six cases had Class III malocclusion and 940 cases did not. These latter groups were used to perform ROC analysis to identify cases with Class III malocclusion. After the cases were grouped according to the presence of the Class II or Class III malocclusion, and the continuous data were converted to categorical data, sample rating category data tables were constructed for each cephalometric measurement and for each type of malocclusion being tested (Tables III and IV). These tables were then used to enter the data into the ROC ANALYZER program. The ROC curves for Class II and Class III malocclusions were constructed for each of the cephalometric measurements with both parametric and nonparametric methods. Statistical comparisons were then made for the areas under ROC curves of different cephalometric measurements. RESULTS
Table V presents the mean, the standard deviation, the maximum and minimum values on age, the molar displacement, and the cephalometric measurements on 976 cases. The sample included 624 (64%) cases with Class II malocclusion (Table II). Therefore, the mean values on SN-Pog angle, facial angle, convexity, AB plane angle, APDI, SNB, ANB, Wits Appraisal, and overjet showed a tendency toward a Class II pattern. The results of the ROC analysis on cephalometric measurements to determine their ability to diagnose Class II malocclusion are shown in Table VI for the parametric method and in Table VII for the
9.6 # 3 7.1 # 3 4.6 # 3 2.1 # 3 20.4 # 3 3 # 20.5 Total
# # # #
9.5 7.0 4.5 2.0
Absent (Normals)
Present (Abnormals)
8 96 341 343 143 9 940
0 0 0 7 21 8 36
nonparametric method. The areas and standard errors are presented in descending order of diagnostic performance. Overall, the nonparametric method yielded lower area estimates than the parametric maximum-likelihood estimation method. These differences, however, were generally small. APDI had the highest area value, and therefore the best diagnostic performance with both the parametric and nonparametric methods. Wits Appraisal and Overjet have also been shown to be highly effective. AB plane, ANB, convexity and SNB angles performed in the middle of the range. SN-Pog, facial and SNA angles showed poor diagnostic performance. Y-axis, gonial, palatal, and mandibular plane angles performed even worse. Two-way comparisons of the areas were performed with the area and standard error values obtained from the nonparametric method. Because APDI had the highest area value, all the other cephalometric measurements were compared against APDI for statistical significance. Two-way comparisons with APDI showed the differences in areas for Wits Appraisal and Overjet were not statistically significant. Two-way comparisons with APDI showed that the differences in areas were significant for AB Plane angle, ANB, Convexity, SNB, SN-Pog, Facial angle, SNA, Gonial angle, Y-axis, PP-MP angle, MP-SN angle and MP-FH angle. The results of the ROC analysis on cephalometric measurements to determine their ability to diagnose Class III malocclusion are shown in Table VIII for the parametric method and in Table IX for the nonparametric method. The areas and standard errors are presented in descending order of diagnostic performance. Overall, the nonparametric method again yielded slightly lower area estimates than the parametric maximum-likelihood estimation method. APDI again had the highest area value, and
542 Han and Kim
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
Table V. Descriptive statistics (N 5 976) Variables
Mean
S.D.
Maximum
Minimum
Age (years, months) Molar displacement (mm) SN-Pog (Deg) Facial angle (Deg) Convexity (Deg) AB Plane angle (Deg) APDI (Deg) SNA (Deg) SNB (Deg) ANB (Deg) Y-Axis (Deg) PP-MP (Deg) MP-SN (Deg) MP-FH (Deg) Gonial angle (Deg) Wits appraisal (mm) Overjet (mm)
11.10 23.00 77.96 86.00 7.71 27.63 77.49 81.48 77.05 4.45 59.83 27.19 34.49 26.35 126.13 1.81 5.75
1.50 2.09 3.59 3.89 5.48 3.59 5.49 3.55 3.24 2.39 3.41 5.48 5.43 5.11 6.07 3.63 2.75
17.00 7.50 94.00 96.50 24.00 9.50 107.00 92.00 90.00 13.00 69.50 46.00 53.50 43.00 146.00 12.00 15.00
8.00 27.60 69.00 78.00 217.00 220.00 62.00 71.00 66.00 26.00 48.00 9.00 14.50 9.00 106.00 211.00 25.00
Table VI. ROC analysis with parametric method for determination of Class II malocclusion
Table VII. ROC analysis with nonparametric method for determination of Class II malocclusion
Cephalometric measurements
Areas and standard errors
Cephalometric measurements
Areas and standard errors
Two-way comparison with APDI
APDI Wits Appraisal Overjet AB Plane Angle ANB Convexity SNB SN-Pog Facial Angle SNA Y-Axis Gonial Angle PP-MP MP-SN MP-FH
0.829 (e 5 0.014) 0.812 (e 5 0.015) 0.805 (e 5 0.015) 0.774 (e 5 0.016) 0.752 (e 5 0.017) 0.722 (e 5 0.017) 0.645 (e 5 0.019) 0.622 (e 5 0.020) 0.621 (e 5 0.020) 0.576 (e 5 0.051) 0.548 (e 5 0.020) 0.543 (e 5 0.020) 0.539 (e 5 0.020) 0.486 (e 5 0.020) 0.464 (e 5 0.020)
APDI Wits Appraisal Overjet AB Plane Angle ANB Convexity SNB SN-Pog Facial Angle SNA Gonial Angle Y-Axis PP-MP MP-SN MP-FH
0.798 (e 5 0.014) 0.787 (e 5 0.015) 0.782 (e 5 0.015) 0.747 (e 5 0.016) 0.722 (e 5 0.017) 0.701 (e 5 0.017) 0.636 (e 5 0.019) 0.608 (e 5 0.019) 0.601 (e 5 0.019) 0.560 (e 5 0.052) 0.542 (e 5 0.019) 0.541 (e 5 0.020) 0.531 (e 5 0.019) 0.480 (e 5 0.019) 0.466 (e 5 0.019)
* * † † † † † † † † † † † †
therefore the best diagnostic performance with the use of both the parametric and nonparametric methods. Wits Appraisal, convexity, AB plane angle, ANB, and overjet have also been shown to be highly effective. Y-axis, SNB, facial, and SN-Pog angle performed in the middle of the range. Gonial, SNA, palatal, and mandibular plane angles showed poor diagnostic performance. Two-way comparisons of the areas were performed with the area and standard error values obtained from the nonparametric method. Because APDI had the highest area value, all the other cephalometric measurements were compared against APDI for statistical significance. Two-way comparisons with APDI showed that the differences in areas for Wits Appraisal, convexity, AB plane angle, ANB and overjet were not statistically significant. Two-way comparisons with APDI showed that
*p . 0.05: A two-way comparison with APDI has determined that the observed difference between the areas can occur by chance. This p-value greater than 0.05 indicates that the difference in areas can be due to chance alone. †p , 0.05: A two-way comparison with APDI has determined that the difference is statistically significant and cannot occur by chance.
the differences in areas were significant for Y-axis, SNB, facial angle, SN-Pog, gonial angle, SNA, PP-MP angle, MP-SN angle, and MP-FH angle. DISCUSSION
Previous studies investigating the relationship between cephalometric measurements and skeletal patterns depended on statistical methods such as mean and standard deviation values of sample groups, t tests, and correlation coefficient analysis. In addition, many cephalometric analyses are proposed based on clinical impression or tradition. Sometimes they were biased to fit the clinician’s
Han and Kim 543
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table VIII. ROC analysis with parametric method determination of Class III malocclusion
Table IX. ROC analysis with nonparametric method determination of Class III malocclusion
Cephalometric measurements
Areas and standard errors
Cephalometric measurements
Areas and standard errors
Two-way comparison with APDI
APDI Wits Appraisal Convexity AB Plane Angle ANB Overjet Y-Axis SNB Facial Angle SN-Pog Gonial Angle SNA PP-MP MP-SN MP-FH
0.937 (e 5 0.018) 0.931 (e 5 0.018) 0.925 (e 5 0.020) 0.920 (e 5 0.022) 0.912 (e 5 0.022) 0.897 (e 5 0.028) 0.789 (e 5 0.045) 0.754 (e 5 0.042) 0.751 (e 5 0.046) 0.717 (e 5 0.050) 0.665 (e 5 0.048) 0.576 (e 5 0.051) 0.476 (e 5 0.050) 0.441 (e 5 0.049) 0.392 (e 5 0.053)
APDI Wits Appraisal Convexity AB Plane Angle Overjet ANB Y-Axis SNB Facial Angle SN-Pog Gonial Angle SNA PP-MP MP-SN MP-FH
0.919 (e 5 0.018) 0.911 (e 5 0.020) 0.901 (e 5 0.021) 0.897 (e 5 0.024) 0.896 (e 5 0.025) 0.882 (e 5 0.023) 0.767 (e 5 0.045) 0.734 (e 5 0.041) 0.723 (e 5 0.045) 0.694 (e 5 0.047) 0.649 (e 5 0.048) 0.560 (e 5 0.052) 0.471 (e 5 0.046) 0.445 (e 5 0.045) 0.395 (e 5 0.048)
* * * * * † † † † † † † † †
treatment mechanics. Receiver Operating Characteristic analysis represents the “state of the art” statistical approach to evaluating the diagnostic performance of cephalometric measurements in determining skeletal patterns. ROC analysis is a superior form of evaluation because it analyzes the diagnostic performance for the full range of cut-off points, and thus eliminates the bias resulting from selection of a single value. The area under the ROC curve provides information about the general “goodness” of a test, not the interpretation of a test result. A perfect test has an area 1.0 whereas a noninformative test has an area of 0.5 or less. The results of the ROC analysis showed the Anteroposterior Dysplasia Indicator (APDI) had the highest diagnostic performance in determining Class II and Class III malocclusions. APDI is the combined angle of facial, AB plane and palatal plane angles and was based on a statistical analysis of cephalometric relationships. As a combination of three features in the orofacial complex, the APDI reflects both skeletal and dentoalveolar characteristics that cannot be reflected by single measurements. According to an earlier study,18 the mean APDI values were 81.4 for the normal sample group (n 5 102), 75.2 for the Class II malocclusion group (n 5 624) and 88.5 for the Class III malocclusion group (n 5 36). The Wits appraisal performed second highest to the APDI according to the ROC analysis. The Wits appraisal was introduced in 1975 by Jacobson and is the distance on the occlusal plane between the contact points of perpendicular lines from A and B points.29 This indicator, however, has been shown to be affected by the great variations in the occlusal plane and the vertical positions of A and B points.30
*p . 0.05: A two-way comparison with APDI has determined that the observed difference between the areas can occur by chance. This p-value greater than 0.05 indicates that the difference in areas can be due to chance alone. †p , 0.05: A two-way comparison with APDI has determined that the difference is statistically significant and cannot occur by chance.
ANB angle is probably the most commonly used measurement of a sagittal skeletal discrepancy. Riedel31 introduced it in 1950 along with SNA and SNB angles. Several studies have claimed that the ANB angle is affected by several environmental factors.32-34 The ROC analysis showed ANB angle performed worse diagnostically than APDI, Wits Appraisal and AB plane angle. SNA and SNB angles performed even worse than ANB angle in determining Class II and Class III malocclusions. The results of this study showed that the nonparametric method yielded slightly lower area estimates than the parametric method. This coincided with the results of the study by Centor and Schwartz.5 They compared the two methods of estimating the area under an ROC curve and the standard error of the area under an ROC curve. They found that for most ROC curves, the two methods yielded similar results. The nonparametric method tended to underestimate the ROC curve area slightly compared with a fitted “binormal area” of the parametric method. But these differences were not thought to be clinically significant for ROC curves that have reasonable numbers of cut-off points that are well separated. The ROC analysis for Class III malocclusion produced higher area estimates than the analysis for Class II malocclusion for most of the cephalometric measurements using both parametric and nonparametric methods. The sample for this study included
544 Han and Kim
624 (64%) cases with Class II malocclusion and 36 (4%) cases with Class III malocclusion. Because the ROC curve plots true-positive ratio versus false-positive ratio, it is completely independent of disease prevalence.3 The differences in area estimates between Class II and Class III groups are not due to the differences in prevalence but due to the severity of the cases selected. The Class II malocclusion group included many borderline cases, whereas Class III malocclusion group included more clear-cut severe cases. All the analytical methods used to evaluate the diagnostic performance of a test depend on a comparison with a “gold standard” or the “truth.” In medicine, if a patient has a suspicious lesion on a diagnostic radiograph, that patient would probably have a biopsy performed of that lesion. If there is a large enough sample, the results of the biopsy can be used as the “gold standard” to evaluate the efficacy of the radiographs in diagnosing a disease or condition. Browner et al.35 stated that the “truth” was only a matter of degree or acceptability. Newhauser and Yin36 said that the “truth” was really the “best there is.” At the present time, there is not a single diagnostic test or cephalometric measurement that has been accepted to be used as the “gold standard” for defining Class II or Class III skeletal patterns. In this study, molar displacement that was measured on the study models was used to classify the malocclusion of each case. These classifications were then used to group cases according to malocclusion for ROC analysis. Perhaps the following assumptions can be made: Class II skeletal patterns tend to yield Class II malocclusions and likewise, Class III skeletal patterns tend to yield Class III malocclusions. Therefore molar displacement measured on study models is perhaps closest to the “truth” or the “best there is” and can be used to evaluate the diagnostic performance of cephalometric measurements in determining Class II and Class III skeletal patterns. There are times when a patient’s dental occlusion does not reflect the actual skeletal pattern. For example, a Class I molar relationship may be present in a patient with Class II skeletal pattern. This may result from shifting of lower molars or mesial inclination of the entire lower dentition. During orthodontic treatment, uprighting of lower molars can immediately transform the molar relationship to Class II. The same phenomenon can occur with a patient who has a Class I molar relationship but possesses a Class III skeletal pattern. What may appear to be an easily treatable Class I malocclusion may shift to a full blown Class III malocclusion during treatment.
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
A differential diagnosis of the skeletal pattern as well as the malocclusion is important in planning a treatment. If a dental relationship were always in agreement with the skeletal pattern, a cephalometric analysis of anteroposterior skeletal discrepancy would be totally unnecessary. This is not true in reality often creating much confusion and frustration during the planning and the treatment process. In general, if an assumption can be made that Class II skeletal pattern produces Class II malocclusion and likewise, Class III skeletal pattern produces Class III malocclusion, then the molar displacement can be used as the “gold standard” in evaluating the diagnostic performance of cephalometric measurements in determining the anteroposterior skeletal discrepancy. The results of the ROC analysis showed that the APDI was the most effective parameter in diagnosing Class II and Class III malocclusions. Therefore, applying the aforementioned assumption, APDI had the best diagnostic performance in determining anteroposterior discrepancy of the skeletal pattern. SUMMARY AND CONCLUSIONS
This study used the ROC analysis to evaluate the diagnostic performance of 15 cephalometric measurements pertinent to the anteroposterior skeletal dysplasia in determiming the presence of Class II and Class III skeletal patterns. The ROC analysis is the “state of the art” statistical approach to evaluating the diagnostic performance for the full range of cut-off values. The results of the study showed: 1. Wits appraisal and overjet were effective in diagnosing Class II malocclusion. 2. Wits appraisal, convexity, AB plane angle, ANB, and overjet were effective in diagnosing Class III malocclusion. 3. APDI was the most effective parameter in diagnosing Class II and Class III malocclusions. Therefore, applying the previously mentioned assumption, APDI had the best diagnostic performance in determining anteroposterior discrepancy of the skeletal pattern. REFERENCES 1. Weinstein MC, Fineberg HV. Clinical decision analysis. Philadelphia: W.B. Saunders; 1980. p. 37-167. 2. Douglass CW. Evaluating diagnostic tests. Adv Dent Res 1993;7:66-9. 3. Metz CE. Basic principles of ROC analysis. Semin Nucl Med 1978;8:283-98. 4. DeLong EB, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating charactenstic curves: a nonparametric approach. Biometrics 1988;44:837-45. 5. Centor RM, Schwartz JS. An evaluation of methods for estimating the area under the Receiver Operating Characteristic (ROC) curve. Med Decis Making 1985;5:149-56.
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5 6. Lusted LB. Signal detectability and medical decision-making. Science 1971;171: 1217-9. 7. Centor RM. Signal detectability: The use of ROC curves and their analyses. Med Decis Making 1991;11:102-6. 8. Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in the emergency room. Med Decis Making 1981;1:239-46. 9. Berbaum KS, Franken EA Jr, Doffman DD, Barloon TJ. Influence of clinical history upon detection of nodules and other lesions. Invest Radiol 1988;23:48-55. 10. Berbaum KS, Dorfman DD, Franken EA Jr. Measuring observer performance by ROC analysis: indications and complications. Invest Radiol 1989;24:228-33. 11. Swets JA. The relative operating characteristic in psychology. Science 1973;182: 990-1000. 12. Berkey CS, Douglass CW, Valachovic RW, Chauncey HH, McNeil BJ. Statistical methods for comparing dental diagnostic procedures. Community Dent Oral Epidemiol 1990;18:169-76. 13. Douglass CW, McNeil BJ. Clinical decision analysis methods applied to diagnostic tests in dentistry. J Dent Edu 1983;47:708-12. 14. Wardlaw DW, Smith RJ, Hertweck DW, Hildebolt CF. Cephalometrics of anterior open bite: a receiver operating characteristic (ROC) analysis. Am J Orthod 1992;101:234-43. 15. Kim YH. Overbite Depth Indicator with particular reference to anterior open bite. Am J Orthod 1974;65:586-611. 16. Choi HY, Chang YI. Comparative study on the horizontal measurements of skeletal Class III malocdusion using the ROC analysis. Korea J Orthod 1995;25: 153-63. 17. Downs WB. Variations in facial relationships: their significance in treatment and prognosis. Am J Orthod 1948;34:812-40. 18. Kim YH, Vietas JJ. Anteroposterior dysplasia indicator: An adjunct to cephalometric differential diagnosis. Am J Orthod 1978;73:619-33. 19. Ogilvie JC, Creelman CD. Maximum-likelihood estimation of ROC curve parameters. J Math Psych 1968;5:377-91. 20. Dorfman DD, Alf E Jr. Maximum-likelihood estimation of parameters of signaldetection theory and determination of confidence intervals-rating method data. J Math Psych 1969;6:487-96.
Han and Kim 545 21. Grey DR, Morgan BJT. Some aspects of ROC curve-fitting: normal and logistic models. J Math Psych 1972;9:128-39. 22. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psych 1975;12:387-415. 23. Hanley JD, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36. 24. Hanley JD, McNeil BJ. The method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148: 839-43. 25. McNeil BJ, Hanley JA. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making 1984;4:137-50. 26. McClish DK. Comparing the areas under more than two independent ROC curves. Med Decis Making 1987;7:149-55. 27. Centor RM. Visicalc program for estimating the area under a receiver operating characteristic (ROC) curve. Med Decis Making 1985;5:139-48. 28. Centor RM, Keightley GE. Receiver operating characteristic (ROC) curve area analysis using the ROC ANALYZER. SCAMC Proceedings 1989;13:222-6. 29. Jacobson A. The “Wits” appraisal of jaw disharmony. Am J Orthod 1975;67: 125-38. 30. Chang HP. Assessment of anteroposterior jaw relationship. Am J Orthod Dentofacial Orthop 1987;92:117-22. 31. Riedel RA. Esthetics and its relation to orthodontic therapy. Angle Orthod 1950;20:168-78. 32. Ferrazzini G. Critical evaluation of the ANB angle. Am J Orthod 1976;69:620-6. 33. Hussels W, Nanda RS. Analysis of factors affecting angle ANB. Am J Orthod 1984;85:620-6. 34. Freeman RS. Adjusting A-N-B angles to reflect the effect of maxillary position. Angle Orthod 1981;51:162-71. 35. Browner WS, Newman TB, Cummings SR. Designing a new study: diagnostic tests. In: Hulley SB, Cummings SR, editors. Designing clinical research. Baltimore (MD): Williams and Wilkins, 1988. p. 87. 36. Newhauser D, Yin XP. Deciding whether a new test measure is useful. Medical Care 1991;29:685-9.
Determination of Class II and Class III skeletal patterns: Receiver operating characteristic (ROC) analysis on various cephalometric measurements Unae Kim Han, DMD, MPH, MS, and Young H. Kim, DMD, MS Lexington and Weston, Mass. Receiver operating characteristic analysis is an excellent method for evaluating and comparing the performance of diagnostic tests. The purpose of this study was to use the receiver operating characteristic analysis to evaluate the diagnostic ability of several cephalometric measurements in determining the presence of Class II and Class III skeletal patterns. Receiver operating characteristic analysis was performed on 976 cases. Fifteen cephalometric measurements were evaluated. A computer software program ROC ANALYZER was used to tabulate the areas under the curves and to perform the statistical comparison between the curves. The results of this study indicated that the Anteroposterior Dysplasia Indicator had the best diagnostic ability in identifying cases with Class II and Class III skeletal patterns. WITS Appraisal and Overjet were highly effective in diagnosing cases with Class II skeletal pattern. WITS Appraisal, Convexity, AB Plane Angle and Overjet also performed well in diagnosing cases with Class III skeletal pattern. (Am J Ortho Dentofacial Orthop 1998;113:538-45.)
S
ince its introduction, lateral cephalometric radiographs have been widely used in the field of orthodontics to diagnose craniofacial anomalies, to determine a diagnosis and treatment plan of an individual patient’s malocclusion, to monitor and predict facial growth, and to evaluate treatment effects. The anteroposterior relationship between the maxilla and the mandible is considered the most important diagnostic criterion in orthodontics. Successful treatment planning and analysis of treatment results depend on reliable diagnostic information. Numerous cephalometric measurements and analyses have been proposed to determine the skeletal pattern of individual patients causing much confusion and controversy. It seems appropriate to ask the following question: How can we measure the quality of diagnostic information provided by cephalometric measurements in determining the skeletal pattern of an individual patient? Efficacy of a cephalometric measurement, e.g., ANB angle, in diagnosing Class II or Class III skeletal pattern can be characterized by its sensitivity, specificity, predictive value positive, predictive value negative, accuracy, and likelihood ratio.1 Methods to calculate these measures can be illustrated with the 2 3 2 decision matrix (Table I). In private practice. Reprint requests to: Unae Kim Han, 789 Massachusetts Ave., Lexington, MA 02173 Copyright © 1998 by the American Association of Orthodontists. 0889-5406/98/$5.00 1 0 8/1/84775
538
The sensitivity is the true-positive ratio and is calculated as (TP/TP1FN). This measures the proportion of diseased patients correctly identified as positive. The specificity is the true-negative ratio and is calculated as (TN/FP1TN). This measures the proportion of disease-free patients correctly identified as negative. The predictive value positive is the probability that a patient with a positive result actually has the disease and is calculated as (TP/TP1FP). The predictive value negative is the probability that a patient with a negative test result actually does not have the disease and is calculated as (TN/TN1FN). The accuracy of a diagnostic measure is the sum of the true-positive and true-negative outcomes divided by all outcomes. This measures the fraction of patients for which the diagnostic test is correct. The likelihood ratio is the true-positive ratio (TP/ TP1FN) divided by the false-positive ratio (FP/ FP1TN). Tests with high likelihood ratios are better than those with low likelihood ratios.2 By observing the decision matrix that is used to calculate the sensitivity, specificity, predictive value positive, predictive value negative, accuracy, and likelihood ratio, it is obvious that the data changes simply by arbitrarily selecting different cut-off value to discriminate the test positive versus the test negative. Therefore the overall assessment of the test can be better made with a receiver operating characteristic (ROC) analysis.3 By changing the cut-off value for a positive (or negative) test several times, several different pairs of false-positive ratio
Han and Kim 539
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table I. Decision matrix Presence of Disease or Condition (e.g., Class II Skeletal Pattern) Present Test Result (e.g., ANB angle)
Absent
Total
Abnormal
TP
FP
TP 1 FP
Normal
FN
TN
FN 1 TN
Total
TP 1 FN
FP 1 TN
Table II. Sample cases N Normal Occlusion Class I Malocclusion Class II Malocclusion Class III Malocclusion Total
102 214 624 36 976
(1-specificity) and true-positive ratio (sensitivity) can be obtained. These pairs are plotted as the X and Y coordinate values on the ROC curve (Fig. 1). Both axes of the graph range in value from 0 to 1. The further upward and to the left this curve lies in the diagram, the better and more diagnostic is the test. This curve is called the ROC curve of a diagnostic test because it describes the inherent detection characteristics of the test and because the receiver of the test information can operate at any point of the curve by using an appropriate cut-off value. Determination of the area under the ROC curve provides a quantitative assessment of a test’s diagnostic performance and allows the comparison of diagnostic ability of different tests.4,5 Since Lee Lusted introduced the concept of ROC curves to medicine in 1971, the use of ROC curves in medical research has become prominent.6-8 They have been used extensively in the field of diagnostic radiology.9.10 Researchers in psychology have used the ROC analysis to study the discrimination behavior by isolating the effects of the observer response bias.11 In dentistry, the ROC analysis has been proposed as the method to evaluate the efficacy of various types of dental radiographs in detecting dental caries and periodontal diseases.12,13 In orthodontics, Wardlaw et al.14 used ROC analysis to evaluate the relationship between several cephalometric measurements and the presence of anterior openbite. They found that the Overbite Depth Indicator15 had the highest diagnostic value in discriminating between patients with and without openbite.15 Choi and Chang16 used the ROC analysis to evaluate the ability of various cephalometric measurements to identify patients with Class III malocclusion. They found Wits Ap-
Fig. 1. Typical conventional ROC curve. Area under curve (gray area) is estimated to evaluate diagnostic performance.
praisal to be highly effective in identifying patients with Class III malocclusion. The ROC analysis is an excellent method for evaluating and comparing the performance of diagnostic tests. The purpose of this study was to use the ROC analysis to evaluate the diagnostic ability of several cephalometric measurements in determining the presence of Class II and Class III skeletal patterns. MATERIAL AND METHODS Data collection
The data for this study were obtained from 976 cases selected from the office of one orthodontist. The age of the sample group ranged from 8 years to 17 years with the average of 11 years and 10 months. The sample included 102 cases with normal occlusion, 214 cases with Class I malocclusion, 624 cases with Class II malocclusion, and 36 cases with Class III malocclusion (Table II). The cases were selected after a careful screening of the study models to exclude any cases with early loss of deciduous molars, with an obvious mesial shift of the posterior teeth, or with a significant functional shift of the mandible. Study models were used to determine the molar displacement of each case. The cuspal relationships of the maxillary and mandibular first molars were obtained by direct measurements made on the study models. The molar displacement was read as zero when the mesiobuccal groove of the mandibular first molar occluded with the tip of the mesiobuccal cusp of the maxillary first molar. The molar displacement
540 Han and Kim
was read as negative when the mandibular first molar deviated in the distal direction. The molar displacement was read as positive when the mandibular first molar deviated mesially. The measurement was read to 0.1 mm with Cenco gauge calipers. An average figure of each case was obtained from the right and left side measurements. Fifteen cephalometric measurements pertinent to the anteroposterior skeletal dysplasia were made on the lateral cephalograms. The measurements were read to 0.5 mm for linear measurements and 0.5° for angular measurements. The following cephalometric measurements were obtained for the ROC Analysis: 1. Facial angle: The angle formed by the Frankfort horizontal plane and the facial plane17 2. Angle of convexity17 3. AB plane angle17 4. SNA angle 5. SNB angle 6. ANB angle 7. SN-Pog angle 8. PP-MP (palatal plane to mandibular plane) angle 9. MP-SN (mandibular plane to sella-nasion) angle 10. MP-FH (mandibular plane to Frankfort Horizontal) angle 11. Anteroposterior dysplasia indicator (APDI): The combined reading of the facial angle 6 AB plane angle 6 palatal plane angle. The palatal plane angle is formed by the palatal plane and the FH plane. This angle is positive when the palatal plane slopes downward anteriorly relative to the FH plane and is negative when the palatal plane slopes upward anteriorly relative to the FH plane.18 12. Gonial angle 13. Y-axis 14. Wits appraisal 15. Overjet ROC analysis
ROC curve plots the true-positive ratio versus false-positive ratio while the cut-off value for a positive (or negative) result is varied (Fig. 1). Determination of the area under the ROC curve provides a quantitative assessment of a test’s diagnostic performance. Evaluating the areas under ROC curves of different diagnostic tests allows comparison of their diagnostic ability.
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
The conventional least-squares curve fitting procedures are inappropriate for ROC curves because both axes represent dependent variables and both are subject to error. Several investigators proposed a parametric curve-fitting technique for ROC curves using maximum-likelihood estimation. This estimation is based on the distributional assumption of binormality in the normal and abnormal populations. This method can estimate the area under a curve and the standard error of that area.19-21 More recently, a nonparametric method has been proposed to estimate the area under ROC curves. This method calculates the area with the trapezoidal rule and is equivalent to Mann-Whitney U-Statistic.22,23 This method also allows calculation of the area and its standard error. It has been known that graphically the better test (of the two) appears closer to the upper left hand corner of the ROC graph. If the area under the curve and its standard error can be measured, a statistical comparison of the two curves can be made. Hanley and McNeil24,25 and McClish26 proposed that given noncorrelated cases, one can simply divide the differences in areas under two ROC curves by the standard error of the area difference to obtain the statistical comparison. The task of measuring the areas under ROC curves and statistically comparing different curves requires computational assistance. Several computer software programs have recently become available facilitating the calculations. The availability of these programs has contributed to the broader acceptance and wider usage of ROC analysis in medical research. For this study, ROC ANALYZER developed by Centor et al.27,28 was used. The ROC ANALYZER allows computation of the area using the parametric maximum-likelihood estimation method or the nonparametric method. The program also performs the statistical comparison of two curves with the methods proposed by Hanley and McNeil.24 The cephalometric measurements obtained for this study represent a set of continuous data. In order to perform ROC analysis with the ROC ANALYZER, these continuous data were converted to categorical data. Ogilvie and Creelman19 said that there is usually little advantage in having many points on the ROC curve. Five or six wellseparated points with suitable sample size should give adequate precision for estimating the area under ROC curves. Six categories were selected using the mean and the standard deviation of each cephalometric measurement.
Han and Kim 541
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table III. Sample rating category data for cephalometric measurement: ANB angle and condition: Class II malocclusion Condition
ANB Angle
Absent (Normals)
Table IV. Sample rating category data for cephalometric measurement: ANB angle and condition: Class III malocclusion Condition
Present (Abnormals) ANB Angle
3 # 20.5 20.4 # 3 2.1 # 3 4.6 # 3 7.1 # 3 9.6 # 3 Total
# # # #
2.0 4.5 7.0 9.5
17 108 139 84 4 0 352
0 56 211 257 92 8 624
The cases were grouped initially on the basis of the presence of Class II malocclusion. Of the total 976 cases, 624 cases had Class II malocclusion and 352 cases did not. These groups were used to perform ROC analysis to identify cases with Class II malocclusion. The cases were then regrouped on the basis of the presence of Class III malocclusion. Thirty-six cases had Class III malocclusion and 940 cases did not. These latter groups were used to perform ROC analysis to identify cases with Class III malocclusion. After the cases were grouped according to the presence of the Class II or Class III malocclusion, and the continuous data were converted to categorical data, sample rating category data tables were constructed for each cephalometric measurement and for each type of malocclusion being tested (Tables III and IV). These tables were then used to enter the data into the ROC ANALYZER program. The ROC curves for Class II and Class III malocclusions were constructed for each of the cephalometric measurements with both parametric and nonparametric methods. Statistical comparisons were then made for the areas under ROC curves of different cephalometric measurements. RESULTS
Table V presents the mean, the standard deviation, the maximum and minimum values on age, the molar displacement, and the cephalometric measurements on 976 cases. The sample included 624 (64%) cases with Class II malocclusion (Table II). Therefore, the mean values on SN-Pog angle, facial angle, convexity, AB plane angle, APDI, SNB, ANB, Wits Appraisal, and overjet showed a tendency toward a Class II pattern. The results of the ROC analysis on cephalometric measurements to determine their ability to diagnose Class II malocclusion are shown in Table VI for the parametric method and in Table VII for the
9.6 # 3 7.1 # 3 4.6 # 3 2.1 # 3 20.4 # 3 3 # 20.5 Total
# # # #
9.5 7.0 4.5 2.0
Absent (Normals)
Present (Abnormals)
8 96 341 343 143 9 940
0 0 0 7 21 8 36
nonparametric method. The areas and standard errors are presented in descending order of diagnostic performance. Overall, the nonparametric method yielded lower area estimates than the parametric maximum-likelihood estimation method. These differences, however, were generally small. APDI had the highest area value, and therefore the best diagnostic performance with both the parametric and nonparametric methods. Wits Appraisal and Overjet have also been shown to be highly effective. AB plane, ANB, convexity and SNB angles performed in the middle of the range. SN-Pog, facial and SNA angles showed poor diagnostic performance. Y-axis, gonial, palatal, and mandibular plane angles performed even worse. Two-way comparisons of the areas were performed with the area and standard error values obtained from the nonparametric method. Because APDI had the highest area value, all the other cephalometric measurements were compared against APDI for statistical significance. Two-way comparisons with APDI showed the differences in areas for Wits Appraisal and Overjet were not statistically significant. Two-way comparisons with APDI showed that the differences in areas were significant for AB Plane angle, ANB, Convexity, SNB, SN-Pog, Facial angle, SNA, Gonial angle, Y-axis, PP-MP angle, MP-SN angle and MP-FH angle. The results of the ROC analysis on cephalometric measurements to determine their ability to diagnose Class III malocclusion are shown in Table VIII for the parametric method and in Table IX for the nonparametric method. The areas and standard errors are presented in descending order of diagnostic performance. Overall, the nonparametric method again yielded slightly lower area estimates than the parametric maximum-likelihood estimation method. APDI again had the highest area value, and
542 Han and Kim
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
Table V. Descriptive statistics (N 5 976) Variables
Mean
S.D.
Maximum
Minimum
Age (years, months) Molar displacement (mm) SN-Pog (Deg) Facial angle (Deg) Convexity (Deg) AB Plane angle (Deg) APDI (Deg) SNA (Deg) SNB (Deg) ANB (Deg) Y-Axis (Deg) PP-MP (Deg) MP-SN (Deg) MP-FH (Deg) Gonial angle (Deg) Wits appraisal (mm) Overjet (mm)
11.10 23.00 77.96 86.00 7.71 27.63 77.49 81.48 77.05 4.45 59.83 27.19 34.49 26.35 126.13 1.81 5.75
1.50 2.09 3.59 3.89 5.48 3.59 5.49 3.55 3.24 2.39 3.41 5.48 5.43 5.11 6.07 3.63 2.75
17.00 7.50 94.00 96.50 24.00 9.50 107.00 92.00 90.00 13.00 69.50 46.00 53.50 43.00 146.00 12.00 15.00
8.00 27.60 69.00 78.00 217.00 220.00 62.00 71.00 66.00 26.00 48.00 9.00 14.50 9.00 106.00 211.00 25.00
Table VI. ROC analysis with parametric method for determination of Class II malocclusion
Table VII. ROC analysis with nonparametric method for determination of Class II malocclusion
Cephalometric measurements
Areas and standard errors
Cephalometric measurements
Areas and standard errors
Two-way comparison with APDI
APDI Wits Appraisal Overjet AB Plane Angle ANB Convexity SNB SN-Pog Facial Angle SNA Y-Axis Gonial Angle PP-MP MP-SN MP-FH
0.829 (e 5 0.014) 0.812 (e 5 0.015) 0.805 (e 5 0.015) 0.774 (e 5 0.016) 0.752 (e 5 0.017) 0.722 (e 5 0.017) 0.645 (e 5 0.019) 0.622 (e 5 0.020) 0.621 (e 5 0.020) 0.576 (e 5 0.051) 0.548 (e 5 0.020) 0.543 (e 5 0.020) 0.539 (e 5 0.020) 0.486 (e 5 0.020) 0.464 (e 5 0.020)
APDI Wits Appraisal Overjet AB Plane Angle ANB Convexity SNB SN-Pog Facial Angle SNA Gonial Angle Y-Axis PP-MP MP-SN MP-FH
0.798 (e 5 0.014) 0.787 (e 5 0.015) 0.782 (e 5 0.015) 0.747 (e 5 0.016) 0.722 (e 5 0.017) 0.701 (e 5 0.017) 0.636 (e 5 0.019) 0.608 (e 5 0.019) 0.601 (e 5 0.019) 0.560 (e 5 0.052) 0.542 (e 5 0.019) 0.541 (e 5 0.020) 0.531 (e 5 0.019) 0.480 (e 5 0.019) 0.466 (e 5 0.019)
* * † † † † † † † † † † † †
therefore the best diagnostic performance with the use of both the parametric and nonparametric methods. Wits Appraisal, convexity, AB plane angle, ANB, and overjet have also been shown to be highly effective. Y-axis, SNB, facial, and SN-Pog angle performed in the middle of the range. Gonial, SNA, palatal, and mandibular plane angles showed poor diagnostic performance. Two-way comparisons of the areas were performed with the area and standard error values obtained from the nonparametric method. Because APDI had the highest area value, all the other cephalometric measurements were compared against APDI for statistical significance. Two-way comparisons with APDI showed that the differences in areas for Wits Appraisal, convexity, AB plane angle, ANB and overjet were not statistically significant. Two-way comparisons with APDI showed that
*p . 0.05: A two-way comparison with APDI has determined that the observed difference between the areas can occur by chance. This p-value greater than 0.05 indicates that the difference in areas can be due to chance alone. †p , 0.05: A two-way comparison with APDI has determined that the difference is statistically significant and cannot occur by chance.
the differences in areas were significant for Y-axis, SNB, facial angle, SN-Pog, gonial angle, SNA, PP-MP angle, MP-SN angle, and MP-FH angle. DISCUSSION
Previous studies investigating the relationship between cephalometric measurements and skeletal patterns depended on statistical methods such as mean and standard deviation values of sample groups, t tests, and correlation coefficient analysis. In addition, many cephalometric analyses are proposed based on clinical impression or tradition. Sometimes they were biased to fit the clinician’s
Han and Kim 543
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5
Table VIII. ROC analysis with parametric method determination of Class III malocclusion
Table IX. ROC analysis with nonparametric method determination of Class III malocclusion
Cephalometric measurements
Areas and standard errors
Cephalometric measurements
Areas and standard errors
Two-way comparison with APDI
APDI Wits Appraisal Convexity AB Plane Angle ANB Overjet Y-Axis SNB Facial Angle SN-Pog Gonial Angle SNA PP-MP MP-SN MP-FH
0.937 (e 5 0.018) 0.931 (e 5 0.018) 0.925 (e 5 0.020) 0.920 (e 5 0.022) 0.912 (e 5 0.022) 0.897 (e 5 0.028) 0.789 (e 5 0.045) 0.754 (e 5 0.042) 0.751 (e 5 0.046) 0.717 (e 5 0.050) 0.665 (e 5 0.048) 0.576 (e 5 0.051) 0.476 (e 5 0.050) 0.441 (e 5 0.049) 0.392 (e 5 0.053)
APDI Wits Appraisal Convexity AB Plane Angle Overjet ANB Y-Axis SNB Facial Angle SN-Pog Gonial Angle SNA PP-MP MP-SN MP-FH
0.919 (e 5 0.018) 0.911 (e 5 0.020) 0.901 (e 5 0.021) 0.897 (e 5 0.024) 0.896 (e 5 0.025) 0.882 (e 5 0.023) 0.767 (e 5 0.045) 0.734 (e 5 0.041) 0.723 (e 5 0.045) 0.694 (e 5 0.047) 0.649 (e 5 0.048) 0.560 (e 5 0.052) 0.471 (e 5 0.046) 0.445 (e 5 0.045) 0.395 (e 5 0.048)
* * * * * † † † † † † † † †
treatment mechanics. Receiver Operating Characteristic analysis represents the “state of the art” statistical approach to evaluating the diagnostic performance of cephalometric measurements in determining skeletal patterns. ROC analysis is a superior form of evaluation because it analyzes the diagnostic performance for the full range of cut-off points, and thus eliminates the bias resulting from selection of a single value. The area under the ROC curve provides information about the general “goodness” of a test, not the interpretation of a test result. A perfect test has an area 1.0 whereas a noninformative test has an area of 0.5 or less. The results of the ROC analysis showed the Anteroposterior Dysplasia Indicator (APDI) had the highest diagnostic performance in determining Class II and Class III malocclusions. APDI is the combined angle of facial, AB plane and palatal plane angles and was based on a statistical analysis of cephalometric relationships. As a combination of three features in the orofacial complex, the APDI reflects both skeletal and dentoalveolar characteristics that cannot be reflected by single measurements. According to an earlier study,18 the mean APDI values were 81.4 for the normal sample group (n 5 102), 75.2 for the Class II malocclusion group (n 5 624) and 88.5 for the Class III malocclusion group (n 5 36). The Wits appraisal performed second highest to the APDI according to the ROC analysis. The Wits appraisal was introduced in 1975 by Jacobson and is the distance on the occlusal plane between the contact points of perpendicular lines from A and B points.29 This indicator, however, has been shown to be affected by the great variations in the occlusal plane and the vertical positions of A and B points.30
*p . 0.05: A two-way comparison with APDI has determined that the observed difference between the areas can occur by chance. This p-value greater than 0.05 indicates that the difference in areas can be due to chance alone. †p , 0.05: A two-way comparison with APDI has determined that the difference is statistically significant and cannot occur by chance.
ANB angle is probably the most commonly used measurement of a sagittal skeletal discrepancy. Riedel31 introduced it in 1950 along with SNA and SNB angles. Several studies have claimed that the ANB angle is affected by several environmental factors.32-34 The ROC analysis showed ANB angle performed worse diagnostically than APDI, Wits Appraisal and AB plane angle. SNA and SNB angles performed even worse than ANB angle in determining Class II and Class III malocclusions. The results of this study showed that the nonparametric method yielded slightly lower area estimates than the parametric method. This coincided with the results of the study by Centor and Schwartz.5 They compared the two methods of estimating the area under an ROC curve and the standard error of the area under an ROC curve. They found that for most ROC curves, the two methods yielded similar results. The nonparametric method tended to underestimate the ROC curve area slightly compared with a fitted “binormal area” of the parametric method. But these differences were not thought to be clinically significant for ROC curves that have reasonable numbers of cut-off points that are well separated. The ROC analysis for Class III malocclusion produced higher area estimates than the analysis for Class II malocclusion for most of the cephalometric measurements using both parametric and nonparametric methods. The sample for this study included
544 Han and Kim
624 (64%) cases with Class II malocclusion and 36 (4%) cases with Class III malocclusion. Because the ROC curve plots true-positive ratio versus false-positive ratio, it is completely independent of disease prevalence.3 The differences in area estimates between Class II and Class III groups are not due to the differences in prevalence but due to the severity of the cases selected. The Class II malocclusion group included many borderline cases, whereas Class III malocclusion group included more clear-cut severe cases. All the analytical methods used to evaluate the diagnostic performance of a test depend on a comparison with a “gold standard” or the “truth.” In medicine, if a patient has a suspicious lesion on a diagnostic radiograph, that patient would probably have a biopsy performed of that lesion. If there is a large enough sample, the results of the biopsy can be used as the “gold standard” to evaluate the efficacy of the radiographs in diagnosing a disease or condition. Browner et al.35 stated that the “truth” was only a matter of degree or acceptability. Newhauser and Yin36 said that the “truth” was really the “best there is.” At the present time, there is not a single diagnostic test or cephalometric measurement that has been accepted to be used as the “gold standard” for defining Class II or Class III skeletal patterns. In this study, molar displacement that was measured on the study models was used to classify the malocclusion of each case. These classifications were then used to group cases according to malocclusion for ROC analysis. Perhaps the following assumptions can be made: Class II skeletal patterns tend to yield Class II malocclusions and likewise, Class III skeletal patterns tend to yield Class III malocclusions. Therefore molar displacement measured on study models is perhaps closest to the “truth” or the “best there is” and can be used to evaluate the diagnostic performance of cephalometric measurements in determining Class II and Class III skeletal patterns. There are times when a patient’s dental occlusion does not reflect the actual skeletal pattern. For example, a Class I molar relationship may be present in a patient with Class II skeletal pattern. This may result from shifting of lower molars or mesial inclination of the entire lower dentition. During orthodontic treatment, uprighting of lower molars can immediately transform the molar relationship to Class II. The same phenomenon can occur with a patient who has a Class I molar relationship but possesses a Class III skeletal pattern. What may appear to be an easily treatable Class I malocclusion may shift to a full blown Class III malocclusion during treatment.
American Journal of Orthodontics and Dentofacial Orthopedics May 1998
A differential diagnosis of the skeletal pattern as well as the malocclusion is important in planning a treatment. If a dental relationship were always in agreement with the skeletal pattern, a cephalometric analysis of anteroposterior skeletal discrepancy would be totally unnecessary. This is not true in reality often creating much confusion and frustration during the planning and the treatment process. In general, if an assumption can be made that Class II skeletal pattern produces Class II malocclusion and likewise, Class III skeletal pattern produces Class III malocclusion, then the molar displacement can be used as the “gold standard” in evaluating the diagnostic performance of cephalometric measurements in determining the anteroposterior skeletal discrepancy. The results of the ROC analysis showed that the APDI was the most effective parameter in diagnosing Class II and Class III malocclusions. Therefore, applying the aforementioned assumption, APDI had the best diagnostic performance in determining anteroposterior discrepancy of the skeletal pattern. SUMMARY AND CONCLUSIONS
This study used the ROC analysis to evaluate the diagnostic performance of 15 cephalometric measurements pertinent to the anteroposterior skeletal dysplasia in determiming the presence of Class II and Class III skeletal patterns. The ROC analysis is the “state of the art” statistical approach to evaluating the diagnostic performance for the full range of cut-off values. The results of the study showed: 1. Wits appraisal and overjet were effective in diagnosing Class II malocclusion. 2. Wits appraisal, convexity, AB plane angle, ANB, and overjet were effective in diagnosing Class III malocclusion. 3. APDI was the most effective parameter in diagnosing Class II and Class III malocclusions. Therefore, applying the previously mentioned assumption, APDI had the best diagnostic performance in determining anteroposterior discrepancy of the skeletal pattern. REFERENCES 1. Weinstein MC, Fineberg HV. Clinical decision analysis. Philadelphia: W.B. Saunders; 1980. p. 37-167. 2. Douglass CW. Evaluating diagnostic tests. Adv Dent Res 1993;7:66-9. 3. Metz CE. Basic principles of ROC analysis. Semin Nucl Med 1978;8:283-98. 4. DeLong EB, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating charactenstic curves: a nonparametric approach. Biometrics 1988;44:837-45. 5. Centor RM, Schwartz JS. An evaluation of methods for estimating the area under the Receiver Operating Characteristic (ROC) curve. Med Decis Making 1985;5:149-56.
American Journal of Orthodontics and Dentofacial Orthopedics Volume 113, No. 5 6. Lusted LB. Signal detectability and medical decision-making. Science 1971;171: 1217-9. 7. Centor RM. Signal detectability: The use of ROC curves and their analyses. Med Decis Making 1991;11:102-6. 8. Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in the emergency room. Med Decis Making 1981;1:239-46. 9. Berbaum KS, Franken EA Jr, Doffman DD, Barloon TJ. Influence of clinical history upon detection of nodules and other lesions. Invest Radiol 1988;23:48-55. 10. Berbaum KS, Dorfman DD, Franken EA Jr. Measuring observer performance by ROC analysis: indications and complications. Invest Radiol 1989;24:228-33. 11. Swets JA. The relative operating characteristic in psychology. Science 1973;182: 990-1000. 12. Berkey CS, Douglass CW, Valachovic RW, Chauncey HH, McNeil BJ. Statistical methods for comparing dental diagnostic procedures. Community Dent Oral Epidemiol 1990;18:169-76. 13. Douglass CW, McNeil BJ. Clinical decision analysis methods applied to diagnostic tests in dentistry. J Dent Edu 1983;47:708-12. 14. Wardlaw DW, Smith RJ, Hertweck DW, Hildebolt CF. Cephalometrics of anterior open bite: a receiver operating characteristic (ROC) analysis. Am J Orthod 1992;101:234-43. 15. Kim YH. Overbite Depth Indicator with particular reference to anterior open bite. Am J Orthod 1974;65:586-611. 16. Choi HY, Chang YI. Comparative study on the horizontal measurements of skeletal Class III malocdusion using the ROC analysis. Korea J Orthod 1995;25: 153-63. 17. Downs WB. Variations in facial relationships: their significance in treatment and prognosis. Am J Orthod 1948;34:812-40. 18. Kim YH, Vietas JJ. Anteroposterior dysplasia indicator: An adjunct to cephalometric differential diagnosis. Am J Orthod 1978;73:619-33. 19. Ogilvie JC, Creelman CD. Maximum-likelihood estimation of ROC curve parameters. J Math Psych 1968;5:377-91. 20. Dorfman DD, Alf E Jr. Maximum-likelihood estimation of parameters of signaldetection theory and determination of confidence intervals-rating method data. J Math Psych 1969;6:487-96.
Han and Kim 545 21. Grey DR, Morgan BJT. Some aspects of ROC curve-fitting: normal and logistic models. J Math Psych 1972;9:128-39. 22. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psych 1975;12:387-415. 23. Hanley JD, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36. 24. Hanley JD, McNeil BJ. The method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148: 839-43. 25. McNeil BJ, Hanley JA. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making 1984;4:137-50. 26. McClish DK. Comparing the areas under more than two independent ROC curves. Med Decis Making 1987;7:149-55. 27. Centor RM. Visicalc program for estimating the area under a receiver operating characteristic (ROC) curve. Med Decis Making 1985;5:139-48. 28. Centor RM, Keightley GE. Receiver operating characteristic (ROC) curve area analysis using the ROC ANALYZER. SCAMC Proceedings 1989;13:222-6. 29. Jacobson A. The “Wits” appraisal of jaw disharmony. Am J Orthod 1975;67: 125-38. 30. Chang HP. Assessment of anteroposterior jaw relationship. Am J Orthod Dentofacial Orthop 1987;92:117-22. 31. Riedel RA. Esthetics and its relation to orthodontic therapy. Angle Orthod 1950;20:168-78. 32. Ferrazzini G. Critical evaluation of the ANB angle. Am J Orthod 1976;69:620-6. 33. Hussels W, Nanda RS. Analysis of factors affecting angle ANB. Am J Orthod 1984;85:620-6. 34. Freeman RS. Adjusting A-N-B angles to reflect the effect of maxillary position. Angle Orthod 1981;51:162-71. 35. Browner WS, Newman TB, Cummings SR. Designing a new study: diagnostic tests. In: Hulley SB, Cummings SR, editors. Designing clinical research. Baltimore (MD): Williams and Wilkins, 1988. p. 87. 36. Newhauser D, Yin XP. Deciding whether a new test measure is useful. Medical Care 1991;29:685-9.