- Email: [email protected]
Researchers should be aware of partial verification bias in diagnostic accuracy studies
Vol. 119 No. 5 May 2015
LETTERS TO THE EDITOR b is the verification for negative cases. BSn is the bias
Researchers should be aware of partial verification bias in diagnostic accuracy studies To the Editor: I wish to comment on the diagnostic accuracy study published by Diaz et al.1 recently published in Oral and Maxillofacial Pathology. In this study, cases were excluded when they were reactive, inflammatory, or metastatic according to cytopathology and were included only if they were verified by histopathology. This type of study design is very common in diagnostic accuracy studies on fine-needle aspiration (FNA) cytology. Typically, cases found positive for malignancy are referred for surgery at a higher rate compared with negative cases. This phenomenon is known as partial verification and leads to a type of bias known as partial verification bias.2,3 Two recent reviews of FNA diagnostic accuracy studies have found that verification bias is common in salivary gland studies and is more commonly found in reports published in surgical journals than in clinical journals.4,5 Under the conditions generally encountered in diagnostic accuracy FNA studies in salivary glands, verification bias generally causes an overestimate of sensitivity and an underestimate of specificity.2 The magnitude of verification bias can be estimated and corrected if the overall patient flow is reported. Unfortunately, as in the Diaz study, patient flows are often not reported, and as a consequence, it is not possible to estimate the magnitude of the bias. However, the bias can be substantial. A recent FNA study on head and neck lesions reported sensitivities of 96.3% (thyroid), 75% (salivary glands), and 85.4% (lymph nodes).6 The sensitivities were changed to 51.8% (thyroid), 68.1% (salivary gland), and 57.4% (lymph nodes) after adjustment for partial verification.7 The degree of bias depends on the difference in the verification rates of negative and positive cases. The bias in sensitivity and specificity are as follows:7
1þ BSp ¼
1
BSn ¼
1Sn Sn
1þ
1
Sn
(1)
Sp
(2)
b a
1Sp Sp
a b
Where Sn and Sp are the true sensitivity and specificity, a is the verification rate for positive cases, and
in sensitivity, and BSp is the bias in specificity. Several measures can be taken to prevent partial verification bias. First, one can use a “brass standard,” such as clinical follow-up, to verify cases without histopathology. Alternatively, one can use statistical adjustment. Unfortunately, these measures are rarely employed. As a consequence, much of the literature on accuracy studies is unreliable. Researchers and clinicians need to be aware of these issues when designing, reporting, and interpreting diagnostic accuracy studies. Robert L. Schmidt, MD, PhD, MBA Assistant Professor Department of Pathology University of Utah School of Medicine and ARUP Laboratories Salt Lake City Utah USA Rachel E. Factor, MD, MHS Department of Pathology University of Utah School of Medicine and ARUP Laboratories Salt Lake City Utah USA REFERENCES 1. Díaz KP, Gerhard R, Domingues RB, et al. High diagnostic accuracy and reproducibility of fine-needle aspiration cytology for diagnosing salivary gland tumors: cytohistologic correlation in 182 cases. Oral Surg Oral Med Oral Pathol Oral Radiol. 2014;118: 226-235. 2. Zhou X-H, Obuchowski N, McLish D. Statistical Methods in Diagnostic Medicine. 2nd ed. Hoboken, New Jersey: John Wiley and Sons; 2011. 3. Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137:558-565. 4. Schmidt RL, Factor RE, Witt BL, Layfield LJ. Quality appraisal of diagnostic accuracy studies in fine-needle aspiration cytology: a survey of risk of bias and comparability. Arch Pathol Lab Med. 2013;137:566-575. 5. Schmidt RL, Jedrzkiewicz JD, Allred RJ, Matsuoka S, Witt BL. Verification bias in diagnostic accuracy studies for fine- and core needle biopsy of salivary gland lesions in otolaryngology journals: a systematic review and analysis. Head Neck. 2014;36:1654-1661. 6. Ahn D, Kim H, Sohn J, Choi J, Na K. Surgeon-performed ultrasound-guided fine-needle aspiration cytology of head and neck mass lesions: sampling adequacy and diagnostic accuracy. Ann Surg Oncol. 2014;2014:1-6. 7. Schmidt R, Cohen MB. Partial verification distorts estimates of sensitivity in diagnostic accuracy studies for fine-needle aspiration cytology. Ann Surg Oncol. 2015;123:193-201. http://dx.doi.org/10.1016/j.oooo.2014.11.027
593
LETTERS TO THE EDITOR b is the verification for negative cases. BSn is the bias
Researchers should be aware of partial verification bias in diagnostic accuracy studies To the Editor: I wish to comment on the diagnostic accuracy study published by Diaz et al.1 recently published in Oral and Maxillofacial Pathology. In this study, cases were excluded when they were reactive, inflammatory, or metastatic according to cytopathology and were included only if they were verified by histopathology. This type of study design is very common in diagnostic accuracy studies on fine-needle aspiration (FNA) cytology. Typically, cases found positive for malignancy are referred for surgery at a higher rate compared with negative cases. This phenomenon is known as partial verification and leads to a type of bias known as partial verification bias.2,3 Two recent reviews of FNA diagnostic accuracy studies have found that verification bias is common in salivary gland studies and is more commonly found in reports published in surgical journals than in clinical journals.4,5 Under the conditions generally encountered in diagnostic accuracy FNA studies in salivary glands, verification bias generally causes an overestimate of sensitivity and an underestimate of specificity.2 The magnitude of verification bias can be estimated and corrected if the overall patient flow is reported. Unfortunately, as in the Diaz study, patient flows are often not reported, and as a consequence, it is not possible to estimate the magnitude of the bias. However, the bias can be substantial. A recent FNA study on head and neck lesions reported sensitivities of 96.3% (thyroid), 75% (salivary glands), and 85.4% (lymph nodes).6 The sensitivities were changed to 51.8% (thyroid), 68.1% (salivary gland), and 57.4% (lymph nodes) after adjustment for partial verification.7 The degree of bias depends on the difference in the verification rates of negative and positive cases. The bias in sensitivity and specificity are as follows:7
1þ BSp ¼
1
BSn ¼
1Sn Sn
1þ
1
Sn
(1)
Sp
(2)
b a
1Sp Sp
a b
Where Sn and Sp are the true sensitivity and specificity, a is the verification rate for positive cases, and
in sensitivity, and BSp is the bias in specificity. Several measures can be taken to prevent partial verification bias. First, one can use a “brass standard,” such as clinical follow-up, to verify cases without histopathology. Alternatively, one can use statistical adjustment. Unfortunately, these measures are rarely employed. As a consequence, much of the literature on accuracy studies is unreliable. Researchers and clinicians need to be aware of these issues when designing, reporting, and interpreting diagnostic accuracy studies. Robert L. Schmidt, MD, PhD, MBA Assistant Professor Department of Pathology University of Utah School of Medicine and ARUP Laboratories Salt Lake City Utah USA Rachel E. Factor, MD, MHS Department of Pathology University of Utah School of Medicine and ARUP Laboratories Salt Lake City Utah USA REFERENCES 1. Díaz KP, Gerhard R, Domingues RB, et al. High diagnostic accuracy and reproducibility of fine-needle aspiration cytology for diagnosing salivary gland tumors: cytohistologic correlation in 182 cases. Oral Surg Oral Med Oral Pathol Oral Radiol. 2014;118: 226-235. 2. Zhou X-H, Obuchowski N, McLish D. Statistical Methods in Diagnostic Medicine. 2nd ed. Hoboken, New Jersey: John Wiley and Sons; 2011. 3. Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137:558-565. 4. Schmidt RL, Factor RE, Witt BL, Layfield LJ. Quality appraisal of diagnostic accuracy studies in fine-needle aspiration cytology: a survey of risk of bias and comparability. Arch Pathol Lab Med. 2013;137:566-575. 5. Schmidt RL, Jedrzkiewicz JD, Allred RJ, Matsuoka S, Witt BL. Verification bias in diagnostic accuracy studies for fine- and core needle biopsy of salivary gland lesions in otolaryngology journals: a systematic review and analysis. Head Neck. 2014;36:1654-1661. 6. Ahn D, Kim H, Sohn J, Choi J, Na K. Surgeon-performed ultrasound-guided fine-needle aspiration cytology of head and neck mass lesions: sampling adequacy and diagnostic accuracy. Ann Surg Oncol. 2014;2014:1-6. 7. Schmidt R, Cohen MB. Partial verification distorts estimates of sensitivity in diagnostic accuracy studies for fine-needle aspiration cytology. Ann Surg Oncol. 2015;123:193-201. http://dx.doi.org/10.1016/j.oooo.2014.11.027
593