- Email: [email protected]
Store-and-forward teledermatology versus in-person visits: A comparison in pediatric teledermatology clinic
Store-and-forward teledermatology versus in-person visits: A comparison in pediatric teledermatology clinic Viday A. Heffner, MD,a,c Valerie B. Lyon, MD,b David C. Brousseau, MD, MS,a,c Kristin E. Holland, MD,b and Kenneth Yen, MD, MSa,c Milwaukee, Wisconsin Background: The role of teledermatology in the diagnosis of pediatric skin conditions has not been studied exclusively. Objective: To determine the ability of a pediatric dermatologist to correctly diagnose rashes by history and digital images. Methods: Consecutive, new referrals to the pediatric dermatology clinic with a rash were enrolled in the study. A history, demographic data, and digital photographs were obtained from each patient. The data were reviewed by a pediatric dermatologist who made a preliminary diagnosis. The child was then seen in person and a final diagnosis was made. Concordance and kappa values were calculated. Cases of diagnostic disagreement were analyzed for their effect on management. Results: One hundred thirty-five patients were enrolled. Diagnostic concordance was 82% (95% confidence interval [CI], 73%-88%), and the kappa value was 0.80. Clinically relevant disagreement occurred in 12% of cases. Limitations: The study was performed at a single site, theoretically limiting generalizability. Conclusion: Teledermatology appears to have a useful role in the care of children with rashes. ( J Am Acad Dermatol 2009;60:956-61.)
INTRODUCTION Telemedicine has become an invaluable healthcare delivery tool in an ever-expanding, medically complex society.1,2 With its emphasis on visual diagnosis, dermatology is a discipline that appears ideally suited to telemedicine. Most research in teledermatology has focused on two types of technology: store-and-forward (SF) and real-time (RT). In SF teledermatology, the data and images from a patient are collected and sent to a specialist to be From the Department of Pediatrics, Section of Emergency Medicinea and the Section of Pediatric Dermatology,b Medical College of Wisconsin; and Children’s Research Institute, Children’s Hospital and Health System.c Funding sources: None. Conflicts of interest: None declared. Accepted for publication November 9, 2008. Reprint requests: Viday A. Heffner, MD, Children’s Corporate Center, Pediatric Emergency Medicine, 999 N 92nd St, Suite C550, Milwaukee, WI 53226. E-mail: [email protected]. Published online April 13, 2009. 0190-9622/$36.00 ª 2009 by the American Academy of Dermatology, Inc. doi:10.1016/j.jaad.2008.11.026
956
Abbreviations used: PI: primary investigator RT: real-time SF: store-and-forward
viewed at a later time. In RT teledermatology, patients and consultants communicate data and images from separate locations in a live format. Most studies show slightly higher interrater agreement rates in RT versus SF; however, SF teledermatology is a more feasible option for clinics unable to operate through live video links.3-7 The initial inquiry which led to this study was whether teledermatology could be a useful tool in the pediatric emergency department. A literature review revealed that the pediatric patient had not been studied exclusively in this discipline. Lack of statistical breakdown between age groups in previously published combined adult/pediatric studies did not allow for isolation of pediatric data.5 With only adult teledermatology background data available, this study was undertaken to determine
Heffner et al 957
J AM ACAD DERMATOL VOLUME 60, NUMBER 6
whether SF teledermatology could be applied to the pediatric patient. The objective of the study was to determine the ability of a board certified pediatric dermatologist to correctly diagnose rashes by history and digital images alone compared with direct visualization of the patient. Our hypothesis was that SF digital photography combined with a brief patient history and rash description would provide sufficient information for the pediatric dermatologist to make the same diagnosis when compared to an in-person clinic visit.
METHODS Study flow This study was conducted at the pediatric dermatology clinic at the Children’s Hospital of Wisconsin. The Children’s Hospital of Wisconsin pediatriic dermatology clinic has an annual census approaching 12,000 and a racially and ethnically diverse population. The study was approved by the institution’s human research committee and parental/patient informed consent/assent was obtained. Consecutive patients on predetermined study days were enrolled during the 13-month period of July 2006 through August 2007. This extensive enrollment period allowed for the seasonality of some common rashes. Patients were enrolled from the clinics of two board-certified pediatric dermatologists. Only new patients with a chief presenting complaint of either a rash or rash descriptors (eg, bumps, spots, patches) were enrolled. No existing patients were enrolled unless the skin condition with which they presented was new and unrelated to their previous clinic visits. Study patients ranged from 3 months through 18 years of age. The primary investigator (PI), a board-certified pediatrician, asked enrolled patients and their parents several basic history and demographic questions (Fig 1: data collection form). Parents were asked to identify the skin shade of their child using a commercially available cosmetic shade scale chart with predetermined cut-offs for light and dark skin. They were also asked to identify the child’s ethnicity. The PI took photographs of every enrolled patient with a Canon Powershot SD450, a 5-megapixel camera in digital macro mode without flash. This camera was chosen for its ease of use and highquality image reproduction. The camera was set in fine detail with image size at 1200 3 1600 pixels. The light source was overhead fluorescent with some natural light through examining room windows. The PI read the basic instruction manual included with the camera, but had no formal photographic training. In the first part of the study, to evaluate the primary outcome of intrarater agreement between in-person
visits and SF images, the PI displayed the photographs on a Dell desktop (Phillips 170B flat screen monitor, 1024 3 768 pixels, 16-bit color) and gave the primary dermatologist a brief verbal patient history. A diagnosis of the skin condition was made and recorded on clinic billing sheets. The child was then seen by the dermatologist and, after a full evaluation including further history and physical examination, an in-person diagnosis was made (Fig 2). In the second part of the study, to evaluate interrater agreement of SF images, a second pediatric dermatologist was asked to view the photographs and the history data sheet from each study patient at a later date. The second dermatologist then rendered a single diagnosis. That diagnosis was compared not only to the primary dermatologist’s photographic diagnosis, but also to the primary dermatologist’s inperson diagnosis. Analysis Percentage of diagnostic concordance was calculated for intrarater as well as both forms of interrater agreement. Cases of diagnostic disagreement were then analyzed by one of the two participating dermatologists to determine if a different diagnosis would require a change in therapy. Cases requiring a therapy change were deemed ‘‘clinically relevant disagreement’’ and those requiring no change in therapy were deemed ‘‘clinically irrelevant dis agreement’’. Data were then analyzed using Stata, version 9.1 (StataCorpLP, College Station, TX) to attain kappa values for intrarater and both forms of interrater agreement, as well as for adjusted agreement considering cases of clinically relevant disagreement alone. Kappa is a statistic used to quantify agreement by removing chance from the equation. Logistic regression was performed to determine whether certain demographic characteristics influenced the ability of the dermatologists to render a correct diagnosis. The demographic characteristics analyzed included the following: age (older or younger than 12 months), dark/light skin color, ethnicity, and presence or absence of previous treatment. Finally, all cases of intrarater disagreement were reviewed at the end of enrollment by one participating dermatologist to determine the etiology of the incorrect diagnoses.
RESULTS Demographics A total of 137 patients were approached for enrollment in this study. One family refused to participate and one patient was removed, having been seen in-person by the dermatologist before
958 Heffner et al
J AM ACAD DERMATOL JUNE 2009
Fig 1. Patient data collection form.
data and picture presentation. The remaining one hundred thirty-five patients gave informed consent and had complete data collection. The mean age was 6 years, 2 months, with a range of 3 months to 18 years, 6 months. The male-to-female ratio was 1.5:1. Forty-three percent of enrolled patients were Caucasian, 38% were African American, 13% were Hispanic, 5% were Asian, and 1% other. The average number of photographs taken per patient was 3.4. Forty-eight different dermatologic diagnoses were found in the study population; the most common diagnoses are listed in Table I. Logistic regression of demographic factors including: ethnicity, dark versus light skin color, age (older or younger than 12 months), and previous treatment showed no statistical significance in diagnostic agreement. Diagnostic agreement Intrarater: Photographic versus in-person. Concordance between the in-person and photographic diagnosis by the primary dermatologist was 82% (95% confidence interval [CI], 75%-89%). The kappa value for intrarater agreement was 0.80 (95% CI, 0.73-0.88). Clinically relevant disagreement,
defined as disagreement where a change in management or therapy was required, occurred in 12% of cases (Table II). Interrater: Photographs alone. Concordance between the two dermatologists receiving photographs and history alone was 73% (95% CI, 64%-80%), with a kappa photographic interrater agreement value of 0.69 (95% CI, 0.61-0.77). Clinically relevant disagreement occurred in 14% of cases (Table III). Interrater: Photographic versus in-person. Concordance between the two dermatologists, one viewing the patient in person and the other viewing photographs alone was 69% (95% CI, 60%-77%). The kappa value for interrater in-person/photographic agreement was 0.65 (95% CI, 0.58-0.73). Clinically relevant disagreement occurred in 16% of cases (Table IV). Missed diagnosis review The 24 cases of disagreement that occurred in the intrarater portion of the study were reviewed by a pediatric dermatologist. In 6 cases (25%), it was thought that a whole-body photograph, instead of only close-up images, would have made the
Heffner et al 959
J AM ACAD DERMATOL VOLUME 60, NUMBER 6
Fig 2. Study flow diagram.
diagnosis easier. In 4% of cases (n = 1) poor photographic quality was identified as the cause of the missed diagnosis. All cases were reviewed to determine the types of dermatoses that were missed. The only scenario represented more than once was three cases of scabies misdiagnosed as atopic dermatitis.
DISCUSSION The results of this study suggest that SF teledermatology leads to the accurate assessment of pediatric rashes by a pediatric dermatologist. Pairing pictures from a simple point-and-shoot camera with
a brief history, the dermatologists were able to correctly treat more than 80% of rashes referred to the dermatology clinic. In previous studies of teledermatology, dermatologist intrarater agreement has been in the range of 31% to 88%.3 Our study showed an intrarater agreement rate of 82%, which falls into the higher end of the range presented in the adult literature, suggesting that teledermatology may be very readily applied to children. Several previous studies that reported higher agreement rates included partial agreement,8,9 which was defined as agreement between the initial
960 Heffner et al
J AM ACAD DERMATOL JUNE 2009
Table I. Most common diagnoses Final diagnosis
Eczema/atopic dermatitis Molluscum contagiosum Seborrheic dermatitis Flat warts Insect bite or sting Contact dermatitis Nevus Perioral dermatitis Scabies Tinea capitis Viral exanthem Vitiligo
No. of cases
% of Total score
33 18 7 6 6 5 5 4 4 4 4 4
24.4 13.3 5.2 4.4 4.4 3.7 3.7 3.0 3.0 3.0 3.0 3.0
Intraobserver
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
No. of patients Percentage Kappa
98 18
73 13
19
14
116 135
86 100
0.68
0.84
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
Table II. Intrarater results (dermatologist vs in-person visit) concordance and agreement rates Intraobserver
Table III. Interrater results (dermatologists viewing photographs alone) concordance and agreement rates
Table IV. Interrater results (primary dermatologist viewing in-person vs secondary dermatologist viewing photographs) concordance and agreement rates
No. of patients Percentage Kappa
111 8
82 6
16
12
119 135
88 100
0.80
0.87
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
differential of several preliminary diagnoses and a final diagnosis. We chose not to allow partial agreement and asked our dermatologists to identify only one preliminary and one final diagnosis. We believed that by making the standards stricter, the agreement rate would be more clinically significant. Diagnostic variability among dermatologists is also a factor that must be considered. We had a second dermatologist review all cases; this ensured that the primary dermatologist on each case was not biased to create higher intrarater agreement. By having another dermatologist review the data and photographs of each patient, we were able to determine an interrater agreement of 73% for photographs alone and 70% with comparison of photographic diagnosis to an in-person visit, which was again in the higher end of the range of adult interrater studies (41%-85%).3 This high rate of interrater agreement supports our belief that SF teledermatology is applicable to pediatric rashes. For both the intrarater and interrater portions of the study, we analyzed the cases of disagreement. Because of the overlap of therapies for many dermatologic conditions, it was imperative to determine whether a change in management or therapy was
Intraobserver
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
No. of patients Percentage Kappa
94 20
70 14
21
16
114 135
84 100
0.65
0.82
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
necessary with the revised final diagnosis. Taking into account the cases of disagreement that did not require a change in management, the intrarater agreement, previously 82%, climbed to 88%. The photographic interrater agreement, previously 73%, climbed to 86%. More telling still was the comparison between an in-person visit with one dermatologist and a photographic diagnosis by another dermatologist. Although agreement was initially the lowest at 69%, it climbed to 84% after removing clinically irrelevant disagreement, putting it into the same range as intrarater agreement. We believe that by showing similar rates between clinically relevant intrarater and interrater agreement, concern for the bias of intrarater agreement has been addressed. Although most studies of teledermatology utilize percentage agreement exclusively, we calculated kappa values for both intrarater and interrater agreement categories to determine whether chance alone could create falsely high rates of concordance. All kappa values were found to be high, at 0.65 and 0.69 for interrater agreement and 0.80 for intrarater agreement; they climbed even higher, to 0.82, 0.84, and 0.87, respectively, when considering only clinically
J AM ACAD DERMATOL
Heffner et al 961
VOLUME 60, NUMBER 6
relevant disagreement. Though a subject of much debate, kappa values greater than 0.81 are generally considered to show very good agreement.10 All cases of intrarater disagreement were examined a third time by one participating dermatologist to analyze the cause of the disagreement between review of the photographs and the in-person visit. In 6 of the 24 cases, it was considered that a picture showing whole-body distribution of the rash, rather than close-up images alone, would have made the diagnosis easier. In 4 of the 24 cases, the skin lesions were too subtle to diagnose by photography. In the remaining 14 cases, no patterns causing misdiagnosis were noted. Before study onset, we hypothesized that certain demographic data would make a photographic diagnosis more challenging. However, diagnostic accuracy was not affected by the age of the child, the skin tone, previous treatment of the rash, nor the duration of symptoms before presentation. The outcome of this study has practical applications within the medical community. In a large geographic area with a small number of specialists, telemedicine has proven to be a useful tool. Knowing that pediatric skin conditions can be diagnosed accurately by SF teledermatology widens treatment options. In conditions such as atopic dermatitis, represented in high numbers in our study, initial therapy and family/patient education could begin before the next available appointment with a dermatologist. In conditions such as tinea capitis, therapy could be instituted by the primary care practitioner with greater certainty at the first visit without having to wait for the result of a fungal culture. There may be some resistance to institution of teledermatology, with regard to reimbursement and liability. Beginning in 2001, Medicare began to cover a wide range of telemedicine services.11 Medicaid reimburses telemedicine on a state-by-state basis; private insurers often follow Medicare/Medicaid guidelines, so practitioners may wish to review the laws of their state.12 In regards to liability, telemedicine services are frequently covered under general malpractice insurance and newer ‘‘telemedicine insurance policies’’ are becoming available.12
LIMITATIONS Although the data were collected in a tertiary care pediatric dermatology clinic in a large city, at the time of this study a 1-day referral system was in place, which allowed for acute conditions to be seen within 24 hours. This system ensured that acute as well as chronic conditions were included.
All of the photographs and initial data collection in this study were obtained only by the PI. However, as the PI had no formal training in medical photography and was using a simple point-and-shoot digital camera, we consider this less a limitation than a benefit with respect to the generalizability of the study. There is concern for bias by the primary dermatologist: seeing patient photographs and later the patient in-person, one would tend to agree with oneself. We addressed this by including the interrater agreement categories, showing that adjusted agreement was very similar in cases where two different dermatologists gave diagnoses on a patient, and one person gave two separate diagnoses on the same patient. Although arguably present, by showing such similar agreement and kappa values for all categories, we believe that this bias is negligible. REFERENCES 1. Dill S, Digiovanna J. Changing paradigms in dermatology: information technology. Clin Dermatol 2003;21:375-82. 2. Maglogiannis I. Design and implementation of a calibrated store and forward imaging system for teledermatology. J Med Syst 2004;28:455-67. 3. Whited J. Teledermatology research review. Int J Dermatol 2006;45:220-9. 4. Pak H, Harden D, Cruess D, Welch M, Poropatich R. National Capital Area Teledermatology Consortium. Teledermatology: an intraobserver diagnostic correlation study, part I. Cutis 2003;71:399-403. 5. High W, Houston M, Calobrisi S, Drage L, McEvoy M. Assessment of the accuracy of low-cost store-and-forward teledermatology consultation. J Am Acad Dermatol 2000;42: 776-83. 6. Phillips C, Burke W, Shechter A, Stone D, Balch D, Gustke S. Reliability of dermatology teleconsultations with the use of teleconferencing technology. J Am Acad Dermatol 1997;37: 398-402. 7. Baba M, Seckin D, Kapdagli S. A comparison of teledermatology using store-and-forward methodology alone, and in combination with Web camera videoconferencing. J Telemed Telecare 2005;11:354-60. 8. Whited J, Hall R, Simel D, Foy M, Stechuchak K, Drugge R, et al. Reliability and accuracy of dermatologists’ clinic-based and digital image consultations. J Am Acad Dermatol 1999;41: 693-702. 9. Gutierrez G. Medicare, the Internet, and the future of telemedicine. Critical Care Med 2001;29:N144-50. 10. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. 11. Lauderdale D, Lacsamama C, Palsbo S. Medicaid and telemedicine in 2002. Telehealth Pract Rep 2003;8:14-5. 12. Wachter G. Malpractice and telemedicine. Telemedicine liability: the uncharted waters of medical risk. Telemedicine Online Archives. Available at: http://tie.telemed.org/articles/article.asp? path=articles&article=malpracticeLiability_gw_tie02.xml. Accessed August 12, 2008.
INTRODUCTION Telemedicine has become an invaluable healthcare delivery tool in an ever-expanding, medically complex society.1,2 With its emphasis on visual diagnosis, dermatology is a discipline that appears ideally suited to telemedicine. Most research in teledermatology has focused on two types of technology: store-and-forward (SF) and real-time (RT). In SF teledermatology, the data and images from a patient are collected and sent to a specialist to be From the Department of Pediatrics, Section of Emergency Medicinea and the Section of Pediatric Dermatology,b Medical College of Wisconsin; and Children’s Research Institute, Children’s Hospital and Health System.c Funding sources: None. Conflicts of interest: None declared. Accepted for publication November 9, 2008. Reprint requests: Viday A. Heffner, MD, Children’s Corporate Center, Pediatric Emergency Medicine, 999 N 92nd St, Suite C550, Milwaukee, WI 53226. E-mail: [email protected]. Published online April 13, 2009. 0190-9622/$36.00 ª 2009 by the American Academy of Dermatology, Inc. doi:10.1016/j.jaad.2008.11.026
956
Abbreviations used: PI: primary investigator RT: real-time SF: store-and-forward
viewed at a later time. In RT teledermatology, patients and consultants communicate data and images from separate locations in a live format. Most studies show slightly higher interrater agreement rates in RT versus SF; however, SF teledermatology is a more feasible option for clinics unable to operate through live video links.3-7 The initial inquiry which led to this study was whether teledermatology could be a useful tool in the pediatric emergency department. A literature review revealed that the pediatric patient had not been studied exclusively in this discipline. Lack of statistical breakdown between age groups in previously published combined adult/pediatric studies did not allow for isolation of pediatric data.5 With only adult teledermatology background data available, this study was undertaken to determine
Heffner et al 957
J AM ACAD DERMATOL VOLUME 60, NUMBER 6
whether SF teledermatology could be applied to the pediatric patient. The objective of the study was to determine the ability of a board certified pediatric dermatologist to correctly diagnose rashes by history and digital images alone compared with direct visualization of the patient. Our hypothesis was that SF digital photography combined with a brief patient history and rash description would provide sufficient information for the pediatric dermatologist to make the same diagnosis when compared to an in-person clinic visit.
METHODS Study flow This study was conducted at the pediatric dermatology clinic at the Children’s Hospital of Wisconsin. The Children’s Hospital of Wisconsin pediatriic dermatology clinic has an annual census approaching 12,000 and a racially and ethnically diverse population. The study was approved by the institution’s human research committee and parental/patient informed consent/assent was obtained. Consecutive patients on predetermined study days were enrolled during the 13-month period of July 2006 through August 2007. This extensive enrollment period allowed for the seasonality of some common rashes. Patients were enrolled from the clinics of two board-certified pediatric dermatologists. Only new patients with a chief presenting complaint of either a rash or rash descriptors (eg, bumps, spots, patches) were enrolled. No existing patients were enrolled unless the skin condition with which they presented was new and unrelated to their previous clinic visits. Study patients ranged from 3 months through 18 years of age. The primary investigator (PI), a board-certified pediatrician, asked enrolled patients and their parents several basic history and demographic questions (Fig 1: data collection form). Parents were asked to identify the skin shade of their child using a commercially available cosmetic shade scale chart with predetermined cut-offs for light and dark skin. They were also asked to identify the child’s ethnicity. The PI took photographs of every enrolled patient with a Canon Powershot SD450, a 5-megapixel camera in digital macro mode without flash. This camera was chosen for its ease of use and highquality image reproduction. The camera was set in fine detail with image size at 1200 3 1600 pixels. The light source was overhead fluorescent with some natural light through examining room windows. The PI read the basic instruction manual included with the camera, but had no formal photographic training. In the first part of the study, to evaluate the primary outcome of intrarater agreement between in-person
visits and SF images, the PI displayed the photographs on a Dell desktop (Phillips 170B flat screen monitor, 1024 3 768 pixels, 16-bit color) and gave the primary dermatologist a brief verbal patient history. A diagnosis of the skin condition was made and recorded on clinic billing sheets. The child was then seen by the dermatologist and, after a full evaluation including further history and physical examination, an in-person diagnosis was made (Fig 2). In the second part of the study, to evaluate interrater agreement of SF images, a second pediatric dermatologist was asked to view the photographs and the history data sheet from each study patient at a later date. The second dermatologist then rendered a single diagnosis. That diagnosis was compared not only to the primary dermatologist’s photographic diagnosis, but also to the primary dermatologist’s inperson diagnosis. Analysis Percentage of diagnostic concordance was calculated for intrarater as well as both forms of interrater agreement. Cases of diagnostic disagreement were then analyzed by one of the two participating dermatologists to determine if a different diagnosis would require a change in therapy. Cases requiring a therapy change were deemed ‘‘clinically relevant disagreement’’ and those requiring no change in therapy were deemed ‘‘clinically irrelevant dis agreement’’. Data were then analyzed using Stata, version 9.1 (StataCorpLP, College Station, TX) to attain kappa values for intrarater and both forms of interrater agreement, as well as for adjusted agreement considering cases of clinically relevant disagreement alone. Kappa is a statistic used to quantify agreement by removing chance from the equation. Logistic regression was performed to determine whether certain demographic characteristics influenced the ability of the dermatologists to render a correct diagnosis. The demographic characteristics analyzed included the following: age (older or younger than 12 months), dark/light skin color, ethnicity, and presence or absence of previous treatment. Finally, all cases of intrarater disagreement were reviewed at the end of enrollment by one participating dermatologist to determine the etiology of the incorrect diagnoses.
RESULTS Demographics A total of 137 patients were approached for enrollment in this study. One family refused to participate and one patient was removed, having been seen in-person by the dermatologist before
958 Heffner et al
J AM ACAD DERMATOL JUNE 2009
Fig 1. Patient data collection form.
data and picture presentation. The remaining one hundred thirty-five patients gave informed consent and had complete data collection. The mean age was 6 years, 2 months, with a range of 3 months to 18 years, 6 months. The male-to-female ratio was 1.5:1. Forty-three percent of enrolled patients were Caucasian, 38% were African American, 13% were Hispanic, 5% were Asian, and 1% other. The average number of photographs taken per patient was 3.4. Forty-eight different dermatologic diagnoses were found in the study population; the most common diagnoses are listed in Table I. Logistic regression of demographic factors including: ethnicity, dark versus light skin color, age (older or younger than 12 months), and previous treatment showed no statistical significance in diagnostic agreement. Diagnostic agreement Intrarater: Photographic versus in-person. Concordance between the in-person and photographic diagnosis by the primary dermatologist was 82% (95% confidence interval [CI], 75%-89%). The kappa value for intrarater agreement was 0.80 (95% CI, 0.73-0.88). Clinically relevant disagreement,
defined as disagreement where a change in management or therapy was required, occurred in 12% of cases (Table II). Interrater: Photographs alone. Concordance between the two dermatologists receiving photographs and history alone was 73% (95% CI, 64%-80%), with a kappa photographic interrater agreement value of 0.69 (95% CI, 0.61-0.77). Clinically relevant disagreement occurred in 14% of cases (Table III). Interrater: Photographic versus in-person. Concordance between the two dermatologists, one viewing the patient in person and the other viewing photographs alone was 69% (95% CI, 60%-77%). The kappa value for interrater in-person/photographic agreement was 0.65 (95% CI, 0.58-0.73). Clinically relevant disagreement occurred in 16% of cases (Table IV). Missed diagnosis review The 24 cases of disagreement that occurred in the intrarater portion of the study were reviewed by a pediatric dermatologist. In 6 cases (25%), it was thought that a whole-body photograph, instead of only close-up images, would have made the
Heffner et al 959
J AM ACAD DERMATOL VOLUME 60, NUMBER 6
Fig 2. Study flow diagram.
diagnosis easier. In 4% of cases (n = 1) poor photographic quality was identified as the cause of the missed diagnosis. All cases were reviewed to determine the types of dermatoses that were missed. The only scenario represented more than once was three cases of scabies misdiagnosed as atopic dermatitis.
DISCUSSION The results of this study suggest that SF teledermatology leads to the accurate assessment of pediatric rashes by a pediatric dermatologist. Pairing pictures from a simple point-and-shoot camera with
a brief history, the dermatologists were able to correctly treat more than 80% of rashes referred to the dermatology clinic. In previous studies of teledermatology, dermatologist intrarater agreement has been in the range of 31% to 88%.3 Our study showed an intrarater agreement rate of 82%, which falls into the higher end of the range presented in the adult literature, suggesting that teledermatology may be very readily applied to children. Several previous studies that reported higher agreement rates included partial agreement,8,9 which was defined as agreement between the initial
960 Heffner et al
J AM ACAD DERMATOL JUNE 2009
Table I. Most common diagnoses Final diagnosis
Eczema/atopic dermatitis Molluscum contagiosum Seborrheic dermatitis Flat warts Insect bite or sting Contact dermatitis Nevus Perioral dermatitis Scabies Tinea capitis Viral exanthem Vitiligo
No. of cases
% of Total score
33 18 7 6 6 5 5 4 4 4 4 4
24.4 13.3 5.2 4.4 4.4 3.7 3.7 3.0 3.0 3.0 3.0 3.0
Intraobserver
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
No. of patients Percentage Kappa
98 18
73 13
19
14
116 135
86 100
0.68
0.84
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
Table II. Intrarater results (dermatologist vs in-person visit) concordance and agreement rates Intraobserver
Table III. Interrater results (dermatologists viewing photographs alone) concordance and agreement rates
Table IV. Interrater results (primary dermatologist viewing in-person vs secondary dermatologist viewing photographs) concordance and agreement rates
No. of patients Percentage Kappa
111 8
82 6
16
12
119 135
88 100
0.80
0.87
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
differential of several preliminary diagnoses and a final diagnosis. We chose not to allow partial agreement and asked our dermatologists to identify only one preliminary and one final diagnosis. We believed that by making the standards stricter, the agreement rate would be more clinically significant. Diagnostic variability among dermatologists is also a factor that must be considered. We had a second dermatologist review all cases; this ensured that the primary dermatologist on each case was not biased to create higher intrarater agreement. By having another dermatologist review the data and photographs of each patient, we were able to determine an interrater agreement of 73% for photographs alone and 70% with comparison of photographic diagnosis to an in-person visit, which was again in the higher end of the range of adult interrater studies (41%-85%).3 This high rate of interrater agreement supports our belief that SF teledermatology is applicable to pediatric rashes. For both the intrarater and interrater portions of the study, we analyzed the cases of disagreement. Because of the overlap of therapies for many dermatologic conditions, it was imperative to determine whether a change in management or therapy was
Intraobserver
Agreement Clinically irrelevant disagreement Clinically relevant disagreement Adjusted agreement* Totals
No. of patients Percentage Kappa
94 20
70 14
21
16
114 135
84 100
0.65
0.82
*Adjusted agreement includes agreement and clinically irrelevant disagreement.
necessary with the revised final diagnosis. Taking into account the cases of disagreement that did not require a change in management, the intrarater agreement, previously 82%, climbed to 88%. The photographic interrater agreement, previously 73%, climbed to 86%. More telling still was the comparison between an in-person visit with one dermatologist and a photographic diagnosis by another dermatologist. Although agreement was initially the lowest at 69%, it climbed to 84% after removing clinically irrelevant disagreement, putting it into the same range as intrarater agreement. We believe that by showing similar rates between clinically relevant intrarater and interrater agreement, concern for the bias of intrarater agreement has been addressed. Although most studies of teledermatology utilize percentage agreement exclusively, we calculated kappa values for both intrarater and interrater agreement categories to determine whether chance alone could create falsely high rates of concordance. All kappa values were found to be high, at 0.65 and 0.69 for interrater agreement and 0.80 for intrarater agreement; they climbed even higher, to 0.82, 0.84, and 0.87, respectively, when considering only clinically
J AM ACAD DERMATOL
Heffner et al 961
VOLUME 60, NUMBER 6
relevant disagreement. Though a subject of much debate, kappa values greater than 0.81 are generally considered to show very good agreement.10 All cases of intrarater disagreement were examined a third time by one participating dermatologist to analyze the cause of the disagreement between review of the photographs and the in-person visit. In 6 of the 24 cases, it was considered that a picture showing whole-body distribution of the rash, rather than close-up images alone, would have made the diagnosis easier. In 4 of the 24 cases, the skin lesions were too subtle to diagnose by photography. In the remaining 14 cases, no patterns causing misdiagnosis were noted. Before study onset, we hypothesized that certain demographic data would make a photographic diagnosis more challenging. However, diagnostic accuracy was not affected by the age of the child, the skin tone, previous treatment of the rash, nor the duration of symptoms before presentation. The outcome of this study has practical applications within the medical community. In a large geographic area with a small number of specialists, telemedicine has proven to be a useful tool. Knowing that pediatric skin conditions can be diagnosed accurately by SF teledermatology widens treatment options. In conditions such as atopic dermatitis, represented in high numbers in our study, initial therapy and family/patient education could begin before the next available appointment with a dermatologist. In conditions such as tinea capitis, therapy could be instituted by the primary care practitioner with greater certainty at the first visit without having to wait for the result of a fungal culture. There may be some resistance to institution of teledermatology, with regard to reimbursement and liability. Beginning in 2001, Medicare began to cover a wide range of telemedicine services.11 Medicaid reimburses telemedicine on a state-by-state basis; private insurers often follow Medicare/Medicaid guidelines, so practitioners may wish to review the laws of their state.12 In regards to liability, telemedicine services are frequently covered under general malpractice insurance and newer ‘‘telemedicine insurance policies’’ are becoming available.12
LIMITATIONS Although the data were collected in a tertiary care pediatric dermatology clinic in a large city, at the time of this study a 1-day referral system was in place, which allowed for acute conditions to be seen within 24 hours. This system ensured that acute as well as chronic conditions were included.
All of the photographs and initial data collection in this study were obtained only by the PI. However, as the PI had no formal training in medical photography and was using a simple point-and-shoot digital camera, we consider this less a limitation than a benefit with respect to the generalizability of the study. There is concern for bias by the primary dermatologist: seeing patient photographs and later the patient in-person, one would tend to agree with oneself. We addressed this by including the interrater agreement categories, showing that adjusted agreement was very similar in cases where two different dermatologists gave diagnoses on a patient, and one person gave two separate diagnoses on the same patient. Although arguably present, by showing such similar agreement and kappa values for all categories, we believe that this bias is negligible. REFERENCES 1. Dill S, Digiovanna J. Changing paradigms in dermatology: information technology. Clin Dermatol 2003;21:375-82. 2. Maglogiannis I. Design and implementation of a calibrated store and forward imaging system for teledermatology. J Med Syst 2004;28:455-67. 3. Whited J. Teledermatology research review. Int J Dermatol 2006;45:220-9. 4. Pak H, Harden D, Cruess D, Welch M, Poropatich R. National Capital Area Teledermatology Consortium. Teledermatology: an intraobserver diagnostic correlation study, part I. Cutis 2003;71:399-403. 5. High W, Houston M, Calobrisi S, Drage L, McEvoy M. Assessment of the accuracy of low-cost store-and-forward teledermatology consultation. J Am Acad Dermatol 2000;42: 776-83. 6. Phillips C, Burke W, Shechter A, Stone D, Balch D, Gustke S. Reliability of dermatology teleconsultations with the use of teleconferencing technology. J Am Acad Dermatol 1997;37: 398-402. 7. Baba M, Seckin D, Kapdagli S. A comparison of teledermatology using store-and-forward methodology alone, and in combination with Web camera videoconferencing. J Telemed Telecare 2005;11:354-60. 8. Whited J, Hall R, Simel D, Foy M, Stechuchak K, Drugge R, et al. Reliability and accuracy of dermatologists’ clinic-based and digital image consultations. J Am Acad Dermatol 1999;41: 693-702. 9. Gutierrez G. Medicare, the Internet, and the future of telemedicine. Critical Care Med 2001;29:N144-50. 10. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. 11. Lauderdale D, Lacsamama C, Palsbo S. Medicaid and telemedicine in 2002. Telehealth Pract Rep 2003;8:14-5. 12. Wachter G. Malpractice and telemedicine. Telemedicine liability: the uncharted waters of medical risk. Telemedicine Online Archives. Available at: http://tie.telemed.org/articles/article.asp? path=articles&article=malpracticeLiability_gw_tie02.xml. Accessed August 12, 2008.