- Email: [email protected]
Reliability of airway obstruction analyses from Sleep MRI sequences
Otolaryngology–Head and Neck Surgery (2010) 142, 526-530
ORIGINAL RESEARCH–SLEEP MEDICINE
Reliability of airway obstruction analyses from Sleep MRI sequences Jose E. Barrera, MD, Ray C. Chang, MD, Gerald R. Popelka, PhD, and Andrew B. Holbrook, MS, Palo Alto, CA; and Lackland AFB, TX Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article. ABSTRACT OBJECTIVE: A real-time MRI “movie” during natural sleep (“Sleep MRI”) allows observation of dynamic airway obstructions in obstructive sleep apnea syndrome (OSAS) patients. The purpose of this article was to determine the reliability of assessing these obstructions. STUDY DESIGN: Cross-sectional diagnostic test evaluation. SETTING: Academic referral center. SUBJECTS AND METHODS: A total of 23 Sleep MRI sequences randomly selected from 20 consecutive OSAS patients were analyzed by two experienced sleep surgeons on two separate occasions separated by at least two weeks. Five dimensions were assessed: presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and duration of an obstructive event. RESULTS: For all dimensions measured, intra-rater reliability coefficients ranged from a low of 0.95 to a high of 1.0 for each rater. Inter-rater reliability coefficients ranged from a low of 0.85 to a high of 1.0. On two separate evaluations separated by at least two weeks, rater 1 identified a retropalatal obstruction in 100 percent of sequences, whereas rater 2 did so in 91 percent and 96 percent of the sequences, respectively. Retroglossal obstruction was identified in 57 percent (rater 1) and 65 percent (rater 2) of sequences. CONCLUSION: Intra-rater and inter-rater reliability coefficients are very high for determination of presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and duration of obstruction from Sleep MRI sequences in OSAS patients. © 2010 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved.
T
he gold standard in the diagnosis of obstructive sleep apnea syndrome (OSAS) is polysomnography (PSG). However, PSG does not provide any direct information regarding the dynamic nature of airway obstructions. This is a significant limitation because multiple structures
in the airway can cause a respiratory event owing to airway narrowing or obstruction, and each structure can be involved to different degrees. More importantly, surgical intervention planning is hampered substantially if the specific site and nature of the obstructions are not known. Certain complementary measures such as druginduced sleep endoscopy (DISE) do provide direct dynamic airway information and can characterize the upper airway collapse that occurs during sedation.1 However, DISE employs measures under sleeping conditions that are less than natural, including sedation, presence of endoscopic instruments, and an artificial surgical environment. The ideal complementary tool would allow natural sleep without sedation and without instruments in the airway, and characterize number, location, type, magnitude, and duration of upper airway obstructions, and narrowing coincident with respiratory events. Sleep magnetic resonance imaging (Sleep MRI) is a real-time imaging procedure that produces a “movie” of the airway in the sagittal plane during natural sleep. This imaging sequence is simultaneously synchronized with MRI-compatible physiological measures of peripheral arterial tone, hemoglobin oxygen saturation, and pulse rate. Sleep MRI has been described previously2 and has been shown to be a safe, feasible approach that can be used to characterize the anatomical site, magnitude, and duration of airway obstruction under dynamic conditions in OSAS patients. However, analysis of images of moving structures is quite different from that obtained from static images. It remains unknown if analysis of the airway structures under dynamic conditions produces repeatable results. The goal of this study was to report the inter- and intrarater reliability of Sleep MRI analyses in OSAS patients for a variety of conditions, including presence or absence of any obstruction, duration of obstruction, presence or absence of a retropalatal (RP) obstruction, presence or absence of a retroglossal (RG) obstruction, and presence or absence of a swallow.
Received August 24, 2009; revised November 17, 2009; accepted January 6, 2010.
0194-5998/$36.00 © 2010 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved. doi:10.1016/j.otohns.2010.01.003
Barrera et al
Reliability of airway obstruction analyses from . . .
Methods Study Design The study design was a prospective, single clinical site, nonrandomized, open enrollment case series of 20 adult OSAS subjects and 15 control subjects without OSA, recruited from the Department of Otolaryngology, Stanford University, Palo Alto, California. Funding for this study was provided by the departments of otolaryngology and radiology. The study protocol was approved by the institutional review board and written informed consent was obtained. This prospective cohort study included patients seen by one author (J.E.B.). Control subjects were randomly chosen from volunteers who responded to advertisements. Control subjects underwent overnight sleep study performed with the Watch-PAT 100 (Itamar Medical, Caesarea, Israel), whereas OSA subjects received a standard overnight PSG. A volumetric MRI was performed in both controls and OSA subjects. The Sleep MRI technique has been described previously.2 All OSA patients underwent a 90-minute Sleep MRI nap conducted in an open magnet MRI system (0.5 T Signa SP; General Electric, Waukesha, WI) after their volumetric MRI. This system has twin magnets oriented vertically and separated by 60 cm, allowing the upper torso and head of the subject to be located in the area between the magnets and providing convenient access to the physiological measurement apparatus. Subject position was based on the subject’s preferred position to facilitate sleep, which was supine for 19 of 20 subjects. Although it is recognized that this position affects the severity of obstruction in PSG and multiple sleep latency tests,3 our goal was to get the subjects to sleep within the MRI scanner. No sedation was used for the evaluation. The mean total sleep time was 63 minutes during the 90-minute period. The measurements resulted in real-time movies of the midsagittal plane showing the complete upper airway and its surrounding structures. Figure 1 demonstrates retropala-
527
tal and retroglossal obstruction in a sample patient. The image resolution in the figure is affected by several factors including frame rate (12 per second) and freezing of the movie frames to show the respective obstruction. These imaging sequences were synchronized with ongoing physiological measures monitored by the Watch-PAT 100.4 Physiological measurement variables included peripheral arterial tone (PAT), oxyhemoglobin saturation level, pulse rate, and wrist actigraphy. The awake versus sleeping state was determined with the actigraphy measures. While asleep, the Sleep MRI recordings were taken every five minutes with a duration of 30 seconds for each recording. The average number of 30second Sleep MRI recordings per patient depended on how long the subject slept and ranged from two to 28 recordings per patient, with an average of 12.3 recordings per patient. PAT attenuation corresponds with respiratory events in Sleep MRI;2 therefore, PAT attenuation was used to determine the most significant respiratory events. To avoid extensive analysis of non-relevant recordings and to randomly sample the total number of Sleep MRI recordings, we evaluated only those recordings associated with three PAT attenuation categories (the smallest, the largest, and the one closest to the mean PAT attenuation value). A total of 23 MRI recordings were rated by two experienced sleep surgeons (authors of this paper, J.E.B. and R.C.C.) on two occasions separated by at least two weeks. The two surgeons were blind to the identification of the patients and all other patient-related information. However, one surgeon (J.E.B.) had potential recall of history and physical examination findings, sleep study results, and treatment plan. An airway obstruction was defined as a narrowing greater than 50 percent of the airway diameter for either one of two airway sites (retropalatal or retroglossal). Five dimensions were assessed. The first dimension was a determination of the presence or absence of any obstruction. For the second and third dimensions, the location of obstruction
Figure 1 (A) An image for a patient with airway obstruction at the retropalatal and retroglossal region (denoted by arrows). (B) An image for a patient with open posterior airway space.
528
Otolaryngology–Head and Neck Surgery, Vol 142, No 4, April 2010
was rated as occurring at the retropalatal area (owing to palate or lateral/posterior pharyngeal wall movement) or at the retroglossal area (secondary to hypopharyngeal obstruction from the tongue, epiglottis, or lateral pharyngeal wall). The fourth dimension was the presence or absence of a false obstruction attributable to a swallow (a short obstruction lasting less than 3 seconds) compared with a true obstruction (a site-specific obstruction lasting greater than 10 seconds). The ratings for obstructions between these two time periods were not obtained during this study. The fifth dimension was a determination of the duration of an obstructive event as measured from the time indicator in the image sequence in seconds. The presence versus absence data were analyzed by comparing the ratings for various conditions using a reliability statistic (Cronbach’s alpha, Aabel 2009; Gigawiz, Ltd., Oklahoma City, OK). The duration estimation data were analyzed by means of a correlation coefficient (Pearson Product-Moment Correlation Coefficient 2009; Aabel). Analysis was performed only on selected portions of the 30-second Sleep MRI recordings for several reasons. First, the temporal relation between these physiological measures and the airway conditions is completely unknown. For example, it is unknown if a drop in oxyhemoglobin saturation occurs at the onset of, for the duration of, or after an apnea. In a seminal study describing the use of PAT for OSA, Schnall et al5 defined a significant arousal as a 33 percent decrease of PAT from baseline for three to 30 seconds or a 15 percent increase in heart rate. This concept was carried over to Sleep MRI analysis in which we analyzed only the MRI recordings temporally surrounding PAT attenuation to identify a respiratory event (RE) (Figs 1 and 2). In addition, it has been previously determined that the precise site of airway obstruction as measured by Sleep MRI is associated with REs in OSAS patients.2 An airway obstruction on Sleep MRI was defined as an isolated palate or base of tongue obstruction or a combined retropalatal and retroglossal complete obstruction, as opposed to a long duration narrowing at these locations or an isolated epiglottis obstruction. An attempt to characterize each obstruction was performed to determine the location of obstruction at both the retropalatal and retro-
Table 1 Intra-rater results for rating 1 versus rating 2
Obstruction Retropalatal Retroglossal Swallow
Rater 1
Rater 2
1.00 1.00 0.95 0.95
1.00 1.00 1.00 0.95
glossal locations. Retropalatal obstruction was evaluated for the presence of palatal and posterior pharyngeal wall displacement. Retroglossal obstruction was evaluated for the presence of tongue base, posterior pharyngeal wall, or epiglottic displacements. In addition, a swallow was distinguished from an airway obstruction by the presence of a bolus in contrast to an airway obstruction, which did not incorporate a bolus but demonstrated dynamic site-specific and repeatable obstruction lasting greater than 10 seconds. Moreover, during Sleep MRI a swallow is characterized by a rise of the soft palate combined with the elevation and anterior movement of the larynx and hyoid bone.
Results Twenty OSA patients and 15 control subjects who underwent MRI evaluation between 2006 and 2008 were prospectively enrolled. All patients were diagnosed by sleep study. The mean OSA patient age was 40.2 ⫾ 10.1 years (range 21-53) years, and gender breakdown was 33 percent female (5 of 15). The mean body mass index was 27.5 ⫾ 4.6 kg/m2. The control group mean age was 32.4 ⫾ 7.0 years (range 24 to 52) years, with a gender breakdown of 20 percent female (4 of 20). The mean body mass index was 23.2 ⫾ 2.9 kg/m2. The control AHI was 3.35 ⫾ 3.1 events per hour with lowest oxyhemoglobin saturation (LSAT) of 93.8 ⫾ 1.5 percent. Control subjects clearly did not have OSA. The Cronbach alpha coefficient is a common statistic used for assessment of reliability of observer ratings, which ranges from 0 (no reliability) to 1.00 (perfect reliability). In cases in which the ratings are identical for two observations, the SD goes to zero and an expected alpha of 1.00 cannot be computed. In these cases an alpha coefficient of 1.00 was reported. The results of intra-rater and inter-rater reliability analyses are shown in Tables 1 and 2.
Table 2 Inter-rater results for rater 1 versus rater 2
Figure 2 Peripheral arterial tone (PAT) amplitude (A) and oxygen saturation (B) during patient sleep over time (in seconds) occurring during a respiratory event associated with Figure 1A.
Obstruction Retropalatal Retroglossal Swallow Duration
1.00 1.00 0.85 0.89 0.81
Barrera et al
Reliability of airway obstruction analyses from . . .
Intra-Rater Reliability Results for Rater 1 Rater 1 determined that airway obstructions were present in 100 percent of the recordings on both the first observation and the second observation in every Sleep MRI sequence (Cronbach alpha ⫽ 1.00). This rater also determined that a retropalatal obstruction was present in 100 percent of the sequences on the first observation and 95 percent of the sequences on the second observation (Cronbach alpha ⫽ 1.00). Rater 1 determined that a retroglossal obstruction was present in 61 percent of the sequences on the first observation and 65 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95) and that a swallow was present in 35 percent of the sequences on the first observation and 39 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95). Intra-rater reliability for rater 1 was very high for all for dimensions.
Intra-Rater Reliability Results for Rater 2 Intra-rater reliability results for rater 2 were similar to those for rater 1. Rater 2 also determined that airway obstructions were present in 100 percent of the sequences on both the first observation and the second observation on every Sleep MRI sequence (Cronbach alpha ⫽ 1.00). Rater 2 also determined that a retropalatal obstruction was present in 100 percent of the sequences on the first observation and 100 percent of the sequences on the second observation (Cronbach alpha ⫽ 1.00). Rater 2 determined that a retroglossal obstruction was present in 57 percent of the sequences on both the first and the second observations (Cronbach alpha ⬎ 0.95) and that a swallow was present in 35 percent of the sequences on the first observation and 39 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95).
Inter-Rater Reliability Results Inter-rater reliability results were obtained by comparing the first observation ratings for each rater. Rater 1 and rater 2 each determined that airway obstructions were present in 100 percent of the sequences (Cronbach alpha ⫽ 1.00) and that a retropalatal obstruction was present in 100 percent of the sequences (Cronbach alpha ⫽ 1.00). Rater 1 determined that a retroglossal obstruction was present in 61 percent of the sequences, whereas rater 2 determined that a retroglossal obstruction was present in 57 percent of the sequences (Cronbach alpha ⬎ 0.85). Rater 1 and rater 2 determined that a swallow was present in 35 percent of the sequences (Cronbach alpha ⬎ 0.89). Each rater determined duration of obstruction only once. These durations ranged from four seconds to 23 seconds for rater 1 and from six seconds to 21 seconds for rater 2. The correlation coefficient for duration determinations for these two raters was 0.81.
Discussion For all dimensions, intra-rater reliability coefficients ranged from a low of 0.95 to a high of 1.0 for each rater. Inter-rater
529
reliability coefficients ranged from a low of 0.85 to a high of 1.0. The ratings consisted of categories (yes or no) for presence or absence of an obstruction at two airway locations (retropalatal, retroglossal) in 23 sequences analyzed separately at least 14 days apart. Even though there was just a single discrepancy between raters for the retropalatal and retroglossal locations, the correlation coefficients were not the same because retropalatal had almost all of one category (yes), whereas retroglossal had a mixture (yes and no). Estimates of the durations of the obstructions were also very reliable between raters. Analyses of Sleep MRI have high test-retest reliability for determination of both retropalatal and retroglossal airway obstructions. Intra-rater and inter-rater reliability for determination of presence or absence of any obstruction, retropalatal obstruction, retroglossal obstruction, swallow, and duration of obstruction from real-time Sleep MRI sequences are also very high. This is true despite multiple points of overlap associated with the categories measured. Correlation was lowest in consideration of analysis of retroglossal obstructions among the two raters separated by 14 days (82% for the initial review, and 87% for the second review). Because increased heterogeneity exists for retroglossal obstructions, the correlation is expected to be lower for retroglossal than retropalatal analyses. Reliability for retropalatal categorization was 91 percent and 96 percent on subsequent trials. Note that intra-rater correlation improved between the first and second trial, demonstrating that a small learning effect may be associated with analyses of Sleep MRI. The test-retest reliability of DISE has been recently published. Rodriguez-Bruno et al1 noted that with DISE it is difficult to simplify the relationships between two separate anatomical regions such as the palate and hypopharynx owing to the dynamic interactions that are not understood completely. In Sleep MRI, we attempt to evaluate these two independent but yet dynamic events with the hope of better understanding the locations and dynamics of airway obstructions in OSAS patients. We also present a regional and site-specific method to better characterize the pattern of airway obstruction and its location, and to improve knowledge to guide surgical treatment and advancements. Advantages of Sleep MRI are its ability to be performed during natural sleep and in less than 90 minutes of recording time. Although not investigated in this study owing to correlation of sympathetic tone and therefore PAT with rapid eye movement sleep,6 Sleep MRI, if measured beyond a 90-minute nap, has the potential to provide information not only on REs, but also their relationship and variation with sleep stage, and sleep architecture disruption. Sleep MRI can be assessed independently of the surgeon, thus decreasing bias concerning specific airway obstruction identification. The goal is not to provide an alternative to PSG, but to establish a reliable tool for pre- and postsurgical evaluation.
530
Otolaryngology–Head and Neck Surgery, Vol 142, No 4, April 2010
Limitations associated with Sleep MRI include cost. The use of the Sleep MRI scanner approximated $1000 for a two-hour study. Therefore, only volumetric MRI analysis was performed on controls, whereas volumetric and dynamic data were obtained on OSA patients. The cost may improve with an overnight study format because MRI facilities usually have a decrease in usage during evening and early morning hours. At our research center, the cost of overnight testing drops substantially to $100 per hour. The disposable cost for the PAT probe is $80 and is designed for single use. The current study was performed on an interventional 0.5-tesla (T) MRI platform. Imaging with this magnet limits resolution because of both the weak magnetic field strength and its weak gradients (a standard clinical system is 1.5 or 3.0 T with maximum gradient strengths of 4 G/cm, compared with 0.5 T with a maximum gradient strength of 1.2 G/cm). Field strength is directly related to the signal-tonoise ratio of the images, which limits the possible image resolution, and improved gradient strengths allow for faster imaging sequences. The current study was limited to the 0.5-T magnet because the Watch-PAT 100 contained some ferrous materials; while the device effects were tolerable at this field strength, at three to six times the field strength the effects could be unsafe.
Conclusion Intra-rater and inter-rater reliability coefficients are very high for determination of presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and estimates of obstruction duration from real-time Sleep MRI sequences.
Author Information From the Division of Sleep Surgery, Department of Otolaryngology (Drs. Barrera, Chang, and Popelka) and the Division of Bioengineering (Dr. Holbrook), Stanford University; and the Division of Sleep Surgery, Department of Otolaryngology, Wilford Hall Medical Center (Dr. Barrera), Lackland AFB.
Corresponding author: Jose E. Barrera, MD, 59 SSS/SGO20, 2200 Bergquist Dr., Ste. 1, Lackland AFB, TX 78236-9908. E-mail address: [email protected]. This article was presented at the 2009 AAO–HNSF Annual Meeting & OTO EXPO, San Diego, CA, October 4-7, 2009.
Author Contributions Jose E. Barrera, conception, design, acquisition of data, analysis, interpretation, drafting, revising, final approval; Ray C. Chang, acquisition of data, analysis, revising, final approval; Gerald R. Popelka, design, acquisition of data, analysis, revising, final approval; Andrew B. Holbrook, acquisition of data, revising, final approval.
Disclosures Competing interests: Jose E. Barrera, developer: Endormir Sleep Solutions, LLC (aims to market Sleep MRI invention; no financial relationship exists), consultant: Ethicon, Inc. (no affiliation with current research), royalty agreement with Stanford University as inventor; Gerald R. Popelka, royalty agreement with Stanford University as inventor; Andrew B. Holbrook, royalty agreement with Stanford University as inventor. Sponsorships: None.
References 1. Rodriguez-Bruno K, Goldberg AN, McCulloch CE, et al. Test-retest reliability of drug-induced sleep endoscopy. Otolaryngol Head Neck Surg 2009;140:646 –51. 2. Barrera JE, Holbrook HS, Santos J, et al. Sleep MRI: novel technique to identify airway obstruction in obstructive sleep apnea. Otolaryngol Head Neck Surg 2009;140:423–5. 3. Oksenberg A, Silverberg DS, Arons E, et al. Positional vs nonpositional obstructive sleep apnea patients: anthropomorphic, nocturnal polysomnographic, and multiple sleep latency test data. Chest 1997;112:629 –39. 4. Ayas NT, Pittman S, MacDonald M, et al. Assessment of a wrist-worn device in the detection of obstructive sleep apnea. Sleep Med 2003;4: 435– 42. 5. Schnall RP, Shlitner A, Sheffy J, et al. Periodic, profound peripheral vasoconstriction. A new marker of obstructive sleep apnea. Sleep 1999; 22:939 – 46. 6. Penzel T, Kesper K, Ploch T, et al. Ambulatory recording of sleep apnea using peripheral arterial tonometry. Conf Proc IEEE Eng Med Biol Soc 2004;5:3856 –9.
ORIGINAL RESEARCH–SLEEP MEDICINE
Reliability of airway obstruction analyses from Sleep MRI sequences Jose E. Barrera, MD, Ray C. Chang, MD, Gerald R. Popelka, PhD, and Andrew B. Holbrook, MS, Palo Alto, CA; and Lackland AFB, TX Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article. ABSTRACT OBJECTIVE: A real-time MRI “movie” during natural sleep (“Sleep MRI”) allows observation of dynamic airway obstructions in obstructive sleep apnea syndrome (OSAS) patients. The purpose of this article was to determine the reliability of assessing these obstructions. STUDY DESIGN: Cross-sectional diagnostic test evaluation. SETTING: Academic referral center. SUBJECTS AND METHODS: A total of 23 Sleep MRI sequences randomly selected from 20 consecutive OSAS patients were analyzed by two experienced sleep surgeons on two separate occasions separated by at least two weeks. Five dimensions were assessed: presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and duration of an obstructive event. RESULTS: For all dimensions measured, intra-rater reliability coefficients ranged from a low of 0.95 to a high of 1.0 for each rater. Inter-rater reliability coefficients ranged from a low of 0.85 to a high of 1.0. On two separate evaluations separated by at least two weeks, rater 1 identified a retropalatal obstruction in 100 percent of sequences, whereas rater 2 did so in 91 percent and 96 percent of the sequences, respectively. Retroglossal obstruction was identified in 57 percent (rater 1) and 65 percent (rater 2) of sequences. CONCLUSION: Intra-rater and inter-rater reliability coefficients are very high for determination of presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and duration of obstruction from Sleep MRI sequences in OSAS patients. © 2010 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved.
T
he gold standard in the diagnosis of obstructive sleep apnea syndrome (OSAS) is polysomnography (PSG). However, PSG does not provide any direct information regarding the dynamic nature of airway obstructions. This is a significant limitation because multiple structures
in the airway can cause a respiratory event owing to airway narrowing or obstruction, and each structure can be involved to different degrees. More importantly, surgical intervention planning is hampered substantially if the specific site and nature of the obstructions are not known. Certain complementary measures such as druginduced sleep endoscopy (DISE) do provide direct dynamic airway information and can characterize the upper airway collapse that occurs during sedation.1 However, DISE employs measures under sleeping conditions that are less than natural, including sedation, presence of endoscopic instruments, and an artificial surgical environment. The ideal complementary tool would allow natural sleep without sedation and without instruments in the airway, and characterize number, location, type, magnitude, and duration of upper airway obstructions, and narrowing coincident with respiratory events. Sleep magnetic resonance imaging (Sleep MRI) is a real-time imaging procedure that produces a “movie” of the airway in the sagittal plane during natural sleep. This imaging sequence is simultaneously synchronized with MRI-compatible physiological measures of peripheral arterial tone, hemoglobin oxygen saturation, and pulse rate. Sleep MRI has been described previously2 and has been shown to be a safe, feasible approach that can be used to characterize the anatomical site, magnitude, and duration of airway obstruction under dynamic conditions in OSAS patients. However, analysis of images of moving structures is quite different from that obtained from static images. It remains unknown if analysis of the airway structures under dynamic conditions produces repeatable results. The goal of this study was to report the inter- and intrarater reliability of Sleep MRI analyses in OSAS patients for a variety of conditions, including presence or absence of any obstruction, duration of obstruction, presence or absence of a retropalatal (RP) obstruction, presence or absence of a retroglossal (RG) obstruction, and presence or absence of a swallow.
Received August 24, 2009; revised November 17, 2009; accepted January 6, 2010.
0194-5998/$36.00 © 2010 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved. doi:10.1016/j.otohns.2010.01.003
Barrera et al
Reliability of airway obstruction analyses from . . .
Methods Study Design The study design was a prospective, single clinical site, nonrandomized, open enrollment case series of 20 adult OSAS subjects and 15 control subjects without OSA, recruited from the Department of Otolaryngology, Stanford University, Palo Alto, California. Funding for this study was provided by the departments of otolaryngology and radiology. The study protocol was approved by the institutional review board and written informed consent was obtained. This prospective cohort study included patients seen by one author (J.E.B.). Control subjects were randomly chosen from volunteers who responded to advertisements. Control subjects underwent overnight sleep study performed with the Watch-PAT 100 (Itamar Medical, Caesarea, Israel), whereas OSA subjects received a standard overnight PSG. A volumetric MRI was performed in both controls and OSA subjects. The Sleep MRI technique has been described previously.2 All OSA patients underwent a 90-minute Sleep MRI nap conducted in an open magnet MRI system (0.5 T Signa SP; General Electric, Waukesha, WI) after their volumetric MRI. This system has twin magnets oriented vertically and separated by 60 cm, allowing the upper torso and head of the subject to be located in the area between the magnets and providing convenient access to the physiological measurement apparatus. Subject position was based on the subject’s preferred position to facilitate sleep, which was supine for 19 of 20 subjects. Although it is recognized that this position affects the severity of obstruction in PSG and multiple sleep latency tests,3 our goal was to get the subjects to sleep within the MRI scanner. No sedation was used for the evaluation. The mean total sleep time was 63 minutes during the 90-minute period. The measurements resulted in real-time movies of the midsagittal plane showing the complete upper airway and its surrounding structures. Figure 1 demonstrates retropala-
527
tal and retroglossal obstruction in a sample patient. The image resolution in the figure is affected by several factors including frame rate (12 per second) and freezing of the movie frames to show the respective obstruction. These imaging sequences were synchronized with ongoing physiological measures monitored by the Watch-PAT 100.4 Physiological measurement variables included peripheral arterial tone (PAT), oxyhemoglobin saturation level, pulse rate, and wrist actigraphy. The awake versus sleeping state was determined with the actigraphy measures. While asleep, the Sleep MRI recordings were taken every five minutes with a duration of 30 seconds for each recording. The average number of 30second Sleep MRI recordings per patient depended on how long the subject slept and ranged from two to 28 recordings per patient, with an average of 12.3 recordings per patient. PAT attenuation corresponds with respiratory events in Sleep MRI;2 therefore, PAT attenuation was used to determine the most significant respiratory events. To avoid extensive analysis of non-relevant recordings and to randomly sample the total number of Sleep MRI recordings, we evaluated only those recordings associated with three PAT attenuation categories (the smallest, the largest, and the one closest to the mean PAT attenuation value). A total of 23 MRI recordings were rated by two experienced sleep surgeons (authors of this paper, J.E.B. and R.C.C.) on two occasions separated by at least two weeks. The two surgeons were blind to the identification of the patients and all other patient-related information. However, one surgeon (J.E.B.) had potential recall of history and physical examination findings, sleep study results, and treatment plan. An airway obstruction was defined as a narrowing greater than 50 percent of the airway diameter for either one of two airway sites (retropalatal or retroglossal). Five dimensions were assessed. The first dimension was a determination of the presence or absence of any obstruction. For the second and third dimensions, the location of obstruction
Figure 1 (A) An image for a patient with airway obstruction at the retropalatal and retroglossal region (denoted by arrows). (B) An image for a patient with open posterior airway space.
528
Otolaryngology–Head and Neck Surgery, Vol 142, No 4, April 2010
was rated as occurring at the retropalatal area (owing to palate or lateral/posterior pharyngeal wall movement) or at the retroglossal area (secondary to hypopharyngeal obstruction from the tongue, epiglottis, or lateral pharyngeal wall). The fourth dimension was the presence or absence of a false obstruction attributable to a swallow (a short obstruction lasting less than 3 seconds) compared with a true obstruction (a site-specific obstruction lasting greater than 10 seconds). The ratings for obstructions between these two time periods were not obtained during this study. The fifth dimension was a determination of the duration of an obstructive event as measured from the time indicator in the image sequence in seconds. The presence versus absence data were analyzed by comparing the ratings for various conditions using a reliability statistic (Cronbach’s alpha, Aabel 2009; Gigawiz, Ltd., Oklahoma City, OK). The duration estimation data were analyzed by means of a correlation coefficient (Pearson Product-Moment Correlation Coefficient 2009; Aabel). Analysis was performed only on selected portions of the 30-second Sleep MRI recordings for several reasons. First, the temporal relation between these physiological measures and the airway conditions is completely unknown. For example, it is unknown if a drop in oxyhemoglobin saturation occurs at the onset of, for the duration of, or after an apnea. In a seminal study describing the use of PAT for OSA, Schnall et al5 defined a significant arousal as a 33 percent decrease of PAT from baseline for three to 30 seconds or a 15 percent increase in heart rate. This concept was carried over to Sleep MRI analysis in which we analyzed only the MRI recordings temporally surrounding PAT attenuation to identify a respiratory event (RE) (Figs 1 and 2). In addition, it has been previously determined that the precise site of airway obstruction as measured by Sleep MRI is associated with REs in OSAS patients.2 An airway obstruction on Sleep MRI was defined as an isolated palate or base of tongue obstruction or a combined retropalatal and retroglossal complete obstruction, as opposed to a long duration narrowing at these locations or an isolated epiglottis obstruction. An attempt to characterize each obstruction was performed to determine the location of obstruction at both the retropalatal and retro-
Table 1 Intra-rater results for rating 1 versus rating 2
Obstruction Retropalatal Retroglossal Swallow
Rater 1
Rater 2
1.00 1.00 0.95 0.95
1.00 1.00 1.00 0.95
glossal locations. Retropalatal obstruction was evaluated for the presence of palatal and posterior pharyngeal wall displacement. Retroglossal obstruction was evaluated for the presence of tongue base, posterior pharyngeal wall, or epiglottic displacements. In addition, a swallow was distinguished from an airway obstruction by the presence of a bolus in contrast to an airway obstruction, which did not incorporate a bolus but demonstrated dynamic site-specific and repeatable obstruction lasting greater than 10 seconds. Moreover, during Sleep MRI a swallow is characterized by a rise of the soft palate combined with the elevation and anterior movement of the larynx and hyoid bone.
Results Twenty OSA patients and 15 control subjects who underwent MRI evaluation between 2006 and 2008 were prospectively enrolled. All patients were diagnosed by sleep study. The mean OSA patient age was 40.2 ⫾ 10.1 years (range 21-53) years, and gender breakdown was 33 percent female (5 of 15). The mean body mass index was 27.5 ⫾ 4.6 kg/m2. The control group mean age was 32.4 ⫾ 7.0 years (range 24 to 52) years, with a gender breakdown of 20 percent female (4 of 20). The mean body mass index was 23.2 ⫾ 2.9 kg/m2. The control AHI was 3.35 ⫾ 3.1 events per hour with lowest oxyhemoglobin saturation (LSAT) of 93.8 ⫾ 1.5 percent. Control subjects clearly did not have OSA. The Cronbach alpha coefficient is a common statistic used for assessment of reliability of observer ratings, which ranges from 0 (no reliability) to 1.00 (perfect reliability). In cases in which the ratings are identical for two observations, the SD goes to zero and an expected alpha of 1.00 cannot be computed. In these cases an alpha coefficient of 1.00 was reported. The results of intra-rater and inter-rater reliability analyses are shown in Tables 1 and 2.
Table 2 Inter-rater results for rater 1 versus rater 2
Figure 2 Peripheral arterial tone (PAT) amplitude (A) and oxygen saturation (B) during patient sleep over time (in seconds) occurring during a respiratory event associated with Figure 1A.
Obstruction Retropalatal Retroglossal Swallow Duration
1.00 1.00 0.85 0.89 0.81
Barrera et al
Reliability of airway obstruction analyses from . . .
Intra-Rater Reliability Results for Rater 1 Rater 1 determined that airway obstructions were present in 100 percent of the recordings on both the first observation and the second observation in every Sleep MRI sequence (Cronbach alpha ⫽ 1.00). This rater also determined that a retropalatal obstruction was present in 100 percent of the sequences on the first observation and 95 percent of the sequences on the second observation (Cronbach alpha ⫽ 1.00). Rater 1 determined that a retroglossal obstruction was present in 61 percent of the sequences on the first observation and 65 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95) and that a swallow was present in 35 percent of the sequences on the first observation and 39 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95). Intra-rater reliability for rater 1 was very high for all for dimensions.
Intra-Rater Reliability Results for Rater 2 Intra-rater reliability results for rater 2 were similar to those for rater 1. Rater 2 also determined that airway obstructions were present in 100 percent of the sequences on both the first observation and the second observation on every Sleep MRI sequence (Cronbach alpha ⫽ 1.00). Rater 2 also determined that a retropalatal obstruction was present in 100 percent of the sequences on the first observation and 100 percent of the sequences on the second observation (Cronbach alpha ⫽ 1.00). Rater 2 determined that a retroglossal obstruction was present in 57 percent of the sequences on both the first and the second observations (Cronbach alpha ⬎ 0.95) and that a swallow was present in 35 percent of the sequences on the first observation and 39 percent of the sequences on the second observation (Cronbach alpha ⬎ 0.95).
Inter-Rater Reliability Results Inter-rater reliability results were obtained by comparing the first observation ratings for each rater. Rater 1 and rater 2 each determined that airway obstructions were present in 100 percent of the sequences (Cronbach alpha ⫽ 1.00) and that a retropalatal obstruction was present in 100 percent of the sequences (Cronbach alpha ⫽ 1.00). Rater 1 determined that a retroglossal obstruction was present in 61 percent of the sequences, whereas rater 2 determined that a retroglossal obstruction was present in 57 percent of the sequences (Cronbach alpha ⬎ 0.85). Rater 1 and rater 2 determined that a swallow was present in 35 percent of the sequences (Cronbach alpha ⬎ 0.89). Each rater determined duration of obstruction only once. These durations ranged from four seconds to 23 seconds for rater 1 and from six seconds to 21 seconds for rater 2. The correlation coefficient for duration determinations for these two raters was 0.81.
Discussion For all dimensions, intra-rater reliability coefficients ranged from a low of 0.95 to a high of 1.0 for each rater. Inter-rater
529
reliability coefficients ranged from a low of 0.85 to a high of 1.0. The ratings consisted of categories (yes or no) for presence or absence of an obstruction at two airway locations (retropalatal, retroglossal) in 23 sequences analyzed separately at least 14 days apart. Even though there was just a single discrepancy between raters for the retropalatal and retroglossal locations, the correlation coefficients were not the same because retropalatal had almost all of one category (yes), whereas retroglossal had a mixture (yes and no). Estimates of the durations of the obstructions were also very reliable between raters. Analyses of Sleep MRI have high test-retest reliability for determination of both retropalatal and retroglossal airway obstructions. Intra-rater and inter-rater reliability for determination of presence or absence of any obstruction, retropalatal obstruction, retroglossal obstruction, swallow, and duration of obstruction from real-time Sleep MRI sequences are also very high. This is true despite multiple points of overlap associated with the categories measured. Correlation was lowest in consideration of analysis of retroglossal obstructions among the two raters separated by 14 days (82% for the initial review, and 87% for the second review). Because increased heterogeneity exists for retroglossal obstructions, the correlation is expected to be lower for retroglossal than retropalatal analyses. Reliability for retropalatal categorization was 91 percent and 96 percent on subsequent trials. Note that intra-rater correlation improved between the first and second trial, demonstrating that a small learning effect may be associated with analyses of Sleep MRI. The test-retest reliability of DISE has been recently published. Rodriguez-Bruno et al1 noted that with DISE it is difficult to simplify the relationships between two separate anatomical regions such as the palate and hypopharynx owing to the dynamic interactions that are not understood completely. In Sleep MRI, we attempt to evaluate these two independent but yet dynamic events with the hope of better understanding the locations and dynamics of airway obstructions in OSAS patients. We also present a regional and site-specific method to better characterize the pattern of airway obstruction and its location, and to improve knowledge to guide surgical treatment and advancements. Advantages of Sleep MRI are its ability to be performed during natural sleep and in less than 90 minutes of recording time. Although not investigated in this study owing to correlation of sympathetic tone and therefore PAT with rapid eye movement sleep,6 Sleep MRI, if measured beyond a 90-minute nap, has the potential to provide information not only on REs, but also their relationship and variation with sleep stage, and sleep architecture disruption. Sleep MRI can be assessed independently of the surgeon, thus decreasing bias concerning specific airway obstruction identification. The goal is not to provide an alternative to PSG, but to establish a reliable tool for pre- and postsurgical evaluation.
530
Otolaryngology–Head and Neck Surgery, Vol 142, No 4, April 2010
Limitations associated with Sleep MRI include cost. The use of the Sleep MRI scanner approximated $1000 for a two-hour study. Therefore, only volumetric MRI analysis was performed on controls, whereas volumetric and dynamic data were obtained on OSA patients. The cost may improve with an overnight study format because MRI facilities usually have a decrease in usage during evening and early morning hours. At our research center, the cost of overnight testing drops substantially to $100 per hour. The disposable cost for the PAT probe is $80 and is designed for single use. The current study was performed on an interventional 0.5-tesla (T) MRI platform. Imaging with this magnet limits resolution because of both the weak magnetic field strength and its weak gradients (a standard clinical system is 1.5 or 3.0 T with maximum gradient strengths of 4 G/cm, compared with 0.5 T with a maximum gradient strength of 1.2 G/cm). Field strength is directly related to the signal-tonoise ratio of the images, which limits the possible image resolution, and improved gradient strengths allow for faster imaging sequences. The current study was limited to the 0.5-T magnet because the Watch-PAT 100 contained some ferrous materials; while the device effects were tolerable at this field strength, at three to six times the field strength the effects could be unsafe.
Conclusion Intra-rater and inter-rater reliability coefficients are very high for determination of presence or absence of any obstruction, presence or absence of a retropalatal obstruction, presence or absence of a retroglossal obstruction, presence or absence of a swallow, and estimates of obstruction duration from real-time Sleep MRI sequences.
Author Information From the Division of Sleep Surgery, Department of Otolaryngology (Drs. Barrera, Chang, and Popelka) and the Division of Bioengineering (Dr. Holbrook), Stanford University; and the Division of Sleep Surgery, Department of Otolaryngology, Wilford Hall Medical Center (Dr. Barrera), Lackland AFB.
Corresponding author: Jose E. Barrera, MD, 59 SSS/SGO20, 2200 Bergquist Dr., Ste. 1, Lackland AFB, TX 78236-9908. E-mail address: [email protected]. This article was presented at the 2009 AAO–HNSF Annual Meeting & OTO EXPO, San Diego, CA, October 4-7, 2009.
Author Contributions Jose E. Barrera, conception, design, acquisition of data, analysis, interpretation, drafting, revising, final approval; Ray C. Chang, acquisition of data, analysis, revising, final approval; Gerald R. Popelka, design, acquisition of data, analysis, revising, final approval; Andrew B. Holbrook, acquisition of data, revising, final approval.
Disclosures Competing interests: Jose E. Barrera, developer: Endormir Sleep Solutions, LLC (aims to market Sleep MRI invention; no financial relationship exists), consultant: Ethicon, Inc. (no affiliation with current research), royalty agreement with Stanford University as inventor; Gerald R. Popelka, royalty agreement with Stanford University as inventor; Andrew B. Holbrook, royalty agreement with Stanford University as inventor. Sponsorships: None.
References 1. Rodriguez-Bruno K, Goldberg AN, McCulloch CE, et al. Test-retest reliability of drug-induced sleep endoscopy. Otolaryngol Head Neck Surg 2009;140:646 –51. 2. Barrera JE, Holbrook HS, Santos J, et al. Sleep MRI: novel technique to identify airway obstruction in obstructive sleep apnea. Otolaryngol Head Neck Surg 2009;140:423–5. 3. Oksenberg A, Silverberg DS, Arons E, et al. Positional vs nonpositional obstructive sleep apnea patients: anthropomorphic, nocturnal polysomnographic, and multiple sleep latency test data. Chest 1997;112:629 –39. 4. Ayas NT, Pittman S, MacDonald M, et al. Assessment of a wrist-worn device in the detection of obstructive sleep apnea. Sleep Med 2003;4: 435– 42. 5. Schnall RP, Shlitner A, Sheffy J, et al. Periodic, profound peripheral vasoconstriction. A new marker of obstructive sleep apnea. Sleep 1999; 22:939 – 46. 6. Penzel T, Kesper K, Ploch T, et al. Ambulatory recording of sleep apnea using peripheral arterial tonometry. Conf Proc IEEE Eng Med Biol Soc 2004;5:3856 –9.