- Email: [email protected]
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers Bruno Bour, MD, Bruno Person, MD, Paul Cal~s, MD, Alain Blanchi, MD, Pascal Burtin, MD Fr~ddric Oberti, MD, Jean Boyer, MD, Mehdi Kaassis, MD, Nourredine Joundy, MD, Jo~l Fort, MD Le Mans, France
Background: The aim of this prospective study was to evaluate the interobserver agreement of stigmata of recent hemorrhage of bleeding peptic ulcers. Methods: Sixty-one consecutive adult patients were enrolled in the study and nine (three junior and six senior) endoscopists reviewed standardized video recordings of endoscopic examinations. Interobserver agreement was evaluated using the kappa (K) index, intraclass correlation coefficient, and proportion of agreement. Observer bias and poorly trained observers were investigated. Results: Interobserver agreement was very good for oozing (K = 0.68), good for clot (K = 0.51), poor for spurting (K = 0.29) and visible vessels (K = 0.33), and excellent for the absence of stigmata (K = 0.82). Observer bias sometimes occurred and the number of poorly trained observers was low. The K indexes were significantly better in senior than in junior investigators: 0.48 -+ 0.16 versus 0.37 - 0.26, respectively, p < 0.05. The agreement between the in vivo evaluation and video tape recordings (intraobserver agreement) was good (K = 0.60 -+ 0.19). There was no training phenomenon between the first and the second half of the patient group. Conclusions: The endoscopic classification of bleeding ulcers might be simplified by limiting grading to a few classes. Special attention should be paid to the training of endoscopists. (Gastrointest Endosc 1997;46:27-32.)
Recent therapeutic strategies for bleeding peptic ulcers rely on ulcer classifications to choose the best option based o~ the risk of rebleeding. 1 Moreover, these classifications can be used t O compare different therapeutic methods and results in different centers. However, the proportion of patients with various stigmata from recent hemorrhages (SRH) and the risk of rebleeding or mortality associated with these stigmata vary widely from study to study. The Forrest classification is most often used in these studies, 2 but a recent investigation has suggested t h a t this classification m a y be limited, s However, the techniques used in t h a t study were limited Received February 10, 1996. For revision April 21, 1996. Accepted January 8, 1997. From the Service d'Hdpato-Gastroentdrologie, Centre Hospitalier G¢n~ral, Le Mans, and the Service d'H@ato-Gastroentdrologie, Centre Hospitalier Universitaire, Angers, France. Reprint requests: Bruno Bout, MD, Endoscopie Digestive, CHG, 72037 Le Mans CEDEX, France. Copyright © 1997 bY the American Society for Gastrointestinal Endoscopy 0016-5107/97/$5.00 + 0 37/1/80770 VO'LUME 46, NO. 1, 1997
(slides, statistical comparisons). Therefore the aim of the present prospective study was to evaluate the interobserver agreement of all aspects of bleeding ulcers, in consecutive patients admitted for bleeding ulcers and using agreement statistics.
PATIENTS AND METHODS Patients Between March 1993 and September 1994, 3746 endoscopic examinations were performed; 61 consecutive adult patients with endoscopically proved acute bleeding due to gastric or duodenal ulcers were eligible for the study. No exclusion criteria were provided. Twenty patients were treated by local injection of epinephrine (1/10,000) during the index endoscopic procedure. Two of these patients rebled. In all, four patients rebled; 2 had red visible vessel with oozing and 2 had adherent black (n = 1) or red (n = 1) clots. None of these patients underwent surgery and none died. Endoscopists Nine endoscopists from two hospital units were enrolled in this study; two of them (BB, BP) performed all endoscopic GASTROINTESTINAL ENDOSCOPY 27
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 1. Proportion of agreement: example for the presence of a sign (yes/no) with two observers Observer 1 Yes Observer 2 Yes No
40 (a) 8 (c)
No 6 (b) 7 (d)
The proportion of agreement for presence is a/(a + b + c)--i.e., 40/(40 + 6 + 8) = 0.74. examinations and videotape recordings. Investigators were divided into junior and senior physicians according to their level of expertise in digestive endoscopy: juniors, less than 3 years of expertise; seniors, more than 3 years of expertise. The three juniors had been at the same institution for a maximum of 1 year and the six seniors had been at the same institution for 7 -+ 2 years. The mean age was 35 + 6 years and experience of senior observers was 9 - 2 years. The endoscopists did not take part in any training program designed to teach diagnosis of ulcer stigmata before or during the study.
Endoscopic examination The endoscopic procedure was performed with local 1% lidocaine anesthesia using a 9 mm diameter videoendoscope with a color CCD (Evis 100 Olympus, Hambourg, Germany). Each examination was recorded on a magnetic tape (U-Matic SP, Sony, Japan) by a standardized method, without indication of the patient's identity, as follows: (1) gastric lavage using suction drainage with a large nasogastric tube before the endoscopic procedure; (2) maximal air inflation of the whole gastric and duodenal cavity during the procedure; (3) optimum endoscope position to provide a better view of the ulcer base and SRH; and (4) recording 1 minute before and 1 minute after water irrigation through the endoscopic channel with a 60 ml water-filled syringe. The "in vivo" judgment of the two senior endoscopists (BB, BP) who performed all examinations was recorded at the time of the procedure and was not revealed to the seven other endoscopists until the study protocol was compiled.
Evaluation design At the end of the recruitment period, each of the nine endoscopists independently evaluated the videoendoscopic recording of each patient on the same video screen (Triniton, Sony), at the same time of day, and at the same distance, during two sessions over a 4-week period. Recordings of the first 30 patients were assessed at the first session and recordings of the last 31 patients were reviewed at the second session. Results of the evaluation of the first 30 patients were not given to the endoscopists prior to the second session. SRH were immediately judged after visualization of the ulcer base recording. Each endoscopic sign had to be classified according to the classification described below. Moreover, observers were required to decide on the quality of videorecordings and the neces28 GASTROINTESTINAL ENDOSCOPY
sity of immediate hemostatic treatment on the basis of a 3 grade scale (yes/no/doubt). Three endoscopists (BP, PB, JB) had previously been involved in an interobserver agreement study. 4
Classification of endoscopic features In this study, the classification as well as the definition of elementary signs were established as follows: spurting bleeding ulcer, oozing hemorrhage, visible vessel, red clot, black clot, flat red pigmented spot, fiat black pigmented spot, or no stigmata. 2 In addition, visible vessels were divided into three categories: white protuberant vessel, or protuberant vessel with firmly attached red clot or black clot. 5 A gold standard was not established before the video endoscopic recording sessions or evaluation or after. So, the SRH were classified on the basis Of the majority opinion of the senior observers.
Statistical analysis Interobserver agreement of qualitative variables was evaluated using the kappa (K) index as proposed by Fleiss. 6 This method measures any agreement above that expected by chance. As a result of this calculation, when observer agreement is greater than chance, the K index ranges from 0 (absence of agreement) to 1 (perfect agreement). Negative values indicate disagreement (complete if K = --1); positive values indicate agreement that is considered poor if K is below 0.40, fair to good from 0.40 to 0.75, and excellent above 0.75. 6 Comparisons between K indexes were made using nonparametric tests for paired data (Wilcoxon's test) or for unpaired data (Mann and Whitney test). The correlation between K indexes was evaluated by the Spearman's coefficient (rs). Moreover, the proportion of agreement was calculated for qualitative variables according to a method recently described by Grant. 7 The proportion of agreement for a sign class is an informative index, expressed in a percentage that measures the number of cases where all observers agree for one class over the number of patients for whom this class was chosen (by at least one observer). An example is depicted in Table 1. Compared to the K index, this index is limited by its inclusion of expected agreement by chance; however, it can be used to determine agreement for the presence and the absence of a sign. This is especially interesting when the prevalence of a sign is low. Indeed, when a sign is rare, the K value can be high because observers naturally agree for the absence of a sign whereas the agreement can be low for the presence of a sign. Finally, observer bias was investigated using a Cochran's test. 7 This test evaluates whether two or more observers consistently assess differently from one another. In other words, there is a bias when the disagreement between observers is not due to hazard. The example in Table 1 tests the null hypothesis that the difference between the two disagreement categories (b and c) is not different from zero. If there was no observer bias--as previously defined--the presence of one or more poorly trained observer(s) was evaluated using the Poisson heterogeneity test.7 A poorly trained observer is defined as an observer who has a number of assessments in complete VOLUME 46, NO. 1, 1997
B Botu, B Person, P Calbs, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 2. Agreement indexes, observer bias, and poorly trained observer tests Agreement proportion
Prevalence (%)
K
Absence
-2 18 2 16 15 8 5 2 31 16 15 32 8 24 10 --
0.385 0.29 0.68 0.25 0.60 0.33 0.27 0.25 0.09 0.52 0.33 0.42 0.475 0.26 0.52 0.82 0.34
. 0.99 0.88 0.965 0.87 0.63 0.78 0.865 0.85 0.73 0.80 0.82 0.68 0.85 0.76 0.96 .
Quality* Spurting Oozing Arterial Venous Visible vessel Red visible vessel White visible vessel Black visible 'vessel Clot Fresh red clot Protuberant black clot Flat pigmented spot Red flat pigmented spot Black flat pigmented spot Clean baser Treatments
Observer bias
Presence .
(p)
. 0.18 0.59 0.16 0.50 0.385 0.24 0.19 0.09 0,50 0.28 0.35 0.50 0.205 0.49 0.725
.
Poorly trained observer
.
. NS NS NS NS <0.001 <0.001 <0.001 <0.001 NS <0.01 NS NS NS NS <0.001 .
(p)
(n)
<0.05 <0.05 <0.05 <0.05 ----<0.05 -<0.05 <0.05 NS <0.05 --
1 1 1 1 ----1 -2 1 -1 --
.
.
*The rio coefficient is given for this unique quantitative variable. SThe difference between the presence of a stigmata (90%) and the sum of the prevalences of each sign (97%) is due to the coexistence of two signs in some patients. SNo additiona:[ tests are available because three responses were possible. disagreement with those of all the other observers that do not occur sporadically. We tested the null hypothesis that there is no difference in complete disagreements for each observer. Quantitative variables were expressed as mean - SD. The interobserver agreement for quantitative variables was calculated using the intraclass correlation coefficient (rlc), s which is interpreted from - 1 to +1 in the same manner as a classic correlation coefficient. The prevalence of a sign was calculated on the basis of t:he majority opinion of the senior observers.
RESULTS T h e r e s u l t s of t h e a g r e e m e n t indexes for all pat i e n t s a n d o b s e r v e r s are p r e s e n t e d in Table 2. F~riefly, t h e i n t e r o b s e r v e r a g r e e m e n t w a s v e r y good fi)r oozing, good for clot a n d flat p i g m e n t e d spot, poor fi)r s p u r t i n g a n d visible vessels, a n d excellent for t h e a b s e n c e of s t i g m a t a . As expected, t h e p r o p o r t i o n of a g r e e m e n t for the a b s e n c e or t h e p r e s e n c e of a sign w a s g e n e r a l l y r e l a t e d to its prevalence, i.e., w h e n a sign is rare, t h e p r o p o r t i o n of a g r e e m e n t is b e t t e r in t h e a b s e n c e class, a n d w h e n a sign is f r e q u e n t , the p r o p o r t i o n of a g r e e m e n t is b e t t e r in t h e p r e s e n c e class. The t e s t s to explain t h e lack of perfect a g r e e m e n t are also p r e s e n t e d in Table 2. P o o r a g r e e m e n t for visible vessels w a s due to a c o n s t a n t o b s e r v e r bias a m o n g t h e different classes. P o o r a g r e e m e n t of s p u r t i n g w a s n o t due to o b s e r v e r bias b u t a low prevalence, t h u s this r e s u l t m u s t be i n t e r p r e t e d VOLUME 46, NO. 2, 1997
with caution. The n u m b e r of poorly t r a i n e d observers w a s low. T h e r e were a significant n u m b e r of complete d i s a g r e e m e n t s b y one o b s e r v e r w i t h the others, i.e., a poorly t r a i n e d observer, 9 t i m e s for 81 c o m p a r i s o n s (= 9 [tests] × 9 [observers], see the second to t h e last c o l u m n in Table 2). T h e s e signifi c a n t complete d i s a g r e e m e n t s were due to five diff e r e n t observers: two out of six senior observers (two t i m e s each), a n d all t h r e e j u n i o r observers, (two twice a n d t h e o t h e r once). The i n t e r o b s e r v e r a g r e e m e n t b e t w e e n senior a n d j u n i o r endoscopists was c o m p a r e d (Table 3). T h e r e was a good c o r r e l a t i o n b e t w e e n the K v a l u e s of senior a n d j u n i o r observers: r S = 0.53, p < 0.05. However, the K indexes were significantly b e t t e r in seniors t h a n in j u n i o r s : 0.49 _+ 0.15 v e r s u s 0.36 + 0 . 2 4 , p < 0.05 (paired data). The s u b g r o u p a n a l y s i s of a g r e e m e n t indexes is also p r e s e n t e d in Table 3. The m e a n a g r e e m e n t b e t w e e n t h e in vivo e v a l u a t i o n a n d t h e e v a l u a t i o n on video r e c o r d i n g s b y the s a m e two observers (BB, BP), w h i c h indicates i n t r a o b s e r v e r a g r e e m e n t , was 0.61 _+ 0.18. This w a s significantly h i g h e r t h a n interobserver agreement. To test the p r e s e n c e of a t r a i n i n g p h e n o m e n o n , we c o m p a r e d the K indexes of t h e first 30 p a t i e n t s w i t h t h o s e of the last 31 p a t i e n t s (Table 3). No statistical difference w a s found: 0.40 _+ 0.20 v e r s u s 0.43 _+ 0.20, in first v e r s u s last p a t i e n t s , r e s p e c t i v e l y ( u n p a i r e d data). Moreover, t h e c o r r e l a t i o n of t h e K indexes GASTROINTESTINAL ENDOSCOPY 29
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 3. Agreement (K or ric): effects of recording evaluation (intraobserver agreement), observer experience, and training phenomenon Half of patients
Quality Spurting Oozing Arterial Venous Visible vessel Red visible vessel White visible vessel Black visible vessel Clots F r e s h red clot P r o t u b e r a n t black clot Flat pigmented spot Red flat pigmented spot Black flat pigmented spot Clean base Treatment Total
Whole*
Intraobserver
Seniors
Juniors
First
Second
0.385 0.29 0.68 0.25 0.60 0.33 0.27 0.25 0.09 0.52 0.33 0.42 0.475 0.26 0.52 0.82 0.34 0.40 + 0.18
--~ --$ 0.92 0.66 0.84 0.68 0.16 0.54 0.56 0.62 0.56 0.51 0.65 0.46 0.60 0.79 --~ 0.61 +-_0.18
0.44 0.60 0.70 0.30 0.60 0.43 0.31 0.40 0.23 0.56 0.4 0.445 0.59 0.37 0.59 0.80 0.55 0.49 _+ 0.15
0.31 0.05 0.74 0.32 0.72 0.33 0,41 0.05 0.185 0.47 0,335 0,32 0,375 0,08 0.48 0.86 0.20 0.36 + 0.24
0.45 --$ 0.675 0.135 0.62 0.40 0.25 0.32 0.06 0.47 0.23 0.42 0.39 0.25 0.49 0.77 --$ 0.40 +_ 0.20
0,395 0,29 0.74 0.26 0.61 0.295 0.27 0.26 0.20 0.55 0.42 0.42 0.56 0.24 0.61 0.875 0.34 0.43 _+ 0.20
I
I
I
I
I
I p < 0.001§ rs = 0.64;p < 0.05
I
I
p < 0.05 rs = 0.53;p < 0.05
I NS tl r s = 0.88;p < 10 4
* T h i s c o l u m n , w h i c h is i d e n t i c a l to t h e s e c o n d c o l u m n of T a b l e 1 ( m a r k e d K i n T a b l e 1), is g i v e n a s a c o m p a r i s o n . ? E v a l u a t i o n n o t p e r f o r m e d i n vivo. $ C a l c u l a t i o n n o t p o s s i b l e ( s i n g l e code). § F o r t h e 14 c o u p l e s of v a r i a b l e s w h e r e t h e c o m p a r i s o n w a s p o s s i b l e . IIFor t h e 15 c o u p l e s of v a r i a b l e s w h e r e t h e c o m p a r i s o n w a s p o s s i b l e .
between these two groups of patients was excellent: r S = 0.88, p < 0.0001. The grade of the quality of the examination ranged from 1.80 +_ 0.72 to 2.12 _+ 0.51 for the nine judges (variance a n a l y s i s - - observer effect: p < 0.01, patient effect: p < 0.0001). The corresponding agreem e n t was poor as reflected by the intraclass correlation coefficient: ric = 0.385. The interobserver agreement was poor for the necessity of immediate endoscopic treatment: K = 0.34; agreement for this variable was better in seniors t h a n in juniors (K: 0.55 +-- 0.05 VS 0.20 --- 0.08, respectively, p < 0.05).
DISCUSSION As expected, agreement seemed to be related to the prevalence of a sign. Indeed, in frequent SRH (i.e., oozing [18%], clots [31%], flat pigmented spots [32%]) agreement was higher t h a n in less prevalent signs (i.e., spurting [2%], visible vessels [15%]). The prevalence of SRH is known to vary from study to study for clots (0% to 43%) fresh clot (30% to 40%), flat pigmented spot (0% to 12%)9--and for visible vessels (4% to 35%). 1° It is very low for active 30
GASTROINTESTINAL ENDOSCOPY
bleeding lesions: spurting, 2% to 8%; oozing, 3% to 15%. 11 This variability can be explained by the design of this type of study, which m a y lead to an overestimation in the number of lesions as a result of unlimited time to visualize videotape recordings. This variability m a y also be influenced by the moment the endoscopic procedure takes place, the technique of irrigation of the ulcer base and, finally, by a lack of consensus for the definition of SRH. When color is considered, i.e., visible vessels and clots, the K value also varied. This does not seem to be due to a recording bias, since intraobserver agreement was excellent, but to an interindividual variation of color definition. 12 To our knowledge, the only bias in this study was the low prevalence of certain signs, especially spurting. We have therefore considered this calculated agreement with caution because its optimal distribution is a prevalence of about 50%. 7 The interobserver agreement was statistically better in senior t h a n in junior observers. Generally, there is no difference between junior and senior observers.4, 13, 14 Moreover, one third of senior and all junior observers were found to be poorly trained VOLUME 46, NO. 1, 1997
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
observers at least once. In this study, no training phenomenon was found between the first and the second half of the patient group, inasmuch as the respective mean K indexes were similar. The very good intraobserver agreement between the in vivo and on video evaluations suggests that the bias due to video recording standardization was negligible. The analysis of additional tests of agreement yields the following information (Table 2). In relation to agreement proportion, agreement was nearly always better for the absence than for the presence of a sign. Because agreement was always high for the absence of a sign, the low calculated agreements were always due to a low agreement for the presence of a sign. The above mentioned analysis was descriptive. The other m e a s u r e m e n t s (observer bias, poorly trained observer) provide additional explanations for the causes of disagreement. Observer bias indicates that the level of disagreement was significantly different among the observers. This bias explains the poor agreement for visible vessels. A poorly trained observer indicates a statistically significant n u m b e r of observations in which the observer completely disagreed with other observers. This occurred in 10% of the cases and helps explain the lack of excellent agreement for several signs. It should be noted[ that this misinterpretation occurred twice in two senior observers and in all junior observers. This test also identifies poorly trained observers for a sign to help improve their judgement. Other studies have investigated the evaluation of bleeding ulcers by observers. The study by Laine et al., 3 had a different method: slides of ulcer features (SRH or clean base) were evaluated independently by three observers instead of video recordings from live examinations, thus creating difficulties for the correct visualization of the bleeding site. In that study 202 participants were asked to correctly identify findings befbre and after a teaching session: the total of correct answers increased from 72% before to 82% after the teaching session. In this study, the classic K index was also used and comparison of the learning curve was made before and after the teaching session, whereas in our study, in addition to the K index, new tests were used such as the proportion of agreement, observer bias, and poorly trained observer to provide an in-depth examination of the cause of disagreement. The primary finding of variability in labeling stigmata is not different in these two studies. Lau et al. 15 reported an interobserver study based on the same methods as ours with a weighted K of 0.453 _+ 0.006 for the entire group. Agreement was poor for visible vessels and a clean VOLUME 46, NO. 1, 1997
B Bour, B Person, P Cal~s, et al.
base, fair for active oozing, adherent clots, and flat pigmented clots, and good for spurts. However, in that study all participants were international experts yielding artificial conditions and K estimates were obtained by pairs of any two experts. Cosentino et al. 16 reported good to excellent interobserver agreement for the whole Forrest classification after 2 months of training for semiological terminology based on videotape learning sessions. However, once again in this preliminary study, the method did not correspond to the clinical situation, inasmuch as endoscopists were all skilled, a consensus was decided before judgement, typical lesions were selected, and the evaluation was made after a training period. The results of this study emphasize the limits of endoscopic diagnosis in this situation. Several hypotheses can be suggested for these limits: an insufficient endoscopic training, a lack of precise definitions or a consensus in the description of lesions, and the oversophistication of some classifications. Finally, we suggest several solutions to these limitations. First, classifications should be simplified (i.e., active vs nonactive hemorrhage), and second, grading should be limited to a few classes (i.e., visible vessel vs clot when nonactive hemorrhages are being considered). Flat spots or a clean base should not be included in S H R because these conditions do not have a risk of rebleeding. In portal hypertension, endoscopist experts agree on the suggestion of simpler classifications 17 and we have observed a significantly higher agreement when using a twoclass classification than a three-class classification. 4 Moreover, special attention should be paid to endoscopic training for these lesions. The quality of this training can be improved by two means: (1) teaching either in vivo or with video tapes, (2) precise definitions of lesions in scientific manuscripts, with high-quality illustrations (drawings or color photographs). REFERENCES 1. N.I.H. Consensus conference. Therapeutic endoscopy and bleeding ulcers. JAMA 1989;262:1369-72. 2. Rosch W. Endoscopy of the upper gastrointestinal tract. Gastroenterology Annual 1986;3:549-67. 3. Laine L, Freeman M, Cohen H. Lack of uniformity in evaluation of endoscopic prognostic features of bleeding ulcers. Gastrointest Endosc 1994;40:411-7. 4. Cal~s P, Burtin P, Person P, Ruget O, Charlois T, Ferrero P, et al. Interobserver agreement on endoscopic diagnosis of diffuse gastric mucosal features in cirrhotic patients. Eur J Gastroenterol Hepatol 1993;5:263-8. 5. Freeman M, Cass OW, Peine CJ, Onstad GR. The nonbleeding visible vessel versus the sentinel clot: natural history and risk of rebleeding. Gastrointest Endosc 1993;39:359-66. 6. Fleiss JL. The measurement of interrater agreement. In: John Wiley. Statistical methods for rates and proportions. New York, 1981:212-36. GASTROINTESTINAL ENDOSCOPY 31
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
7. Grant JM. The fetal heart rate trace is normal, isn't it? Observer agreement of categorical assessments. Lancet 1991; 337:215-8. 8. Snedecor GW, Cochran WG. Statistical methods. 6th ed. Ames, Iowa: Iowa State University Press, 1967:328-30. 9. Branicki FJ, Coleman SY, Fok PJ, Pritchett CJ, Fan ST, Lai ECS, et al. Bleeding duodenal ulcer: a prospective evaluation of risk factors for rebleeding and death. Ann Surg 1990;211: 411-8. 10. Hsu PI, Lin XZ, Chan SH, Lin CY, Chang TT, Shin JS, et al. Bleeding peptic ulcer--risk factors for rebleeding and sequential changes in endoscopic finding. Gut 1994;35:746-9. 11. Lin HJ, Perng CL, Lee FY, Lee CH, Lee SD. Clinical courses and predictors for rebleeding in patients with peptic ulcers and nonbleeding visible vessels: a prospective study. Gut 1994;35:1389-93. 12. Uno Y. Color vision and endoscopic diagnosis. Gastrointest Endosc 1995;42:377-8.
13. Bendtsen F, Skovgaard LT, Sorensen TIA, Matzen P. Agreement among multiple observers on endoscopic diagnosis of esophageal varices before bleeding. Hepatology 1990;11: 341-7. 14. Tissot O, Aub~ C, Namour A, Blanc F, Dauver A, Cal~s P. S~miologie ~chographique de la cirrhose: concordance interobservateur. Gastroenterol Clin Biol 1995;19:291-6. 15. Lau JYW, Sung JY, Lau JFT, Chan ACW, Ng EKW, Chung SCS. Stigmata of recent hemorrhage in peptic ulcer bleeding: is there inter-observer agreement among international experts? [abstract] Gastrointest Endosc 1995;41:368. 16. Cosentino F, Arcidiacono PG, Rubis G, Passoni G, Morandi E, Motta R, et al. Observer variation in the assessment of stigmata of recent haemorrhage: a multicenter study by 13 skilled endoscopists [abstract]. Gut 1994;35:$80. 17. Spina GP, Arcidiacono R, Bosch J, Pagliaro L, Burroughs AT{, Santambrogio R, et al. Gastric endoscopic features in portal hypertension: final report of a consensus conference, Milan, Italy, September 19, 1992. J Hepatol 1994;21:461-7.
Availability of Journal back issues As a service to our subscribers, copies of b a c k issues of Gastrointestinal Endoscopy for the preceding 5 y e a r s a r e m a i n t a i n e d a n d are available for p u r c h a s e from Mosby u n t i l i n v e n t o r y is d e p l e t e d a t a cost of $12.00 pe r issue. The following q u a n t i t y discounts a r e available: 25% off on q u a n t i t i e s of 12 to 23, a n d one t h i r d off on q u a n t i t i e s of 24 or more. P l e a s e w r i t e to M o s b y - Y e a r Book, Inc., Subscription Services, 11830 W e s t l i n e Indust r i a l Dr., St. Louis, MO 63146-3318, or call (800) 453-4351 or (314) 453-4351 for i n f o r m a t i o n on a v a i l a b i l i t y of p a r t i c u l a r issues. If u n a v a i l a b l e from the publisher, photocopies of complete issues m a y be p u r c h a s e d from UMI, 300 N. Zeeb Rd., A n n Arbor, M I 48106 (313) 761-4700.
32 GASTROINTESTINAL ENDOSCOPY
VOLUME 46, NO. 1, 1997
Background: The aim of this prospective study was to evaluate the interobserver agreement of stigmata of recent hemorrhage of bleeding peptic ulcers. Methods: Sixty-one consecutive adult patients were enrolled in the study and nine (three junior and six senior) endoscopists reviewed standardized video recordings of endoscopic examinations. Interobserver agreement was evaluated using the kappa (K) index, intraclass correlation coefficient, and proportion of agreement. Observer bias and poorly trained observers were investigated. Results: Interobserver agreement was very good for oozing (K = 0.68), good for clot (K = 0.51), poor for spurting (K = 0.29) and visible vessels (K = 0.33), and excellent for the absence of stigmata (K = 0.82). Observer bias sometimes occurred and the number of poorly trained observers was low. The K indexes were significantly better in senior than in junior investigators: 0.48 -+ 0.16 versus 0.37 - 0.26, respectively, p < 0.05. The agreement between the in vivo evaluation and video tape recordings (intraobserver agreement) was good (K = 0.60 -+ 0.19). There was no training phenomenon between the first and the second half of the patient group. Conclusions: The endoscopic classification of bleeding ulcers might be simplified by limiting grading to a few classes. Special attention should be paid to the training of endoscopists. (Gastrointest Endosc 1997;46:27-32.)
Recent therapeutic strategies for bleeding peptic ulcers rely on ulcer classifications to choose the best option based o~ the risk of rebleeding. 1 Moreover, these classifications can be used t O compare different therapeutic methods and results in different centers. However, the proportion of patients with various stigmata from recent hemorrhages (SRH) and the risk of rebleeding or mortality associated with these stigmata vary widely from study to study. The Forrest classification is most often used in these studies, 2 but a recent investigation has suggested t h a t this classification m a y be limited, s However, the techniques used in t h a t study were limited Received February 10, 1996. For revision April 21, 1996. Accepted January 8, 1997. From the Service d'Hdpato-Gastroentdrologie, Centre Hospitalier G¢n~ral, Le Mans, and the Service d'H@ato-Gastroentdrologie, Centre Hospitalier Universitaire, Angers, France. Reprint requests: Bruno Bout, MD, Endoscopie Digestive, CHG, 72037 Le Mans CEDEX, France. Copyright © 1997 bY the American Society for Gastrointestinal Endoscopy 0016-5107/97/$5.00 + 0 37/1/80770 VO'LUME 46, NO. 1, 1997
(slides, statistical comparisons). Therefore the aim of the present prospective study was to evaluate the interobserver agreement of all aspects of bleeding ulcers, in consecutive patients admitted for bleeding ulcers and using agreement statistics.
PATIENTS AND METHODS Patients Between March 1993 and September 1994, 3746 endoscopic examinations were performed; 61 consecutive adult patients with endoscopically proved acute bleeding due to gastric or duodenal ulcers were eligible for the study. No exclusion criteria were provided. Twenty patients were treated by local injection of epinephrine (1/10,000) during the index endoscopic procedure. Two of these patients rebled. In all, four patients rebled; 2 had red visible vessel with oozing and 2 had adherent black (n = 1) or red (n = 1) clots. None of these patients underwent surgery and none died. Endoscopists Nine endoscopists from two hospital units were enrolled in this study; two of them (BB, BP) performed all endoscopic GASTROINTESTINAL ENDOSCOPY 27
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 1. Proportion of agreement: example for the presence of a sign (yes/no) with two observers Observer 1 Yes Observer 2 Yes No
40 (a) 8 (c)
No 6 (b) 7 (d)
The proportion of agreement for presence is a/(a + b + c)--i.e., 40/(40 + 6 + 8) = 0.74. examinations and videotape recordings. Investigators were divided into junior and senior physicians according to their level of expertise in digestive endoscopy: juniors, less than 3 years of expertise; seniors, more than 3 years of expertise. The three juniors had been at the same institution for a maximum of 1 year and the six seniors had been at the same institution for 7 -+ 2 years. The mean age was 35 + 6 years and experience of senior observers was 9 - 2 years. The endoscopists did not take part in any training program designed to teach diagnosis of ulcer stigmata before or during the study.
Endoscopic examination The endoscopic procedure was performed with local 1% lidocaine anesthesia using a 9 mm diameter videoendoscope with a color CCD (Evis 100 Olympus, Hambourg, Germany). Each examination was recorded on a magnetic tape (U-Matic SP, Sony, Japan) by a standardized method, without indication of the patient's identity, as follows: (1) gastric lavage using suction drainage with a large nasogastric tube before the endoscopic procedure; (2) maximal air inflation of the whole gastric and duodenal cavity during the procedure; (3) optimum endoscope position to provide a better view of the ulcer base and SRH; and (4) recording 1 minute before and 1 minute after water irrigation through the endoscopic channel with a 60 ml water-filled syringe. The "in vivo" judgment of the two senior endoscopists (BB, BP) who performed all examinations was recorded at the time of the procedure and was not revealed to the seven other endoscopists until the study protocol was compiled.
Evaluation design At the end of the recruitment period, each of the nine endoscopists independently evaluated the videoendoscopic recording of each patient on the same video screen (Triniton, Sony), at the same time of day, and at the same distance, during two sessions over a 4-week period. Recordings of the first 30 patients were assessed at the first session and recordings of the last 31 patients were reviewed at the second session. Results of the evaluation of the first 30 patients were not given to the endoscopists prior to the second session. SRH were immediately judged after visualization of the ulcer base recording. Each endoscopic sign had to be classified according to the classification described below. Moreover, observers were required to decide on the quality of videorecordings and the neces28 GASTROINTESTINAL ENDOSCOPY
sity of immediate hemostatic treatment on the basis of a 3 grade scale (yes/no/doubt). Three endoscopists (BP, PB, JB) had previously been involved in an interobserver agreement study. 4
Classification of endoscopic features In this study, the classification as well as the definition of elementary signs were established as follows: spurting bleeding ulcer, oozing hemorrhage, visible vessel, red clot, black clot, flat red pigmented spot, fiat black pigmented spot, or no stigmata. 2 In addition, visible vessels were divided into three categories: white protuberant vessel, or protuberant vessel with firmly attached red clot or black clot. 5 A gold standard was not established before the video endoscopic recording sessions or evaluation or after. So, the SRH were classified on the basis Of the majority opinion of the senior observers.
Statistical analysis Interobserver agreement of qualitative variables was evaluated using the kappa (K) index as proposed by Fleiss. 6 This method measures any agreement above that expected by chance. As a result of this calculation, when observer agreement is greater than chance, the K index ranges from 0 (absence of agreement) to 1 (perfect agreement). Negative values indicate disagreement (complete if K = --1); positive values indicate agreement that is considered poor if K is below 0.40, fair to good from 0.40 to 0.75, and excellent above 0.75. 6 Comparisons between K indexes were made using nonparametric tests for paired data (Wilcoxon's test) or for unpaired data (Mann and Whitney test). The correlation between K indexes was evaluated by the Spearman's coefficient (rs). Moreover, the proportion of agreement was calculated for qualitative variables according to a method recently described by Grant. 7 The proportion of agreement for a sign class is an informative index, expressed in a percentage that measures the number of cases where all observers agree for one class over the number of patients for whom this class was chosen (by at least one observer). An example is depicted in Table 1. Compared to the K index, this index is limited by its inclusion of expected agreement by chance; however, it can be used to determine agreement for the presence and the absence of a sign. This is especially interesting when the prevalence of a sign is low. Indeed, when a sign is rare, the K value can be high because observers naturally agree for the absence of a sign whereas the agreement can be low for the presence of a sign. Finally, observer bias was investigated using a Cochran's test. 7 This test evaluates whether two or more observers consistently assess differently from one another. In other words, there is a bias when the disagreement between observers is not due to hazard. The example in Table 1 tests the null hypothesis that the difference between the two disagreement categories (b and c) is not different from zero. If there was no observer bias--as previously defined--the presence of one or more poorly trained observer(s) was evaluated using the Poisson heterogeneity test.7 A poorly trained observer is defined as an observer who has a number of assessments in complete VOLUME 46, NO. 1, 1997
B Botu, B Person, P Calbs, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 2. Agreement indexes, observer bias, and poorly trained observer tests Agreement proportion
Prevalence (%)
K
Absence
-2 18 2 16 15 8 5 2 31 16 15 32 8 24 10 --
0.385 0.29 0.68 0.25 0.60 0.33 0.27 0.25 0.09 0.52 0.33 0.42 0.475 0.26 0.52 0.82 0.34
. 0.99 0.88 0.965 0.87 0.63 0.78 0.865 0.85 0.73 0.80 0.82 0.68 0.85 0.76 0.96 .
Quality* Spurting Oozing Arterial Venous Visible vessel Red visible vessel White visible vessel Black visible 'vessel Clot Fresh red clot Protuberant black clot Flat pigmented spot Red flat pigmented spot Black flat pigmented spot Clean baser Treatments
Observer bias
Presence .
(p)
. 0.18 0.59 0.16 0.50 0.385 0.24 0.19 0.09 0,50 0.28 0.35 0.50 0.205 0.49 0.725
.
Poorly trained observer
.
. NS NS NS NS <0.001 <0.001 <0.001 <0.001 NS <0.01 NS NS NS NS <0.001 .
(p)
(n)
<0.05 <0.05 <0.05 <0.05 ----<0.05 -<0.05 <0.05 NS <0.05 --
1 1 1 1 ----1 -2 1 -1 --
.
.
*The rio coefficient is given for this unique quantitative variable. SThe difference between the presence of a stigmata (90%) and the sum of the prevalences of each sign (97%) is due to the coexistence of two signs in some patients. SNo additiona:[ tests are available because three responses were possible. disagreement with those of all the other observers that do not occur sporadically. We tested the null hypothesis that there is no difference in complete disagreements for each observer. Quantitative variables were expressed as mean - SD. The interobserver agreement for quantitative variables was calculated using the intraclass correlation coefficient (rlc), s which is interpreted from - 1 to +1 in the same manner as a classic correlation coefficient. The prevalence of a sign was calculated on the basis of t:he majority opinion of the senior observers.
RESULTS T h e r e s u l t s of t h e a g r e e m e n t indexes for all pat i e n t s a n d o b s e r v e r s are p r e s e n t e d in Table 2. F~riefly, t h e i n t e r o b s e r v e r a g r e e m e n t w a s v e r y good fi)r oozing, good for clot a n d flat p i g m e n t e d spot, poor fi)r s p u r t i n g a n d visible vessels, a n d excellent for t h e a b s e n c e of s t i g m a t a . As expected, t h e p r o p o r t i o n of a g r e e m e n t for the a b s e n c e or t h e p r e s e n c e of a sign w a s g e n e r a l l y r e l a t e d to its prevalence, i.e., w h e n a sign is rare, t h e p r o p o r t i o n of a g r e e m e n t is b e t t e r in t h e a b s e n c e class, a n d w h e n a sign is f r e q u e n t , the p r o p o r t i o n of a g r e e m e n t is b e t t e r in t h e p r e s e n c e class. The t e s t s to explain t h e lack of perfect a g r e e m e n t are also p r e s e n t e d in Table 2. P o o r a g r e e m e n t for visible vessels w a s due to a c o n s t a n t o b s e r v e r bias a m o n g t h e different classes. P o o r a g r e e m e n t of s p u r t i n g w a s n o t due to o b s e r v e r bias b u t a low prevalence, t h u s this r e s u l t m u s t be i n t e r p r e t e d VOLUME 46, NO. 2, 1997
with caution. The n u m b e r of poorly t r a i n e d observers w a s low. T h e r e were a significant n u m b e r of complete d i s a g r e e m e n t s b y one o b s e r v e r w i t h the others, i.e., a poorly t r a i n e d observer, 9 t i m e s for 81 c o m p a r i s o n s (= 9 [tests] × 9 [observers], see the second to t h e last c o l u m n in Table 2). T h e s e signifi c a n t complete d i s a g r e e m e n t s were due to five diff e r e n t observers: two out of six senior observers (two t i m e s each), a n d all t h r e e j u n i o r observers, (two twice a n d t h e o t h e r once). The i n t e r o b s e r v e r a g r e e m e n t b e t w e e n senior a n d j u n i o r endoscopists was c o m p a r e d (Table 3). T h e r e was a good c o r r e l a t i o n b e t w e e n the K v a l u e s of senior a n d j u n i o r observers: r S = 0.53, p < 0.05. However, the K indexes were significantly b e t t e r in seniors t h a n in j u n i o r s : 0.49 _+ 0.15 v e r s u s 0.36 + 0 . 2 4 , p < 0.05 (paired data). The s u b g r o u p a n a l y s i s of a g r e e m e n t indexes is also p r e s e n t e d in Table 3. The m e a n a g r e e m e n t b e t w e e n t h e in vivo e v a l u a t i o n a n d t h e e v a l u a t i o n on video r e c o r d i n g s b y the s a m e two observers (BB, BP), w h i c h indicates i n t r a o b s e r v e r a g r e e m e n t , was 0.61 _+ 0.18. This w a s significantly h i g h e r t h a n interobserver agreement. To test the p r e s e n c e of a t r a i n i n g p h e n o m e n o n , we c o m p a r e d the K indexes of t h e first 30 p a t i e n t s w i t h t h o s e of the last 31 p a t i e n t s (Table 3). No statistical difference w a s found: 0.40 _+ 0.20 v e r s u s 0.43 _+ 0.20, in first v e r s u s last p a t i e n t s , r e s p e c t i v e l y ( u n p a i r e d data). Moreover, t h e c o r r e l a t i o n of t h e K indexes GASTROINTESTINAL ENDOSCOPY 29
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
Table 3. Agreement (K or ric): effects of recording evaluation (intraobserver agreement), observer experience, and training phenomenon Half of patients
Quality Spurting Oozing Arterial Venous Visible vessel Red visible vessel White visible vessel Black visible vessel Clots F r e s h red clot P r o t u b e r a n t black clot Flat pigmented spot Red flat pigmented spot Black flat pigmented spot Clean base Treatment Total
Whole*
Intraobserver
Seniors
Juniors
First
Second
0.385 0.29 0.68 0.25 0.60 0.33 0.27 0.25 0.09 0.52 0.33 0.42 0.475 0.26 0.52 0.82 0.34 0.40 + 0.18
--~ --$ 0.92 0.66 0.84 0.68 0.16 0.54 0.56 0.62 0.56 0.51 0.65 0.46 0.60 0.79 --~ 0.61 +-_0.18
0.44 0.60 0.70 0.30 0.60 0.43 0.31 0.40 0.23 0.56 0.4 0.445 0.59 0.37 0.59 0.80 0.55 0.49 _+ 0.15
0.31 0.05 0.74 0.32 0.72 0.33 0,41 0.05 0.185 0.47 0,335 0,32 0,375 0,08 0.48 0.86 0.20 0.36 + 0.24
0.45 --$ 0.675 0.135 0.62 0.40 0.25 0.32 0.06 0.47 0.23 0.42 0.39 0.25 0.49 0.77 --$ 0.40 +_ 0.20
0,395 0,29 0.74 0.26 0.61 0.295 0.27 0.26 0.20 0.55 0.42 0.42 0.56 0.24 0.61 0.875 0.34 0.43 _+ 0.20
I
I
I
I
I
I p < 0.001§ rs = 0.64;p < 0.05
I
I
p < 0.05 rs = 0.53;p < 0.05
I NS tl r s = 0.88;p < 10 4
* T h i s c o l u m n , w h i c h is i d e n t i c a l to t h e s e c o n d c o l u m n of T a b l e 1 ( m a r k e d K i n T a b l e 1), is g i v e n a s a c o m p a r i s o n . ? E v a l u a t i o n n o t p e r f o r m e d i n vivo. $ C a l c u l a t i o n n o t p o s s i b l e ( s i n g l e code). § F o r t h e 14 c o u p l e s of v a r i a b l e s w h e r e t h e c o m p a r i s o n w a s p o s s i b l e . IIFor t h e 15 c o u p l e s of v a r i a b l e s w h e r e t h e c o m p a r i s o n w a s p o s s i b l e .
between these two groups of patients was excellent: r S = 0.88, p < 0.0001. The grade of the quality of the examination ranged from 1.80 +_ 0.72 to 2.12 _+ 0.51 for the nine judges (variance a n a l y s i s - - observer effect: p < 0.01, patient effect: p < 0.0001). The corresponding agreem e n t was poor as reflected by the intraclass correlation coefficient: ric = 0.385. The interobserver agreement was poor for the necessity of immediate endoscopic treatment: K = 0.34; agreement for this variable was better in seniors t h a n in juniors (K: 0.55 +-- 0.05 VS 0.20 --- 0.08, respectively, p < 0.05).
DISCUSSION As expected, agreement seemed to be related to the prevalence of a sign. Indeed, in frequent SRH (i.e., oozing [18%], clots [31%], flat pigmented spots [32%]) agreement was higher t h a n in less prevalent signs (i.e., spurting [2%], visible vessels [15%]). The prevalence of SRH is known to vary from study to study for clots (0% to 43%) fresh clot (30% to 40%), flat pigmented spot (0% to 12%)9--and for visible vessels (4% to 35%). 1° It is very low for active 30
GASTROINTESTINAL ENDOSCOPY
bleeding lesions: spurting, 2% to 8%; oozing, 3% to 15%. 11 This variability can be explained by the design of this type of study, which m a y lead to an overestimation in the number of lesions as a result of unlimited time to visualize videotape recordings. This variability m a y also be influenced by the moment the endoscopic procedure takes place, the technique of irrigation of the ulcer base and, finally, by a lack of consensus for the definition of SRH. When color is considered, i.e., visible vessels and clots, the K value also varied. This does not seem to be due to a recording bias, since intraobserver agreement was excellent, but to an interindividual variation of color definition. 12 To our knowledge, the only bias in this study was the low prevalence of certain signs, especially spurting. We have therefore considered this calculated agreement with caution because its optimal distribution is a prevalence of about 50%. 7 The interobserver agreement was statistically better in senior t h a n in junior observers. Generally, there is no difference between junior and senior observers.4, 13, 14 Moreover, one third of senior and all junior observers were found to be poorly trained VOLUME 46, NO. 1, 1997
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
observers at least once. In this study, no training phenomenon was found between the first and the second half of the patient group, inasmuch as the respective mean K indexes were similar. The very good intraobserver agreement between the in vivo and on video evaluations suggests that the bias due to video recording standardization was negligible. The analysis of additional tests of agreement yields the following information (Table 2). In relation to agreement proportion, agreement was nearly always better for the absence than for the presence of a sign. Because agreement was always high for the absence of a sign, the low calculated agreements were always due to a low agreement for the presence of a sign. The above mentioned analysis was descriptive. The other m e a s u r e m e n t s (observer bias, poorly trained observer) provide additional explanations for the causes of disagreement. Observer bias indicates that the level of disagreement was significantly different among the observers. This bias explains the poor agreement for visible vessels. A poorly trained observer indicates a statistically significant n u m b e r of observations in which the observer completely disagreed with other observers. This occurred in 10% of the cases and helps explain the lack of excellent agreement for several signs. It should be noted[ that this misinterpretation occurred twice in two senior observers and in all junior observers. This test also identifies poorly trained observers for a sign to help improve their judgement. Other studies have investigated the evaluation of bleeding ulcers by observers. The study by Laine et al., 3 had a different method: slides of ulcer features (SRH or clean base) were evaluated independently by three observers instead of video recordings from live examinations, thus creating difficulties for the correct visualization of the bleeding site. In that study 202 participants were asked to correctly identify findings befbre and after a teaching session: the total of correct answers increased from 72% before to 82% after the teaching session. In this study, the classic K index was also used and comparison of the learning curve was made before and after the teaching session, whereas in our study, in addition to the K index, new tests were used such as the proportion of agreement, observer bias, and poorly trained observer to provide an in-depth examination of the cause of disagreement. The primary finding of variability in labeling stigmata is not different in these two studies. Lau et al. 15 reported an interobserver study based on the same methods as ours with a weighted K of 0.453 _+ 0.006 for the entire group. Agreement was poor for visible vessels and a clean VOLUME 46, NO. 1, 1997
B Bour, B Person, P Cal~s, et al.
base, fair for active oozing, adherent clots, and flat pigmented clots, and good for spurts. However, in that study all participants were international experts yielding artificial conditions and K estimates were obtained by pairs of any two experts. Cosentino et al. 16 reported good to excellent interobserver agreement for the whole Forrest classification after 2 months of training for semiological terminology based on videotape learning sessions. However, once again in this preliminary study, the method did not correspond to the clinical situation, inasmuch as endoscopists were all skilled, a consensus was decided before judgement, typical lesions were selected, and the evaluation was made after a training period. The results of this study emphasize the limits of endoscopic diagnosis in this situation. Several hypotheses can be suggested for these limits: an insufficient endoscopic training, a lack of precise definitions or a consensus in the description of lesions, and the oversophistication of some classifications. Finally, we suggest several solutions to these limitations. First, classifications should be simplified (i.e., active vs nonactive hemorrhage), and second, grading should be limited to a few classes (i.e., visible vessel vs clot when nonactive hemorrhages are being considered). Flat spots or a clean base should not be included in S H R because these conditions do not have a risk of rebleeding. In portal hypertension, endoscopist experts agree on the suggestion of simpler classifications 17 and we have observed a significantly higher agreement when using a twoclass classification than a three-class classification. 4 Moreover, special attention should be paid to endoscopic training for these lesions. The quality of this training can be improved by two means: (1) teaching either in vivo or with video tapes, (2) precise definitions of lesions in scientific manuscripts, with high-quality illustrations (drawings or color photographs). REFERENCES 1. N.I.H. Consensus conference. Therapeutic endoscopy and bleeding ulcers. JAMA 1989;262:1369-72. 2. Rosch W. Endoscopy of the upper gastrointestinal tract. Gastroenterology Annual 1986;3:549-67. 3. Laine L, Freeman M, Cohen H. Lack of uniformity in evaluation of endoscopic prognostic features of bleeding ulcers. Gastrointest Endosc 1994;40:411-7. 4. Cal~s P, Burtin P, Person P, Ruget O, Charlois T, Ferrero P, et al. Interobserver agreement on endoscopic diagnosis of diffuse gastric mucosal features in cirrhotic patients. Eur J Gastroenterol Hepatol 1993;5:263-8. 5. Freeman M, Cass OW, Peine CJ, Onstad GR. The nonbleeding visible vessel versus the sentinel clot: natural history and risk of rebleeding. Gastrointest Endosc 1993;39:359-66. 6. Fleiss JL. The measurement of interrater agreement. In: John Wiley. Statistical methods for rates and proportions. New York, 1981:212-36. GASTROINTESTINAL ENDOSCOPY 31
B Bour, B Person, P Cal~s, et al.
Interobserver agreement on endoscopic diagnosis of bleeding peptic ulcers
7. Grant JM. The fetal heart rate trace is normal, isn't it? Observer agreement of categorical assessments. Lancet 1991; 337:215-8. 8. Snedecor GW, Cochran WG. Statistical methods. 6th ed. Ames, Iowa: Iowa State University Press, 1967:328-30. 9. Branicki FJ, Coleman SY, Fok PJ, Pritchett CJ, Fan ST, Lai ECS, et al. Bleeding duodenal ulcer: a prospective evaluation of risk factors for rebleeding and death. Ann Surg 1990;211: 411-8. 10. Hsu PI, Lin XZ, Chan SH, Lin CY, Chang TT, Shin JS, et al. Bleeding peptic ulcer--risk factors for rebleeding and sequential changes in endoscopic finding. Gut 1994;35:746-9. 11. Lin HJ, Perng CL, Lee FY, Lee CH, Lee SD. Clinical courses and predictors for rebleeding in patients with peptic ulcers and nonbleeding visible vessels: a prospective study. Gut 1994;35:1389-93. 12. Uno Y. Color vision and endoscopic diagnosis. Gastrointest Endosc 1995;42:377-8.
13. Bendtsen F, Skovgaard LT, Sorensen TIA, Matzen P. Agreement among multiple observers on endoscopic diagnosis of esophageal varices before bleeding. Hepatology 1990;11: 341-7. 14. Tissot O, Aub~ C, Namour A, Blanc F, Dauver A, Cal~s P. S~miologie ~chographique de la cirrhose: concordance interobservateur. Gastroenterol Clin Biol 1995;19:291-6. 15. Lau JYW, Sung JY, Lau JFT, Chan ACW, Ng EKW, Chung SCS. Stigmata of recent hemorrhage in peptic ulcer bleeding: is there inter-observer agreement among international experts? [abstract] Gastrointest Endosc 1995;41:368. 16. Cosentino F, Arcidiacono PG, Rubis G, Passoni G, Morandi E, Motta R, et al. Observer variation in the assessment of stigmata of recent haemorrhage: a multicenter study by 13 skilled endoscopists [abstract]. Gut 1994;35:$80. 17. Spina GP, Arcidiacono R, Bosch J, Pagliaro L, Burroughs AT{, Santambrogio R, et al. Gastric endoscopic features in portal hypertension: final report of a consensus conference, Milan, Italy, September 19, 1992. J Hepatol 1994;21:461-7.
Availability of Journal back issues As a service to our subscribers, copies of b a c k issues of Gastrointestinal Endoscopy for the preceding 5 y e a r s a r e m a i n t a i n e d a n d are available for p u r c h a s e from Mosby u n t i l i n v e n t o r y is d e p l e t e d a t a cost of $12.00 pe r issue. The following q u a n t i t y discounts a r e available: 25% off on q u a n t i t i e s of 12 to 23, a n d one t h i r d off on q u a n t i t i e s of 24 or more. P l e a s e w r i t e to M o s b y - Y e a r Book, Inc., Subscription Services, 11830 W e s t l i n e Indust r i a l Dr., St. Louis, MO 63146-3318, or call (800) 453-4351 or (314) 453-4351 for i n f o r m a t i o n on a v a i l a b i l i t y of p a r t i c u l a r issues. If u n a v a i l a b l e from the publisher, photocopies of complete issues m a y be p u r c h a s e d from UMI, 300 N. Zeeb Rd., A n n Arbor, M I 48106 (313) 761-4700.
32 GASTROINTESTINAL ENDOSCOPY
VOLUME 46, NO. 1, 1997