Criterion validity of the Language Background Questionnaire: A self-assessment instrument

E LS EV 1ER

CRITERION VALIDITY OF THE LANGUAGE BACKGROUND QUESTIONNAIRE: A SELF-ASSESSMENT INSTRUMENT D A L E EVAN M E T Z State Universi~, of New York, Geneseo, NY

FRANK CACCAMISE and M A R I A N N E S. G U S T A F S O N National Technical Institute for the Deaf at Rochester Institute of Technology, Rochester, NY

The present investigation examined relationships between the Language Background Questionnaire (LBQ) self-assessed sign language skills, LBQ self-assessed spoken communication skills, and formal independent sign and spoken communication skills assessments of young adults who are deaf or hard of hearing. Results indicated a high degree of congruence between self-assessed communication skills and formal independent assessments of communication skills. The findings establish reasonable criterion validity for the LBQ regarding self-assessed sign language and spoken communication skills. © 1997 by Elsevier Science Inc. Educational Objectives: (1) acquire knowledge and understanding of the Language Background Questionnaire (LBQ); and (2) understand the relationship between LBQ communication skill self-assessment scores and formal communication test scores.

INTRODUCTION Self-assessment instruments have recently gained widespread popularity and frequently are used to evaluate communication performance of persons who are deaf or hard of hearing (c.f. Kaplan, Bally, & Brandt, 1991; Owens & Raggio, 1988; Skinner, 1988). Several advantages of self-assessment instruments have been outlined by Erdman (1994): (a) they are easy and inexpensive to administer; (b) they can be used for a wide variety of purposes; (c) they

Address correspondence to Dale Evan Metz, Ph.D, Department of Communicative Disorders and Sciences, 208-C Sturges Hall, State University of New York-Geneseo, Geneseo, New York 14454. J. COMMUN. DISORD. 30 (1997), 23-32 © 1997 by Elsevier Science Inc. 655 Avenue of the Americas, New York, NY 10010

0021-9924/97/$17.00 SSDI 0021-9924(95)00056-J

24

METZ et al.

LANGUAGE B A C K G R O U N D QUESTIONNAIRE (I.BQ) National Technical lrmitute for the Deaf Rochester Institute of Technology 52 Lomb Memorial Dri~ Rochester, INry 1462~-5604 Note: This questionnaire should be filled-out by the student. Name

Social Security Number

Date

Date of Birth month / day'/year

month I day / year

Instructions: Please read the followingiquestions carefullyand circlethe appropriateanswers. It is very important that you answer these quesUons as honestly as possible, When you "have completed this questionnaire,please return it in the enclosed envelope as soon as possible. Thank you. I, Do you most preferusing (pleasecircleonly one): A.

Sign alone?

B.

Speech alone?

C.

Sign and speech together

(s1~muhaneouscommunication)? 2.

Pleaserate your signingskills:

5 Excellent 3.

4

4 Almost everything I say

3 About half of what I say

2 Only a few words

1 None of my words

Both deaf?

B.

Both hearing?

C.

One de.d, one hearing?

How old were you when you became deaf or hard-of-heor/ng? A.

6.

1 No skills

Are your parents: A.

5.

2

Please rate your speech skills. Hearing people understand:

5 Everything I say 4.

3 Fair

At birth

B.

- - years old

C.

Unknown

Has your hearing loss become worse since age 7 years? A.

Yes

B.

No

C.

Don't know

'Adapted from Caecamlseand poor (1956l,l-~didd (1992),and Haff~ld at Jl. (1978)veeliom of the LBQ by F. C~ami~ G. Poor, T. Armour, K, eagle, D. Burfield,and 5. Holcomb (1911~, Item 13 added in 1992~ G. Otmaf~. and D. Mt~ 099z3.

on information provided by

Figure 1. Language Background Questionnaire (LBQ).

are noninvasive and nonthreatening; and (d) they have good face validity because self-report instruments reflect personal experiences. Erdman further agrees that self-assessments engender a cooperative attitude among respondents "because the task is perceived as relevant..." (Erdman, 1994). This article focuses on a self-assessment instrument, the Language Background Questionnaire (LBQ) developed at the National Technical Institute for the Deaf (Hatfield, 1982; Hatfield, Caccamise, & Siple, 1978; see Figure 1). The development of the LBQ was based on the sociolinguistic factors discussed by Woodward (1973), and was designed to permit self-assessment of sign language skills by young adults who are deaf or hard of hearing. Additionally, the LBQ has other self-referent questions designed to provide pertinent background information that could be useful in educational planning. The original work of Hatfield (1982) and Hatfield et al. (1978) demonstrated that the LBQ provided accurate information regarding preferred mode of com-

LANGUAGE BACKGROUND QUESTIONNAIRE (LBQ)

25

munication (PMC) and reasonable congruence between self-assessed sign language skills and independent formal assessments of sign language skills of young adults who are deaf and hard of hearing. Because of the putative accuracy and simplicity of administration, the LBQ is currently used in lieu of formal tests of sign language at NTID (Caccamise, 1995). In 1992, the LBQ was modified to provide self-assessed estimates of spoken communication skills. This modified version of the LBQ was based on the preliminary research of Gustafson and Metz (1992, 1994), who demonstrated an extremely high degree of congruence between ordinal level self-assessed estimates of spoken communication skills and percentage estimates of speech intelligibility derived from judgments of trained listeners. The findings of Gustafson and Metz were consistent with two previous investigations that suggested that young adults who are deaf and hard of hearing had realistic and accurate perceptions regarding their spoken communication skills (McKee, Stinson, & Blake, 1984; Subtelny, 1982). Based on the preliminary work of Gustafson and Metz (1992), one item (#3, Figure 1) was added to the LBQ. The new item was worded in such a manner as to retain the simplicity and self-referent item format of the original LBQ in an effort to maintain face validity. It was deemed necessary, however, to insure that the previously observed levels of congruence between selfassessed sign language and spoken communication skills and formal measures of these skills were maintained in the new LBQ format. As such, the specific purpose of this research was to determine empirically the magnitude of congruence between: (a) the LBQ self-assessed sign language skills response item and formal assessments of sign language skills, and (b) the LBQ self-assessed spoken communication skills response item and formal assessments of spoken communication skills (i.e., measured speech intelligibility).

METHOD This investigation was conducted to examine the relationships among self-assessed sign language skills and spoken communication skills, and estimates of both these skills as determined by formal independent measures. In essence, this is a criterion validity study regarding specific LBQ response items #2, self ratings of sign language skills, and #3, self ratings of spoken communication skills (see Figure 1). The subjects for this study were 231 students who were enrolled at NTID and participating in orientation programs conducted during the summers of 1993 and 1994. Of these students, 89 were female and 142 were male; their ages ranged from 17 years, 3 months to 29 years, 6 months (mean = 19 years, 5 months), and their better ear mean pure tone average was 94.1 dB HL. Each student completed the experimental version of the LBQ and each student was

26

METZ et al.

formally assessed with respect to sign language skills and spoken communication skills. Measured sign language skills estimates were obtained by individual administrations of the Sign Instruction Placement Interview (SIPI). Details regarding the administration and scoring of the SIPI can be found in Caccamise, Poor, and Holcomb (1992). Briefly, each student was interviewed in a one-toone format by a highly skilled signer knowledgeable about SIPI protocol. Following the interview, the skilled signer assigned a numerical value as an estimate of the student's sign language skills. The assigned numbers ranged from 1 to 5 with 1 indicating no or minimal sign language skills and 5 indicating a high skill level in both American Sign Language and meaning based English word order signing. Measurement of spoken communication skills (i.e, write down speech intelligibility estimates) were obtained using the transcription procedures developed by Magner (1972). Details regarding the administration of this procedure can be found in Samar and Metz (1988). Briefly, for each student, ten sentences were audio recorded as the student read them aloud from a printed script. A trained judge subsequently audited the recorded sentences and transcribed verbatim what s/he heard. Fifty target words embedded in the ten sentences were subsequently scored for their correctness by an independent scorer and a percentage score was computed. This percentage score was taken as the estimate of speech intelligibility. The resulting percentage scores for each student were subsequently partitioned into quintiles (i.e., 0%-20%, 21%-40%, and so on). This data transformation was done to provide correspondence between the percentage scores and the five response levels of LBQ item #3. Responses to items #2 and #3 are at the ordinal level of measurement, and the estimates of sign language skills derived from the SIPI and the quintile groupings of speech intelligibility scores are also at the ordinal level of measurement. The statistical procedure employed to evaluate the above data was the Contingency Coefficient (C), a function of the Chi-square statistic (c.f. Siegel, 1956). One can roughly interpret C as one would interpret any other correlation coefficient (e.g., Pearson r), but the upper limit for C is not 1.0 or unity as it is for the Pearson r. The upper limit of C is determined by the size of the contingency table from which it was derived. For example, the upper limit for C derived from a 2 x 2 contingency table is .707, for a 3 x 3 contingency table the upper limit is .816 (Siegel, 1956; Ventry & Schiavetti, 1986). The upper limit for the contingency coefficients derived from the 5 x 5 contingency tables employed in the present study is .894. Three separate 5 X 5 contingency tables were constructed to examine the relationships between (a) self-assessed sign language skills and the SIPI, (b) self-assessed spoken communication skills and write down speech intelligibil-

LANGUAGE BACKGROUND QUESTIONNAIRE (LBQ)

27

ity, and (c) self-assessed sign language skills and self-assessed spoken communication skills.

RESULTS The results of this investigation reflect a high degree of systematic congruence (covariance) between the formal measures of communication skills and sell'assessed communication skills.

Self-Assessed Sign Language Skills and SIPI The relationship between self-assessed sign language skills and the SIPI ratings was strong and significant (X2 = 166.4; C = ,726; df = 16; p < .001). The direction of this relationship was positive indicating that those students who rated themselves as good to excellent signers (i.e., 4 or 5 on question #2 of the LBQ) also received advanced to highly skilled ratings on the SIPI. Those students who rated themselves as having low or no sign language skills (i.e., 2 or 1 on question #2 of the LBQ) had commensurate SIPI ratings.

Self-Assessed Spoken Communication Skills and Write Down Speech Intelligibility The relationship between self-assessed spoken communication skills and write down speech intelligibility was strong and significant (×2 = 154.9; C = .659; df = 16; p < .001). The direction of this relationship was positive indicating that those students who had rated themselves as having good to excellent spoken communication skills (i.e., 4 or 5 on question #3 of the LBQ) scored in the highest quintile on the write down intelligibility measure and those students who rated themselves as having poor or no spoken communication skills (i.e., 2 or 1 on question #3 of the LBQ) scored in the lowest two quintiles on the write down measure.

Self-Assessed Sign Language Skills and Self-Assessed Spoken Communication Skills The relationship between self-assessed sign language skills and self-assessed spoken communication skills was also significant (X2 = 75.9; C = .507; df = 16; p < .001). The direction of this relationship was positive indicating that the preponderance of students who rated themselves as good to excellent sign-

t The raw data response matrices that were used to derive the three contingency tables reported in this study are available by request from the senior author.

28

METZ et al.

ers (i.e., 4 or 5 on question #2 of the LBQ) also rated themselves as having fair to excellent spoken communication skills (i.e., 3, 4, or 5 on question #3 of the LBQ). The preponderance of students who rated themselves as having good to excellent spoken communication skills generally rated themselves as having fair to excellent sign language skills.

DISCUSSION The present findings indicate a high degree of congruence between self-assessed communication skills and formal measures of communication skills of young adults who are deaf or hard of hearing. These findings establish a reasonable level of criterion validity for the LBQ self assessments of sign language and spoken communication skills. The results further suggest that forreal assessment procedures may not be required if one is interested in a simple indexing of relative communication skills of young adults who are deaf and hard of heating. Their knowledge of their communication skills appears to provide an accurat e index of skill level estimates obtained from formal assessment procedures. Self-assessments are not intended to constructed to supplant detailed diagnostic testing for rehabilitation or educational purposes. However, the simplicity, ease, and economy of administration and interpretation of LBQ responses support its use for general classification purposes similar to a screening instrument. The present results are in concert with the notions of Bess and Humes (1995), who suggest that screening instruments must be evaluated against independent tests (gold standards) that are accepted as valid estimates of the dimension being measured. The LBQ compares very favorably to the "gold standards" (SIPI and write-down speech intelligibility) employed in this investigation suggesting that it could be used effectively as a screening device that has the additional benefit of clear face validity. As Anastasi (1976, p. 139) points out, face validity fundamentally involves "public relations" and it pertains directly to "whether the test 'looks valid' to the examinees who take it, the administrative personnel who decide on its use, and other technically untrained observers." A test that lacks face validity (i.e., it appears irrelevant or inappropriate) will result in poor cooperation among the respondents. The LBQ has good face validity and the present findings establish a reasonable level of criterion validity. As such, the LBQ provides an inexpensive, noninvasive approach to obtaining accurate information about the sign language and spoken communication skills of young adults who are deaf or hard of hearing. Finally, the contingency coefficient relating self-assessed sign and selfassessed spoken communication skills is not directly related to the criterion validity of the LBQ. Rather, this coefficient reflects the potential degree to which one mode of communication (sign or spoken) may be developed at the

LANGUAGE BACKGROUND QUESTIONNAIRE (LBQ)

29

expense of the other among deaf or hard of hearing persons. Conventional wisdom might suggest that highly advanced sign language skills would be associated with low spoken communication skills and vice versa. The observed moderate contingency coefficient (C = .507) indicates that the relationship between sign and spoken communication skills is not as straightforward as conventional wisdom might suggest. For example, 70 of the 231 students in this study rated themselves as having "excellent sign skills" (i.e., 5 on question #2 of the LBQ). Of that 70, 37 rated themselves as having fair to excellent spoken communication skills (i.e., 3, 4, or 5 on question #3 of the LBQ), 30 rated themselves as having poor skills (i.e., 2 on question #3 of the LBQ) and only 3 rated themselves as having no spoken communication skills (i.e., 1 on question #3 of the LBQ). Similarly, 30 of the 231 students rated themselves as having excellent spoken communication skills (i.e., 5 on question #3 of the LBQ). Of that 30, 19 rated themselves as having fair to excellent sign language skills (i.e., 3, 4, or 5 on question #2 of the LBQ), five as having poor skills (i.e., 2 on question #2 of the LBQ), and six rated themselves as having no sign language skills (1 on question #2 of the LBQ). In short, for these students, proficiency in one mode of communication does not necessarily indicate deficiency in the other. We are currently exploring the nature of the relationship between sign language skills and spoken communication skills with respect to the individual's preferred mode of communication. This work was conducted under the course of an agreement with the National Technical Institute for the Deaf, Rochester Institute of Technology and the United States Department of Education. The authors thank Dr. Nicholas Schiavetti for his pertinent comments on a previous version of the manuscript.

REFERENCES Anastasi, A. (1976) Psychological testing (4th Ed.), New York: Macmillan. Bess, F.H., & Humes, L.E. (1995). Audiology: The fundamentals (2nd Ed.). Baltimore, MD: Williams and Wilkins. Caccamise, F., & Poor, G. (1986). Language background questionnaire. Rochester, NY: National Technical Institute for the Deaf. Caccamise, F., Poor, G., Armour, T., Cagle, K., Burfield, D., & Holcomb, S. (1989). Language background questionnaire. Rochester, NY: National Technical Institute for the Deaf. Caccamise, F. (1995). Student sign language education: Recommendations for the NTID 1995 Summer Vestibule Program (SVP) and Summer Support

30

METZ et al.

Team Orientation Program (STOP). Unpublished Manuscript, National Technical Institute for the Deaf, Rochester, NY. Caccamise, F., Poor, G., & Holcomb, B.R. (1992). Interviewer-rater instructions for the Sign Instruction Placement Interview (SIPI). Manuscript, National Technical Institute for the Deaf, Rochester, NY. Erdman, S.A. (1994). Self-assessment: From research focus to research tool. In J. Gangn6 & N. Tye-Murray (Eds). Research in audiological rehabilitation: Current trends and future directions, Journal of the Academy of Rehabilitative Audiology, Monograph Supplement XXVII, 7-90. Gustafson, M.S., & Metz, D.E. (1992). Comparison of self-ratings of speech intelligibility and measured speech intelligibility estimates: A working paper. Unpublished Manuscript, National Technical Institute for the Deaf, Rochester, NY. Gustafson, M.S., & Metz, D.E. (1994). Comparison of self-assessed and measured speech intelligibility estimates. Paper presented at the International Biannual Convention of The Alexander Graham Bell Association for the Deaf, Rochester, NY. Hatfield, N. (1982). An investigation of bilingualism in two signed languages: American Sign Language and Manually Coded English. Unpublished doctoral dissertation, University of Rochester, Rochester, NY. Hatfield, N., Caccamise, F., & Siple, P. (1978). Deaf students language competency: A bilingual perspective. American Annals of the Deaf 123, 847851. Kaplan, H., Bally, S.J., & Brandt, F. (1991). Communication self-assessment scale inventory for deaf adults. Journal of the American Academy of Audiology 2, 164-182. Magner, N.E. (1972). A speech intelligibility test for deaf children. Northampton, MA: Clarke School for the Deaf. McKee, B., Stinson, M., & Blake, R. (1984). Perceived versus measured communication skills of hearing impaired college students. Journal of Rehabilitation of the Deaf lS, 19-24. Owens, E., & Raggio, M. (1988). Performance inventory for profound to severe loss (PIPSL). Journal of Speech and Hearing Disorders 53, 355-357. Samar, V.J., & Metz, D.E. (1988). Criterion validity of speech rating-scale procedures for the hearing-impaired population. Journal of Speech and Hearing Research 31,307-316. Siegel, S. (1956). Nonparametric statistics. New York: McGraw-Hill.

LANGUAGE BACKGROUND QUESTIONNAIRE (LBQ)

31

Skinner, M.W. (1988). Hearing aid evaluation. Englewood Cliffs, NJ.: Prentice-Hall. Subtelny, J.D. (1982). Speech assessment of the adolescent with impaired hearing. In D. Sims, G. Walter, & R.L. Whitehead (Eds), Deafness and communication (pp 155-176). Baltimore, MD: Williams and Wilkins. Ventry, I., & Schiavetti, N. (1986). Evaluating research in speech pathology and audiology (2nd Ed.). New York: Macmillan. Woodward, J. (1973). Some observations on sociolinguistic variation and ASL. Kansas Journal of Sociology 9, 191-200. Manuscript receivedFebruary 24, 1995; revisedJuly 22, 1995.

CONTINUING EDUCATION

Self-Examination Study Questions for Criterion Validity of the Language Background Questionnaire: A Self-Assessment Instrument QUESTIONS 1. Some benefits of communication self-assessment protocols are that: a. They are easy to use b. They are inexpensive c. They can be used for a wide variety of purposes d. They are noninvasive e. All the above 2. The original version of the Language Background Questionnaire was designed to permit self-assessment of: a. Spoken communication skills b. Fingerspelling skills c. Sign language communication skills d. a a n d b e. a a n d c 3. The modified version of the Language Background Questionnaire, as reported in the article, is designed to permit self-assessment of: a. Spoken communication skills b. Fingerspelling skills c. Sign language communication skills d. a a n d b e. a a n d c

32

METZ et al.

4. The positive relationship between the Language Background Questionnaire's self-rated sign language skill scores and scores on the Sign Instruction Placement Interview suggests that: a. Young persons who are deaf and hard of hearing have a good appreciation of their sign language communication skills b. Young persons who are deaf and hard of hearing have a good appreciation of their spoken communication skills c. Young persons who are deaf and hard of hearing do not have a good appreciation of their sign language communication skills d. Young persons who are deaf and hard of heating do not have a good appreciation of their spoken communication skills e. None of the above . The positive relationship between the Language Background Questionnaire's self-rated spoken communication skill scores and scores on the write-down procedure used to estimate speech intelligibility suggests that: a. Young persons who are deaf and hard of hearing have a good appreciation of their sign language communication skills b. Young persons who are deaf and hard of hearing have a good appreciation of their spoken communication skills c. Young persons who are deaf and hard of hearing do not have a good appreciation of their sign language communication skills d. Young persons who are deaf and hard of hearing do not have a good appreciation of their spoken communication skills e. None of the above