- Email: [email protected]
Designing research survey design—Part two
Designing Research Survey Design Part Two M A U R E E N GIUFFRE, PhD, RN Surveys are essential for discovering the incidence, distribution, and interrelationship of variables within a population. As such, it is important that we are able to critically read published surveys. This manuscript is the second of two dealing with survey research. This manuscript deals with the advantages and disadvantages of different types of surveys and what the reader should be looking for when reading survey research. @ 1997 by American Society of PeriAnesthesia Nurses.
HE TYPE of survey a researcher will select depends on the type of information that is being sought. If the researcher is looking for the incidence of a certain variable, eg, "Are you employed in nursing," a simple, mailed questionnaire might do. If the researcher is interested in a deeper understanding of feelings about something, a face-to-face interview would be more appropriate. Interviews conducted by telephone fall in between face-to-face and mailed questionnaires in the areas of type of data that might be obtained, advantages, and disadvantages. Surveys, regardless of type, usually begin by asking the respondent factual, non-threatening information, eg, age, gender, marital status, years of education, and so on. Factual information can also be referred to as demographic information (affectionately referred to as demo). This information is gathered for two reasons. First, it is
T
Maureen Giuffre, PhD, RN is a Clinical Research Consultant in Private Practice, in Salisbury, MD. Address correspondence to Maureen Giuffre, PhD, RN, 26361 High Banks Rd, Salisbury, MD 21801. 9 1997 by American Society of PeriAnesthesia Nurses. 1089-9472/97/1205-0007503.00/0
358
useful in assessing the adequacy of the sampling technique. Should your author be gathering information about the American public (more or less 50% female and 50% male), but the respondents to the survey are 25% female and 75% male, you should be alerted that something was not right in the sampling methodology. Alternatively, the author might have adequately represented his community, which happens to be predominately suburban, white, and upper income, but your population is not. As the consumer of this research, you might accept the validity of the findings for the specific population surveyed or the generalizability of the findings to similar populations, but would also recognize that those findings might not be generalizable to your inner city, lower socioeconomic clientele. The other use for demographic information is to associate that data (age, gender, etc) with the issue under investigation in the survey. Once the author discovers that 60% of the respondents are totally sedentary she would then be able to determine if a particular age or gender group was more sedentary than another. In this way, when interventions are designed, they can be tailored to selected subgroups. Another general point about surveys is that the
Journal of PeriAnesthesia Nursing, Vol 12, No 5 (October), 1997: pp 358-362
SURVEY D E S I G N - - P A R T TWO data being collected is what the subject wishes to reveal about his opinions, beliefs, or behaviors, not the behaviors themselves. Whatever method is likely to produce the clearest, most complete picture of what the behaviors would have been if you directly observed them is the type of survey that should be used. When reading a survey, ask yourself to what extent the nature of the questions and the nature of the survey will interact. MAILED QUESTIONNAIRE
A mailed questionnaire or a questionnaire distributed through some system that does not allow contact between the researcher and the subject has many advantages. First and foremost is that it is the least expensive type of survey. For the price of two stamps and some paper it is possible to get a great deal of information. Because mailed questionnaires are relatively inexpensive, respondents from a wide geographic area can be sought. Although this can also be accomplished with a telephone survey, the expense will be greater and time zones will need to be considered. Another great advantage of mailed surveys is ease of analysis. Do not confuse this with ease of interpretation. By ease of analysis I mean that if the researcher sends out a questionnaire with four option questions (ie, chose from a, b, c, or d) it is a simple matter to code the responses and feed the data into a computer. In addition, responses from a great number of people can be handled with a mailed questionnaire. Mailed questionnaires do not require the services of a skilled interviewer. A researcher who is pathologically shy might have a very difficult time with a research project that required interviewing people face-to-face. A subject matter that might be very personal or embarrassing might be best studied via a mailed questionnaire. Along the same lines, anonymity of the respondent is more likely assured with the use of a mailed questionnaire, and as a result, may produce information that is less socially desirable, but more truthful, than a face-to-face questionnaire. People may be more likely to show their sexist, racist, or other bias feelings in a mailed questionnaire than they are in a face-to-face interview. Another advantage of a mailed questionnaire is that it allows for standardized wording. Each
359
respondent is presented the question in exactly the same manner. On the other hand, if there is some confusion with the meaning of the question, a mailed questionnaire does not have any option for clarification. Disadvantages include not knowing who the respondent really is. If you were to send a questionnaire to my house and ask for the opinions of the male member of the household, i would invariably give them to you. You would believe that the questionnaire was completed by a male when in fact it was completed by a female who believes she knows what the male would have responded. Depending on the portion of society canvassed and the level of interest the subjects may have for the nature of the study, response rates from mailed questionnaires can be very low. The large Boston-based longitudinal study of nurses that is being conducted out of the Harvard School of Public Health is interesting not only for the valuable information that has been obtained but because of the very high response rates.l At this point it appears that the only reason one of their subjects would not send back one of her biannual or annual questionnaires is because she has died. But this is a notoriously unusual response rate. High response rates are achievable when the targeted audience feels as though they have a personal stake in the outcomes of the research. Questionnaires sent to a narrow and select portion of the population, eg, "You have been selected to receive this questionnaire because you are a PACU nurse" are more likely to result in a response than a purely random questionnaire sent out to all Americans. Along the same lines, all mailed surveys must be carefully interpreted. Let us say that a researcher sends out a survey on sexual harassment to every registered nurse in the state and get a response rate of 25 %. The respondents report that 75% of them have experienced on the job sexual harassment. Does this mean that 75% of nurses experience sexual harassment? No, it means that 75% of the respondents experienced sexual harassment. The researchers will go to some lengths to show you that the respondents and the nonrespondents did not differ on age, gender, education, or employment status, but in all likelihood the respondents and non-respondents probably
360
M A U R E E N GIUFFRE
differed greatly on one important i s s u e - - t h a t of having experienced sexual harassment. Those nurses who have experienced sexual harassment are much more likely to return a questionnaire on the subject than those who did not. One alternative to mailed questionnaires is a survey that is conducted "in house." A typical situation is one in which the department of nursing at a particular hospital will not give the researcher the names and addresses of the employees but agrees to distribute the questionnaires and place a box in nursing service for their return. (Graduate students without the cost of two stamps per potential subject might also select this method.) There are two problems here. How are the surveys distributed? If they are sent out attached to each nurse's pay check or some similar manner that is acceptable. But if the nursing supervisor distributes them on her rounds, the chances of her deliberately or subconsciously selecting who will receive them must be considered by the reader. If the method of distribution is found to be acceptable, the method of collection is also important. Even if the hospital distributes the questionnaire, the researcher should enclose a stamped, self-addressed envelope so that the surveys can be directly returned to her. It is not uncommon to read that the surveys are dropped in a sealed box in nursing service. I am not suggesting that nursing administration sneaks in at night and reads what the nurses are writing about them. But if you were a dissatisfied employee, would you be willing to express your dissatisfaction in a survey and then leave that survey, with potentially identifying information on it. under administration's nose? No. Surveys distributed and collected in this manner are likely to result in sanitized responses, because the respondents are usually not comfortable being honest. FACE-TO-FACE INTERVIEW
The major advantage of face-to-face or personal interviews is that the interviewer can probe more deeply the responses and clarify confusing questions. If the subject matter is conceptually a bit more complicated, face-to-face may be the way to go. If, for example, the researcher is interested in the needs of family members waiting in the inten-
sive care unit waiting room while the patient is undergoing open heart surgery, a questionnaire handed out by the waiting room attendant might be ignored or used as a distraction by the family member. On the other hand, if the researcher decides the best way to go is to insert herself into the situation and do a face-to-face interview, to what extent is her presence actually effecting the data? When reading a study that used a face-to-face interview, this latter question needs to be addressed. To what extent does the way the interviewer behavior or even looks (age, gender, race, manner of dress) effect the responses the subject makes? When reading a face-to-face study, are the characteristics of the interviewer made apparent? This might not be important if the subject matter is "preferences in a new car," but if the subject matter is in anyway "sensitive" it may be very important. Earlier it was stated that a person might be more likely to show his less socially desirable sentiments in a mailed questionnaire. Face-to-face also can be very successful in showing true sentiments if the interviewer is perceived by the subject to be sympathetic to his views. As mentioned earlier, the major advantage of face-to-face interviewing is the opportunity to probe and clarify. When reading the study, is the extent to which this was done explained? If more than one person conducted the interviews, were they instructed to explain questions the same way or is it possible that in the long run interviewer A was really interpreting a question one way and interviewer B was interpreting it another way? A final question to ask yourself when reading a face-to-face study,was " H o w was the population obtained?" Lots of people do not want to be bothered. Is it likely that people who do not want to be bother are different from people who would love to give you their opinion about anything. TELEPHONE INTERVIEW OR QUESTIONNAIRE
Historically, telephone interviews have been characterized as tending to fall somewhere in between mailed questionnaires and personal interviews when both advantages and disadvantages are concerned. However, telephone interviews
SURVEY D E S I G N - - P A R T T W O may be changing significantly because of the changing technology. When I read a survey where the author tells me that 100 people were interviewed by phone after being contacted via random dialing, the first thing I know about the respondents is that they are not like me. Ten years ago, I would have politely listened to the researcher's appeal for information, but after years of dinnertime interruptions, I will not let a random caller get through the basic introduction before I hang up. Of course not everyone is like me, lots of people have caller ID. When reading a study that involved random telephone surveys, be very skeptical about generalizing from the findings. Follow-up telephone surveys are different from random telephone surveys in that the subject is aware that they will be receiving a call 1 week after an event (eg, hospitalization) to query them about certain aspects of the event, their disease, or their recovery. This type of interview may result in valuable and potentially generalizable information. RISKS TO INTERNAL VALIDITY
Political pollings or surveys have nearly made a mockery out of the practice of surveys. It is not uncommon to hear on the morning news that the American public believes that special interest groups have an impact on every Senator and Representative in Washington. On the evening news we are told that the American public is not terribly concerned with the situation of special interests in Washington. By carefully wording the questions, it is possible to get these apparently disparate results even if the exact same populations were used for the survey. It is no longer acceptable simply to receive the researchers' summation of the findings of a survey. To some extent the consumer needs to understand the content of the questionnaire so they will be able to evaluate the questionnaire's internal validity. Not long ago, I read a survey by a local nurse researcher in which teenagers were questioned about health habits. The questions were all worded so that there would be no reasonable alternative but to find that teenagers would engage in healthy habits if given more community funded opportunities. After reading the questionnaire, it was very clear what the re-
361
searcher's agenda was. This is very much like the " D o you still beat your wife?" question. The question to be asked by the reader is " I s there any alternative explanation for the findings of this survey?" The answer may have been that the wording of the questions predetermines the response from most people. RISKS TO EXTERNAL VALIDITY
Let us assume that the findings of the study are internally valid. What is reported is an accurate representations of the opinions or behaviors of the population studied. The question now becomes a bit of a " s o what?" To what extent do the opinions or reported behaviors of that population represent the opinions or behavior of another population? This would be referred to as the study' s external validity or generalizability. Generalizability is affected by a number of things, the most important being demographics and response rate. Response rate can be thought of as affecting either the internal or external validity or both. For example, the author is interested in the opinions of RNs. Rather than send out a questionnaire to all RNs, she will send out questionnaires to a sample. Let us say she reports that she has sent out 100 questionnaires with 75 returned, for a 75% response rate. The author will then make an attempt at determining how well that 75 represent the 100. Before we progress any further let us be clear that we are dealing with three sets of people here. The RNs, surveyed nurses (versus non-surveyed), and respondents (versus non-respondents). At some level the author is going to make the case that the respondents represent the RNs. Simply because the subjects selected to receive the questionnaire are randomly selected from a population that does not necessarily make the responses generalizable to that population or any other population. Common sense must be invoked here. What percentage of the surveyed population responded? If it is a very small percentage, to what extent did the subject matter of the questionnaire have special meaning to them? What percentage is sufficient? These questions will have no readily apparent answer. Clearly a 75% response rate is good. And even though it is a good response rate you must still ask yourself
362
M A U R E E N GIUFFRE
why the other 25% did not respond. A 25% response rate is not sufficient from which to generalize, although not an uncommon response rate in published research. Everything in between is a judgement call. These comments on response rate only concern generalizability. The authors may not be attempting to generalize. It is very possible that they only received a 25% response rate, but of those people who responded they received some very interesting comments which are discussed. RISKS TO RELIABILITY
Risks to reliability are most often a concern when face-to-face interviews are used. Does the interviewer interpret the question to each subject the same way? Do different interviewers interpret the question the same way? The researcher needs to explain to the reader what steps were taken to
ensure that each subject heard the question the same way. The researchers may discuss guidelines for interpreting the question to subjects. The reader needs some explanation to evaluate the reliability of the questionnaire. Surveys can be powerful tools in showing the incidence, distribution, and/or inter-relationship of variables within a population or feelings and behaviors of a selected group of people. When reading a survey, it is very important to keep in mind that the type of data available is limited. Surveys can never prove a point, but can be used as the base from which interventions can be designed. REFERENCE 1. Colditz GA, Willett WC, Hunter DJ, et al: Family history, age, and risk of breast cancer. Prospective data from the nurses' health study. JAMA 270:338-343, 1993
HE TYPE of survey a researcher will select depends on the type of information that is being sought. If the researcher is looking for the incidence of a certain variable, eg, "Are you employed in nursing," a simple, mailed questionnaire might do. If the researcher is interested in a deeper understanding of feelings about something, a face-to-face interview would be more appropriate. Interviews conducted by telephone fall in between face-to-face and mailed questionnaires in the areas of type of data that might be obtained, advantages, and disadvantages. Surveys, regardless of type, usually begin by asking the respondent factual, non-threatening information, eg, age, gender, marital status, years of education, and so on. Factual information can also be referred to as demographic information (affectionately referred to as demo). This information is gathered for two reasons. First, it is
T
Maureen Giuffre, PhD, RN is a Clinical Research Consultant in Private Practice, in Salisbury, MD. Address correspondence to Maureen Giuffre, PhD, RN, 26361 High Banks Rd, Salisbury, MD 21801. 9 1997 by American Society of PeriAnesthesia Nurses. 1089-9472/97/1205-0007503.00/0
358
useful in assessing the adequacy of the sampling technique. Should your author be gathering information about the American public (more or less 50% female and 50% male), but the respondents to the survey are 25% female and 75% male, you should be alerted that something was not right in the sampling methodology. Alternatively, the author might have adequately represented his community, which happens to be predominately suburban, white, and upper income, but your population is not. As the consumer of this research, you might accept the validity of the findings for the specific population surveyed or the generalizability of the findings to similar populations, but would also recognize that those findings might not be generalizable to your inner city, lower socioeconomic clientele. The other use for demographic information is to associate that data (age, gender, etc) with the issue under investigation in the survey. Once the author discovers that 60% of the respondents are totally sedentary she would then be able to determine if a particular age or gender group was more sedentary than another. In this way, when interventions are designed, they can be tailored to selected subgroups. Another general point about surveys is that the
Journal of PeriAnesthesia Nursing, Vol 12, No 5 (October), 1997: pp 358-362
SURVEY D E S I G N - - P A R T TWO data being collected is what the subject wishes to reveal about his opinions, beliefs, or behaviors, not the behaviors themselves. Whatever method is likely to produce the clearest, most complete picture of what the behaviors would have been if you directly observed them is the type of survey that should be used. When reading a survey, ask yourself to what extent the nature of the questions and the nature of the survey will interact. MAILED QUESTIONNAIRE
A mailed questionnaire or a questionnaire distributed through some system that does not allow contact between the researcher and the subject has many advantages. First and foremost is that it is the least expensive type of survey. For the price of two stamps and some paper it is possible to get a great deal of information. Because mailed questionnaires are relatively inexpensive, respondents from a wide geographic area can be sought. Although this can also be accomplished with a telephone survey, the expense will be greater and time zones will need to be considered. Another great advantage of mailed surveys is ease of analysis. Do not confuse this with ease of interpretation. By ease of analysis I mean that if the researcher sends out a questionnaire with four option questions (ie, chose from a, b, c, or d) it is a simple matter to code the responses and feed the data into a computer. In addition, responses from a great number of people can be handled with a mailed questionnaire. Mailed questionnaires do not require the services of a skilled interviewer. A researcher who is pathologically shy might have a very difficult time with a research project that required interviewing people face-to-face. A subject matter that might be very personal or embarrassing might be best studied via a mailed questionnaire. Along the same lines, anonymity of the respondent is more likely assured with the use of a mailed questionnaire, and as a result, may produce information that is less socially desirable, but more truthful, than a face-to-face questionnaire. People may be more likely to show their sexist, racist, or other bias feelings in a mailed questionnaire than they are in a face-to-face interview. Another advantage of a mailed questionnaire is that it allows for standardized wording. Each
359
respondent is presented the question in exactly the same manner. On the other hand, if there is some confusion with the meaning of the question, a mailed questionnaire does not have any option for clarification. Disadvantages include not knowing who the respondent really is. If you were to send a questionnaire to my house and ask for the opinions of the male member of the household, i would invariably give them to you. You would believe that the questionnaire was completed by a male when in fact it was completed by a female who believes she knows what the male would have responded. Depending on the portion of society canvassed and the level of interest the subjects may have for the nature of the study, response rates from mailed questionnaires can be very low. The large Boston-based longitudinal study of nurses that is being conducted out of the Harvard School of Public Health is interesting not only for the valuable information that has been obtained but because of the very high response rates.l At this point it appears that the only reason one of their subjects would not send back one of her biannual or annual questionnaires is because she has died. But this is a notoriously unusual response rate. High response rates are achievable when the targeted audience feels as though they have a personal stake in the outcomes of the research. Questionnaires sent to a narrow and select portion of the population, eg, "You have been selected to receive this questionnaire because you are a PACU nurse" are more likely to result in a response than a purely random questionnaire sent out to all Americans. Along the same lines, all mailed surveys must be carefully interpreted. Let us say that a researcher sends out a survey on sexual harassment to every registered nurse in the state and get a response rate of 25 %. The respondents report that 75% of them have experienced on the job sexual harassment. Does this mean that 75% of nurses experience sexual harassment? No, it means that 75% of the respondents experienced sexual harassment. The researchers will go to some lengths to show you that the respondents and the nonrespondents did not differ on age, gender, education, or employment status, but in all likelihood the respondents and non-respondents probably
360
M A U R E E N GIUFFRE
differed greatly on one important i s s u e - - t h a t of having experienced sexual harassment. Those nurses who have experienced sexual harassment are much more likely to return a questionnaire on the subject than those who did not. One alternative to mailed questionnaires is a survey that is conducted "in house." A typical situation is one in which the department of nursing at a particular hospital will not give the researcher the names and addresses of the employees but agrees to distribute the questionnaires and place a box in nursing service for their return. (Graduate students without the cost of two stamps per potential subject might also select this method.) There are two problems here. How are the surveys distributed? If they are sent out attached to each nurse's pay check or some similar manner that is acceptable. But if the nursing supervisor distributes them on her rounds, the chances of her deliberately or subconsciously selecting who will receive them must be considered by the reader. If the method of distribution is found to be acceptable, the method of collection is also important. Even if the hospital distributes the questionnaire, the researcher should enclose a stamped, self-addressed envelope so that the surveys can be directly returned to her. It is not uncommon to read that the surveys are dropped in a sealed box in nursing service. I am not suggesting that nursing administration sneaks in at night and reads what the nurses are writing about them. But if you were a dissatisfied employee, would you be willing to express your dissatisfaction in a survey and then leave that survey, with potentially identifying information on it. under administration's nose? No. Surveys distributed and collected in this manner are likely to result in sanitized responses, because the respondents are usually not comfortable being honest. FACE-TO-FACE INTERVIEW
The major advantage of face-to-face or personal interviews is that the interviewer can probe more deeply the responses and clarify confusing questions. If the subject matter is conceptually a bit more complicated, face-to-face may be the way to go. If, for example, the researcher is interested in the needs of family members waiting in the inten-
sive care unit waiting room while the patient is undergoing open heart surgery, a questionnaire handed out by the waiting room attendant might be ignored or used as a distraction by the family member. On the other hand, if the researcher decides the best way to go is to insert herself into the situation and do a face-to-face interview, to what extent is her presence actually effecting the data? When reading a study that used a face-to-face interview, this latter question needs to be addressed. To what extent does the way the interviewer behavior or even looks (age, gender, race, manner of dress) effect the responses the subject makes? When reading a face-to-face study, are the characteristics of the interviewer made apparent? This might not be important if the subject matter is "preferences in a new car," but if the subject matter is in anyway "sensitive" it may be very important. Earlier it was stated that a person might be more likely to show his less socially desirable sentiments in a mailed questionnaire. Face-to-face also can be very successful in showing true sentiments if the interviewer is perceived by the subject to be sympathetic to his views. As mentioned earlier, the major advantage of face-to-face interviewing is the opportunity to probe and clarify. When reading the study, is the extent to which this was done explained? If more than one person conducted the interviews, were they instructed to explain questions the same way or is it possible that in the long run interviewer A was really interpreting a question one way and interviewer B was interpreting it another way? A final question to ask yourself when reading a face-to-face study,was " H o w was the population obtained?" Lots of people do not want to be bothered. Is it likely that people who do not want to be bother are different from people who would love to give you their opinion about anything. TELEPHONE INTERVIEW OR QUESTIONNAIRE
Historically, telephone interviews have been characterized as tending to fall somewhere in between mailed questionnaires and personal interviews when both advantages and disadvantages are concerned. However, telephone interviews
SURVEY D E S I G N - - P A R T T W O may be changing significantly because of the changing technology. When I read a survey where the author tells me that 100 people were interviewed by phone after being contacted via random dialing, the first thing I know about the respondents is that they are not like me. Ten years ago, I would have politely listened to the researcher's appeal for information, but after years of dinnertime interruptions, I will not let a random caller get through the basic introduction before I hang up. Of course not everyone is like me, lots of people have caller ID. When reading a study that involved random telephone surveys, be very skeptical about generalizing from the findings. Follow-up telephone surveys are different from random telephone surveys in that the subject is aware that they will be receiving a call 1 week after an event (eg, hospitalization) to query them about certain aspects of the event, their disease, or their recovery. This type of interview may result in valuable and potentially generalizable information. RISKS TO INTERNAL VALIDITY
Political pollings or surveys have nearly made a mockery out of the practice of surveys. It is not uncommon to hear on the morning news that the American public believes that special interest groups have an impact on every Senator and Representative in Washington. On the evening news we are told that the American public is not terribly concerned with the situation of special interests in Washington. By carefully wording the questions, it is possible to get these apparently disparate results even if the exact same populations were used for the survey. It is no longer acceptable simply to receive the researchers' summation of the findings of a survey. To some extent the consumer needs to understand the content of the questionnaire so they will be able to evaluate the questionnaire's internal validity. Not long ago, I read a survey by a local nurse researcher in which teenagers were questioned about health habits. The questions were all worded so that there would be no reasonable alternative but to find that teenagers would engage in healthy habits if given more community funded opportunities. After reading the questionnaire, it was very clear what the re-
361
searcher's agenda was. This is very much like the " D o you still beat your wife?" question. The question to be asked by the reader is " I s there any alternative explanation for the findings of this survey?" The answer may have been that the wording of the questions predetermines the response from most people. RISKS TO EXTERNAL VALIDITY
Let us assume that the findings of the study are internally valid. What is reported is an accurate representations of the opinions or behaviors of the population studied. The question now becomes a bit of a " s o what?" To what extent do the opinions or reported behaviors of that population represent the opinions or behavior of another population? This would be referred to as the study' s external validity or generalizability. Generalizability is affected by a number of things, the most important being demographics and response rate. Response rate can be thought of as affecting either the internal or external validity or both. For example, the author is interested in the opinions of RNs. Rather than send out a questionnaire to all RNs, she will send out questionnaires to a sample. Let us say she reports that she has sent out 100 questionnaires with 75 returned, for a 75% response rate. The author will then make an attempt at determining how well that 75 represent the 100. Before we progress any further let us be clear that we are dealing with three sets of people here. The RNs, surveyed nurses (versus non-surveyed), and respondents (versus non-respondents). At some level the author is going to make the case that the respondents represent the RNs. Simply because the subjects selected to receive the questionnaire are randomly selected from a population that does not necessarily make the responses generalizable to that population or any other population. Common sense must be invoked here. What percentage of the surveyed population responded? If it is a very small percentage, to what extent did the subject matter of the questionnaire have special meaning to them? What percentage is sufficient? These questions will have no readily apparent answer. Clearly a 75% response rate is good. And even though it is a good response rate you must still ask yourself
362
M A U R E E N GIUFFRE
why the other 25% did not respond. A 25% response rate is not sufficient from which to generalize, although not an uncommon response rate in published research. Everything in between is a judgement call. These comments on response rate only concern generalizability. The authors may not be attempting to generalize. It is very possible that they only received a 25% response rate, but of those people who responded they received some very interesting comments which are discussed. RISKS TO RELIABILITY
Risks to reliability are most often a concern when face-to-face interviews are used. Does the interviewer interpret the question to each subject the same way? Do different interviewers interpret the question the same way? The researcher needs to explain to the reader what steps were taken to
ensure that each subject heard the question the same way. The researchers may discuss guidelines for interpreting the question to subjects. The reader needs some explanation to evaluate the reliability of the questionnaire. Surveys can be powerful tools in showing the incidence, distribution, and/or inter-relationship of variables within a population or feelings and behaviors of a selected group of people. When reading a survey, it is very important to keep in mind that the type of data available is limited. Surveys can never prove a point, but can be used as the base from which interventions can be designed. REFERENCE 1. Colditz GA, Willett WC, Hunter DJ, et al: Family history, age, and risk of breast cancer. Prospective data from the nurses' health study. JAMA 270:338-343, 1993