- Email: [email protected]
A hard look at the reference check
A HARD LOOK AT THE REFERENCE CHECK Its modest worth can be improved
ALLAN N. NASH AND STEPHEN J. CARROLL, JR.
Both authors are faculty members in the College o f Business and Public Administration, University o f Maryland.
How valuable is the widely used reference check in predicting performance on the new job? A number of studies indicate that the value is modest at best. Previous employers may fail to respond to a reference request or be reluctant to return negative statements. A prospective employer can improve the usefulness o f the written reference questionnaire by bearing in mind that personal characteristics o f the reference giver may affect the validity of his report. Reference givers tend to rate their own sex differently than they rate the opposite sex, raters ap[Sear overly harsh on short tenure former employees, and national and racial differences apparently offset the usefulness of the response. The questionnaire's usefulness can be improved by incorporating the forced-choice format to cut down on leniency. The response rate is affected by the approach used in requesting cooperation from reference givers.
Virtually all employers use the reference check as part of their selection process. There are two basic types of references: the
OCTOBER, 1970
character reference, typically obtained from a friend or relative of the applicant, and the previous employer reference, which is most widely used in the industrial situation. One survey found that 90 percent of all responding employers contacted former employers of their job applicants. 1 The reference check with former employers has two purposes. It is used to check the validity of information given by the job applicant on the application blank, and it is used to predict success on the new job. The second purpose seems to be more important since several research studies have indicated that job applicants rarely lie when filling out an application blank. 2 Most of the organizations that contact former employers believe that they obtain useful information. For example, one survey reports that 76 percent of responding companies believed that their
1. W.A. Spriegel and V. A. James, "Trends in Recruitment and Selection Practice," Personnel, VI (1958), pp. 35-45. 2. For example, D. Yoder, H. Henemen, J. Turnbull, and C. Stone, Handbook o f Personnel Management and Labor Relations (New York: McGraw-Hill Book Company, 1958), pp. 8-17.
4."
ALLAN N. NASH AND STEPHEN J. CARROLL
selection procedures would suffer if written references were not available to them. 3
HOW VALUABLE ARE REFERENCES?
44
Despite this widespread acceptance of references by employers, a survey we have made of the ten most widely used personnel management and employee selection textbooks indicates that the writers of these books believe that the reference check does not provide enough valid information to be of practical value. In addition, a number of research studies have examined the effectiveness or validity of the reference check in predicting job performance. In the most extensive study of this type, Mosel and Goheen compared the information in reference checks obtained for 1,193 civil service employees in twelve / skilled occupations with specially devised performance ratings made b y their present supervisors. 4 The information analyzed dealt with occupational ability, character and reputation, and employability. The reference check information showed no consistent or sizable relationship with present job success (no relationships were higher than a correlation of r = +.29 and most were near zero). Another study in a nonindustrial setting collected references from many sources for 508 school teachers, s These references were correlated to v a r i o u s ratings o f teaching effectiveness in the first year in the new teaching position. The references were generally unpredictive of teaching effectiveness (average correlation was r = .13). References were most predictive when they were completed by the last supervisor of the
3. j . N. Mosel and H. W. Goheen, "Use of the ERQ in Hiring," Personnel Journal, XXXVI (1958B), pp. 338-40. 4. J . N . Mosel and H. W. Goheen, "The Validity of the Employment Recommendation Questionnaire in Personnel Selection: Skilled Trades," Personnel Psychology, XL (1958A), pp. 481-90. 5. R.C. Browning, "Validity of Reference Ratings from Previous Employers," Personnel Psychology, XXI (1968), pp. 389-93.
teacher in a teaching position, but the relationship here was modest (r = .23). Still another study examined the relationship of references received for Peace Corps candidates with ratings of their performance in overseas positions. Again the relationship was modest.
REFERENCE CHECK PROBLEMS The Rate of Return The reference check, unlike other selection approaches, has the problem of the response rate. If previous employers exercise their right to reject a reference request for a former employee, the employing organization must get along without this information. There are little data available to indicate what the rate of return is likely to be. Mosel and Goheen report a 56 percent return for 4,000 reference questionnaires sent out by a federal government agency for twelve skilled occupations, and a 64 percent rate of return for 16,000 questionnaires sent out for professional and semiprofessional
groups. 6
Research conducted by the present authors indicates that the m e t h o d used to solicit the reference checks can substantially influence the response rate. Nearly 85 percent of written, objective, one-page questionnaires were completed and r e t u r n e d by former employers of more than a hundred 1967-68 clerical applicants at the University of Maryland, However, this was achieved after a previous effort had resulted in a return rate of only 35 percent. The first approach involved a mimeographed f o r m letter requesting cooperation, a stamped addressed envelope, and a copy of the questionnaire. A switch was made to an individually typed letter with basically the same content sent out under the title "Director of Personnel" and signed in ink by one of the authors. The response rate
6. Mosel and Goheen, "The Validity o f . . . , 481-90.
pp.
BUSINESS HORIZONS
A Hard Look at the Reference Check
improved after this change was made. However, this approach was not nearly as successful in obtaining a satisfactory return rate when it was used for former employers of maintenance and custodial applicants at the university. The finding that the m e t h o d used has differential effects depending on the occupation involved is consistent with findings reported by Mosel and Goheen, who also found a higher response rate for applicants for higher level jobs. It now appears that, with care, a high rate of return can be obtained for reference checks for job applicants, and that somewhat higher rates of return can be expected for occupations at higher levels. However, a fairly recent development may reduce the rate of return for reference checks in the future. Several management reporting services have described a case in which a company was successfully sued when it failed to keep a reference confidential, but referred to a negative one it had received as reason for rejecting an applicant. 7 It is conceivable that this may frighten personnel departments into adopting a policy prohibiting company managers from filling out reference requests.
The Leniency Problem An adequate response rate, of course, does not mean that the references received are of much value in making e m p l o y m e n t decisions about job applicants. The research study conducted by Mosel and Goheen indicate that employers may be reluctant to furnish other employers with negative information about former employees. For this reason, the reference check generally lacks high validity. On a four-category scale (poor, satisfactory, good, and outstanding), they found almost 92 percent of the responses were either good or outstanding w h e n occupational ability was
7.
White Collar Management (New York: Man and
Manager, Inc., 1967).
OCTOBER, 1970
being rated, and about 95 percent when character and reputation were being rated. Over 97 percent of their respondents said they would reemploy the applicant. These findings may be distorted somewhat by unique circumstances existing in that study. First, it is possible that the typical government r e f e r e n c e form a n d the impersonal manner of collecting references contribute to an indifferent attitude on the part of the reference giver. It is also possible that the typical reference giver does not mind passing off a borderline employee onto the government as much as he might to a smaller less bureaucratic private organization, which might send him a critical follow-up c o m m e n t if t h e r e c o m m e n d e d applicant does not live up to his reference. The reference giver probably feels that the government is less likely to send such a follow-up, especially if the reference is sent to a department or institution rather than a specific person, as is usually the case. He may even feel that the government can better absorb s u c h an employee. The feeling may exist that the bureaucracy has more than its share of similar kinds of people anyhow, and "one more won't hurt it." Evidence collected b y the authors in conducting a study of the predictive validity of a new type of reference check (forced choice) for clerical employees also suggests that the degree of leniency in the Mosel and Goheen results is not necessarily to be expected in industrial applications. Notes and comments frequently written in the margin of a new forced-choice reference form being tested, which included only pairs of positivesounding statements, suggested that a significant minority of reference givers would have checked negative statements if they had been given the opportunity to do so on the form. A small pilot study attempted to test this proposition. 8 In this study, seven personnel
8. Allan N. Nash, "A Methodological Suggestion for Constructing Forced Choice Scales," Experimental Journal o f the APA (1970).
45
ALLAN N. NASH AND STEPHEN J. CARROLL
46
managers were contacted and asked to provide the names of at least two former employees "who were borderline performers at best." The personnel managers furnished these names along with the names of the immediate supervisors of these employees. A reference form a n d l e t t e r soliciting cooperation was sent to each former supervisor of the inferior employee. F i f t e e n reference check forms were sent out and thirteen were returned, of which eleven had negative statements checked for the employee in question. This study indicates that supervisors, when asked to give a reference for a former relatively unsatisfactory employee, were willing to make negative comments. Results of an older study by Bingham in w h i c h o n e - h u n d r e d sales managers were surveyed also supports this contention. 9 He found that although 85 percent of them said they would give the former employee the benefit of the doubt in writing letters of recommendation, 39 percent also said they would point out the man's failings and weaknesses as well as his strong points.
Characteristics of the Reference Giver A few studies suggest that there are marked differences in the validity of references depending on who gives them and the relationship of the giver to the applicant. For example, the studies by Mosel and Goheen and Browning indicated that references from immediate supervisors are somewhat more valid than those from other sources, including various character sources not related to the job. Browning also found that the most recent immediate supervisor was better than an older one. Kornhauser found references given by former teachers and employers were better at predicting academic success than references given by a group of friends and professional 9. W. V. Bingham, "The Three Functions of the Interview in Employment," Management Review, V X (1926), p. 36.
acquaintances, which is similar to findings reported above by Mosel and Goheen and Browning. Yoder suggests that the reference is most useful if the job being sought is similar to jobs previously held. 10 Data collected by the authors in a study of a forced-choice reference check also support the theory that the nature of the reference giver may affect the validity of the references. We found the following personal characteristics to be important. First, leniency seems related to the sex of the reference giver. Women tended to be more lenient than men in rating both former employees and present employees. In this situation, the w o m e n were rating women. This finding is congruent with previous research, which indicates that individuals of a particular sex are more lenient in rating members of their own sex. 11 However, in this study the men rating w o m e n held higher and more responsible jobs than the w o m e n rating women. Second, the usefulness of the reference depends on how much the reference giver knows about, the applicant. We found that references given by a supervisor for a job applicant who had performed nonclerical work were not predictive of success in doing clerical work in another organization. This supports the suggestion of Yoder that references from former employers should be discounted unless the work done in the old job is similar to the work to be done on the new job. We also found that references received for former employees who had held their old jobs for less than two months were unpredictive of performance on the second job. This may have occurred not only because of sketchy information, but also because of a tendency for former supervisors to be disgusted with previous employees who quit shortly after being hired. 10. D. Yoder, Personnel Management and Industrial Relations (Sth ed.; Englewood Cliffs, N. J.: Prentice-Hall, Inc., 1962), p. 346. 11. Stephen J. Carroll and Allan N. Nash, "The Evaluation of Performance," unpublished manuscript, 1970.
BUSINESS HORIZONS
A Hard Look at the Reference Check
Third, references given b y a person in a foreign country were not predictive of performance on a j o b in this country. In addition, references given b y a supervisor of one race were n o t predictive of j o b performance as rated b y a supervisor on the new j o b of another race. This was especially true when the applicant was of the same race as the reference giver b u t of a different race from the supervisor evaluating his performance on the new job. This finding is congruent with that in other studies, which generally show that individuals have a tendency to rate members of their own race higher than they rate members of another race. In general, then, it does appear that the characteristics of the reference giver must be considered in deciding h o w reliable their references are. References are probably more accurate if received from immediate supervisors who supervised the applicant for a considerable length of time in a j o b similar to that for which the applicant is being considered. References received from supervisors of the same sex, race, or country of origin as the applicant are likely to be more favorable than performance ratings b y a supervisor of a different sex, race, or country of origin.
Method of Obtaining References References can be obtained from former employers b y letter, telephone, and the use of specially structured objective type questionnaires or field investigations which use interviews to collect i n f o r m a t i o n from the reference givers. It is not k n o w n what percentage of all employers use these alternative methods, b u t one survey indicated that about 50 percent of all firms require some t y p e of written recommendation. The majority of authors in the field suggest that the telephone check is better than a written reference considering cost, accuracy, and time. With the telephone check, one is reasonably certain that the reference
OCTOBER, 1970
giver is the person named as the reference and not some secretary or clerk. The reference can also be obtained quickly and inexpensively. In addition, some feel that the reference giver finds it more difficult to stretch the truth about a j o b applicant in oral conversation than he would in providing the reference in writing. Also, oral answers may contain hesitations and voice inflections that can be revealing as to what the reference giver really thinks about the person under consideration. The reference giver may feel more secure in making negative statements informally than he would in making them formally in a w r i t t e n reference check. Despite these apparent advantages of the telephone check no research study has demonstrated its superiority over the other methods of obtaining references. The field investigation, which involves interviewing reference givers in their homes or places of e m p l o y m e n t , has some of the obvious advantages of the telephone check. One can be certain the named reference is involved in the check, and the oral conversation may reveal information that might not be presented in written form. However, the field investigation is obviously considerably more expensive and time consuming than the telephone check. One study compared the field investigation with the written reference check. In this study, written reference questionnaires and field investigation r e p o r t s were obtained for 109 applicants for three professional jobs in the federal service. 12 Each field investigation consisted of from three to six interviews with individuals who knew the applicants. Only a moderate relationship was established b e t w e e n ratings made of the applicants on the basis of these two different types of reference checks. An investigation was also made of the success of
12. J. N. Mosel and H. W. Goheen, "The Employment Recommendation Questionnaire: Validity of Different Types of References," Personnel Psychology, XII (1959), pp. 469-77.
47
ALLAN N. NASH AND STEPHEN J. CARROLL
48
these methods in identifying applicants who were ineligible for e m p l o y m e n t in the federal service because of alcoholism, homosexuality, or g r o s s incompetence. Only the field investigation identified the seven individuals who were n o t acceptable for these reasons. The written reference form did not identify these individuals primarily because written references were not sent in by the persons who were sent questionnaires. Perhaps, however, the telephone check could collect the information on such clearly disqualified applicants under the standards used, since the problem of nonresponse is greatly reduced under this method. The telephone check would be far cheaper and less t i m e consuming than the field investigation. A research study should be conducted in this area, perhaps by the federal government which obviously spends a great deal of m o n e y on field investigations. The disadvantages of the written reference questionnaire have already been discussed; it is subject to the problems of poor response and to low validity because of the problem of leniency. Our research has shown the poor response problem can be reduced considerably by the use of individually typed and signed letters. We also have developed a new type of reference check questionnaire to eliminate or alleviate the problem of low validity due to the error of leniency. The new check incorporates the forced-choice format developed for the prediction of promotability in the armed forces in the mid-1940's, a 3 The forced-choice m e t h o d was introduced in order to eliminate or reduce the error of leniency in ratings of performance. The rater is presented with a group of behavioral statements. Such statements may be in groups of two, three, or four. Through a fairly 13. Allan N. Nash and Stephen J. Carroll, Jr., "Improving the Validity of a Forced Choice Reference Check with Rater and Job Characteristics Moderators," Proceedings of the 78th Annual Convention, American Psychological Association, 1970, pp. 577-78.
complex research procedure, it is determined how well each behavioral statement used in the form predicts success in a particular job and also how favorable each statement sounds to the raters using the form. The behavioral statements are grouped so that they are similar to other statements in their group with respect to "apparent" favorability, but differ from the other statements in their group with respect to ability to predict success in a particular job. The rater is asked to choose the statement(s) from a group of statements that most apply to the person being rated. Thus the rater is forced to choose from among alternative behavioral statements. We have developed forced-choice rating forms for both maintenance workers and for clerical workers, but only the form for the clerical workers was tested in order to establish its validity. The form consists of twenty-four pairs of behavioral statements descriptive of clerical workers. All the statements were constructed in a positive manner because of the apparent desire of reference givers to give positive references. The pairs of statements are constructed so that the two statements in each pair sound equally favorable, are equally characteristic of clerical workers in general, but differ in their ability to predict success in clerical work. The form was developed for clerical workers in an electronics firm in Baltimore and then was tested at the University of Maryland. The test involved the use of forced-choice reference checks completed by former employers of applicants for clerical jobs at the university. These reference checks were then put aside and not examined until the 122 clerical workers studied had been on the job for at least four months; they were then rated by the supervisors with respect to many aspects of their over-all work performance. It was found that when the reference giver on the previous job had adequate time to observe the worker and was of the same sex, race, and nationality as the supervisor on the
BUSINESS HORIZONS
A Hard L o o k at the Reference Check
new job, and where the old and new jobs were similar in content, the forced choice reference check was quite highly predictive of success (r = +.64 between reference check scores and performance ratings).
IMPROVING USEFULNESS REFERENCE CHECK
OF
THE
The research indicates that organizations can improve the usefulness of the reference check in selecting organizational personnel by following certain suggestions derived from this research.
Survey only previous supervisors of the job applicant. The research indicates that only references from former supervisors of the job applicant are predictive of success on a new job. Less valid references are provided by friends, neighbors, and other organizational personnel in the old organization such as coworkers and higher level organizational personnel who were not in immediate contact with the applicant.
Consider the characteristics o f the immediate supervisor giving the reference. References from supervisors who have not had adequate opportunity to observe the job performance of the applicant are less pre-
OCTOBER, 1970
dictive of job success than references from supervisors who have had an adequate observational opportunity. In addition, references from supervisors on previous jobs differing in sex, country of employment, or race from supervisors on the prospective job are less predictive of job success than when these supervisors are similar.
Use individual typed letters to request the reference. Our research indicated that a return of 85 percent can be obtained with individually typed letters. Choose the appropriate method. There are several alternative methods of obtaining references, including the written reference form, a forced-choice variation of this developed by the authors, the telephone check, and the field investigation. Research has established that the forced-choice reference check format can be predictive of job success. Research can be carried out by organizations themselves on the relative validity of the other reference check methods. The personnel department should find it easy to compare, in a systematic way, alternative reference check methods for the various types of employees the organization employs. Certainly such research would be worthwhile considering the cost, time, and present usefulness of reference check methods.
49
ALLAN N. NASH AND STEPHEN J. CARROLL, JR.
Both authors are faculty members in the College o f Business and Public Administration, University o f Maryland.
How valuable is the widely used reference check in predicting performance on the new job? A number of studies indicate that the value is modest at best. Previous employers may fail to respond to a reference request or be reluctant to return negative statements. A prospective employer can improve the usefulness o f the written reference questionnaire by bearing in mind that personal characteristics o f the reference giver may affect the validity of his report. Reference givers tend to rate their own sex differently than they rate the opposite sex, raters ap[Sear overly harsh on short tenure former employees, and national and racial differences apparently offset the usefulness of the response. The questionnaire's usefulness can be improved by incorporating the forced-choice format to cut down on leniency. The response rate is affected by the approach used in requesting cooperation from reference givers.
Virtually all employers use the reference check as part of their selection process. There are two basic types of references: the
OCTOBER, 1970
character reference, typically obtained from a friend or relative of the applicant, and the previous employer reference, which is most widely used in the industrial situation. One survey found that 90 percent of all responding employers contacted former employers of their job applicants. 1 The reference check with former employers has two purposes. It is used to check the validity of information given by the job applicant on the application blank, and it is used to predict success on the new job. The second purpose seems to be more important since several research studies have indicated that job applicants rarely lie when filling out an application blank. 2 Most of the organizations that contact former employers believe that they obtain useful information. For example, one survey reports that 76 percent of responding companies believed that their
1. W.A. Spriegel and V. A. James, "Trends in Recruitment and Selection Practice," Personnel, VI (1958), pp. 35-45. 2. For example, D. Yoder, H. Henemen, J. Turnbull, and C. Stone, Handbook o f Personnel Management and Labor Relations (New York: McGraw-Hill Book Company, 1958), pp. 8-17.
4."
ALLAN N. NASH AND STEPHEN J. CARROLL
selection procedures would suffer if written references were not available to them. 3
HOW VALUABLE ARE REFERENCES?
44
Despite this widespread acceptance of references by employers, a survey we have made of the ten most widely used personnel management and employee selection textbooks indicates that the writers of these books believe that the reference check does not provide enough valid information to be of practical value. In addition, a number of research studies have examined the effectiveness or validity of the reference check in predicting job performance. In the most extensive study of this type, Mosel and Goheen compared the information in reference checks obtained for 1,193 civil service employees in twelve / skilled occupations with specially devised performance ratings made b y their present supervisors. 4 The information analyzed dealt with occupational ability, character and reputation, and employability. The reference check information showed no consistent or sizable relationship with present job success (no relationships were higher than a correlation of r = +.29 and most were near zero). Another study in a nonindustrial setting collected references from many sources for 508 school teachers, s These references were correlated to v a r i o u s ratings o f teaching effectiveness in the first year in the new teaching position. The references were generally unpredictive of teaching effectiveness (average correlation was r = .13). References were most predictive when they were completed by the last supervisor of the
3. j . N. Mosel and H. W. Goheen, "Use of the ERQ in Hiring," Personnel Journal, XXXVI (1958B), pp. 338-40. 4. J . N . Mosel and H. W. Goheen, "The Validity of the Employment Recommendation Questionnaire in Personnel Selection: Skilled Trades," Personnel Psychology, XL (1958A), pp. 481-90. 5. R.C. Browning, "Validity of Reference Ratings from Previous Employers," Personnel Psychology, XXI (1968), pp. 389-93.
teacher in a teaching position, but the relationship here was modest (r = .23). Still another study examined the relationship of references received for Peace Corps candidates with ratings of their performance in overseas positions. Again the relationship was modest.
REFERENCE CHECK PROBLEMS The Rate of Return The reference check, unlike other selection approaches, has the problem of the response rate. If previous employers exercise their right to reject a reference request for a former employee, the employing organization must get along without this information. There are little data available to indicate what the rate of return is likely to be. Mosel and Goheen report a 56 percent return for 4,000 reference questionnaires sent out by a federal government agency for twelve skilled occupations, and a 64 percent rate of return for 16,000 questionnaires sent out for professional and semiprofessional
groups. 6
Research conducted by the present authors indicates that the m e t h o d used to solicit the reference checks can substantially influence the response rate. Nearly 85 percent of written, objective, one-page questionnaires were completed and r e t u r n e d by former employers of more than a hundred 1967-68 clerical applicants at the University of Maryland, However, this was achieved after a previous effort had resulted in a return rate of only 35 percent. The first approach involved a mimeographed f o r m letter requesting cooperation, a stamped addressed envelope, and a copy of the questionnaire. A switch was made to an individually typed letter with basically the same content sent out under the title "Director of Personnel" and signed in ink by one of the authors. The response rate
6. Mosel and Goheen, "The Validity o f . . . , 481-90.
pp.
BUSINESS HORIZONS
A Hard Look at the Reference Check
improved after this change was made. However, this approach was not nearly as successful in obtaining a satisfactory return rate when it was used for former employers of maintenance and custodial applicants at the university. The finding that the m e t h o d used has differential effects depending on the occupation involved is consistent with findings reported by Mosel and Goheen, who also found a higher response rate for applicants for higher level jobs. It now appears that, with care, a high rate of return can be obtained for reference checks for job applicants, and that somewhat higher rates of return can be expected for occupations at higher levels. However, a fairly recent development may reduce the rate of return for reference checks in the future. Several management reporting services have described a case in which a company was successfully sued when it failed to keep a reference confidential, but referred to a negative one it had received as reason for rejecting an applicant. 7 It is conceivable that this may frighten personnel departments into adopting a policy prohibiting company managers from filling out reference requests.
The Leniency Problem An adequate response rate, of course, does not mean that the references received are of much value in making e m p l o y m e n t decisions about job applicants. The research study conducted by Mosel and Goheen indicate that employers may be reluctant to furnish other employers with negative information about former employees. For this reason, the reference check generally lacks high validity. On a four-category scale (poor, satisfactory, good, and outstanding), they found almost 92 percent of the responses were either good or outstanding w h e n occupational ability was
7.
White Collar Management (New York: Man and
Manager, Inc., 1967).
OCTOBER, 1970
being rated, and about 95 percent when character and reputation were being rated. Over 97 percent of their respondents said they would reemploy the applicant. These findings may be distorted somewhat by unique circumstances existing in that study. First, it is possible that the typical government r e f e r e n c e form a n d the impersonal manner of collecting references contribute to an indifferent attitude on the part of the reference giver. It is also possible that the typical reference giver does not mind passing off a borderline employee onto the government as much as he might to a smaller less bureaucratic private organization, which might send him a critical follow-up c o m m e n t if t h e r e c o m m e n d e d applicant does not live up to his reference. The reference giver probably feels that the government is less likely to send such a follow-up, especially if the reference is sent to a department or institution rather than a specific person, as is usually the case. He may even feel that the government can better absorb s u c h an employee. The feeling may exist that the bureaucracy has more than its share of similar kinds of people anyhow, and "one more won't hurt it." Evidence collected b y the authors in conducting a study of the predictive validity of a new type of reference check (forced choice) for clerical employees also suggests that the degree of leniency in the Mosel and Goheen results is not necessarily to be expected in industrial applications. Notes and comments frequently written in the margin of a new forced-choice reference form being tested, which included only pairs of positivesounding statements, suggested that a significant minority of reference givers would have checked negative statements if they had been given the opportunity to do so on the form. A small pilot study attempted to test this proposition. 8 In this study, seven personnel
8. Allan N. Nash, "A Methodological Suggestion for Constructing Forced Choice Scales," Experimental Journal o f the APA (1970).
45
ALLAN N. NASH AND STEPHEN J. CARROLL
46
managers were contacted and asked to provide the names of at least two former employees "who were borderline performers at best." The personnel managers furnished these names along with the names of the immediate supervisors of these employees. A reference form a n d l e t t e r soliciting cooperation was sent to each former supervisor of the inferior employee. F i f t e e n reference check forms were sent out and thirteen were returned, of which eleven had negative statements checked for the employee in question. This study indicates that supervisors, when asked to give a reference for a former relatively unsatisfactory employee, were willing to make negative comments. Results of an older study by Bingham in w h i c h o n e - h u n d r e d sales managers were surveyed also supports this contention. 9 He found that although 85 percent of them said they would give the former employee the benefit of the doubt in writing letters of recommendation, 39 percent also said they would point out the man's failings and weaknesses as well as his strong points.
Characteristics of the Reference Giver A few studies suggest that there are marked differences in the validity of references depending on who gives them and the relationship of the giver to the applicant. For example, the studies by Mosel and Goheen and Browning indicated that references from immediate supervisors are somewhat more valid than those from other sources, including various character sources not related to the job. Browning also found that the most recent immediate supervisor was better than an older one. Kornhauser found references given by former teachers and employers were better at predicting academic success than references given by a group of friends and professional 9. W. V. Bingham, "The Three Functions of the Interview in Employment," Management Review, V X (1926), p. 36.
acquaintances, which is similar to findings reported above by Mosel and Goheen and Browning. Yoder suggests that the reference is most useful if the job being sought is similar to jobs previously held. 10 Data collected by the authors in a study of a forced-choice reference check also support the theory that the nature of the reference giver may affect the validity of the references. We found the following personal characteristics to be important. First, leniency seems related to the sex of the reference giver. Women tended to be more lenient than men in rating both former employees and present employees. In this situation, the w o m e n were rating women. This finding is congruent with previous research, which indicates that individuals of a particular sex are more lenient in rating members of their own sex. 11 However, in this study the men rating w o m e n held higher and more responsible jobs than the w o m e n rating women. Second, the usefulness of the reference depends on how much the reference giver knows about, the applicant. We found that references given by a supervisor for a job applicant who had performed nonclerical work were not predictive of success in doing clerical work in another organization. This supports the suggestion of Yoder that references from former employers should be discounted unless the work done in the old job is similar to the work to be done on the new job. We also found that references received for former employees who had held their old jobs for less than two months were unpredictive of performance on the second job. This may have occurred not only because of sketchy information, but also because of a tendency for former supervisors to be disgusted with previous employees who quit shortly after being hired. 10. D. Yoder, Personnel Management and Industrial Relations (Sth ed.; Englewood Cliffs, N. J.: Prentice-Hall, Inc., 1962), p. 346. 11. Stephen J. Carroll and Allan N. Nash, "The Evaluation of Performance," unpublished manuscript, 1970.
BUSINESS HORIZONS
A Hard Look at the Reference Check
Third, references given b y a person in a foreign country were not predictive of performance on a j o b in this country. In addition, references given b y a supervisor of one race were n o t predictive of j o b performance as rated b y a supervisor on the new j o b of another race. This was especially true when the applicant was of the same race as the reference giver b u t of a different race from the supervisor evaluating his performance on the new job. This finding is congruent with that in other studies, which generally show that individuals have a tendency to rate members of their own race higher than they rate members of another race. In general, then, it does appear that the characteristics of the reference giver must be considered in deciding h o w reliable their references are. References are probably more accurate if received from immediate supervisors who supervised the applicant for a considerable length of time in a j o b similar to that for which the applicant is being considered. References received from supervisors of the same sex, race, or country of origin as the applicant are likely to be more favorable than performance ratings b y a supervisor of a different sex, race, or country of origin.
Method of Obtaining References References can be obtained from former employers b y letter, telephone, and the use of specially structured objective type questionnaires or field investigations which use interviews to collect i n f o r m a t i o n from the reference givers. It is not k n o w n what percentage of all employers use these alternative methods, b u t one survey indicated that about 50 percent of all firms require some t y p e of written recommendation. The majority of authors in the field suggest that the telephone check is better than a written reference considering cost, accuracy, and time. With the telephone check, one is reasonably certain that the reference
OCTOBER, 1970
giver is the person named as the reference and not some secretary or clerk. The reference can also be obtained quickly and inexpensively. In addition, some feel that the reference giver finds it more difficult to stretch the truth about a j o b applicant in oral conversation than he would in providing the reference in writing. Also, oral answers may contain hesitations and voice inflections that can be revealing as to what the reference giver really thinks about the person under consideration. The reference giver may feel more secure in making negative statements informally than he would in making them formally in a w r i t t e n reference check. Despite these apparent advantages of the telephone check no research study has demonstrated its superiority over the other methods of obtaining references. The field investigation, which involves interviewing reference givers in their homes or places of e m p l o y m e n t , has some of the obvious advantages of the telephone check. One can be certain the named reference is involved in the check, and the oral conversation may reveal information that might not be presented in written form. However, the field investigation is obviously considerably more expensive and time consuming than the telephone check. One study compared the field investigation with the written reference check. In this study, written reference questionnaires and field investigation r e p o r t s were obtained for 109 applicants for three professional jobs in the federal service. 12 Each field investigation consisted of from three to six interviews with individuals who knew the applicants. Only a moderate relationship was established b e t w e e n ratings made of the applicants on the basis of these two different types of reference checks. An investigation was also made of the success of
12. J. N. Mosel and H. W. Goheen, "The Employment Recommendation Questionnaire: Validity of Different Types of References," Personnel Psychology, XII (1959), pp. 469-77.
47
ALLAN N. NASH AND STEPHEN J. CARROLL
48
these methods in identifying applicants who were ineligible for e m p l o y m e n t in the federal service because of alcoholism, homosexuality, or g r o s s incompetence. Only the field investigation identified the seven individuals who were n o t acceptable for these reasons. The written reference form did not identify these individuals primarily because written references were not sent in by the persons who were sent questionnaires. Perhaps, however, the telephone check could collect the information on such clearly disqualified applicants under the standards used, since the problem of nonresponse is greatly reduced under this method. The telephone check would be far cheaper and less t i m e consuming than the field investigation. A research study should be conducted in this area, perhaps by the federal government which obviously spends a great deal of m o n e y on field investigations. The disadvantages of the written reference questionnaire have already been discussed; it is subject to the problems of poor response and to low validity because of the problem of leniency. Our research has shown the poor response problem can be reduced considerably by the use of individually typed and signed letters. We also have developed a new type of reference check questionnaire to eliminate or alleviate the problem of low validity due to the error of leniency. The new check incorporates the forced-choice format developed for the prediction of promotability in the armed forces in the mid-1940's, a 3 The forced-choice m e t h o d was introduced in order to eliminate or reduce the error of leniency in ratings of performance. The rater is presented with a group of behavioral statements. Such statements may be in groups of two, three, or four. Through a fairly 13. Allan N. Nash and Stephen J. Carroll, Jr., "Improving the Validity of a Forced Choice Reference Check with Rater and Job Characteristics Moderators," Proceedings of the 78th Annual Convention, American Psychological Association, 1970, pp. 577-78.
complex research procedure, it is determined how well each behavioral statement used in the form predicts success in a particular job and also how favorable each statement sounds to the raters using the form. The behavioral statements are grouped so that they are similar to other statements in their group with respect to "apparent" favorability, but differ from the other statements in their group with respect to ability to predict success in a particular job. The rater is asked to choose the statement(s) from a group of statements that most apply to the person being rated. Thus the rater is forced to choose from among alternative behavioral statements. We have developed forced-choice rating forms for both maintenance workers and for clerical workers, but only the form for the clerical workers was tested in order to establish its validity. The form consists of twenty-four pairs of behavioral statements descriptive of clerical workers. All the statements were constructed in a positive manner because of the apparent desire of reference givers to give positive references. The pairs of statements are constructed so that the two statements in each pair sound equally favorable, are equally characteristic of clerical workers in general, but differ in their ability to predict success in clerical work. The form was developed for clerical workers in an electronics firm in Baltimore and then was tested at the University of Maryland. The test involved the use of forced-choice reference checks completed by former employers of applicants for clerical jobs at the university. These reference checks were then put aside and not examined until the 122 clerical workers studied had been on the job for at least four months; they were then rated by the supervisors with respect to many aspects of their over-all work performance. It was found that when the reference giver on the previous job had adequate time to observe the worker and was of the same sex, race, and nationality as the supervisor on the
BUSINESS HORIZONS
A Hard L o o k at the Reference Check
new job, and where the old and new jobs were similar in content, the forced choice reference check was quite highly predictive of success (r = +.64 between reference check scores and performance ratings).
IMPROVING USEFULNESS REFERENCE CHECK
OF
THE
The research indicates that organizations can improve the usefulness of the reference check in selecting organizational personnel by following certain suggestions derived from this research.
Survey only previous supervisors of the job applicant. The research indicates that only references from former supervisors of the job applicant are predictive of success on a new job. Less valid references are provided by friends, neighbors, and other organizational personnel in the old organization such as coworkers and higher level organizational personnel who were not in immediate contact with the applicant.
Consider the characteristics o f the immediate supervisor giving the reference. References from supervisors who have not had adequate opportunity to observe the job performance of the applicant are less pre-
OCTOBER, 1970
dictive of job success than references from supervisors who have had an adequate observational opportunity. In addition, references from supervisors on previous jobs differing in sex, country of employment, or race from supervisors on the prospective job are less predictive of job success than when these supervisors are similar.
Use individual typed letters to request the reference. Our research indicated that a return of 85 percent can be obtained with individually typed letters. Choose the appropriate method. There are several alternative methods of obtaining references, including the written reference form, a forced-choice variation of this developed by the authors, the telephone check, and the field investigation. Research has established that the forced-choice reference check format can be predictive of job success. Research can be carried out by organizations themselves on the relative validity of the other reference check methods. The personnel department should find it easy to compare, in a systematic way, alternative reference check methods for the various types of employees the organization employs. Certainly such research would be worthwhile considering the cost, time, and present usefulness of reference check methods.
49