- Email: [email protected]
An experimental evaluation of driver license manuals and written tests
4cc~d. 4haL l Prer Vol. 14. N o 3. pp 187-192, 1982 Primed
:n
0001-1575,82.'030187-9~.~3 0Oi0 Pergamon Pre~,~, Ltd
Great Britain
AN EXPERIMENTAL EVALUATION OF DRIVER LICENSE MANUALS AND WRITTEN TESTSt A. JAMES MCKNIGHT National Public Services Research Institute. Alexandria. VA 2231a, U.S.A. and RICHARD EDWARDS Virginia Division of Motor Vehicles, Richmond, VA 23220, U.S.A.
(Receired 2 March 1981: in rerised forrn 20 June 1981) Abstract--Written manuals and tests dealing with safe driving practices were designed for licence applicants in three categories: New Drivers, Renewals and Older Drivers. Contents were based upon an analysis of the critical information requirements of each group. The manuals and tests were evaluated in an experiment involving over 30,00t? license applicants. Among New Drivers. the treatment group had significantly fewer accidents than a control group administered the regular driver's manual. Among Renewals, the treatment group had significantly fewer accidents with convictions than a control group not required to take a test. No consistent effects were found for Older Drivers. It was concluded that welI-designed manuals and tests are a cost-effective accident countermeasure.
Written driver license tests have evidenced very little validity in predicting accidents. Significant but extremely low correlations have been reported by Kaestner [1964], McRae [1968], Creech and Grandy [1974] and Dreyer [1976], all of whom concluded that written tests have too little validity to be utilized in screening drivers. What is most questionable about these studies is not their results but the assumption that written license tests are even expected to serve as part of a screening process. As defined by the American Association of Motor Vehicle Administrators, the purpose of a written test is "to force the applicant to prepare himself.., to meet the challenge of modern day driving" [AAMVA, 1967]. Certainly the fact that over 95% of drivers seeking a license ultimately obtain one [Dreyer, 1976] suggests that little screening takes place. This view is supported by Miller and Dimling [1969] who, after an exhaustive review of license tests, concluded that the only way they can contribute to safety is by improving the performance of all drivers rather than screening out poor ones. None of the studies cited offered evidence of the ability of written tests to force drivers to prepare to become better drivers. First, correlation is an inappropriate statistic for assessing the effectiveness of any process in producing change. The correlation between tests scores and accidents is mediated by too many factors, such as education, verbal ability and age, to assess causative effect. Evidence of this comes from a recent experiment in which use of a motorcycle skill and knowledge test was associated with a 16% reduction in accidents despite the fact that there was no correlation between accidents and test scores [Anderson, 1980; Jonah et al., 1980]. Second, none of the studies attempted to evaluate the resources available to drivers to prepare themselves for the written test. The content of most state license tests is drawn from the driver's manual. Nuckols[1972] reviewed the driver's manuals of all states and found them to be deficient in both relevance to driving safety and their ability to communicate. The National Public Services Research Institute, with the cooperation of the Virginia Division of Motor Vehicles, undertook an experimental evaluation of written tests and manuals in preparing license applicants to become better drivers.
D E V E L O P M E N T OF M A N U A L S AND TESTS
Most states use a single manual and test for all people who apply for a driver's license, regardless of age or experience. In the experiment described, three different manuals and tests were developed to accommodate the differing characteristics of the following three populations: New Drivers--Drivers not previously licensed in any state. Renewals--Drivers under the age of tThe work described in this report was carried out under contract to the National Highway Traffic Safety Administration. 187
188
A.J. McK,",IGHTand R. EDWARDS
60 seeking to renew licenses. Older Drivers--Drivers 60 yr of age or older seeking to renew licenses.
New drivers The content of the New Driver manual and test was based upon information underlying driving tasks related as "above average" in criticality in a previously developed Driver Task Analysis [McKnight and Adams, 1970]. Content areas included rules of the road. observing other drivers, communicating with other drivers, adjusting speed to conditions, maintaining adequate intervehicle separation, handling emergencies, alcohol and drugs, maintaining the vehicle in safe operating condition, and licensing procedures. A 5-6th grade reading level was maintained throughout, with illustrations used where necessary to communicate effectively. A pool of 140 multiple choice test items was developed by sampling representatively from the content of the manual. These items were divided into several 20-item alternative forms equated for content area and level of difficulty. The overwhelming majority of new drivers are teenagers who have recently completed driver education. In many schools, the state driver manual serves as the driver education text. To facilitate use of the New Driver manual in this context, a special supplement was prepared. It provided a series of questions, problems, and exercises calling for information from the manual. Renewals The contents of the Renewal manual and test were drawn from the content selected for New Drivers. However, to account for the greater experience of the renewal applicants, two modifications were made: Common Knowledge--Tests were administered to renewal applicants to find out how much of the content they already knew. Inclusion of information in the Renewal manual was then based upon a combination of criticality and current level of knowledge. As a result of the process, information already possessed by the large majority of renewal applicants was retained only if it was of extremely high criticality. Self Tests--Experienced drivers are much less likely to recognize the need for driving information than are new drivers. Therefore, the Renewal manual began with a self-test designed to convince applicants that they could profit from the content of the manual. Older drivers A number of driving problems accompany the aging process, including difficulty in maintaining the speed of traffic, difficulty in looking behind the vehicle, confusion resulting from traffic conditions, visual and hearing impairment, fatigue, health problems, drugqnduced problems and the introduction of new traffic signs, signals and regulations. An Older Driver manual was designed to provide the information needed by older drivers to enable them to recognize problems and correct or compensate for them. The manual also described alternate forms of transportation and sources of assistance in transportation for those who are unable to drive safely. The style of the manual was nonthreatening and problems were introduced as being general correlates of age rather than necessary consequences of the aging process. A large type size was employed for both the manual and test. Their limited content permitted only one form of the test to be developed. EVALUATION METHOD The three manuals and tests were evaluated in a experiment carried out by the Virginia Division of Motor Vehicles (DMV). The experiment involved randon assignment of applicants to treatment and control groups. Treatment groups received the experimental manuals and tests while control groups were processed in the regular manner. Accident records for all groups were followed up at 6-month intervals for a period of 2 yr following licensing.
Processing of new drivers The New Driver manual and test were administered to 6000 students in 28 Northern Virginia
An experimental evaluation of driver license manuals and written tests
189
high schools. Administration through high schools allowed better control of randomization than was possible for "walk-in" license applicants. Driver education classes in each of the schools were randomly divided into treatment and control groups. Students in the treatment group were issued the New Driver manual and supplement, while those in the control group were issued the regular state manual. Both manuals were used in class as prescribed by the driver education curriculum. At the completion of driver education, the students seeking a license were administered the appropriate written test (New Driver test or regular state test) by a DMV examiner at the school. All students were issued color-coded driver education certificates to allow DMV personnel to identify their experimental assignment and record it in their driver records. Students who failed to obtain their license within 6 months were not included in the follow-up. Processing of renewals and older drivers The State of Virginia does not require a written test for license renewal. However, legislation was enacted to allow the DMV to impose such a requirement for research purposes. Since drivers had to appear at a branch office to take an eye test, the additional requirement to take a written test did not impose a great hardship. Renewals and Older Drivers are administered manuals and tests by the same procedure. Drivers in the Northern Virginia area whose licenses were about to expire were identified and randomly assigned to treatment and control groups. A total of 24,776 Renewals and 5975 Older Drivers were randomly divided between each of the two experimental groups. The group assignment was immediately entered into the driver record. The control group received only the regular renewal application. Those in the treatment group received, along with the application, a notice that they would be required to take a written test and were provided a copy of the Renewal or Older Driver manual with which to prepare. Renewal applications for all four experimental groups were color-coded to allow DMV personnel to identify group assignment. If drivers failed to bring the renewal application with them, their group assignment was ascertained through terminal access to the driver's record. Licenses of applicants in the treatment group were renewed regardless of their score on the test. Since they had no way of knowing this in advance, the procedure presumably did not affect their preparation. A cciden t folio w- up Drivers in Virginia are required to report all accidents involving damage in excess of $200 (at that time) to the DMV. The date of each accident is registered in the driver's record. The accidents of all drivers were totaled at 6-month intervals for 2 yr following the date of license issue or renewal. Separate totals were obtained for accidents in which the driver was convicted of a traffic violation. The processing of data was handled by the Virginia DMV.
RESULTS
The percentages of drivers sustaining accidents and accidents with convictions were tabulated for the two experimental groups within each of the three target groups. Significance was tested through a 1-tailed test of the difference between proportions. New drivers Results obtained from New Driver target group appear in Table I. Differences in favor of the treatment group appear at all intervals on both criteria. However, significance is attained only after 12 months for total accidents and 18 months for accidents with convictions. It should be noted that significance tests are not exact since randomization took place at the level of classes while analysis is at the level of students. Unfortunately, available driver records did not permit an analysis of accidents by high school class. Slightly less than half of the original 3000 students assigned to the treatment and control groups applied for and received licenses within 6 months after completing driver education. Approximately 100 more treatments obtained licenses than did controls. However, the treatments had fewer total accidents despite their large numbers. This is certainly promising,
A. J. McKNIGHT and R EDWARDS
190
Table I. Per cent of New Drivers having accidents and accidents with convictions. (Treatments: N = 1~.86. Controls: N = l~.~) .... Accidents Interval
Treat.
~ccidents ~ith Convictions Cont.
P MS
Treat.
Cont.
.135%
P
3 months
.675%
.02%
.292%-
NS
6 months
2.52 %
3.57%
NS
.703%
.656%
NS
12 months
7.55 %
9.54%
.05
1.55 :
1.82 %
NS
]8 months
]2.45%
16.02%
.Of
2.56 %
3.86 %
.05
24 months
17.36%
21.41%
.01
3.57 ~
5.24 %
.01
particularly in view of the facts that the effects of the manual and test were superimposed upon that of a 30 hr classroom driver education program. Approximately half of the drivers were not licensed and were therefore not a part of the final sample. This limits the generality of the findings since the drivers who were licensed are not representative of the entire New Driver target group. Unfortunately there were no data available to permit differences between licensed and non-licensed students to be ascertained. However, unless results obtained from those who were licensed after the study ended were such as to reverse earlier trends, the New Driver manual and test would appear to have had a favorable effect.
Renewals The percentages of Renewals involved in accidents and accidents with convictions appear in Table 2. Differences between treatments and controls in total accidents are negligible an(l nonsignificant. However, substantial and significant differences appear in accidents with convictions after 6 months. Why the results differ for the two different categories of accidents cannot be determined from the data. It is possible that the manual and test were more effective in keeping drivers from being responsible for accidents than it was in keeping them from simply being in accident~. Or, it may be that accidents with convictions, having been investigated by police, are a more reliable criterion than accidents in general, most of which are self-reported. If accidents with convictions are subtracted from total accidents, the resultant accidents without convictions favor the controls. However, the differences are well within chance variation. Therefore, a reduction in accidents with convictions can be treated as a benefit by itself. Under the procedures employed by the Virginia DMV, drivers who refused to take the renewal test were not required to do so. Unfortunately, this fact was publicized in a local newspaper with the result that approx. 25% of the treatment group did not participate in testing. The nonparticipants were retained in the same sample to avoid introducing any bias. Since the subsequent accident records of nonparticipants were worse than those of participants, total participation might have yielded an even greater overall effect.
Table 2. Per cent of Renewals having accidents and accidents with convictions. (Treatments: N = 12,335. Controls: N = I2,441) Accidents
Accidents with Convictions
Intervat
Treat.
Cont.
P
3 months
].34 %
1.42%
NS
Treat. .057%
.195%
6 months
2.57 %
2.56%
NS
.207%
.385%
.01
12 months
5.23 %
5.12%
NS
.5ti%
.739%
.O1
18 months
7.38 %
7,19%
NS
.76 %
24 months
9.56%
9.42%
NS
1.09 %
Cont.
I.II
P NS
%
.01
1.37 %
.Of
An experimental evaluation of driver license manuals and ~ritten tests
191
Older drivers Results compiled for Older Drivers appear in Table 3. Differences in favor of the treatment group appear for both criteria at all levels. The only cumulative differences that are statistically significant are those appearing at 6 and 12 months for accidents without conL'ictions. Some 17% of the treatment group refused to take the test. These nonparticipants, like the Renewa!s, were retained in the sample to avoid introducing a bias. While overall differences are always in favor of the treatment group, the control group actually had significantly fewer accidents during the second year, largely offsetting the advantage enjoyed by the treatment group during the first year. It is possible that the Older Driver manual and test had an effect that was beneficial during the first year and then boomeranged. It is hard to imagine just what such an effect would be. A more parsimonious hypothesis is that the manual and test really had no effect over the 2 yr period and that what appeared to be significant effects were really only due to chance. In any case, the results are not favorable to the manual and test. While the results do not prove the manual and test for Older Drivers is effective, they do not prove it is ineffective either. First, the sample size was not large enough to establish the significance of very small effects. Secondly, the content of the manual and test was limited to the special problems of the target group. Had this content been added to that prepared for the Renewals and had a sample of the same size been employed, an equally favorable result might have been found. DISCUSSION The results establish that a combination of driver license tests and manuals can reduce certain types of accidents for certain categories of drivers. The magnitude of any reduction is obviously very small. However, the low cost of manuals and tests can make quite small differences cost-beneficial. A cost-benefit evaluation of the New Driver program is certainly favorable. Sparing one of every 40 new drivers in accident by means of a manual that costs no more to produce than the regular state manual cannot help but be advantageous. Among the Renewals, benefit is somewhat marginal. With the total costs of accidents estimated to be in the neighborhood of $5000, a program that prevented one accident for every 350 people would be cost-beneficial as long as it did not exceed $14 per applicant. The costs to the Virginia DMV were clearly well below this. Only if the cost of the applicant's time came into consideration would the cost benefit be questionable. Few people at the present time question the value of a written test for New Drivers. However, requiring written tests of renewals has been widely challenged, particularly when the drivers involved have good records. Most of the states that require written examinations of licensed drivers administer them at every renewal, typical every 4 yr. The results obtained from this present study, while supporting the testing of licensed drivers generally, cannot be taken as validating periodic reexamination. The Renewals in the experiment had not been given a written test since their original Virginia license has been issued. Whether they would have shown the same gain had they been tested every 4 yr in the interim is a matter of conjecture. In short, while the study results support renewal knowledge testing, they do not establish a need to do it every time the license is renewed. The results of this study are in some conflict with those obtained by Carpenter [1978] and Table 3. Per cent of Older Drivers having accidents and accidents with convictions.(Treatments: N = 2987. Controls: N = 2988) Accidents
Accidents with Convictions
Interval
Treat.
Cont.
P
Treat.
3 months
].07 %
1.48%
NS
0
6 months
1.91%
2.27%
NS
.106%
12 months
3.6] %
4.31%
NS
18 months
5.52 %
5.62%
NS
24 months
7.56%
7.20%
NS
Cont. .168%
P NS
.334%
.05
.368%
.803%
.Of
.74 %
].Of %
NS
1.04 %
1.17 %
NS
192
A.J. McKsIoH'rand R. EDWARDS
Stoke [1978], neither of whom found a program of information and testing to improve upon subsequent accident records of drivers. However, both of those studies focused more upon testing for knowledge than providing it in the first place. In Stoke's investigation, renewal applicants received only the state driver's manual they had used to obtain their original licenses. While Carpenter refers to providing "supplemental safe driving information," the information itself is not described and appears to have been provided after the tests were scored. The emphasis in this effort was upon providing information. The function of tests was simply to create an incentive for drivers to acquire the information presented. The information that serves as a basis for manuals and tests was drawn from an analysis of critical driving tasks and a rigorous determination of the information needs of various groups of drivers. The results appear to indicate that small but marginally cost-effective gains can be achieved through a comprehensive program of information-giving and testing. CONCLUSIONS The results of the experiment described lead to the following conclusions: (1) A combination of tests and manuals is capable of reducing certain types of accidents for certain categories of drivers. (2) Because of the relatively low cost of manuals and tests, even a small reduction in accidents can be cost-effective. (3) Because very small reductions in accidents can be cost-effective, evaluation of license manuals and tests must employ large enough samples (e.g. over 10,000 per group) to accurately detect small reductions. REFERENCES American Association of Motor Vehicle Administrators (AAMVA). Testing Drivers: A Manual for Driver-License Examiners and Administrators, Washington, D.C., 1967. Anderson J. W., The effect of new motorcycle licensing programs and skills training on the driver records of original applicants. Proc. Int. Motorcycle Safety Conf., Vol. 1, May 1980. Carpenter D. W., The Effects of Written Licensing Tests Stressing Knowledge of Safe Driving Principlesfor Intermediate Record Renewal Applicants. Sacramento: California Department of Motor Vehicles, 1978. Carpenter D. W., The Effects of Adding Safe Driving Content to Written Law Test Administered to OriginalDrivers License Applicants. Sacramento: California Department of Motor Vehicles, 1978. Creech F. and Grandy J., An Analysis of the Relationships Between the Renewal Rules Tests and the Accident and Violation Histories. Princeton, New Jersey, Educational Testing Service, 1974. Dreyer D. R., An Evaluation of California's Driver Licensing Examination. Sacramento: California Department of Motor Vehicles, 1976. Jonah B. A., et al. Predictive validity of the motorcycle operator license skill test. Proc. Int. Motorcycle Safety Conf., Vol. 1, May 1980. Kaestner N., A Study of Licensed Drivers in Oregon: Part H-Analysis of Tra95c Involvement Records. Salem: Oregon Department of Motor Vehicles, 1964. Miller L. and Dimling J. A., Jr. Driver Licensing and Performance, VoL 1, Research Review and Recommendations. Lexington, KY: Spindletop Research Incorporated, 1969. McKnight A. J. and Adams B. G., Driver Education Task Analysis, Vol. I: Task Descriptions, HumRROTechnical Report 70-103, U.S. Departmentof TransportationTechnical Report HS 800 367, DOTContract No. FHH 11-7336,November 1970. McRae D., The Relation of Licensing Test Scores to Subsequent Driver Performance. Chapel Hill: The L. L. Thurstone Psychometric Laboratory, University of North Carolina, 1968. Nuckols H. C., An Analysis of the Contents of State Driver Manuals. Department of Civil Engineering, Clemson University, South Carolina, May 1971. Stoke C. B., The Short-Term Effectiveness of Written Driver Knowledge Tests. Virginia Highway and Transportation Research Council. 1978.
:n
0001-1575,82.'030187-9~.~3 0Oi0 Pergamon Pre~,~, Ltd
Great Britain
AN EXPERIMENTAL EVALUATION OF DRIVER LICENSE MANUALS AND WRITTEN TESTSt A. JAMES MCKNIGHT National Public Services Research Institute. Alexandria. VA 2231a, U.S.A. and RICHARD EDWARDS Virginia Division of Motor Vehicles, Richmond, VA 23220, U.S.A.
(Receired 2 March 1981: in rerised forrn 20 June 1981) Abstract--Written manuals and tests dealing with safe driving practices were designed for licence applicants in three categories: New Drivers, Renewals and Older Drivers. Contents were based upon an analysis of the critical information requirements of each group. The manuals and tests were evaluated in an experiment involving over 30,00t? license applicants. Among New Drivers. the treatment group had significantly fewer accidents than a control group administered the regular driver's manual. Among Renewals, the treatment group had significantly fewer accidents with convictions than a control group not required to take a test. No consistent effects were found for Older Drivers. It was concluded that welI-designed manuals and tests are a cost-effective accident countermeasure.
Written driver license tests have evidenced very little validity in predicting accidents. Significant but extremely low correlations have been reported by Kaestner [1964], McRae [1968], Creech and Grandy [1974] and Dreyer [1976], all of whom concluded that written tests have too little validity to be utilized in screening drivers. What is most questionable about these studies is not their results but the assumption that written license tests are even expected to serve as part of a screening process. As defined by the American Association of Motor Vehicle Administrators, the purpose of a written test is "to force the applicant to prepare himself.., to meet the challenge of modern day driving" [AAMVA, 1967]. Certainly the fact that over 95% of drivers seeking a license ultimately obtain one [Dreyer, 1976] suggests that little screening takes place. This view is supported by Miller and Dimling [1969] who, after an exhaustive review of license tests, concluded that the only way they can contribute to safety is by improving the performance of all drivers rather than screening out poor ones. None of the studies cited offered evidence of the ability of written tests to force drivers to prepare to become better drivers. First, correlation is an inappropriate statistic for assessing the effectiveness of any process in producing change. The correlation between tests scores and accidents is mediated by too many factors, such as education, verbal ability and age, to assess causative effect. Evidence of this comes from a recent experiment in which use of a motorcycle skill and knowledge test was associated with a 16% reduction in accidents despite the fact that there was no correlation between accidents and test scores [Anderson, 1980; Jonah et al., 1980]. Second, none of the studies attempted to evaluate the resources available to drivers to prepare themselves for the written test. The content of most state license tests is drawn from the driver's manual. Nuckols[1972] reviewed the driver's manuals of all states and found them to be deficient in both relevance to driving safety and their ability to communicate. The National Public Services Research Institute, with the cooperation of the Virginia Division of Motor Vehicles, undertook an experimental evaluation of written tests and manuals in preparing license applicants to become better drivers.
D E V E L O P M E N T OF M A N U A L S AND TESTS
Most states use a single manual and test for all people who apply for a driver's license, regardless of age or experience. In the experiment described, three different manuals and tests were developed to accommodate the differing characteristics of the following three populations: New Drivers--Drivers not previously licensed in any state. Renewals--Drivers under the age of tThe work described in this report was carried out under contract to the National Highway Traffic Safety Administration. 187
188
A.J. McK,",IGHTand R. EDWARDS
60 seeking to renew licenses. Older Drivers--Drivers 60 yr of age or older seeking to renew licenses.
New drivers The content of the New Driver manual and test was based upon information underlying driving tasks related as "above average" in criticality in a previously developed Driver Task Analysis [McKnight and Adams, 1970]. Content areas included rules of the road. observing other drivers, communicating with other drivers, adjusting speed to conditions, maintaining adequate intervehicle separation, handling emergencies, alcohol and drugs, maintaining the vehicle in safe operating condition, and licensing procedures. A 5-6th grade reading level was maintained throughout, with illustrations used where necessary to communicate effectively. A pool of 140 multiple choice test items was developed by sampling representatively from the content of the manual. These items were divided into several 20-item alternative forms equated for content area and level of difficulty. The overwhelming majority of new drivers are teenagers who have recently completed driver education. In many schools, the state driver manual serves as the driver education text. To facilitate use of the New Driver manual in this context, a special supplement was prepared. It provided a series of questions, problems, and exercises calling for information from the manual. Renewals The contents of the Renewal manual and test were drawn from the content selected for New Drivers. However, to account for the greater experience of the renewal applicants, two modifications were made: Common Knowledge--Tests were administered to renewal applicants to find out how much of the content they already knew. Inclusion of information in the Renewal manual was then based upon a combination of criticality and current level of knowledge. As a result of the process, information already possessed by the large majority of renewal applicants was retained only if it was of extremely high criticality. Self Tests--Experienced drivers are much less likely to recognize the need for driving information than are new drivers. Therefore, the Renewal manual began with a self-test designed to convince applicants that they could profit from the content of the manual. Older drivers A number of driving problems accompany the aging process, including difficulty in maintaining the speed of traffic, difficulty in looking behind the vehicle, confusion resulting from traffic conditions, visual and hearing impairment, fatigue, health problems, drugqnduced problems and the introduction of new traffic signs, signals and regulations. An Older Driver manual was designed to provide the information needed by older drivers to enable them to recognize problems and correct or compensate for them. The manual also described alternate forms of transportation and sources of assistance in transportation for those who are unable to drive safely. The style of the manual was nonthreatening and problems were introduced as being general correlates of age rather than necessary consequences of the aging process. A large type size was employed for both the manual and test. Their limited content permitted only one form of the test to be developed. EVALUATION METHOD The three manuals and tests were evaluated in a experiment carried out by the Virginia Division of Motor Vehicles (DMV). The experiment involved randon assignment of applicants to treatment and control groups. Treatment groups received the experimental manuals and tests while control groups were processed in the regular manner. Accident records for all groups were followed up at 6-month intervals for a period of 2 yr following licensing.
Processing of new drivers The New Driver manual and test were administered to 6000 students in 28 Northern Virginia
An experimental evaluation of driver license manuals and written tests
189
high schools. Administration through high schools allowed better control of randomization than was possible for "walk-in" license applicants. Driver education classes in each of the schools were randomly divided into treatment and control groups. Students in the treatment group were issued the New Driver manual and supplement, while those in the control group were issued the regular state manual. Both manuals were used in class as prescribed by the driver education curriculum. At the completion of driver education, the students seeking a license were administered the appropriate written test (New Driver test or regular state test) by a DMV examiner at the school. All students were issued color-coded driver education certificates to allow DMV personnel to identify their experimental assignment and record it in their driver records. Students who failed to obtain their license within 6 months were not included in the follow-up. Processing of renewals and older drivers The State of Virginia does not require a written test for license renewal. However, legislation was enacted to allow the DMV to impose such a requirement for research purposes. Since drivers had to appear at a branch office to take an eye test, the additional requirement to take a written test did not impose a great hardship. Renewals and Older Drivers are administered manuals and tests by the same procedure. Drivers in the Northern Virginia area whose licenses were about to expire were identified and randomly assigned to treatment and control groups. A total of 24,776 Renewals and 5975 Older Drivers were randomly divided between each of the two experimental groups. The group assignment was immediately entered into the driver record. The control group received only the regular renewal application. Those in the treatment group received, along with the application, a notice that they would be required to take a written test and were provided a copy of the Renewal or Older Driver manual with which to prepare. Renewal applications for all four experimental groups were color-coded to allow DMV personnel to identify group assignment. If drivers failed to bring the renewal application with them, their group assignment was ascertained through terminal access to the driver's record. Licenses of applicants in the treatment group were renewed regardless of their score on the test. Since they had no way of knowing this in advance, the procedure presumably did not affect their preparation. A cciden t folio w- up Drivers in Virginia are required to report all accidents involving damage in excess of $200 (at that time) to the DMV. The date of each accident is registered in the driver's record. The accidents of all drivers were totaled at 6-month intervals for 2 yr following the date of license issue or renewal. Separate totals were obtained for accidents in which the driver was convicted of a traffic violation. The processing of data was handled by the Virginia DMV.
RESULTS
The percentages of drivers sustaining accidents and accidents with convictions were tabulated for the two experimental groups within each of the three target groups. Significance was tested through a 1-tailed test of the difference between proportions. New drivers Results obtained from New Driver target group appear in Table I. Differences in favor of the treatment group appear at all intervals on both criteria. However, significance is attained only after 12 months for total accidents and 18 months for accidents with convictions. It should be noted that significance tests are not exact since randomization took place at the level of classes while analysis is at the level of students. Unfortunately, available driver records did not permit an analysis of accidents by high school class. Slightly less than half of the original 3000 students assigned to the treatment and control groups applied for and received licenses within 6 months after completing driver education. Approximately 100 more treatments obtained licenses than did controls. However, the treatments had fewer total accidents despite their large numbers. This is certainly promising,
A. J. McKNIGHT and R EDWARDS
190
Table I. Per cent of New Drivers having accidents and accidents with convictions. (Treatments: N = 1~.86. Controls: N = l~.~) .... Accidents Interval
Treat.
~ccidents ~ith Convictions Cont.
P MS
Treat.
Cont.
.135%
P
3 months
.675%
.02%
.292%-
NS
6 months
2.52 %
3.57%
NS
.703%
.656%
NS
12 months
7.55 %
9.54%
.05
1.55 :
1.82 %
NS
]8 months
]2.45%
16.02%
.Of
2.56 %
3.86 %
.05
24 months
17.36%
21.41%
.01
3.57 ~
5.24 %
.01
particularly in view of the facts that the effects of the manual and test were superimposed upon that of a 30 hr classroom driver education program. Approximately half of the drivers were not licensed and were therefore not a part of the final sample. This limits the generality of the findings since the drivers who were licensed are not representative of the entire New Driver target group. Unfortunately there were no data available to permit differences between licensed and non-licensed students to be ascertained. However, unless results obtained from those who were licensed after the study ended were such as to reverse earlier trends, the New Driver manual and test would appear to have had a favorable effect.
Renewals The percentages of Renewals involved in accidents and accidents with convictions appear in Table 2. Differences between treatments and controls in total accidents are negligible an(l nonsignificant. However, substantial and significant differences appear in accidents with convictions after 6 months. Why the results differ for the two different categories of accidents cannot be determined from the data. It is possible that the manual and test were more effective in keeping drivers from being responsible for accidents than it was in keeping them from simply being in accident~. Or, it may be that accidents with convictions, having been investigated by police, are a more reliable criterion than accidents in general, most of which are self-reported. If accidents with convictions are subtracted from total accidents, the resultant accidents without convictions favor the controls. However, the differences are well within chance variation. Therefore, a reduction in accidents with convictions can be treated as a benefit by itself. Under the procedures employed by the Virginia DMV, drivers who refused to take the renewal test were not required to do so. Unfortunately, this fact was publicized in a local newspaper with the result that approx. 25% of the treatment group did not participate in testing. The nonparticipants were retained in the same sample to avoid introducing any bias. Since the subsequent accident records of nonparticipants were worse than those of participants, total participation might have yielded an even greater overall effect.
Table 2. Per cent of Renewals having accidents and accidents with convictions. (Treatments: N = 12,335. Controls: N = I2,441) Accidents
Accidents with Convictions
Intervat
Treat.
Cont.
P
3 months
].34 %
1.42%
NS
Treat. .057%
.195%
6 months
2.57 %
2.56%
NS
.207%
.385%
.01
12 months
5.23 %
5.12%
NS
.5ti%
.739%
.O1
18 months
7.38 %
7,19%
NS
.76 %
24 months
9.56%
9.42%
NS
1.09 %
Cont.
I.II
P NS
%
.01
1.37 %
.Of
An experimental evaluation of driver license manuals and ~ritten tests
191
Older drivers Results compiled for Older Drivers appear in Table 3. Differences in favor of the treatment group appear for both criteria at all levels. The only cumulative differences that are statistically significant are those appearing at 6 and 12 months for accidents without conL'ictions. Some 17% of the treatment group refused to take the test. These nonparticipants, like the Renewa!s, were retained in the sample to avoid introducing a bias. While overall differences are always in favor of the treatment group, the control group actually had significantly fewer accidents during the second year, largely offsetting the advantage enjoyed by the treatment group during the first year. It is possible that the Older Driver manual and test had an effect that was beneficial during the first year and then boomeranged. It is hard to imagine just what such an effect would be. A more parsimonious hypothesis is that the manual and test really had no effect over the 2 yr period and that what appeared to be significant effects were really only due to chance. In any case, the results are not favorable to the manual and test. While the results do not prove the manual and test for Older Drivers is effective, they do not prove it is ineffective either. First, the sample size was not large enough to establish the significance of very small effects. Secondly, the content of the manual and test was limited to the special problems of the target group. Had this content been added to that prepared for the Renewals and had a sample of the same size been employed, an equally favorable result might have been found. DISCUSSION The results establish that a combination of driver license tests and manuals can reduce certain types of accidents for certain categories of drivers. The magnitude of any reduction is obviously very small. However, the low cost of manuals and tests can make quite small differences cost-beneficial. A cost-benefit evaluation of the New Driver program is certainly favorable. Sparing one of every 40 new drivers in accident by means of a manual that costs no more to produce than the regular state manual cannot help but be advantageous. Among the Renewals, benefit is somewhat marginal. With the total costs of accidents estimated to be in the neighborhood of $5000, a program that prevented one accident for every 350 people would be cost-beneficial as long as it did not exceed $14 per applicant. The costs to the Virginia DMV were clearly well below this. Only if the cost of the applicant's time came into consideration would the cost benefit be questionable. Few people at the present time question the value of a written test for New Drivers. However, requiring written tests of renewals has been widely challenged, particularly when the drivers involved have good records. Most of the states that require written examinations of licensed drivers administer them at every renewal, typical every 4 yr. The results obtained from this present study, while supporting the testing of licensed drivers generally, cannot be taken as validating periodic reexamination. The Renewals in the experiment had not been given a written test since their original Virginia license has been issued. Whether they would have shown the same gain had they been tested every 4 yr in the interim is a matter of conjecture. In short, while the study results support renewal knowledge testing, they do not establish a need to do it every time the license is renewed. The results of this study are in some conflict with those obtained by Carpenter [1978] and Table 3. Per cent of Older Drivers having accidents and accidents with convictions.(Treatments: N = 2987. Controls: N = 2988) Accidents
Accidents with Convictions
Interval
Treat.
Cont.
P
Treat.
3 months
].07 %
1.48%
NS
0
6 months
1.91%
2.27%
NS
.106%
12 months
3.6] %
4.31%
NS
18 months
5.52 %
5.62%
NS
24 months
7.56%
7.20%
NS
Cont. .168%
P NS
.334%
.05
.368%
.803%
.Of
.74 %
].Of %
NS
1.04 %
1.17 %
NS
192
A.J. McKsIoH'rand R. EDWARDS
Stoke [1978], neither of whom found a program of information and testing to improve upon subsequent accident records of drivers. However, both of those studies focused more upon testing for knowledge than providing it in the first place. In Stoke's investigation, renewal applicants received only the state driver's manual they had used to obtain their original licenses. While Carpenter refers to providing "supplemental safe driving information," the information itself is not described and appears to have been provided after the tests were scored. The emphasis in this effort was upon providing information. The function of tests was simply to create an incentive for drivers to acquire the information presented. The information that serves as a basis for manuals and tests was drawn from an analysis of critical driving tasks and a rigorous determination of the information needs of various groups of drivers. The results appear to indicate that small but marginally cost-effective gains can be achieved through a comprehensive program of information-giving and testing. CONCLUSIONS The results of the experiment described lead to the following conclusions: (1) A combination of tests and manuals is capable of reducing certain types of accidents for certain categories of drivers. (2) Because of the relatively low cost of manuals and tests, even a small reduction in accidents can be cost-effective. (3) Because very small reductions in accidents can be cost-effective, evaluation of license manuals and tests must employ large enough samples (e.g. over 10,000 per group) to accurately detect small reductions. REFERENCES American Association of Motor Vehicle Administrators (AAMVA). Testing Drivers: A Manual for Driver-License Examiners and Administrators, Washington, D.C., 1967. Anderson J. W., The effect of new motorcycle licensing programs and skills training on the driver records of original applicants. Proc. Int. Motorcycle Safety Conf., Vol. 1, May 1980. Carpenter D. W., The Effects of Written Licensing Tests Stressing Knowledge of Safe Driving Principlesfor Intermediate Record Renewal Applicants. Sacramento: California Department of Motor Vehicles, 1978. Carpenter D. W., The Effects of Adding Safe Driving Content to Written Law Test Administered to OriginalDrivers License Applicants. Sacramento: California Department of Motor Vehicles, 1978. Creech F. and Grandy J., An Analysis of the Relationships Between the Renewal Rules Tests and the Accident and Violation Histories. Princeton, New Jersey, Educational Testing Service, 1974. Dreyer D. R., An Evaluation of California's Driver Licensing Examination. Sacramento: California Department of Motor Vehicles, 1976. Jonah B. A., et al. Predictive validity of the motorcycle operator license skill test. Proc. Int. Motorcycle Safety Conf., Vol. 1, May 1980. Kaestner N., A Study of Licensed Drivers in Oregon: Part H-Analysis of Tra95c Involvement Records. Salem: Oregon Department of Motor Vehicles, 1964. Miller L. and Dimling J. A., Jr. Driver Licensing and Performance, VoL 1, Research Review and Recommendations. Lexington, KY: Spindletop Research Incorporated, 1969. McKnight A. J. and Adams B. G., Driver Education Task Analysis, Vol. I: Task Descriptions, HumRROTechnical Report 70-103, U.S. Departmentof TransportationTechnical Report HS 800 367, DOTContract No. FHH 11-7336,November 1970. McRae D., The Relation of Licensing Test Scores to Subsequent Driver Performance. Chapel Hill: The L. L. Thurstone Psychometric Laboratory, University of North Carolina, 1968. Nuckols H. C., An Analysis of the Contents of State Driver Manuals. Department of Civil Engineering, Clemson University, South Carolina, May 1971. Stoke C. B., The Short-Term Effectiveness of Written Driver Knowledge Tests. Virginia Highway and Transportation Research Council. 1978.