- Email: [email protected]
Caution: A comment on Alena Erke's red light for red-light cameras? A meta-analysis of the effects of red-light cameras on crashes
Accident Analysis and Prevention 41 (2009) 895–896
Contents lists available at ScienceDirect
Accident Analysis and Prevention journal homepage: www.elsevier.com/locate/aap
Brief communication
Caution: A comment on Alena Erke’s red light for red-light cameras? A meta-analysis of the effects of red-light cameras on crashes Adrian K. Lund a , Sergey Y. Kyrychenko a , Richard A. Retting b,∗ a b
Insurance Institute for Highway Safety, United States Sam Schwartz Engineering, Washington DC, 1100 N. Glebe Road, Suite 1010, Arlington, VA 22201, United States
a r t i c l e
i n f o
Article history: Received 23 January 2009 Received in revised form 13 March 2009 Accepted 22 March 2009 Keywords: Red light cameras Meta-analysis
a b s t r a c t In Red Light for Red-Light Cameras?, Alena Erke concludes that “. . . on the whole, redlight cameras do not seem to be a successful safety measure.” Although Erke’s survey of the literature appears to have been comprehensive, her review of the studies was not critical. She appears to have accepted the authors’ descriptions of their analyses rather than providing readers with her own considered opinion of how valid those analyses were and what their true implications might be. For the meta-analysis leading to her final conclusion, Erke combines data from two questionable studies with three other “well-controlled” studies. Non-peer-reviewed studies received substantial statistical weight in the meta-analysis. These problems likely produce misleading results. If the highway safety field is to succeed in identifying for policymakers those strategies that are most likely to reduce the human tragedy of motor vehicle crashes, we need first to focus on conducting valid research and analysis. Adding precision to the estimated benefits of those strategies through meta-analysis is important, but secondary, and cannot replace the function of a systematic and critical review. © 2009 Elsevier Ltd. All rights reserved.
In Red Light for Red-Light Cameras? in the current issue of Accident Analysis and Prevention, Erke, 2009 that “. . . on the whole, RLCs [red-light cameras] do not seem to be a successful safety measure.” To the average highway safety professional reading this respected, peer-reviewed journal, this conclusion must be taken seriously. Not only is Erke’s conclusion based on a comprehensive survey of available research reports on the topic of red-light cameras and their effect on motor vehicle crashes (81 effect estimates from 21 research reports), but her use of a powerful, statistical procedure like meta-analysis lends her study a particular impression of scientific authority. Unfortunately, that impression is misleading because Erke’s application of meta-analysis omits a key component: a critical review that evaluates the variety of studies, their methods, and whether those methods address the stated research questions in a valid way. Without this critical review, Erke’s metaanalysis says nothing convincing about the effects of red-light cameras on the tragedy of motor vehicle crashes. Rather, it is reduced to an academic exercise of powerful mathematical techniques whose publication in AAP may be more informative about how researchers sometimes are so caught up in the application of these techniques that they ignore key conceptual and methodological issues.
∗ Corresponding author. E-mail address: [email protected] (R.A. Retting). 0001-4575/$ – see front matter © 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.aap.2009.03.018
For example, Erke’s description of the application of metaanalytic mathematics is thorough. There is discussion of how effects are calculated for each study and how confidence intervals are derived for estimates of effects combined from different studies. However, there is no comparable discussion of whether the effects estimated by the different studies are comparable enough in nature to justify averaging them, which is essentially what meta-analysis does. Some estimates of the effect of red-light cameras came from comparisons with intersections without cameras in the same community; others came from comparisons with intersections outside that community. Do these comparisons estimate the same effect? That seems unlikely, given that potential spillover of camera effects to intersections without cameras is controlled in one study and not the other. Thus, the measured effect differs in systematic, nonrandom ways between studies. Yet Erke averages across estimates of different effects, a clear contradiction of the intent of meta-analysis. Erke may argue that her meta-analysis was structured specifically to check for such potential problems. Indeed, her conclusion cited above is based mainly on the five studies she lists as controlling for not only the spillover effect but also the potential for regression to the mean. However, if these studies are examined critically, it becomes apparent that two should not have been listed as such. In one (Burkey and Obeng, 2004), the control intersections were in the same city as the camera intersections; spillover of camera effects was possible and likely. In addition, although Burkey and Obeng claimed to have controlled for regression to the mean, in fact their statistical models did nothing to control for the fact
896
A.K. Lund et al. / Accident Analysis and Prevention 41 (2009) 895–896
that cameras were likely installed at high crash locations. Further details on the methodological flaws of this research can be found in Kyrychenko and Retting (2004). Another “well-controlled” study according to Erke, Garber et al. (2007), also used signalized intersections without red-light cameras in the same communities as controls. Again, there was no design control over the choice of intersections receiving cameras, so the design of the study is subject to both spillover and regression to mean effects. Among different analyses conducted by the authors, Erke focuses on estimates from an empirical Bayes approach, which can adjust statistically for regression to the mean if appropriate data are available and the statistical model is developed appropriately. Unfortunately, neither was the case. Traffic volume data, which were key to the model, fluctuated hugely and inexplicably between the periods before and after camera installation. The model itself resulted in highly illogical effects of intersection characteristics on crashes, and spillover remained a problem. A more detailed critique of this study is available in Persaud et al. (2008). For the meta-analysis leading to her final conclusion, Erke combines the data from these two questionable studies with three other “well-controlled” studies. Two of these are published in peer-reviewed journals, providing some confidence that their study designs and methods were reasonable. In one (Retting and Kyrychenko, 2002), all signalized intersections in the study community provided the before–after estimate of crashes, not just those equipped with red-light cameras. The control intersections were non-signalized intersections in the community, so spillover was not an issue. Nearby comparison communities without any camera enforcement were studied to control further for possible weather or other statewide factors that might have affected crashes before or after camera installation. This did not provide perfect control of confounding effects – such perfect control rarely is achieved in motor vehicle safety studies because random assignment of treatment usually is impossible – but the design controlled for many potentially confounding factors. In the second published study (Shin and Washington, 2007), a before–after matched control group design was used, coupled with appropriate application of empirical Bayes analysis to statistically control for factors not controlled in the study design, although the potential for spillover effects still was there. The third of these studies has not benefited from peer review for journal publication (Council et al., 2005) and was in need of more thorough review by Erke. For the purposes of this comment, we note only that this study was conducted for the U.S. Federal Highway Administration and that the empirical Bayes analysis applied in this research most likely benefited from the inclusion among the authors of a leading expert in that statistical technique (Persaud, second author). These latter three studies, which appear to have been the strongest methodologically, all reported that red-light cameras yield beneficial reductions in the most serious motor vehicle crashes at signalized intersections. It is the two weaker studies, which Erke erroneously classifies as well-controlled, that produce the inconsistent results that lead to her conclusion that red-light cameras have not been a successful safety measure. The ability of these two studies to overwhelm the positive effects of the three other, stronger studies reflects the fact that Erke’s statistical weighting of the studies in her meta-analysis is based on mathematical considerations of statistical precision – how many crashes are available for analysis? – rather than on methodological considerations of effect validity – to what extent have confounding factors been controlled? Among the five studies Erke labels “strong,” the two weaker studies received more statistical weight than the three stronger studies.
1. What can be learned from this comment? The principal problem with the Erke report is not her use of meta-analysis, which can be a very powerful procedure, but rather her omission of a careful review of the research she summarized. In other words, statistical sophistication trumps the issue of methodological validity. Elvik (2005) noted that the mathematical procedures involved in meta-analysis are “an extension of traditional narrative literature reviews” (pg 230), and meta-analysis does not replace or substitute for that review. More specifically Elvik said, “Meta-analysis is based on a systematic review of relevant studies,” where a “systematic review is a comprehensive and critical review of studies dealing with a certain topic” (pg 230). Although Erke’s survey of the literature appears to have been comprehensive, her review of the studies was not critical. She appears to have accepted the authors’ descriptions of their analyses rather than providing readers with her own considered opinion of how valid those analyses were and what their true implications might be. That might be somewhat understandable, though still troubling, if all the studies had been in peer-reviewed publications. However, three of Erke’s “well-controlled” studies are simply reports to the agencies contracting for the research, and the authors’ responses to any reviews are unknown. Making matters worse, these non-peer-reviewed studies received substantial statistical weight in the meta-analysis. If Erke’s target had been a more traditional literature review with a premium on analytical thought and synthesis of disparate findings rather than mathematical techniques, two of the studies would receive little weight in reaching conclusions about the effectiveness of red-light cameras. If the highway safety field is to succeed in identifying for policymakers those strategies that are most likely to reduce the human tragedy of motor vehicle crashes, we need first to focus on conducting valid research and analysis. Adding precision to the estimated benefits of those strategies through meta-analysis is important, but secondary, and cannot replace the function of a systematic and critical review. References Burkey, M., Obeng, K., 2004. A Detailed Investigation of Crash Risk Reduction Resulting from Red Light Cameras in Small Urban Areas. North Carolina Agricultural and Technical State University, Greensboro, NC. Council, F.M., Persaud, B., Eccles, K., Lyon, C., Griffith, M.S., 2005. Safety Evaluation of Red-Light Cameras. Federal Highway Administration, Washington, DC, Report no. FHWA-HRT-05-048. Elvik, R., 2005. Introductory guide to systematic reviews and meta-analysis. Transportation Research Record 1908, 230–235. Erke, A., 2009. Red light for red-light cameras? A meta-analysis of the effects of red-light cameras on crashes. Accident Analysis and Prevention, doi:10.1016/j.aap.2008.08.011. Garber, N.J., Miller, J.S., Abel, R.E., Eslambolchi, S., Korukonda, S.K., 2007. The Impact of Red Light Cameras (Photo-Red Enforcement) on Crashes in Virginia. Virginia Transportation Research Council, Charlottesville, VA, Report no. VTRC 07-R2. Kyrychenko, S.Y., Retting, R.A., 2004. Review of “A detailed investigation of crash risk reduction resulting from red light cameras in small urban areas” by M. Burkey and K. Obeng. Insurance Institute for Highway Safety, Arlington, VA. Available: http://www.iihs.org/research/topics/pdf/r1034.pdf. Persaud, B.N., Retting, R.A., Lyon, C., McCartt, A.T., 2008. Review of “The impact of red light cameras (photo-red enforcement) on crashes in Virginia” by Nicholas J. Garber, John S. Miller, R. Elizabeth Abel, Saeed Eslambolchi, and Santhosh K. Korukonda. Insurance Institute for Highway Safety, Arlington, VA. Available: http://www.iihs.org/research/topics/pdf/r1100.pdf. Retting, R.A., Kyrychenko, S.Y., 2002. Reductions in injury crashes associated with red light camera enforcement in Oxnard California. American Journal of Public Health 92, 1822–1825. Shin, K., Washington, S., 2007. The impact of red light cameras on safety in Arizona. Accident Analysis and Prevention 39, 1212–1221.
Contents lists available at ScienceDirect
Accident Analysis and Prevention journal homepage: www.elsevier.com/locate/aap
Brief communication
Caution: A comment on Alena Erke’s red light for red-light cameras? A meta-analysis of the effects of red-light cameras on crashes Adrian K. Lund a , Sergey Y. Kyrychenko a , Richard A. Retting b,∗ a b
Insurance Institute for Highway Safety, United States Sam Schwartz Engineering, Washington DC, 1100 N. Glebe Road, Suite 1010, Arlington, VA 22201, United States
a r t i c l e
i n f o
Article history: Received 23 January 2009 Received in revised form 13 March 2009 Accepted 22 March 2009 Keywords: Red light cameras Meta-analysis
a b s t r a c t In Red Light for Red-Light Cameras?, Alena Erke concludes that “. . . on the whole, redlight cameras do not seem to be a successful safety measure.” Although Erke’s survey of the literature appears to have been comprehensive, her review of the studies was not critical. She appears to have accepted the authors’ descriptions of their analyses rather than providing readers with her own considered opinion of how valid those analyses were and what their true implications might be. For the meta-analysis leading to her final conclusion, Erke combines data from two questionable studies with three other “well-controlled” studies. Non-peer-reviewed studies received substantial statistical weight in the meta-analysis. These problems likely produce misleading results. If the highway safety field is to succeed in identifying for policymakers those strategies that are most likely to reduce the human tragedy of motor vehicle crashes, we need first to focus on conducting valid research and analysis. Adding precision to the estimated benefits of those strategies through meta-analysis is important, but secondary, and cannot replace the function of a systematic and critical review. © 2009 Elsevier Ltd. All rights reserved.
In Red Light for Red-Light Cameras? in the current issue of Accident Analysis and Prevention, Erke, 2009 that “. . . on the whole, RLCs [red-light cameras] do not seem to be a successful safety measure.” To the average highway safety professional reading this respected, peer-reviewed journal, this conclusion must be taken seriously. Not only is Erke’s conclusion based on a comprehensive survey of available research reports on the topic of red-light cameras and their effect on motor vehicle crashes (81 effect estimates from 21 research reports), but her use of a powerful, statistical procedure like meta-analysis lends her study a particular impression of scientific authority. Unfortunately, that impression is misleading because Erke’s application of meta-analysis omits a key component: a critical review that evaluates the variety of studies, their methods, and whether those methods address the stated research questions in a valid way. Without this critical review, Erke’s metaanalysis says nothing convincing about the effects of red-light cameras on the tragedy of motor vehicle crashes. Rather, it is reduced to an academic exercise of powerful mathematical techniques whose publication in AAP may be more informative about how researchers sometimes are so caught up in the application of these techniques that they ignore key conceptual and methodological issues.
∗ Corresponding author. E-mail address: [email protected] (R.A. Retting). 0001-4575/$ – see front matter © 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.aap.2009.03.018
For example, Erke’s description of the application of metaanalytic mathematics is thorough. There is discussion of how effects are calculated for each study and how confidence intervals are derived for estimates of effects combined from different studies. However, there is no comparable discussion of whether the effects estimated by the different studies are comparable enough in nature to justify averaging them, which is essentially what meta-analysis does. Some estimates of the effect of red-light cameras came from comparisons with intersections without cameras in the same community; others came from comparisons with intersections outside that community. Do these comparisons estimate the same effect? That seems unlikely, given that potential spillover of camera effects to intersections without cameras is controlled in one study and not the other. Thus, the measured effect differs in systematic, nonrandom ways between studies. Yet Erke averages across estimates of different effects, a clear contradiction of the intent of meta-analysis. Erke may argue that her meta-analysis was structured specifically to check for such potential problems. Indeed, her conclusion cited above is based mainly on the five studies she lists as controlling for not only the spillover effect but also the potential for regression to the mean. However, if these studies are examined critically, it becomes apparent that two should not have been listed as such. In one (Burkey and Obeng, 2004), the control intersections were in the same city as the camera intersections; spillover of camera effects was possible and likely. In addition, although Burkey and Obeng claimed to have controlled for regression to the mean, in fact their statistical models did nothing to control for the fact
896
A.K. Lund et al. / Accident Analysis and Prevention 41 (2009) 895–896
that cameras were likely installed at high crash locations. Further details on the methodological flaws of this research can be found in Kyrychenko and Retting (2004). Another “well-controlled” study according to Erke, Garber et al. (2007), also used signalized intersections without red-light cameras in the same communities as controls. Again, there was no design control over the choice of intersections receiving cameras, so the design of the study is subject to both spillover and regression to mean effects. Among different analyses conducted by the authors, Erke focuses on estimates from an empirical Bayes approach, which can adjust statistically for regression to the mean if appropriate data are available and the statistical model is developed appropriately. Unfortunately, neither was the case. Traffic volume data, which were key to the model, fluctuated hugely and inexplicably between the periods before and after camera installation. The model itself resulted in highly illogical effects of intersection characteristics on crashes, and spillover remained a problem. A more detailed critique of this study is available in Persaud et al. (2008). For the meta-analysis leading to her final conclusion, Erke combines the data from these two questionable studies with three other “well-controlled” studies. Two of these are published in peer-reviewed journals, providing some confidence that their study designs and methods were reasonable. In one (Retting and Kyrychenko, 2002), all signalized intersections in the study community provided the before–after estimate of crashes, not just those equipped with red-light cameras. The control intersections were non-signalized intersections in the community, so spillover was not an issue. Nearby comparison communities without any camera enforcement were studied to control further for possible weather or other statewide factors that might have affected crashes before or after camera installation. This did not provide perfect control of confounding effects – such perfect control rarely is achieved in motor vehicle safety studies because random assignment of treatment usually is impossible – but the design controlled for many potentially confounding factors. In the second published study (Shin and Washington, 2007), a before–after matched control group design was used, coupled with appropriate application of empirical Bayes analysis to statistically control for factors not controlled in the study design, although the potential for spillover effects still was there. The third of these studies has not benefited from peer review for journal publication (Council et al., 2005) and was in need of more thorough review by Erke. For the purposes of this comment, we note only that this study was conducted for the U.S. Federal Highway Administration and that the empirical Bayes analysis applied in this research most likely benefited from the inclusion among the authors of a leading expert in that statistical technique (Persaud, second author). These latter three studies, which appear to have been the strongest methodologically, all reported that red-light cameras yield beneficial reductions in the most serious motor vehicle crashes at signalized intersections. It is the two weaker studies, which Erke erroneously classifies as well-controlled, that produce the inconsistent results that lead to her conclusion that red-light cameras have not been a successful safety measure. The ability of these two studies to overwhelm the positive effects of the three other, stronger studies reflects the fact that Erke’s statistical weighting of the studies in her meta-analysis is based on mathematical considerations of statistical precision – how many crashes are available for analysis? – rather than on methodological considerations of effect validity – to what extent have confounding factors been controlled? Among the five studies Erke labels “strong,” the two weaker studies received more statistical weight than the three stronger studies.
1. What can be learned from this comment? The principal problem with the Erke report is not her use of meta-analysis, which can be a very powerful procedure, but rather her omission of a careful review of the research she summarized. In other words, statistical sophistication trumps the issue of methodological validity. Elvik (2005) noted that the mathematical procedures involved in meta-analysis are “an extension of traditional narrative literature reviews” (pg 230), and meta-analysis does not replace or substitute for that review. More specifically Elvik said, “Meta-analysis is based on a systematic review of relevant studies,” where a “systematic review is a comprehensive and critical review of studies dealing with a certain topic” (pg 230). Although Erke’s survey of the literature appears to have been comprehensive, her review of the studies was not critical. She appears to have accepted the authors’ descriptions of their analyses rather than providing readers with her own considered opinion of how valid those analyses were and what their true implications might be. That might be somewhat understandable, though still troubling, if all the studies had been in peer-reviewed publications. However, three of Erke’s “well-controlled” studies are simply reports to the agencies contracting for the research, and the authors’ responses to any reviews are unknown. Making matters worse, these non-peer-reviewed studies received substantial statistical weight in the meta-analysis. If Erke’s target had been a more traditional literature review with a premium on analytical thought and synthesis of disparate findings rather than mathematical techniques, two of the studies would receive little weight in reaching conclusions about the effectiveness of red-light cameras. If the highway safety field is to succeed in identifying for policymakers those strategies that are most likely to reduce the human tragedy of motor vehicle crashes, we need first to focus on conducting valid research and analysis. Adding precision to the estimated benefits of those strategies through meta-analysis is important, but secondary, and cannot replace the function of a systematic and critical review. References Burkey, M., Obeng, K., 2004. A Detailed Investigation of Crash Risk Reduction Resulting from Red Light Cameras in Small Urban Areas. North Carolina Agricultural and Technical State University, Greensboro, NC. Council, F.M., Persaud, B., Eccles, K., Lyon, C., Griffith, M.S., 2005. Safety Evaluation of Red-Light Cameras. Federal Highway Administration, Washington, DC, Report no. FHWA-HRT-05-048. Elvik, R., 2005. Introductory guide to systematic reviews and meta-analysis. Transportation Research Record 1908, 230–235. Erke, A., 2009. Red light for red-light cameras? A meta-analysis of the effects of red-light cameras on crashes. Accident Analysis and Prevention, doi:10.1016/j.aap.2008.08.011. Garber, N.J., Miller, J.S., Abel, R.E., Eslambolchi, S., Korukonda, S.K., 2007. The Impact of Red Light Cameras (Photo-Red Enforcement) on Crashes in Virginia. Virginia Transportation Research Council, Charlottesville, VA, Report no. VTRC 07-R2. Kyrychenko, S.Y., Retting, R.A., 2004. Review of “A detailed investigation of crash risk reduction resulting from red light cameras in small urban areas” by M. Burkey and K. Obeng. Insurance Institute for Highway Safety, Arlington, VA. Available: http://www.iihs.org/research/topics/pdf/r1034.pdf. Persaud, B.N., Retting, R.A., Lyon, C., McCartt, A.T., 2008. Review of “The impact of red light cameras (photo-red enforcement) on crashes in Virginia” by Nicholas J. Garber, John S. Miller, R. Elizabeth Abel, Saeed Eslambolchi, and Santhosh K. Korukonda. Insurance Institute for Highway Safety, Arlington, VA. Available: http://www.iihs.org/research/topics/pdf/r1100.pdf. Retting, R.A., Kyrychenko, S.Y., 2002. Reductions in injury crashes associated with red light camera enforcement in Oxnard California. American Journal of Public Health 92, 1822–1825. Shin, K., Washington, S., 2007. The impact of red light cameras on safety in Arizona. Accident Analysis and Prevention 39, 1212–1221.