- Email: [email protected]
OxRec model for assessing risk of recidivism: ethics
Correspondence
Mental health policies and suicide rate I read with interest the Article by Nav Kapur and colleagues (June, 2016).1 The authors acknowledge the limitations of a purely ecological study without consideration of individual equivalents. Additionally, certain other methodological limitations seem to restrict the reliability of the study results. Although the authors analysed data from England only, heterogeneity in provision of mental health services is documented even within England.2 In the absence of any quality or process indicators, the comprehensiveness and quality of implementation of the mental health service changes cannot be commented on and hence cannot be considered a homogenous entity for the purpose of comparison. Differences pertaining to ethnic variation determining the access and pathway to specialist mental health services, as well as admission rates in England, is a well-reported confounding variable that might influence the study results. 3 This confounder, in addition to other confounding variables such as economic status, employment, substance use, and other socio-economic variables, as well as variables related to service delivery such as funding and manpower, merits attention before the study results are interpreted. Also, it is worth noting that some service provisions or policies might benefit a subset of the target population better than others.4 Though some of the policy measures were introduced as packages, 1 some were mutually or partially exclusive and sometimes introduced sequentially, thereby producing unknown interactions that might be difficult to quantify. Some of the less potent policy measures might appear more powerful and vice versa, simply because of differences in the sequence of implementation or by simultaneous implementation. 808
Use of regression analysis or modelling might be helpful in discerning the effect of different policy measures on suicide rate. Addition of components of qualitative analysis and use of multi-level analysis, which take into consideration individual equivalents wherever possible, might add to the robustness of study results. The study would then provide a template for other nations to replicate similar successful measures in their local context, as suicide rate is often considered a surrogate marker of the effectiveness of mental health services.5 I declare no competing interests.
Sundar Gnanavel [email protected] Northumberland, Tyne and Wear NHS Foundation Trust, Morpeth, Northumberland, UK 1
2
3
4
5
Kapur N, Ibrahim S, While D, et al. Mental health service changes, organisational factors, and patient suicide in England in 1997–2012: a before-and-after study. Lancet Psychiatry 2016; 3: 526–34. Social Care Local Government and Care Partnership Directorate. Closing the gap: priorities for essential change in mental health. London: Department of Health, 2014. Bhui K, Stansfeld S, Hull S, et al. Ethnic variations in pathways to and use of specialist mental health services in the UK: a systematic review. Br J Psychiatry 2003; 182: 105–16. Burgess P, Pirkis J, Jolley D, Whiteford H, Saxena S. Do nations’ mental health policies, programs and legislation influence their suicide rates? An ecological study of 100 countries. Aust N Z J Psychiatry 2004; 38: 933–39. Shah A, Bhat R. Are elderly suicide rates improved by increased provision of mental health service resources? A cross-national study. Int Psychogeriatr 2008; 20: 1230–37.
OxRec model for assessing risk of recidivism: ethics Concerned by the increasing use of actuarial risk assessments in our criminal justice system and worldwide, we read with interest the Article ( June, 2016) 1 by Seena Fazel and colleagues presenting the derivation and validation of one such model, OxRec, in Sweden. Typically, these tools use a variety
of factors, such as criminal, medical, and demographic information, to calculate an individual’s risk of either recidivism generally or committing specific, usually violent, crimes. 2 Assessments are then used to set sentences, determine conditions and time of parole, and target post-release interventions, among other applications depending on local laws.2,3 Furthermore, the Sentencing Reform and Corrections Act of 2015, which is under review by the US Congress, would enshrine these assessments in the federal penal system. This expansion continues despite serious ethical and legal challenges, conflicting evidence on the predictive superiority of actuarial assessments to clinical assessments, and negligible evidence on whether their application translates into reduced recidivism.2,3 Because broad ethical analyses that critique these models’ applications are common in published literature2,3 and lay press,4,5 we will focus on several distinctive aspects of OxRec. First, we appreciate the authors’ transparency. Many models, some currently in use, do not disclose their derivation or validation procedures, the weighting associated with risk factors, or even which factors are used to evaluate risk.4 Withholding such information hinders external enquiry and individuals’ ability to contest their risk assessments. Second, we are troubled by the inclusion of disposable income, as well as factors that track socioeconomic status and race, such as education, employment, and neighbourhood deprivation, as components of the tool. Experimenting with the OxRec calculator, we found that the smallest allowable shift in any one of these variables—for example, from medium to low income—can alter a person’s risk assessment from low to medium or medium to high. Although most US models do not explicitly include class and
www.thelancet.com/psychiatry Vol 3 September 2016
Correspondence
race, 5 some do include associated characteristics.2,3 The incorporation of these factors is especially concerning if the assessment influences the provision of advantages such as early release to low-risk individuals or disadvantages such as greater supervision to high-risk individuals. These applications run counter to the principles of justice and fairness. We expect our legal system to treat people equally, especially when it comes to features outside one’s control.3 Moreover, certain tools now used in sentencing in the USA have been found to falsely identify more black people as high risk and fewer as low risk than white people;4,5 this bias could further institutionalise racism in the US criminal justice system, compounding the status quo of mass incarceration. Consequently, we disagree with the authors that their model could appropriately be used “to assist in decisions about the timing of parole and conditions associated with it”1 in Sweden or any other country, though particularly in the USA, given the socioeconomic factors the model includes. Third, we are apprehensive about unintended consequences of releasing OxRec online as a simple and free tool. Since the model is validated only in Sweden, use in other countries would likely increase the already substantial error rate: based on the threshold of 20% risk over 2 years, 33% of Swedish people who violently reoffend would not be identified, and 63% of those identified as risky would not violently reoffend.1 Accordingly, any application in other countries, and certain applications in Sweden, could produce serious harms, including unequal treatment of incarcerated people on the basis of class, extended sentences for the poor, and exacerbation of pre-existing societal inequities. Therefore, we believe the risks of releasing the tool without substantial guidance on appropriate use outweigh the benefits.
DWB, SND, CPR, and DSH are supported by the Intramural Research Program of the National Institutes of Health, where DWB, SND, and DSH are fellows and CPR is a research associate in the Clinical Center Department of Bioethics. The views expressed are the authors’ own. They do not represent the position or policy of the US National Institutes of Health, Public Health Service, or Department of Health and Human Services. We declare no other competing interests.
*Derek W Braverman, Samuel N Doernberg, Carlisle P Runge, Dana S Howard [email protected] Department of Bioethics, National Institutes of Health, Bethesda, MD 20892, USA 1
2
3
4
5
Fazel S, Chang Z, Fanshawe T, et al. Prediction of violent reoffending on release from prison: derivation and external validation of a scalable tool. Lancet Psychiatry 2016; 3: 535–43. McGuire J. Minimising harm in violence risk assessment: practical solutions to ethical problems? Health Risk Soc 2004; 6: 327–45. Starr SB. Evidence-based sentencing and the scientific rationalization of discrimination. Stan L Rev 2014; 66: 803–72. Angwin J, Larson J, Mattu S, Kirchner L. Machine bias. New York: ProPublica, 2016. http://www.propublica.org/article/machinebias-risk-assessments-in-criminal-sentencing (accessed June 23, 2016). Barry-Jester AM, Casselman B, Goldstein D. Should prison sentences be based on crimes that haven’t been committed yet? New York: FiveThirtyEight, 2015. http://www. fivethirtyeight.com/features/prison-reformrisk-assessment/ (accessed June 23, 2016).
Authors’ reply In our paper,1 we used transparent and robust methods to derive and externally validate a short, scalable tool (OxRec) for the prediction of violent reoffending in released prisoners. We made it clear that it should be used as an adjunct to professional judgement because individual factors and circumstances might need to be considered. However, OxRec provides a framework with which to anchor such judgements in evidence. Additionally, OxRec could be used as a basis to prioritise non-harmful interventions towards modifiable risk factors such as alcohol and drug use disorders, and other psychiatric disorders. Thus, OxRec has some key advantages over many existing instruments—a transparent methodology, prespecified protocol, large representative sample to develop
www.thelancet.com/psychiatry Vol 3 September 2016
the tool, and reporting of a wide range of performance measures for its external validation. We have also made available an online calculator that could be used in Sweden, and for research in other countries. We agree with Derek Braverman and colleagues that validation in other countries is required. They suggest that the socioeconomic variables in our tool “track socioeconomic status and race”. If track means associated with, then this is the case for all the variables we have used, including criminal history that has the strongest links with reoffending risk. In fact, the socioeconomic variables are relatively weak risk factors, and as we provide an actuarial (or probability) score, the effect of changing individual risk factors can be seen. Although it is correct to say that a change in income category could lead to a change in risk category, the effect of income on risk is small, so this will not typically be the case, and the impact of changing an individual from low to medium to high income can be seen. We disagree, however, with Braverman and colleagues’ comment that inclusion of socioeconomic variables is discriminatory. As others have pointed out, this would entirely depend on the baseline criminal justice context—in other words, all things being equal, what effect does using a potential risk calculator have on the system as it is currently practised?2 We suggest two issues need to be considered. First, whether current approaches are more or less transparent, consistent, and accurate than evidence-based risk assessment. In fact, many reviews have shown that unstructured clinical approaches do worse than structured violence risk assessments.3 Second, do decisions based purely on other apparently non-discriminatory variables reduce or exacerbate social disparities? As we have stated above, if socioeconomic variables are thought to be discriminatory because they are associated with
For more on OxRec see http://oxrisk.com/oxrec
809
Mental health policies and suicide rate I read with interest the Article by Nav Kapur and colleagues (June, 2016).1 The authors acknowledge the limitations of a purely ecological study without consideration of individual equivalents. Additionally, certain other methodological limitations seem to restrict the reliability of the study results. Although the authors analysed data from England only, heterogeneity in provision of mental health services is documented even within England.2 In the absence of any quality or process indicators, the comprehensiveness and quality of implementation of the mental health service changes cannot be commented on and hence cannot be considered a homogenous entity for the purpose of comparison. Differences pertaining to ethnic variation determining the access and pathway to specialist mental health services, as well as admission rates in England, is a well-reported confounding variable that might influence the study results. 3 This confounder, in addition to other confounding variables such as economic status, employment, substance use, and other socio-economic variables, as well as variables related to service delivery such as funding and manpower, merits attention before the study results are interpreted. Also, it is worth noting that some service provisions or policies might benefit a subset of the target population better than others.4 Though some of the policy measures were introduced as packages, 1 some were mutually or partially exclusive and sometimes introduced sequentially, thereby producing unknown interactions that might be difficult to quantify. Some of the less potent policy measures might appear more powerful and vice versa, simply because of differences in the sequence of implementation or by simultaneous implementation. 808
Use of regression analysis or modelling might be helpful in discerning the effect of different policy measures on suicide rate. Addition of components of qualitative analysis and use of multi-level analysis, which take into consideration individual equivalents wherever possible, might add to the robustness of study results. The study would then provide a template for other nations to replicate similar successful measures in their local context, as suicide rate is often considered a surrogate marker of the effectiveness of mental health services.5 I declare no competing interests.
Sundar Gnanavel [email protected] Northumberland, Tyne and Wear NHS Foundation Trust, Morpeth, Northumberland, UK 1
2
3
4
5
Kapur N, Ibrahim S, While D, et al. Mental health service changes, organisational factors, and patient suicide in England in 1997–2012: a before-and-after study. Lancet Psychiatry 2016; 3: 526–34. Social Care Local Government and Care Partnership Directorate. Closing the gap: priorities for essential change in mental health. London: Department of Health, 2014. Bhui K, Stansfeld S, Hull S, et al. Ethnic variations in pathways to and use of specialist mental health services in the UK: a systematic review. Br J Psychiatry 2003; 182: 105–16. Burgess P, Pirkis J, Jolley D, Whiteford H, Saxena S. Do nations’ mental health policies, programs and legislation influence their suicide rates? An ecological study of 100 countries. Aust N Z J Psychiatry 2004; 38: 933–39. Shah A, Bhat R. Are elderly suicide rates improved by increased provision of mental health service resources? A cross-national study. Int Psychogeriatr 2008; 20: 1230–37.
OxRec model for assessing risk of recidivism: ethics Concerned by the increasing use of actuarial risk assessments in our criminal justice system and worldwide, we read with interest the Article ( June, 2016) 1 by Seena Fazel and colleagues presenting the derivation and validation of one such model, OxRec, in Sweden. Typically, these tools use a variety
of factors, such as criminal, medical, and demographic information, to calculate an individual’s risk of either recidivism generally or committing specific, usually violent, crimes. 2 Assessments are then used to set sentences, determine conditions and time of parole, and target post-release interventions, among other applications depending on local laws.2,3 Furthermore, the Sentencing Reform and Corrections Act of 2015, which is under review by the US Congress, would enshrine these assessments in the federal penal system. This expansion continues despite serious ethical and legal challenges, conflicting evidence on the predictive superiority of actuarial assessments to clinical assessments, and negligible evidence on whether their application translates into reduced recidivism.2,3 Because broad ethical analyses that critique these models’ applications are common in published literature2,3 and lay press,4,5 we will focus on several distinctive aspects of OxRec. First, we appreciate the authors’ transparency. Many models, some currently in use, do not disclose their derivation or validation procedures, the weighting associated with risk factors, or even which factors are used to evaluate risk.4 Withholding such information hinders external enquiry and individuals’ ability to contest their risk assessments. Second, we are troubled by the inclusion of disposable income, as well as factors that track socioeconomic status and race, such as education, employment, and neighbourhood deprivation, as components of the tool. Experimenting with the OxRec calculator, we found that the smallest allowable shift in any one of these variables—for example, from medium to low income—can alter a person’s risk assessment from low to medium or medium to high. Although most US models do not explicitly include class and
www.thelancet.com/psychiatry Vol 3 September 2016
Correspondence
race, 5 some do include associated characteristics.2,3 The incorporation of these factors is especially concerning if the assessment influences the provision of advantages such as early release to low-risk individuals or disadvantages such as greater supervision to high-risk individuals. These applications run counter to the principles of justice and fairness. We expect our legal system to treat people equally, especially when it comes to features outside one’s control.3 Moreover, certain tools now used in sentencing in the USA have been found to falsely identify more black people as high risk and fewer as low risk than white people;4,5 this bias could further institutionalise racism in the US criminal justice system, compounding the status quo of mass incarceration. Consequently, we disagree with the authors that their model could appropriately be used “to assist in decisions about the timing of parole and conditions associated with it”1 in Sweden or any other country, though particularly in the USA, given the socioeconomic factors the model includes. Third, we are apprehensive about unintended consequences of releasing OxRec online as a simple and free tool. Since the model is validated only in Sweden, use in other countries would likely increase the already substantial error rate: based on the threshold of 20% risk over 2 years, 33% of Swedish people who violently reoffend would not be identified, and 63% of those identified as risky would not violently reoffend.1 Accordingly, any application in other countries, and certain applications in Sweden, could produce serious harms, including unequal treatment of incarcerated people on the basis of class, extended sentences for the poor, and exacerbation of pre-existing societal inequities. Therefore, we believe the risks of releasing the tool without substantial guidance on appropriate use outweigh the benefits.
DWB, SND, CPR, and DSH are supported by the Intramural Research Program of the National Institutes of Health, where DWB, SND, and DSH are fellows and CPR is a research associate in the Clinical Center Department of Bioethics. The views expressed are the authors’ own. They do not represent the position or policy of the US National Institutes of Health, Public Health Service, or Department of Health and Human Services. We declare no other competing interests.
*Derek W Braverman, Samuel N Doernberg, Carlisle P Runge, Dana S Howard [email protected] Department of Bioethics, National Institutes of Health, Bethesda, MD 20892, USA 1
2
3
4
5
Fazel S, Chang Z, Fanshawe T, et al. Prediction of violent reoffending on release from prison: derivation and external validation of a scalable tool. Lancet Psychiatry 2016; 3: 535–43. McGuire J. Minimising harm in violence risk assessment: practical solutions to ethical problems? Health Risk Soc 2004; 6: 327–45. Starr SB. Evidence-based sentencing and the scientific rationalization of discrimination. Stan L Rev 2014; 66: 803–72. Angwin J, Larson J, Mattu S, Kirchner L. Machine bias. New York: ProPublica, 2016. http://www.propublica.org/article/machinebias-risk-assessments-in-criminal-sentencing (accessed June 23, 2016). Barry-Jester AM, Casselman B, Goldstein D. Should prison sentences be based on crimes that haven’t been committed yet? New York: FiveThirtyEight, 2015. http://www. fivethirtyeight.com/features/prison-reformrisk-assessment/ (accessed June 23, 2016).
Authors’ reply In our paper,1 we used transparent and robust methods to derive and externally validate a short, scalable tool (OxRec) for the prediction of violent reoffending in released prisoners. We made it clear that it should be used as an adjunct to professional judgement because individual factors and circumstances might need to be considered. However, OxRec provides a framework with which to anchor such judgements in evidence. Additionally, OxRec could be used as a basis to prioritise non-harmful interventions towards modifiable risk factors such as alcohol and drug use disorders, and other psychiatric disorders. Thus, OxRec has some key advantages over many existing instruments—a transparent methodology, prespecified protocol, large representative sample to develop
www.thelancet.com/psychiatry Vol 3 September 2016
the tool, and reporting of a wide range of performance measures for its external validation. We have also made available an online calculator that could be used in Sweden, and for research in other countries. We agree with Derek Braverman and colleagues that validation in other countries is required. They suggest that the socioeconomic variables in our tool “track socioeconomic status and race”. If track means associated with, then this is the case for all the variables we have used, including criminal history that has the strongest links with reoffending risk. In fact, the socioeconomic variables are relatively weak risk factors, and as we provide an actuarial (or probability) score, the effect of changing individual risk factors can be seen. Although it is correct to say that a change in income category could lead to a change in risk category, the effect of income on risk is small, so this will not typically be the case, and the impact of changing an individual from low to medium to high income can be seen. We disagree, however, with Braverman and colleagues’ comment that inclusion of socioeconomic variables is discriminatory. As others have pointed out, this would entirely depend on the baseline criminal justice context—in other words, all things being equal, what effect does using a potential risk calculator have on the system as it is currently practised?2 We suggest two issues need to be considered. First, whether current approaches are more or less transparent, consistent, and accurate than evidence-based risk assessment. In fact, many reviews have shown that unstructured clinical approaches do worse than structured violence risk assessments.3 Second, do decisions based purely on other apparently non-discriminatory variables reduce or exacerbate social disparities? As we have stated above, if socioeconomic variables are thought to be discriminatory because they are associated with
For more on OxRec see http://oxrisk.com/oxrec
809