- Email: [email protected]
Training child welfare citizen review panel members: A promising approach?
Children and Youth Services Review 86 (2018) 94–97
Contents lists available at ScienceDirect
Children and Youth Services Review journal homepage: www.elsevier.com/locate/childyouth
Training child welfare citizen review panel members: A promising approach?
T
⁎
J. Jay Miller , LaToya Burns Vaughn Training Resource Center, University of Kentucky, United States
A R T I C L E I N F O
A B S T R A C T
Keywords: Citizen review panels Training Child welfare
Citizen Review Panels (CRPS) are groups of volunteers mandated by United States federal law to examine policies, procedures, and practices of public (e.g., state) child welfare agencies, and make recommendations for systemic improvements. To date, several researchers have made clarion calls for training frameworks and approaches for CRP members. Despite the federal mandate, millions of dollars in resources allocated to these panels, and the potential to positively impact the child welfare system, no published training frameworks exist. This brief documents the evaluation of an online training for CRP members (N = 21) in one southeastern state. The training was developed based on needs identified in existing CRP literature, and delivered via an online learning platform. A pre-experimental (pre/post) approach was used to evaluate the training. Results indicate a significant improvement in knowledge associated with serving on the CRP, and overall, participants viewed the training as being positively impactful to their work as a CRP member. The brief will provide an overview of the training, evaluation approach, and briefly discuss salient implications derived from the results.
1. Introduction
via an online learning platform. Training modules focused on building member knowledge associated with CRPs. A pre-experimental (pre/ post) approach was used to evaluate the training. After a brief review of literature, this brief will provide an overview of the training, explicate training results, and discuss salient implications derived from the training evaluation.
An engaged citizenry has long been a desirable component of public child welfare programs (Kinney, 2008; Miller & Jones, 2015; Stivers, 1990). In the United States (U.S.), this participation has manifested via the development of child welfare Citizen Review Panels (CRPs). CRPs are groups of volunteers mandated by U.S. federal law to examine public (e.g., state) child welfare agencies. Specifically, CRPs review policies, procedures, and practices associated with the public child welfare system (U.S. Department for Health and Human Services, 2014). These reviews culminate with reports that make recommendations for systemic improvements. Several studies have examined CRPs. The majority of these works have discussed the need for effective training initiatives and frameworks (e.g., Bryan, Jones, & Lawson, 2010; Jones, 2004; Jones & Royse, 2008; Miller, Collins-Camargo, & Jones, 2017; Miller, Collins-Camargo, Niu, & Jones, 2017; Miller & Jones, 2015). Despite these clarion calls related to training for CRP members, a comprehensive literature review revealed no published studies that explicitly examine training models for CRP members. This study seeks to address this limitation in the current literature. This brief documents the evaluation of an online training for CRP members (N = 21) in one southeastern state. The training was developed based on needs identified in existing CRP literature, and delivered
⁎
2. Background 2.1. CRP overview Citizen Review Panels (CRPs) are groups of citizen volunteers charged with evaluating the public child welfare system of the state in which they are formed (Kot, Bruner, & Scott, 1998). These groups were mandated via a 1996 reauthorization of the Child Abuse Prevention and Treatment Act (CAPTA; P.L. 104–235). This reauthorization focused on public oversight of state child welfare agencies. Thus, congress mandated that states seeking federal funds for child welfare services form CRPs (Collins, 1998). Per CAPTA, CRPs were to be formed and implemented by July 1999. In essence, CRPs are to monitor state compliance with CAPTA, review child fatalities, and monitor Title IV-E (e.g., adoption/foster care) programs. As well, CRPs are able to select, review, and/or evaluate other aspects of the public child welfare system, as they see fit (Jones &
Corresponding author. E-mail address: [email protected] (J.J. Miller).
https://doi.org/10.1016/j.childyouth.2018.01.022 Received 24 October 2017; Received in revised form 16 January 2018; Accepted 16 January 2018 Available online 31 January 2018 0190-7409/ © 2018 Elsevier Ltd. All rights reserved.
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
2.4. Purpose of the evaluation
Royse, 2008). Based on this work, CRPs make recommendations, via an annual report, related to systemic improvements. In turn, per a 2003 reauthorization of CAPTA, called the Keeping Children and Families Safe Act (P.L. 108–36), states are required to respond to the written CRP report within six months. Today, CRPs have been implemented in all states and the District of Columbia (Miller & Jones, 2015).
The evaluation of the training was guided by three (3) distinct queries: (1) Does participation in the initiative increase perceived member knowledge about CRPs; (2) Were CRP members satisfied with the training; and, (3) Did participants view the training as potentially impactful to the work of their panel? By answering these questions, and documenting processes associated with the training, this brief uniquely addresses limitations in the current literature.
2.2. CRP research Despite the federal mandate associated with CRPs and the resources, financial and otherwise, allocated to these panels, there is sparse research related to CRPs (Collins-Camargo, Buckwalter, & Jones, 2016; Miller, Collins-Camargo, Niu, et al., 2017). Limitations notwithstanding, several researchers have examined various aspects of CRPs. Many of these studies have explored member perceptions related to CRP participation. For instance, in national studies, both Jones (2004) and Jones and Royse (2008) found that CRP members reported a lack of communication and ambiguous goals as impediments to CRP work. Buckwalter (2014) made similar findings. In the most recent national examination of CRPs, Miller, Collins-Camargo, Niu, et al. (2017) concluded that members lack knowledge related to the federal mandate guiding their work. These authors also concluded that panels need to more effectively recruit and retain members representative of the communities in which the panels serve. A common leitmotif in the current research literature is the need for training frameworks for CRP members. In fact, most of the published studies associated with CRPs have discussed the training needs of CRP members. Over a decade ago, Jones (2004) explicitly argued about the need for member training. In their national study related to CRPs, Jones and Royse (2008) found that some participants reported needing “better training” related to CRPs (p. 155). Likewise, Bryan, CollinsCamargo, and Jones (2011) asserted that CRP members should be offered “more direct training and access to knowledge” (p. 618). Bryan, Jones, Allen, and Collins-Camargo (2007) called for trainings associated with developing knowledge about CRP members. These authors asserted that training for CRP members should be a “priority” for all stakeholders involved in CRP work (p. 1299). Both Jones, Litzelfelner, and Ford (2003) and Miller, Collins-Camargo, and Jones (2017) concluded that CRP training framework are needed to ensure that members have the requisite knowledge to effectively serve on the panels. The implications derived from the literature are clear. Whilst the training needs of CRPs have been identified in the literature, there are very few, if any, published examinations of training models. If CRPs are to meet their full potential, training endeavors must be examined and results disseminated. This brief seeks to meet this limitation in the current child welfare CRP literature.
3. Evaluation approach 3.1. Participants All CRP members in this southeastern state were invited to participate in the training. Each participant was sent a link pertaining to the training and registered for the training via the OLP. Participation in the training was optional and participants were able to discontinue participation at any time. Twenty-one of the 29 CRP members completed the training. A part of the training evaluation, CRP members who took part in the training did provide some basic demographic and general information. The typical member who completed the training was aged 46.33 (sd = 13.9), identified as Caucasian/White (n = 19) or AfricanAmerican/Black (n = 2) and had served on the panel 2.27 years (sd = 3.1; Mdn = 1). 3.2. Method Data were collected from each participant who took part in the training. Data were collected before participants started the training (e.g., pre-test) and again immediately upon completion of the training (post-test). Once collected, data were analyzed for the explicit purpose of answering the previously proposed evaluation queries. 3.3. Instrument Based on a literature review on CRPs (Bryan et al., 2010; Jones et al., 2003; Jones & Royse, 2008; Miller & Jones, 2015), the workgroup developed a questionnaire intended to collect primary data related to evaluating the training. The survey entailed three distinct sections and was piloted with a small group of former CRP members (n = 5) for item clarity and readability. The sections of the survey are as follows: (1) CRP Knowledge. To assess member knowledge associated with CRPs, the workgroup employed a knowledge subscale utilized in previous CRP research (see Miller, Collins-Camargo, Niu, et al., 2017). This subscale consists of five items designed to assess general knowledge related to CRPs, including the federal mandate (e.g., CAPTA) guiding the panels. Items are measured via Likert-type scale anchored at 1 with “Not knowledgeable” to 5 with “Very Knowledgeable.” An example item is as follows: In general, how knowledgeable are you about the federal legislation mandating CRPs? The knowledge score is comprised of the mean across all items. (2) Satisfaction with Training. Satisfaction was measured using a five-item scale developed by the workgroup. Items in this scale were designed to measure the extent to which participants were satisfied with the training. Items were measured via Likert-type scale anchored at 1 with “Not at All” to 5 with “Extremely.” Items for this scale are included in Table 2. (3) Impact of Training on CRP Work. Impact was measured via a three-item scale. Items in this scale were designed to measure the perceived impact of the training of CRP work. Items were measured via Likert-type scale anchored at 1 with “Not at All” to 5 with “Extremely.” As with the scale above, items for this scale are included in Table 2.
2.3. Training description A workgroup developed the training for CRP members in one southeastern state. This workgroup included university researchers, the state CRP-liaison and two former CRP members with extensive CRP experience. The training is rooted in literature that identifies the training needs of CRP members (e.g., Jones & Royse, 2008; Bryan et al., 2011; Miller, Collins-Camargo, & Jones, 2017). These studies in mind, the training was designed to meet three distinct, yet interconnected goals: (1) Educate members about the federal mandates associated with CAPTA; (2) Familiarize CRP members with the kind of work that CRPs undertake; and, (3) Inform members of contemporary state-level child welfare issues (e.g., practices and policies) that may influence the work of the CRP. The training was delivered via an online learning platform (OLP). Research suggests that online delivery may be ideal for volunteer groups (Cravens, 2001). The training was structured via four modules, conducive to meeting the afore-mentioned goals. Table 1 denotes each module and the overarching focus of the module. 95
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
Table 1 CRP training modules. Module
Overarching foci
Module 1. The Child Abuse Prevention and Treatment Act (CAPTA) and other Federal and State Child Welfare Laws
This module focuses on providing participants foundational information associated with CAPTA, in general, and sections that mandate CRPs and the focus of their work, specifically. Additionally, this module provides introductory information related to other seminal pieces of child welfare legislation that may impact the work of CRPs. This module examines the history and evolution of CRPs; research related to their work and impact; and identifies potential challenges to CRP work. Additionally, the module provides examples of previous projects that CRPs have engaged in and strategies for developing annual CRP reports. This module focuses on providing introductory information about child maltreatment. Specifically, the module concentrates on the classification, etiology, and diagnosis of child maltreatment. The module concludes by delineating cultural factors impacting child maltreatment and responses for addressing child maltreatment. This module provides an overview of state-specific child welfare practices, including state standards of practice (SOP) and legislative mandates. The module concludes by providing participants with information about contemporary child welfare issues plaguing the state and current/ongoing statelevel child welfare initiatives.
Module 2. Citizen Review Panels 101
Module 3. Overview of child maltreatment
Module 4. Public child welfare system
4. Evaluation results
knowledge scores at post were 3.70 (sd = 0.61), indicating a level of neutrality related to knowledge associated with CRPs. This finding suggests that future research should continue to examine training frameworks and delivery approaches. Notably, educational theorists maintain that sustained changes in knowledge are not achieved simply through providing content, but rather require repetition, implementation, and meaningful connection (e.g., Driscoll, 2004; Weimer, 2002). As such, CRP training frameworks that offer “refreshers” at specified intervals or utilize approaches that connect training components to the pragmatic realities of CRP work may for continuing knowledge development for CRP members (e.g., Bryan et al., 2007). Data from this evaluation also suggest the need for broad-based training frameworks for CRPs. Despite the federal mandate for CRPs, currently, there is no federal guidance associated with training CRP members (Vadapalli, 2017). The development of a CRP competency framework or training standards may help CRP members in being more knowledgeable about the child welfare issues impacting their work, thus addressing challenges detailed in the current literature. Entities such as the National Citizen Review Panel Advisory Board may certainly play a role in promulgating these standards. As indicated in Table 2, it appears that CRP members were satisfied with the training and viewed the content of the training as positively impactful to their work. Individual responses to items associated with these areas of the survey ranged between 4 and 5. Overall, these data bode well for the structure, format, and content associated with the training. One element of the training that may have impacted satisfaction with the training is the fact that the training was administered via an online learning platform. State-level CRP trainings in previous years had been delivered via face-to-face meetings, usually conducted during an annual meeting. Anecdotal evidence suggests that members often found it difficult to travel to a central location. The online format
As indicated, CRP Knowledge was assessed at pre (before starting the training) and at post (after the participant finished the training). A paired-samples t-test was conducted to compare the mean CRP Knowledge pre and post scores, respectively. The mean on the pretest was 2.46 (sd = 0.64), and the mean on the posttest was 3.70 (sd = 0.61). A significant effect was detected, whereby the posttest scores were significantly higher than the pretest scores (t(20) = −7.16, p = .000; Cohen's d = −1.5). In addition to the pre/post assessment, participants were asked to respond to several specific items associated with the training. These items were completed by participants after participating in the training (post-only). Table 2 includes the items and item means for these areas of the survey.
5. Discussion and implications Previous research has shown knowledge deficits among CRP members (Miller, Collins-Camargo, & Jones, 2017), and most every study associated with CRPs has stressed the need for research about CRP training endeavors. The purpose of this brief was to examine the impact of an online training for CRP members, specifically in the area of knowledge associated with CRPs. The following paragraphs identify salient discussion points, rooted in the evaluative results presented above. It appears that the training offered to CRP members did meet the overarching goal of improving CRP members' perceived knowledge associated with CRP work. Knowledge scores from pre to posttest were significantly higher, indicating an increase an overall knowledge related to CRPs. Despite this increase, data suggest there is room for improvement related to the training, specifically in the area of knowledge. Mean Table 2 Descriptive data for Satisfaction with Training and Impact of Training on CRP Work. Item
Survey area: Satisfaction with Training To what extent are you satisfied with the training you received? How credible did you find the information? How well organized do you believe the training program to be? How satisfied are you with the content of the training? How satisfied are you with the complexity of the training? Survey area: Impact of Training on CRP Work How useful do you think the information will be to CRP members as they engage in their work this year? To what extent do you expect to be able to incorporate the information from the training into your work as a CRP member? How compatible do you expect the practices from the training to be with the practical realities of serving on the CRP?
96
Min
Max
Mean (SD)
4 4 4 4 4
5 5 5 5 5
4.57(.50) 4.62(.49) 4.43(.50) 4.90(.30) 4.43(.50)
4 4 4
5 5 5
4.43(.50) 4.90(.30) 4.43(.50)
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
References
documented here allowed participants to access and complete the training at a time convenient for them. The use of online tools and educational platforms can certainly be a benefit for existing CRPs. Certainly, future studies should continue to examine training modules. Specially, studies that investigate the efficacy of online trainings, in comparison to other delivery models (e.g., face-to-face training, hybrid, etc.) may be ideal. As well, longitudinal aspects that examine the change of variables, specifically knowledge, overtime are warranted. An initial step may be to replicate the study documented here in other states and U.S. territories.
Bryan, V., Collins-Camargo, C., & Jones, B. (2011). Reflections on citizen-state child welfare partnerships: Listening to citizen review panel volunteers and agency liaisons. Children and Youth Services Review, 32(1), 986–1010. Bryan, V., Jones, B. L., Allen, E., & Collins-Camargo, C. (2007). Civic engagement or token participation? Perceived impact of the citizen review panel initiative in Kentucky. Child and Youth Services Review, 29, 1286–1300. Bryan, V., Jones, B. L., & Lawson (2010). Key features of effective citizen–state child welfare partnerships: Findings from a national study of citizen review panels. Children and Youth Services Review, 32(4), 595–603. Buckwalter, N. (2014). The potential for public empowerment through government-organized participation. Public Administration Review, 74(5), 573–584. Collins, W. (1998). Citizen Review Panels: An opportunity to improve child protection? Spokane, Washington: Washington State Institute for Public Policy. Collins-Camargo, C., Buckwalter, N., & Jones, B. L. (2016). Perceptions of state child welfare administrators regarding federally-mandated citizen review panels. Children and Youth Services Review. 62, 83–89. Cravens, J. (2001). Online mentoring: Programs and suggested practices as of February 2001. Journal of Technology in Human Services, 21(1–2), 85–109. Driscoll, M. P. (2004). Psychology of learning for instruction. Boston: Allyn & Bacon. Jones, B., Litzelfelner, P., & Ford, J. (2003). The value and role of Citizen Review Panels in child welfare: Perceptions of citizens review panel members and child protection workers. Child Abuse & Neglect, 27, 699–704. Jones, B. L. (2004). Variables impacting the effectiveness of citizens review panels for child protective services: A multi-state study. Children and Youth Services Review, 26(12), 1117–1127. Jones, B. L., & Royse, D. (2008). Citizen review panels: The connection between training and perceived effectiveness. Child Abuse & Neglect: The International Journal, 32, 918–919. Kinney, A. (2008). Current approaches to citizen involvement in performance measurement and questions they raise. National Civic Review, 46–54. http://dx.doi.org/10. 1002/ncr.206. Kot, V., Bruner, C., & Scott, S. (1998). Citizen review panels for the child protective services system: Guidelines and protocols. Chicago, IL: National Committee to Prevent Child Abuse. Miller, J., Collins-Camargo, C., & Jones, B. (2017). Exploring the university partnership model for child welfare Citizen Review Panels: A research brief. Child and Youth Services Review, 81, 1–4. http://www.sciencedirect.com/science/article/pii/ S019074091730405X. Miller, J., Collins-Camargo, C., Niu, C., & Jones, B. (2017). Exploring member perspectives on participation on child welfare Citizen Review Panels: A national study. Child Abuse and Neglect: The International Journal, 72, 352–359. Miller, J., & Jones, B. (2015). Using concept mapping as a planning tool: Child welfare Citizen Review Panels. Evaluation and Planning: The International Journal, 53, 99–106. Stivers, C. (1990). Active citizenship in public administration. In G. L. Wamsley, R. N. Bacher, C. T. Goodsell, P. S. Kronenberg, J. A. Rohr, C. M. Stivbers, O. F. White, & J. F. Wolf (Eds.). Refounding public administration (pp. 246–273). Thousand Oaks, CA: Sage Publications. U.S. Department for Health and Human Services (2014). Laws and policies. Retrieved from http://www.acf.hhs.gov/cwpm/programs/cb/laws_policies/laws/cwpm/ policy_dsp.jsp?citID=70. Vadapalli, D. (2017). Citizen review panels in child protection: Misunderstood, neglected, and underutilized. Children and Youth Services Review, 79, 539–546. Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. San Francisco: Jossey-Bass.
5.1. Limitations As with any evaluation effort, this one is not without limitations. Eight CRP members elected not to participate in the training. Given the sample size, inclusion of these individuals may have impacted the findings. As well, this evaluation assessed perceived knowledge about CRPs, not actual knowledge. Future evaluation efforts should look to discern between the two, and/or any knowledge discrepancy. This aspect can be vital for any citizen volunteer group. All participants in this training were CRP members in one state. Because CRPs are highly contextual, implementing this training in other jurisdictions may yield different results. Additionally, examining effects, by demographic (e.g., race, age, etc.) and general CRP information (e.g., length of time on the panel, etc.) may prove useful in developing the knowledge base associated with training and CRPs. Given these limitations, and others, generalizations associated with this evaluation should be considered carefully and critically. 6. Conclusion If the potential of CRPs in the area of child welfare is to be realized, than adept training models that increase the knowledge of CRP members must be actualized. This brief offers an initial framework for developing and evaluating a CRP training. Data associated with the evaluation of the training suggest that the framework has promise. It is imperative that models, such as the one documented here, continue to be developed, tested, and information disseminated. Conflict of interest declaration The authors report no conflict of interests associated with this work.
97
Contents lists available at ScienceDirect
Children and Youth Services Review journal homepage: www.elsevier.com/locate/childyouth
Training child welfare citizen review panel members: A promising approach?
T
⁎
J. Jay Miller , LaToya Burns Vaughn Training Resource Center, University of Kentucky, United States
A R T I C L E I N F O
A B S T R A C T
Keywords: Citizen review panels Training Child welfare
Citizen Review Panels (CRPS) are groups of volunteers mandated by United States federal law to examine policies, procedures, and practices of public (e.g., state) child welfare agencies, and make recommendations for systemic improvements. To date, several researchers have made clarion calls for training frameworks and approaches for CRP members. Despite the federal mandate, millions of dollars in resources allocated to these panels, and the potential to positively impact the child welfare system, no published training frameworks exist. This brief documents the evaluation of an online training for CRP members (N = 21) in one southeastern state. The training was developed based on needs identified in existing CRP literature, and delivered via an online learning platform. A pre-experimental (pre/post) approach was used to evaluate the training. Results indicate a significant improvement in knowledge associated with serving on the CRP, and overall, participants viewed the training as being positively impactful to their work as a CRP member. The brief will provide an overview of the training, evaluation approach, and briefly discuss salient implications derived from the results.
1. Introduction
via an online learning platform. Training modules focused on building member knowledge associated with CRPs. A pre-experimental (pre/ post) approach was used to evaluate the training. After a brief review of literature, this brief will provide an overview of the training, explicate training results, and discuss salient implications derived from the training evaluation.
An engaged citizenry has long been a desirable component of public child welfare programs (Kinney, 2008; Miller & Jones, 2015; Stivers, 1990). In the United States (U.S.), this participation has manifested via the development of child welfare Citizen Review Panels (CRPs). CRPs are groups of volunteers mandated by U.S. federal law to examine public (e.g., state) child welfare agencies. Specifically, CRPs review policies, procedures, and practices associated with the public child welfare system (U.S. Department for Health and Human Services, 2014). These reviews culminate with reports that make recommendations for systemic improvements. Several studies have examined CRPs. The majority of these works have discussed the need for effective training initiatives and frameworks (e.g., Bryan, Jones, & Lawson, 2010; Jones, 2004; Jones & Royse, 2008; Miller, Collins-Camargo, & Jones, 2017; Miller, Collins-Camargo, Niu, & Jones, 2017; Miller & Jones, 2015). Despite these clarion calls related to training for CRP members, a comprehensive literature review revealed no published studies that explicitly examine training models for CRP members. This study seeks to address this limitation in the current literature. This brief documents the evaluation of an online training for CRP members (N = 21) in one southeastern state. The training was developed based on needs identified in existing CRP literature, and delivered
⁎
2. Background 2.1. CRP overview Citizen Review Panels (CRPs) are groups of citizen volunteers charged with evaluating the public child welfare system of the state in which they are formed (Kot, Bruner, & Scott, 1998). These groups were mandated via a 1996 reauthorization of the Child Abuse Prevention and Treatment Act (CAPTA; P.L. 104–235). This reauthorization focused on public oversight of state child welfare agencies. Thus, congress mandated that states seeking federal funds for child welfare services form CRPs (Collins, 1998). Per CAPTA, CRPs were to be formed and implemented by July 1999. In essence, CRPs are to monitor state compliance with CAPTA, review child fatalities, and monitor Title IV-E (e.g., adoption/foster care) programs. As well, CRPs are able to select, review, and/or evaluate other aspects of the public child welfare system, as they see fit (Jones &
Corresponding author. E-mail address: [email protected] (J.J. Miller).
https://doi.org/10.1016/j.childyouth.2018.01.022 Received 24 October 2017; Received in revised form 16 January 2018; Accepted 16 January 2018 Available online 31 January 2018 0190-7409/ © 2018 Elsevier Ltd. All rights reserved.
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
2.4. Purpose of the evaluation
Royse, 2008). Based on this work, CRPs make recommendations, via an annual report, related to systemic improvements. In turn, per a 2003 reauthorization of CAPTA, called the Keeping Children and Families Safe Act (P.L. 108–36), states are required to respond to the written CRP report within six months. Today, CRPs have been implemented in all states and the District of Columbia (Miller & Jones, 2015).
The evaluation of the training was guided by three (3) distinct queries: (1) Does participation in the initiative increase perceived member knowledge about CRPs; (2) Were CRP members satisfied with the training; and, (3) Did participants view the training as potentially impactful to the work of their panel? By answering these questions, and documenting processes associated with the training, this brief uniquely addresses limitations in the current literature.
2.2. CRP research Despite the federal mandate associated with CRPs and the resources, financial and otherwise, allocated to these panels, there is sparse research related to CRPs (Collins-Camargo, Buckwalter, & Jones, 2016; Miller, Collins-Camargo, Niu, et al., 2017). Limitations notwithstanding, several researchers have examined various aspects of CRPs. Many of these studies have explored member perceptions related to CRP participation. For instance, in national studies, both Jones (2004) and Jones and Royse (2008) found that CRP members reported a lack of communication and ambiguous goals as impediments to CRP work. Buckwalter (2014) made similar findings. In the most recent national examination of CRPs, Miller, Collins-Camargo, Niu, et al. (2017) concluded that members lack knowledge related to the federal mandate guiding their work. These authors also concluded that panels need to more effectively recruit and retain members representative of the communities in which the panels serve. A common leitmotif in the current research literature is the need for training frameworks for CRP members. In fact, most of the published studies associated with CRPs have discussed the training needs of CRP members. Over a decade ago, Jones (2004) explicitly argued about the need for member training. In their national study related to CRPs, Jones and Royse (2008) found that some participants reported needing “better training” related to CRPs (p. 155). Likewise, Bryan, CollinsCamargo, and Jones (2011) asserted that CRP members should be offered “more direct training and access to knowledge” (p. 618). Bryan, Jones, Allen, and Collins-Camargo (2007) called for trainings associated with developing knowledge about CRP members. These authors asserted that training for CRP members should be a “priority” for all stakeholders involved in CRP work (p. 1299). Both Jones, Litzelfelner, and Ford (2003) and Miller, Collins-Camargo, and Jones (2017) concluded that CRP training framework are needed to ensure that members have the requisite knowledge to effectively serve on the panels. The implications derived from the literature are clear. Whilst the training needs of CRPs have been identified in the literature, there are very few, if any, published examinations of training models. If CRPs are to meet their full potential, training endeavors must be examined and results disseminated. This brief seeks to meet this limitation in the current child welfare CRP literature.
3. Evaluation approach 3.1. Participants All CRP members in this southeastern state were invited to participate in the training. Each participant was sent a link pertaining to the training and registered for the training via the OLP. Participation in the training was optional and participants were able to discontinue participation at any time. Twenty-one of the 29 CRP members completed the training. A part of the training evaluation, CRP members who took part in the training did provide some basic demographic and general information. The typical member who completed the training was aged 46.33 (sd = 13.9), identified as Caucasian/White (n = 19) or AfricanAmerican/Black (n = 2) and had served on the panel 2.27 years (sd = 3.1; Mdn = 1). 3.2. Method Data were collected from each participant who took part in the training. Data were collected before participants started the training (e.g., pre-test) and again immediately upon completion of the training (post-test). Once collected, data were analyzed for the explicit purpose of answering the previously proposed evaluation queries. 3.3. Instrument Based on a literature review on CRPs (Bryan et al., 2010; Jones et al., 2003; Jones & Royse, 2008; Miller & Jones, 2015), the workgroup developed a questionnaire intended to collect primary data related to evaluating the training. The survey entailed three distinct sections and was piloted with a small group of former CRP members (n = 5) for item clarity and readability. The sections of the survey are as follows: (1) CRP Knowledge. To assess member knowledge associated with CRPs, the workgroup employed a knowledge subscale utilized in previous CRP research (see Miller, Collins-Camargo, Niu, et al., 2017). This subscale consists of five items designed to assess general knowledge related to CRPs, including the federal mandate (e.g., CAPTA) guiding the panels. Items are measured via Likert-type scale anchored at 1 with “Not knowledgeable” to 5 with “Very Knowledgeable.” An example item is as follows: In general, how knowledgeable are you about the federal legislation mandating CRPs? The knowledge score is comprised of the mean across all items. (2) Satisfaction with Training. Satisfaction was measured using a five-item scale developed by the workgroup. Items in this scale were designed to measure the extent to which participants were satisfied with the training. Items were measured via Likert-type scale anchored at 1 with “Not at All” to 5 with “Extremely.” Items for this scale are included in Table 2. (3) Impact of Training on CRP Work. Impact was measured via a three-item scale. Items in this scale were designed to measure the perceived impact of the training of CRP work. Items were measured via Likert-type scale anchored at 1 with “Not at All” to 5 with “Extremely.” As with the scale above, items for this scale are included in Table 2.
2.3. Training description A workgroup developed the training for CRP members in one southeastern state. This workgroup included university researchers, the state CRP-liaison and two former CRP members with extensive CRP experience. The training is rooted in literature that identifies the training needs of CRP members (e.g., Jones & Royse, 2008; Bryan et al., 2011; Miller, Collins-Camargo, & Jones, 2017). These studies in mind, the training was designed to meet three distinct, yet interconnected goals: (1) Educate members about the federal mandates associated with CAPTA; (2) Familiarize CRP members with the kind of work that CRPs undertake; and, (3) Inform members of contemporary state-level child welfare issues (e.g., practices and policies) that may influence the work of the CRP. The training was delivered via an online learning platform (OLP). Research suggests that online delivery may be ideal for volunteer groups (Cravens, 2001). The training was structured via four modules, conducive to meeting the afore-mentioned goals. Table 1 denotes each module and the overarching focus of the module. 95
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
Table 1 CRP training modules. Module
Overarching foci
Module 1. The Child Abuse Prevention and Treatment Act (CAPTA) and other Federal and State Child Welfare Laws
This module focuses on providing participants foundational information associated with CAPTA, in general, and sections that mandate CRPs and the focus of their work, specifically. Additionally, this module provides introductory information related to other seminal pieces of child welfare legislation that may impact the work of CRPs. This module examines the history and evolution of CRPs; research related to their work and impact; and identifies potential challenges to CRP work. Additionally, the module provides examples of previous projects that CRPs have engaged in and strategies for developing annual CRP reports. This module focuses on providing introductory information about child maltreatment. Specifically, the module concentrates on the classification, etiology, and diagnosis of child maltreatment. The module concludes by delineating cultural factors impacting child maltreatment and responses for addressing child maltreatment. This module provides an overview of state-specific child welfare practices, including state standards of practice (SOP) and legislative mandates. The module concludes by providing participants with information about contemporary child welfare issues plaguing the state and current/ongoing statelevel child welfare initiatives.
Module 2. Citizen Review Panels 101
Module 3. Overview of child maltreatment
Module 4. Public child welfare system
4. Evaluation results
knowledge scores at post were 3.70 (sd = 0.61), indicating a level of neutrality related to knowledge associated with CRPs. This finding suggests that future research should continue to examine training frameworks and delivery approaches. Notably, educational theorists maintain that sustained changes in knowledge are not achieved simply through providing content, but rather require repetition, implementation, and meaningful connection (e.g., Driscoll, 2004; Weimer, 2002). As such, CRP training frameworks that offer “refreshers” at specified intervals or utilize approaches that connect training components to the pragmatic realities of CRP work may for continuing knowledge development for CRP members (e.g., Bryan et al., 2007). Data from this evaluation also suggest the need for broad-based training frameworks for CRPs. Despite the federal mandate for CRPs, currently, there is no federal guidance associated with training CRP members (Vadapalli, 2017). The development of a CRP competency framework or training standards may help CRP members in being more knowledgeable about the child welfare issues impacting their work, thus addressing challenges detailed in the current literature. Entities such as the National Citizen Review Panel Advisory Board may certainly play a role in promulgating these standards. As indicated in Table 2, it appears that CRP members were satisfied with the training and viewed the content of the training as positively impactful to their work. Individual responses to items associated with these areas of the survey ranged between 4 and 5. Overall, these data bode well for the structure, format, and content associated with the training. One element of the training that may have impacted satisfaction with the training is the fact that the training was administered via an online learning platform. State-level CRP trainings in previous years had been delivered via face-to-face meetings, usually conducted during an annual meeting. Anecdotal evidence suggests that members often found it difficult to travel to a central location. The online format
As indicated, CRP Knowledge was assessed at pre (before starting the training) and at post (after the participant finished the training). A paired-samples t-test was conducted to compare the mean CRP Knowledge pre and post scores, respectively. The mean on the pretest was 2.46 (sd = 0.64), and the mean on the posttest was 3.70 (sd = 0.61). A significant effect was detected, whereby the posttest scores were significantly higher than the pretest scores (t(20) = −7.16, p = .000; Cohen's d = −1.5). In addition to the pre/post assessment, participants were asked to respond to several specific items associated with the training. These items were completed by participants after participating in the training (post-only). Table 2 includes the items and item means for these areas of the survey.
5. Discussion and implications Previous research has shown knowledge deficits among CRP members (Miller, Collins-Camargo, & Jones, 2017), and most every study associated with CRPs has stressed the need for research about CRP training endeavors. The purpose of this brief was to examine the impact of an online training for CRP members, specifically in the area of knowledge associated with CRPs. The following paragraphs identify salient discussion points, rooted in the evaluative results presented above. It appears that the training offered to CRP members did meet the overarching goal of improving CRP members' perceived knowledge associated with CRP work. Knowledge scores from pre to posttest were significantly higher, indicating an increase an overall knowledge related to CRPs. Despite this increase, data suggest there is room for improvement related to the training, specifically in the area of knowledge. Mean Table 2 Descriptive data for Satisfaction with Training and Impact of Training on CRP Work. Item
Survey area: Satisfaction with Training To what extent are you satisfied with the training you received? How credible did you find the information? How well organized do you believe the training program to be? How satisfied are you with the content of the training? How satisfied are you with the complexity of the training? Survey area: Impact of Training on CRP Work How useful do you think the information will be to CRP members as they engage in their work this year? To what extent do you expect to be able to incorporate the information from the training into your work as a CRP member? How compatible do you expect the practices from the training to be with the practical realities of serving on the CRP?
96
Min
Max
Mean (SD)
4 4 4 4 4
5 5 5 5 5
4.57(.50) 4.62(.49) 4.43(.50) 4.90(.30) 4.43(.50)
4 4 4
5 5 5
4.43(.50) 4.90(.30) 4.43(.50)
Children and Youth Services Review 86 (2018) 94–97
J.J. Miller, L.B. Vaughn
References
documented here allowed participants to access and complete the training at a time convenient for them. The use of online tools and educational platforms can certainly be a benefit for existing CRPs. Certainly, future studies should continue to examine training modules. Specially, studies that investigate the efficacy of online trainings, in comparison to other delivery models (e.g., face-to-face training, hybrid, etc.) may be ideal. As well, longitudinal aspects that examine the change of variables, specifically knowledge, overtime are warranted. An initial step may be to replicate the study documented here in other states and U.S. territories.
Bryan, V., Collins-Camargo, C., & Jones, B. (2011). Reflections on citizen-state child welfare partnerships: Listening to citizen review panel volunteers and agency liaisons. Children and Youth Services Review, 32(1), 986–1010. Bryan, V., Jones, B. L., Allen, E., & Collins-Camargo, C. (2007). Civic engagement or token participation? Perceived impact of the citizen review panel initiative in Kentucky. Child and Youth Services Review, 29, 1286–1300. Bryan, V., Jones, B. L., & Lawson (2010). Key features of effective citizen–state child welfare partnerships: Findings from a national study of citizen review panels. Children and Youth Services Review, 32(4), 595–603. Buckwalter, N. (2014). The potential for public empowerment through government-organized participation. Public Administration Review, 74(5), 573–584. Collins, W. (1998). Citizen Review Panels: An opportunity to improve child protection? Spokane, Washington: Washington State Institute for Public Policy. Collins-Camargo, C., Buckwalter, N., & Jones, B. L. (2016). Perceptions of state child welfare administrators regarding federally-mandated citizen review panels. Children and Youth Services Review. 62, 83–89. Cravens, J. (2001). Online mentoring: Programs and suggested practices as of February 2001. Journal of Technology in Human Services, 21(1–2), 85–109. Driscoll, M. P. (2004). Psychology of learning for instruction. Boston: Allyn & Bacon. Jones, B., Litzelfelner, P., & Ford, J. (2003). The value and role of Citizen Review Panels in child welfare: Perceptions of citizens review panel members and child protection workers. Child Abuse & Neglect, 27, 699–704. Jones, B. L. (2004). Variables impacting the effectiveness of citizens review panels for child protective services: A multi-state study. Children and Youth Services Review, 26(12), 1117–1127. Jones, B. L., & Royse, D. (2008). Citizen review panels: The connection between training and perceived effectiveness. Child Abuse & Neglect: The International Journal, 32, 918–919. Kinney, A. (2008). Current approaches to citizen involvement in performance measurement and questions they raise. National Civic Review, 46–54. http://dx.doi.org/10. 1002/ncr.206. Kot, V., Bruner, C., & Scott, S. (1998). Citizen review panels for the child protective services system: Guidelines and protocols. Chicago, IL: National Committee to Prevent Child Abuse. Miller, J., Collins-Camargo, C., & Jones, B. (2017). Exploring the university partnership model for child welfare Citizen Review Panels: A research brief. Child and Youth Services Review, 81, 1–4. http://www.sciencedirect.com/science/article/pii/ S019074091730405X. Miller, J., Collins-Camargo, C., Niu, C., & Jones, B. (2017). Exploring member perspectives on participation on child welfare Citizen Review Panels: A national study. Child Abuse and Neglect: The International Journal, 72, 352–359. Miller, J., & Jones, B. (2015). Using concept mapping as a planning tool: Child welfare Citizen Review Panels. Evaluation and Planning: The International Journal, 53, 99–106. Stivers, C. (1990). Active citizenship in public administration. In G. L. Wamsley, R. N. Bacher, C. T. Goodsell, P. S. Kronenberg, J. A. Rohr, C. M. Stivbers, O. F. White, & J. F. Wolf (Eds.). Refounding public administration (pp. 246–273). Thousand Oaks, CA: Sage Publications. U.S. Department for Health and Human Services (2014). Laws and policies. Retrieved from http://www.acf.hhs.gov/cwpm/programs/cb/laws_policies/laws/cwpm/ policy_dsp.jsp?citID=70. Vadapalli, D. (2017). Citizen review panels in child protection: Misunderstood, neglected, and underutilized. Children and Youth Services Review, 79, 539–546. Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. San Francisco: Jossey-Bass.
5.1. Limitations As with any evaluation effort, this one is not without limitations. Eight CRP members elected not to participate in the training. Given the sample size, inclusion of these individuals may have impacted the findings. As well, this evaluation assessed perceived knowledge about CRPs, not actual knowledge. Future evaluation efforts should look to discern between the two, and/or any knowledge discrepancy. This aspect can be vital for any citizen volunteer group. All participants in this training were CRP members in one state. Because CRPs are highly contextual, implementing this training in other jurisdictions may yield different results. Additionally, examining effects, by demographic (e.g., race, age, etc.) and general CRP information (e.g., length of time on the panel, etc.) may prove useful in developing the knowledge base associated with training and CRPs. Given these limitations, and others, generalizations associated with this evaluation should be considered carefully and critically. 6. Conclusion If the potential of CRPs in the area of child welfare is to be realized, than adept training models that increase the knowledge of CRP members must be actualized. This brief offers an initial framework for developing and evaluating a CRP training. Data associated with the evaluation of the training suggest that the framework has promise. It is imperative that models, such as the one documented here, continue to be developed, tested, and information disseminated. Conflict of interest declaration The authors report no conflict of interests associated with this work.
97