- Email: [email protected]
An investigation of perceptions of programme quality support of adult basic education programmes
Evaluation and Program Planning 61 (2017) 106–112
Contents lists available at ScienceDirect
Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan
An investigation of perceptions of programme quality support of adult basic education programmes Gary Udouj Jr.a , Kenda Groverb,* , Greg Belcherc,d, Kit Kacirekb a
Fort Smith Public Schools, Fort Smith, AR, United States Adult and Lifelong Learning Program, University of Arkansas, United States Department of Technology and Workforce Learning, United States d Career and Technical Education, Pittsburg State University, United States b c
A R T I C L E I N F O
Article history: Received 22 September 2016 Received in revised form 21 November 2016 Accepted 23 November 2016 Available online 13 December 2016 Keywords: Adult basic education Self-assessment Evaluation Workforce development Programme management
A B S T R A C T
This study was designed to identify the degree to which the directors of adult basic education programs perceive they have program quality support, as evidenced by a well-defined mission and role in the community, a management system, human resources management, and a suitable learning environment. NSCALL’s Evidence-based program self-assessment (2006) was modified and administered electronically to administrators of adult education programs in a mid-southern state. Findings indicated that most directors perceive they are implementing the indicators of program quality support in all of the areas surveyed. A research-based annual self-study that considers the quality indicators is recommended, leaving a need for an update to the NCSALL assessment for use as a program assessment instrument. © 2016 Elsevier Ltd. All rights reserved.
1. Introduction What does it take for a local adult education programme to operate successfully? Although state and national standards for adult basic education programmes in the United States exist, there is currently no established structure in place to help programmes meet these standards. In addition, while there are performance indicators that are measured and used by state and federal agencies to determine funding and services at the state and local levels, they do not indicate how programmes and practitioners of adult basic education might best achieve improved outcomes. These outcome-based accountability systems do not measure programme processes or the operations that define programme quality, as defined by Comings, Soricone, Santos, & National Centre for the Study of Adult Learning and Literacy (2006). Programmes should have a structure in place that supports continuous improvement, allowing them to identify areas of need, develop strategies to address the needs, pilot test the strategies, integrate solutions programme-wide, and evaluate the impact of the strategies (McLendon & Polis, 2009). The accountability system in place for federally funded, state administered adult education programmes is The National
* Corresponding author. E-mail address: [email protected] (K. Grover). http://dx.doi.org/10.1016/j.evalprogplan.2016.11.015 0149-7189/© 2016 Elsevier Ltd. All rights reserved.
Reporting System (NRS). It addresses the accountability requirements of the Adult Education and Family Literacy Act (AEFLA), which is Title II of the Workforce Investment Act (WIA) (1998), and identifies three types of core measures: outcome measures, descriptive measures, and participation measures. The Workforce Innovation and Opportunity Act of 2014 (WIOA) updates and reauthorizes WIA, and includes six primary indicators of individual programme performance, including the percentage of programme participants in unsubsidized employment; median earnings of participants; percentage of participants who obtain a postsecondary credential or diploma; participants achieving skills gains; and effectiveness in serving employers (U.S. Department of Education, 2014). These shared performance measures will be used to gage successes in the core WIOA programmes (Adult, Dislocated Worker and Youth formula programmes; Adult Education and Literacy Act programmes; Wagner-Peyser Act employment services; and Rehabilitation Act Title I programmes), and strengthen coordination between adult education, postsecondary institutions, and employers. While these performance indicators are crucial to measuring student outcomes needed for success in postsecondary training and the workforce, they do not address the programme level systems needed to achieve these successes. To meet performance indicators, it is necessary to evaluate adult education programmes to ensure they are meeting the needs of their clientele. These evaluations can be formative or summative. Formative evaluations
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
are conducted to find where programmes need to be improved and summative evaluations are conducted to measure the overall merit and value added of the programme. Studies suggest that adult education programmes should engage in a regular process of evaluation for programme development and accountability, assessing the effectiveness of recruitment, intake, orientation, instruction, counseling, transition, and support services (Comings et al., 2006). Along those lines, this study explored adult education programme directors’ perceptions of best practices related to programme quality support in adult basic education programmes in a southern state in the U.S. Specifically, the study examined the degree to which directors perceive that their programmes have a well-defined role in the community, a quality management system, an adequate system to manage their human resources, and a suitable environment for learning for adult students. Data was gathered using a modified survey based on the practices identified by Comings et al. (2006) in their study An Evidence-based Adult Education Model Appropriate for Research. 2. Context of the study To date, there has been limited research conducted on adult basic education in the area of programme quality indicators. The seminal work, An Evidence-based Adult Education Model Appropriate for Research (2006) is an extension of the 2003 National Centre for the Study of Adult Learning and Literacy (NCSALL) publication, Establishing an Evidence-based Adult Education System, which described the steps adult basic education programmes can take to improve their practice (Comings et al., 2006). Since the development of this model, no others have been introduced, leaving room for an update to the NCSALL piece to provide states with a programme assessment instrument. Although the model was designed to help identify benchmark sites with similar quality practices for further research in adult education methods or practices, it is also to be used as an instrument for guiding programme improvement. It serves as the basis for NSCALL’s Evidence-based Programme Self-Assessment survey instrument, which allows adult basic education programmes to identify the principles of best practice that they currently employ, and to identify which areas need improvement. In the Research Site Identification Protocol (RSIP) appendix, Comings et al. (2006) also allow for the principles to be addressed to the degree to which a programme quality or practice is evident, thus offering a more in-depth analysis of programme components and activities. When used as a self-assessment tool, respondents can rate the degree to which a certain principle, such as a welldefined role, an effective management system, human resource management, and a suitable environment for learning, is evident in their programme, allowing administrators and researchers to diagnose programme strengths and areas for improvement. The research model designed by Comings et al. (2006) identifies the principles of best practice of four components: programme quality, entrance into a programme, participation in a programme, and reengagement in learning. Programme quality includes the local programme’s role in the community, management and human resources systems, and environment for learning. Entrance into a programme includes a programme’s approach to student recruitment, intake, and orientation. Participating in a programme includes a programme’s approach to classroom management, instruction, and supporting student persistence. Reengagement in learning includes practices for supporting students who have returned from a break in study. These components explore the principles derived from empirical evidence and professional wisdom that support them, allowing researchers to more accurately seek, develop, or evaluate interventions by studying programmes that have the ability to deliver services.
107
2.1. Programme mission, goals, and role A programme with a well-defined role will have both a clear organisational mission and an awareness of the programme environment (Comings et al., 2006). According to the Teachers of English to Speakers of other Languages (TESOL), an adult education programme needs a mission statement, with goals developed with input from stakeholders (TESOL, 2003). The mission of adult education as a field is to reengage adults who do not have the knowledge and skills to hold living wage jobs (Jones & Kelly, 2007). Missions for adult education programmes vary by programme and state, but generally focus on improving adults’ capacity to participate in society and improve their lives (Comings et al., 2006). An adult education programme should organise its instructional offerings to be consistent with the programme’s mission and the goals of the learners being served by the programme (TESOL, 2003). Adult education programmes are required to have a welldefined role under Title II of the Workforce Innovation and Opportunity Act (WIOA). Local adult basic education programmes should be responsive to the regional and local needs of the community, serving individuals in the community who are identified as most in need of adult education and literacy services, including adults who have low literacy levels and adults who are English language learners. Adult education programmes should also demonstrate alignment between their proposed activities and services and the strategies and goals of the local WIOA plan, providing opportunities for cooperation between adult education, youth services, rehabilitation, and other workforce services (H.R. 803, 2014). 2.2. Components of an effective management system The second main component of programme quality support identified by Comings et al. (2006) is an effective management system. An effective management system includes governance of a programme, data collection and use, a planning process, regular evaluation, and financial management. Governance of a programme must include a governing body that is representative of the local community and meets regularly to oversee programme activities. A programme should have an advisory group and bylaws or a board of directors that ensures accountability, administration of programme activities, and stakeholder participation (TESOL, 2003). An advisory board can advocate for the programme and provide programme allies, inform programme directors regarding potential funding sources, advise on industry trends, offer expertise regarding professional credentials, and provide ideas for programme improvement (Hicks, Hancher-Rauch, Vansickle, & Satterblom, 2011). A programme should also gather and use stakeholder data for programme improvement and accountability (Comings et al., 2006). The final component of an effective management system identified by Comings et al. (2006) is financial management. A programme should have sound financial management procedures to collect and maintain fiscal information, guide programme budgeting, ensure continuity of funding, and meet reporting requirements (TESOL, 2003). Additionally, a programme should have sustainable funding and manage resources effectively by maintaining records, establishing and monitoring a budget, and engaging in fundraising (Comings et al., 2006). The Workforce Innovation Opportunity Act (WIOA) of 2014 requires states to implement fiscal management and accountability information systems based on guidelines to be established by the Secretary of Labor and Secretary of Education, in consultation with state governors, elected officials, and one-stop partners (H.R. 803, 2014). The guidelines promote “efficient collection and use of
108
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
fiscal and management information for reporting and monitoring use of funds authorized under the core programmes” (H.R. 803, 2014: 155). States are required to use some WIOA funds to align adult education activities with other core programmes, including training services and career pathways. States need to establish high quality professional development programmes to improve instruction related to the specific needs of adult learners, and to disseminate “information about models and promising practices related to such programmes” (H.R. 803, 2014,: 522). 2.3. Human resource management and professional development Human resource management is another area of programme quality improvement identified by Comings et al. (2006), encompassing staff selection, working conditions, and professional development. A clear process is needed for recruiting and hiring qualified instructional and support staff (Comings et al., 2006; TESOL, 2003). While qualifications may vary according to local educational agency requirements, staff should reflect the cultural diversity of the programme learners (TESOL, 2003). While many programmes rely on part-time staff, research suggests making adult education teaching positions full-time; this may be important to maximizing outcomes, such as educational functioning level gains or entrance into post-secondary training (Chisman, 2002). Danielson (2007) found that recruitment and hiring of K-12 teachers is facilitated by a coherent definition of good teaching, aligned with mentoring, professional development, and teacher evaluation. Bingham and Smith (2007) point to a need for more research in what contributes to teacher quality specifically in the field of adult education. Older research on adult education suggests that adult education teachers should have personal qualities that include general knowledge, sensitivity to people and good communication skills, as well as professional qualities that include a philosophy about adult education, the ability to translate that philosophy into practice, and organisational and managerial skills (Soifer et al., 1990). In terms of working conditions, programmes should provide a safe and clean working environment, with compensation and benefits commensurate with other professional staff at similar institutions (TESOL, 2003). Expectations need to be set that all teachers should continue to learn and be provided opportunities to interact and collaborate (Taylor, Smith, & Bingham, 2005). Research indicates that adult educators want to work in an environment that provides opportunities to dialogue with colleagues and directors, where they can exchange ideas and get feedback on their teaching (Marceau, 2003). One way adult education programmes can improve teachers’ working conditions is to pay teachers to attend professional development events (Smith, Hofer, Gillespie, Solomon, & Rowe, 2003). A professional development plan should meet the needs of the teachers and provide opportunities to receive training, practice new skills, and receive feedback (Comings et al., 2006). Such a plan should be created with input from staff and stakeholders and appropriate resources to implement the plan should be acquired (TESOL, 2003; Taylor et al., 2005). Much of the discussion of the effectiveness of professional development comes from K-12 research, but the need for meaningful professional development is widely accepted. Darling-Hammond (2000) suggests that policy investments in the quality of teachers, including teacher education, licensing, professional development, and hiring, may be related to improvements in student performance. Smith et al. (2003) found that quality professional development for adult education instructors is characterized by effective facilitation and group dynamics, and is designed to meet the needs of the participants. Bingham and Smith
(2007) found that adult education professional development must include activities that have been shown to increase teachers’ understanding, access, and use of research. Opportunities should be provided for teachers to examine current trends, best practices, and technology in adult education and English as a Second Language (TESOL, 2003). 2.4. The learning environment The final component of programme quality support identified by Comings et al. (2006) is a suitable environment for learning. Programmes must offer flexible hours and convenient locations, a physically safe and psychologically safe environment, and learning materials and resources relevant to adults. In his foundational work, Knowles (1973) presented a “process model” in which an adult education teacher establishes a climate for learning, establishes opportunities for mutual planning, and works with adult students to diagnose student needs, formulate objectives, design and conduct learning experiences with appropriate techniques and materials, and evaluate outcomes. To meet the needs of adult learners and the community, programmes should provide courses with flexible schedules in convenient locations (TESOL, 2003), offering classes both day and night to accommodate working adults, within the constraints of programme resources (Comings et al., 2006). According to most researchers, there are two main barriers to adult learning: internal and external. Internal barriers could include an adult student’s disposition and level of anxiety (Falasca, 2011). External barriers arise from life circumstances of adults, and could include home and job responsibilities (Cross, 1981). Programmes should make provisions for differences in style, time, place, and pace of learning (Knowles, 1973). Research demonstrates that learning materials should be up-to-date, culturally sensitive, and suitable for a variety of learning styles (TESOL, 2003). K-12 research suggests that resources may include texts, Supplementary materials, and print materials, but also may include human resources such as experts from the community (Danielson 2007). Another component of providing a suitable environment for learning is the physical environment of a programme. Adult education programmes should maintain a safe and comfortable physical environment that supports adult learning (Comings et al., 2006). The physical environment can include temperature, ventilation, access to bathrooms, adequate light, comfortable chairs, and access to refreshments (Knowles, 1973). A learning space that lacks a comfortable degree of personal space will detract from the learning process (Merriam & Brockett, 1997). Danielson (2007) stresses classroom safety and accessibility to learning. No exits should be blocked and traffic flow should be efficient. Students need access to learning resources and the teacher. Classroom furniture should be best arranged to suit the classroom activities, including group work, pair work, and class discussions. A programme should also be psychologically safe for adult learners by protecting student confidentiality and demonstrating respect for student, staff, and stakeholder cultures (Comings et al., 2006). Additionally, procedures should be established for ensuring internal and external stakeholder confidentiality (TESOL, 2003). 3. Methods Participants in this study included administrators of state funded adult education programmes in one southern state in the U. S. The adult education programmes varied in size and were located in both rural and urban areas of the state. Data were collected during the spring of 2015 using a modified version of NSCALL’s Evidence-based Program Self-assessment (2006). The original survey was designed as a self-assessment tool to be used by adult
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
education programmes to determine whether they are using principles of best practices based on research, theory, and professional wisdom. The original instrument included four components: programme quality support, entrance into a programme, participation in a programme, and reengagement in learning, and allowed the respondents to indicate whether or not they practice those principles. Respondents could also comment on each principle. The survey included multiple components in each survey item, and responses were limited to “we need help” or “we have or don’t need help.” For this study the modified survey instrument focused specifically on the first element, perceptions of programme quality support, and was revised to separate out individual principles that were combined in the original instrument. Additionally, the survey was modified to include a 5-point Likert-type scale. The instrument was pilot and field tested with eight adult education administrators who were asked to provide feedback via email regarding the clarity and readability of the questions and to make suggestions for improvement. Additional modifications were made based on their observations. The modified survey instrument was divided into five sections. In the first four sections, participants were asked to respond using a 5-point Likert-type scale, with 5 being strongly agree and 1 being strongly disagree. For the purposes of data analysis, the 5-point Likert-type scale questions were interpreted as: 1–1.49 equals “strongly disagree,” 1.5–2.49 equals “agree,” 2.5–3.49 is “neither agree or disagree,” 3.5–4.49 is “agree,” and 4.5–5.0 is “strongly agree.” Section one of the survey included four questions pertaining to the administrators’ perceptions of their centre’s role in the community. Section two included 13 questions pertaining to the participants’ perceptions of the effectiveness of the planning, evaluation, and governance of their programmes. Section three contained 12 questions related to how human resources are managed. Section four contained 11 questions related to the degree to which participants perceive they provide a suitable environment for learning for adult students in their programmes. The final section contained questions related to the demographics of individual programmes, such as whether the programme is urban or rural, and how many counties the programme serves (1–5 or more). The final question asked the participant to describe whether their Local Educational Administrator (LEA) is a public school district, a 2-year college, or other (part of a trade school). A Cronbach’s alpha performed on the completed survey resulted in a reliability of 0.883, which is above the 0.7 level of acceptability (Gliem & Gliem, 2003). All Likert-type items were included. A Cronbach’s Alpha is used to measure the internal consistency of the instrument (Cronbach, 1951). 4. Findings Of the 39 directors, 36 (92%) responded to the invitation to participate. As shown in Table 1, almost two-thirds (61.1%) described their adult education centre as rural, while the remaining described their programme as urban. One-half of the programmes served one county, while 19.4% served two counties, 13.9% served three counties, 2.8% served four counties, and 13.9% served five counties. Just over one-half (52.8) of local programmes in the state were administrated by 2-year colleges, and more than one-third (38.9%) by public school districts, with the remaining programmes part of trade or other types of schools. As shown in Table 2, responses on the first four statements on the survey related to a programme’s role in the community. The respondents agreed that their programmes are designed to meet community needs, and that they use their mission statements to help guide services and direction. The means on these four questions ranged from 4.08 to 4.36, indicating that all directors surveyed agreed or strongly agreed that they were implementing
109
Table 1 Demographic Information for Adult Education programs in Arkansas. Which best describes your program? Urban Rural Total
Frequency 14 22 36
How many counties does your program serve? 18 1 2 7 3 5 4 1 5 5 Total 36
Percent 38.9 61.1 100.0
50.0 19.4 13.9 2.8 13.9 100.0
Which best describes your Local Educational Administrator (LEA): Public School district 14 38.9 2-year College 19 52.8 Missing 3 8.3 Total 36 100.0
these practices. Standard deviation ranged from 0.639 to 0.770. The written responses suggested that directors use mission statements to help guide practice. Thirteen statements on the survey referred to a programme’s management system (Table 3). Results from the survey revealed that the respondents strongly agreed that their programmes monitor their budgets (M = 4.67, SD = 0.478), maintain accurate financial records (M = 4.64, SD = 0.639), and adhere to state and national policies (M = 4.61, SD = 0.494). They agreed that they have a planning process that facilitates programme development and accountability (M = 4.22, SD = 0.485). The survey results indicated that adult education directors perceive they have a regularly scheduled evaluation process (M = 4.19, SD = 0.749) and planning process (M = 4.06, SD = 0.674). While the programme directors surveyed perceive that they have advisory boards that are fairly representative of the local community (M = 4.08, SD = 0.604) and meet regularly (M = 4.08, SD = 0.841), responses indicated that directors had less agreement on whether the advisory boards provide direction to the programmes (M = 3.94, SD = 1.194). Respondents neither agreed or disagreed with the survey statement, “I think we have an adequate budget for our adult education programme.” This statement had the lowest mean of the 40 Likert-type questions (M = 2.58, SD = 1.402), with 31% of respondents choosing “strongly disagree,” indicating that almost a third of respondents perceive that their programmes are underfunded. Respondents also reported being less likely to seek funding for their programmes from multiple sources (M = 3.17, SD = 1.108). Twelve statements on the survey referred to human resources management, encompassing staff selection, working conditions, and professional development. The respondents agreed (M = 4.00, SD = 0.956) that they have a clear process for recruiting and hiring qualified instructional and support staff (Comings et al., 2006; TESOL, 2003). The respondents agreed that their programme’s hiring policy considers the skills, knowledge, and life experiences of applicants (M = 4.44, SD = 0.504). Results from the survey revealed that adult education directors agreed that they provide working conditions that support the work and growth of the staff (M = 4.44, SD = 0.735). They strongly agreed that they treat staff as professionals (M = 4.69, SD = 0.467), and agreed that the working conditions for faculty and staff contribute positively to student progress (M = 4.44, SD = 0.652). The survey results indicated that the directors agreed that they provide opportunities for training (M = 4.33, SD = 0.586), opportunities to practice new skills (M = 4.17, SD = 0.775), and provide staff opportunities to receive constructive feedback (M = 3.94,
110
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
Table 2 Descriptive Statistics for a Well-defined Role.
I think that our adult education program’s services are designed to meet community needs. I believe we have a clear mission statement that is used to provide direction for the adult education program. Our adult education program’s mission is used to guide most aspects of the program’s services. I believe our adult education program has a clear statement of goals that helps us meet our mission.
Mean
SD
4.36 4.25 4.14 4.08
0.639 0.692 0.683 0.770
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Table 3 Descriptive Statistics for Quality Management System.
I I I I I I I I I I I I I
feel we monitor our budget on a regular basis within our adult education program. am confident the financial records in our adult education program are accurately maintained. believe our program’s services adhere to policies set by state and national funding agencies. think our adult education program has adequate management of the financial resources. feel our program’s planning process facilitates program development and accountability. believe our program has a regularly scheduled evaluation process. believe our advisory board has regularly scheduled meetings. believe our advisory board is fairly representative of the local community. believe our program has a regularly scheduled planning process. think our adult education program uses its advisory board or board of directors to provide direction to the program. believe our advisory board properly guides the activities of the adult education program. believe we seek out fundraising opportunities from multiple sources for our adult education program. think we have an adequate budget for our adult education program.
Mean
SD
4.67 4.64 4.61 4.47 4.22 4.19 4.08 4.08 4.06 3.94 3.39 3.17 2.58
0.478 0.639 0.494 0.696 0.485 0.749 0.841 0.604 0.674 1.194 0.994 1.108 1.402
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
SD = 0.630). The directors surveyed agreed that their programmes’ professional development plans take into account the needs of the programme and staff (M = 4.19, SD = 0.624). There was a greater standard deviation related to staff salaries (M = 3.61, SD = 1.337) on the statement “Staff is provided with adequate pay and benefits,” with 11 respondents disagreeing or strongly disagreeing with that statement (Table 4). Ten statements on the survey related to providing a suitable environment for learning, which includes offering flexible hours and convenient locations, a both physically safe and psychologically safe environment, and learning materials and resources relevant to adults. Directors strongly agreed that their programmes make efforts to protect the confidentiality of students and staff (M = 4.83, SD = 0.878), and that they provide both physically safe (M = 4.56, SD = 0.652) and psychologically safe (M = 4.53, SD = 0.654) environments for students and staff, and that this supports student learning (M = 4.53, SD = 0.696). Directors were in agreement that their programmes offer flexible scheduling (M = 4.36, SD = 0.639) and convenient locations for students to learn (M = 4.36, SD = 0.639). Programme directors agreed that they maintain materials and resources appropriate for learning and for
adult students (M = 4.49, SD = 0.562), including software that was designed for use by adult students (M = 4.44, SD = 0.504) (Table 5). 5. Discussion, implications for practice, and recommendations for future research The survey results provide a current snapshot of the practices of local programmes in this southern state, and the challenges that the leaders in these programmes are facing. The single most important finding related to the funding of local programmes. Adequate funding for adult education programmes remains a concern for many directors, especially with the increased accountability measures coming from the new WIOA legislation. For example, in awarding grants to local programmes, federal WIOA legislation requires states to consider the degree to which the local programme would be responsive to regional needs; to serving individuals identified as most in need of adult education services (students with low literacy levels and English language learners); the ability of the programme to serve students with disabilities; the programme’s past effectiveness; the programme’s alignment activities; and the goals of local one-stop partners.
Table 4 Descriptive Statistics for Human Resource Management.
I I I I I I I I I I I I
believe staff within our program are treated as professionals. believe our program’s working conditions support staff improvement. believe working conditions for our faculty and staff contribute positively to student progress. think our adult education program’s hiring policy equally considers the skills, knowledge, and life experiences of applicants. believe our professional development plan provides staff opportunities to receive proper training. believe our adult education program has a clear process for hiring instructional and support staff. believe our staffs' professional development plan takes into account the needs of the program and staff. believe our professional development plan provides staff opportunities to practice new skills. feel our adult education program helps employees create a professional development plan that enables them to grow professionally. believe our adult education program has a clear process for recruiting instructional and support staff. believe our professional development plan provides staff opportunities to receive constructive feedback. believe our staff is provided with adequate pay and benefits.
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Mean
SD
4.69 4.44 4.44 4.44 4.33 4.22 4.19 4.17 4.17 4.00 3.94 3.61
0.467 0.735 0.652 0.504 0.586 0.898 0.624 0.775 0.609 0.956 0.630 1.337
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
111
Table 5 Descriptive Statistics for Adult Education Learning Environment.
I I I I I I I I I I
believe believe believe believe believe believe believe believe believe believe
our our our our our our our our our our
program makes appropriate efforts to protect the confidentiality of students and staff. program demonstrates proper respect for the cultures of students and staff. program’s physical environment is safe for staff and students. program’s physical environment supports adult learning. program environment is psychologically safe for staff and students. program uses appropriate/relevant written materials that were designed for use by adult students. program’s physical environment is comfortable for staff and students. program uses software that was designed for use by adult students. program provides instructional services at locations that are convenient for our students. program provides instructional services at hours convenient for students.
Mean
SD
4.83 4.78 4.56 4.53 4.53 4.49 4.47 4.44 4.36 4.36
0.878 0.866 0.652 0.696 0.654 0.562 0.696 0.504 0.639 0.639
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Other considerations impacted by funding include whether the local programme bases instruction on research-based best practices and whether technology is used to increase the amount and quality of learning. Programmes should also use well-trained instructors and administrators who have access to high quality professional development (H.R. 803, 2014). Hiring more full-time staff, as research suggests programmes should do (Chisman, 2002), is another substantial drain on already limited resources. With many states’ shrinking budgets for adult education services, meeting these considerations will prove difficult for local programmes. While this state’s Adult Education Division examines these considerations when awarding grants to local programmes, the amount available from the state budget for distribution to local programmes needs to increase to keep up with the pace of salary increases, costs, and the demand for adult basic education services. Further research could investigate how the lack of or decrease in adult basic education funding affects programme outcomes. The survey also indicated (M = 3.39, SD = 0.994) that directors are ambivalent about how advisory boards help guide the activities of their adult education programmes, indicating that they may not have a clear understanding of how boards can be useful in helping them reach their goals. An effective advisory group addresses issues related to programme administration and ensures accountability (TESOL, 2003). Board configuration may be different for every local programme and their influence and involvement of members likely varies significantly. Directors of adult education programmes might benefit from professional development focused on selecting suitable board members who are stakeholders in the programme or have some level of expertise in working with similar organisations, and effectively using the board to promote, guide, and improve their programme. Stakeholders might include business leaders, other service agency representatives, students, human resource managers, staff, and others. While this survey looks at the self-reported perceptions of directors of adult education programmes, future research might also look at the perceptions of programme quality of teachers and staff of these same programmes, and compare those perceptions to those of the administrators. While a local programme leader may feel he or she is using best practices efficiently, teachers, paraprofessionals, board members, and other stakeholders such as students and community members may see it differently. Exploring the perception of others who have a vested interest in the programme would provide a more accurate picture of areas that need attention and improvement. A limitation of this study was the small population. This study could be replicated and include adult education programme directors across the United States, yielding a broader perspective of what is occurring nationally. Additionally, a qualitative study might use focus groups and in-depth interviews to examine the perceptions of both directors and staff and the reasons behind their responses.
Quality programme support on a local level includes the existence of a well-defined role of the adult education programme, an effective local management system, a local human resource management system, and a suitable environment for learning (Comings et al., 2006). Many state assessment tools look at some or all of these areas, but without a standardized instrument, local programmes may not be judged using consistent criteria from state to state and programme to programme. There is a need for a research-based, standardized instrument that can be used as a guide for self-assessment by local programmes, and also for programme monitoring by the states, to include not only quality programme support, but also the components not addressed in this study: student entrance into a programme, participation in a programme, and reengagement in learning. A next step would be to modify and update the instrument developed by Comings et al. (2006), focusing on the three remaining areas, to create a comprehensive assessment tool that measures the four components identified in the original study.
References Bingham, M., & Smith, C. (2007). Professional development and evidence-based practice in adult education. In A. Belzer (Ed.), Toward defining and improving quality in adult basic education (pp. 69–84).New York: Routledge. Chisman, F. P. (2002). Leading from the middle: The state role in adult education & literacy. . . Retrieved from http://eric.ed.gov. Comings, J., Soricone, L., Santos, M., & National Centre for the Study of Adult Learning and Literacy (2006). An evidence-based adult education programme model appropriate for research. NCSALL occasional paper. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from http://www. ncsall.net. Cronbach, L. (1951). Cronbach alpha and the internal structure of tests. Psychmerika, 16, 297–334. Cross, K. P. (1981). Adults as learners increasing participation and facilitating learning. San Francisco: Jossey-Bass. Danielson, C. (2007). Enhancing professional practice: A framework for teaching, Alexandria, VA : Association for supervision & curriculum development. . Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives 8, 1. . Retrieved from http://www.politicalscience.uncc.edu. Falasca, M. (2011). Barriers to adult learning: Bridging the gap. Australian Journal of Adult Learning 51(3), 583–590. . Retrieved from http://files.eric.ed.gov/fulltext/ EJ954482.pdf. Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Midwest Research to Practice Conference in Adult, Continuing, and Community Education . . Retrieved December 13, 2016 https://www.google.com/webhp?sourceid=chromeinstant&ion=1&espv=2&ie=UTF-8#q=gliem%20and%20gliem%202003. Hicks, L., Hancher-Rauch, H., Vansickle, J., & Satterblom, A. (2011). Creating a programme advisory board: A key advocacy and marketing strategy for longevity in turbulent times. Journal of Physical Education Recreation & Dance, 82 (3), 46–52. Jones, D., & Kelly, P. (2007). Mounting pressures facing the U.S. workforce and the increasing need for adult education and literacy. New York, NY: National Commission on Adult Literacy. . Retrieved from http://www. nationalcommissiononadultliteracy.org/content/nchemspresentation.pdf. Knowles, M. (1973). The adult learner: A neglected species. Houston TX: Gulf 1990. Marceau, G. (2003). Professional development in adult basic education. New Directions for Adult and Continuing Education, 98, 67–74.
112
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
McLendon, L., & Polis, K. (2009). A Resource packet for using stakeholder assessments to identify strengths and needs. Syracuse, NY: NAEPDC- Proliteracy America Joint Initiative. . Retrieved from http://lea.proliteracy.org/external/lpl/download/ Workshop%201%20Participant%20Pa cket%20Rev%207-12. pdf. Merriam, S. B., & Brockett, R. G. (1997). The profession and practice of adult education: An introduction. San Francisco: Jossey-Bass. Smith, C., Hofer, J., Gillespie, M., Solomon, M., & Rowe, K. (2003). How teachers change: A study of professional development in adult education. NCSALL reports number 25. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from www.proquest.com. Soifer, R., Irwin, M., Crumrine, B., Honzaki, E., Simmons, B., & Young, D. (1990). The complete theory-to-practice handbook of adult literacy: Curriculum design and teaching approaches. Columbia University, NY: Teachers College Press. Teachers of English to Speakers of Other Languages (TESOL) (2003). Standards for adult education ESOL programs. Alexandria, VA: Teachers of English to Speakers of other Languages, Inc.. Taylor, J., Smith, C., & Bingman, B. (2005). Programme administrators’ sourcebook. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from http://lea.proliteracy.org. U.S. Department of Education (2014). The workforce innovation and opportunity act, overview of title II: Adult education and literacy. . . Retrieved from http://www2. ed.gov/about/offices/list/ovae/pi/AdultEd/wioa-overview.pdf. Workforce Investment Act (WIA) (1998). Public law. 105–220. Workforce Investment, & Opportunity Act (WIOA) (2014). H.R. 803, 113th Cong. (2014). . Gary Udouj, Jr. is the Director of Adult Education for the Fort Smith Public Schools Fort Smith, Arkansas. He has served as chair of the Arkansas Adult Education Advisory Board and President of the Arkansas Association of Administrators of Adult
Education. He is the recipient of the 2015 Outstanding Doctoral Student in Adult and Lifelong Learning at the College of Education and Health Professions at the University of Arkansas.
Kenda Grover is an assistant professor in the Adult and Lifelong Learning Program at the University of Arkansas. She serves as the program coordinator for the master of education degree in ADLL. Her research interests include the professional development of faculty members who teach in online programs, leadership in adult education programs, self-directed learning and healthcare, and communities of practice in nonformal learning environments.
Greg Belcher is a professor in the Department of Technology and Workforce Learning and the director of the Kansas Center for Career and Technical Education at Pittsburg State University. His research interests includes the retention of new Career and Technical Education instructors, how to use student assessment outcomes in making instructional decisions and the evaluation of Career and Technical Education programs.
Kit Kacirek is an associate professor in the Adult and Lifelong Learning Program at the University of Arkansas. Her research agenda focuses on bridging the gap between theory and practice in learning theory and leadership development. Dr. Kacirek also serves as the program coordinator for the doctoral program in Adult and Lifelong Learning.
Contents lists available at ScienceDirect
Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan
An investigation of perceptions of programme quality support of adult basic education programmes Gary Udouj Jr.a , Kenda Groverb,* , Greg Belcherc,d, Kit Kacirekb a
Fort Smith Public Schools, Fort Smith, AR, United States Adult and Lifelong Learning Program, University of Arkansas, United States Department of Technology and Workforce Learning, United States d Career and Technical Education, Pittsburg State University, United States b c
A R T I C L E I N F O
Article history: Received 22 September 2016 Received in revised form 21 November 2016 Accepted 23 November 2016 Available online 13 December 2016 Keywords: Adult basic education Self-assessment Evaluation Workforce development Programme management
A B S T R A C T
This study was designed to identify the degree to which the directors of adult basic education programs perceive they have program quality support, as evidenced by a well-defined mission and role in the community, a management system, human resources management, and a suitable learning environment. NSCALL’s Evidence-based program self-assessment (2006) was modified and administered electronically to administrators of adult education programs in a mid-southern state. Findings indicated that most directors perceive they are implementing the indicators of program quality support in all of the areas surveyed. A research-based annual self-study that considers the quality indicators is recommended, leaving a need for an update to the NCSALL assessment for use as a program assessment instrument. © 2016 Elsevier Ltd. All rights reserved.
1. Introduction What does it take for a local adult education programme to operate successfully? Although state and national standards for adult basic education programmes in the United States exist, there is currently no established structure in place to help programmes meet these standards. In addition, while there are performance indicators that are measured and used by state and federal agencies to determine funding and services at the state and local levels, they do not indicate how programmes and practitioners of adult basic education might best achieve improved outcomes. These outcome-based accountability systems do not measure programme processes or the operations that define programme quality, as defined by Comings, Soricone, Santos, & National Centre for the Study of Adult Learning and Literacy (2006). Programmes should have a structure in place that supports continuous improvement, allowing them to identify areas of need, develop strategies to address the needs, pilot test the strategies, integrate solutions programme-wide, and evaluate the impact of the strategies (McLendon & Polis, 2009). The accountability system in place for federally funded, state administered adult education programmes is The National
* Corresponding author. E-mail address: [email protected] (K. Grover). http://dx.doi.org/10.1016/j.evalprogplan.2016.11.015 0149-7189/© 2016 Elsevier Ltd. All rights reserved.
Reporting System (NRS). It addresses the accountability requirements of the Adult Education and Family Literacy Act (AEFLA), which is Title II of the Workforce Investment Act (WIA) (1998), and identifies three types of core measures: outcome measures, descriptive measures, and participation measures. The Workforce Innovation and Opportunity Act of 2014 (WIOA) updates and reauthorizes WIA, and includes six primary indicators of individual programme performance, including the percentage of programme participants in unsubsidized employment; median earnings of participants; percentage of participants who obtain a postsecondary credential or diploma; participants achieving skills gains; and effectiveness in serving employers (U.S. Department of Education, 2014). These shared performance measures will be used to gage successes in the core WIOA programmes (Adult, Dislocated Worker and Youth formula programmes; Adult Education and Literacy Act programmes; Wagner-Peyser Act employment services; and Rehabilitation Act Title I programmes), and strengthen coordination between adult education, postsecondary institutions, and employers. While these performance indicators are crucial to measuring student outcomes needed for success in postsecondary training and the workforce, they do not address the programme level systems needed to achieve these successes. To meet performance indicators, it is necessary to evaluate adult education programmes to ensure they are meeting the needs of their clientele. These evaluations can be formative or summative. Formative evaluations
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
are conducted to find where programmes need to be improved and summative evaluations are conducted to measure the overall merit and value added of the programme. Studies suggest that adult education programmes should engage in a regular process of evaluation for programme development and accountability, assessing the effectiveness of recruitment, intake, orientation, instruction, counseling, transition, and support services (Comings et al., 2006). Along those lines, this study explored adult education programme directors’ perceptions of best practices related to programme quality support in adult basic education programmes in a southern state in the U.S. Specifically, the study examined the degree to which directors perceive that their programmes have a well-defined role in the community, a quality management system, an adequate system to manage their human resources, and a suitable environment for learning for adult students. Data was gathered using a modified survey based on the practices identified by Comings et al. (2006) in their study An Evidence-based Adult Education Model Appropriate for Research. 2. Context of the study To date, there has been limited research conducted on adult basic education in the area of programme quality indicators. The seminal work, An Evidence-based Adult Education Model Appropriate for Research (2006) is an extension of the 2003 National Centre for the Study of Adult Learning and Literacy (NCSALL) publication, Establishing an Evidence-based Adult Education System, which described the steps adult basic education programmes can take to improve their practice (Comings et al., 2006). Since the development of this model, no others have been introduced, leaving room for an update to the NCSALL piece to provide states with a programme assessment instrument. Although the model was designed to help identify benchmark sites with similar quality practices for further research in adult education methods or practices, it is also to be used as an instrument for guiding programme improvement. It serves as the basis for NSCALL’s Evidence-based Programme Self-Assessment survey instrument, which allows adult basic education programmes to identify the principles of best practice that they currently employ, and to identify which areas need improvement. In the Research Site Identification Protocol (RSIP) appendix, Comings et al. (2006) also allow for the principles to be addressed to the degree to which a programme quality or practice is evident, thus offering a more in-depth analysis of programme components and activities. When used as a self-assessment tool, respondents can rate the degree to which a certain principle, such as a welldefined role, an effective management system, human resource management, and a suitable environment for learning, is evident in their programme, allowing administrators and researchers to diagnose programme strengths and areas for improvement. The research model designed by Comings et al. (2006) identifies the principles of best practice of four components: programme quality, entrance into a programme, participation in a programme, and reengagement in learning. Programme quality includes the local programme’s role in the community, management and human resources systems, and environment for learning. Entrance into a programme includes a programme’s approach to student recruitment, intake, and orientation. Participating in a programme includes a programme’s approach to classroom management, instruction, and supporting student persistence. Reengagement in learning includes practices for supporting students who have returned from a break in study. These components explore the principles derived from empirical evidence and professional wisdom that support them, allowing researchers to more accurately seek, develop, or evaluate interventions by studying programmes that have the ability to deliver services.
107
2.1. Programme mission, goals, and role A programme with a well-defined role will have both a clear organisational mission and an awareness of the programme environment (Comings et al., 2006). According to the Teachers of English to Speakers of other Languages (TESOL), an adult education programme needs a mission statement, with goals developed with input from stakeholders (TESOL, 2003). The mission of adult education as a field is to reengage adults who do not have the knowledge and skills to hold living wage jobs (Jones & Kelly, 2007). Missions for adult education programmes vary by programme and state, but generally focus on improving adults’ capacity to participate in society and improve their lives (Comings et al., 2006). An adult education programme should organise its instructional offerings to be consistent with the programme’s mission and the goals of the learners being served by the programme (TESOL, 2003). Adult education programmes are required to have a welldefined role under Title II of the Workforce Innovation and Opportunity Act (WIOA). Local adult basic education programmes should be responsive to the regional and local needs of the community, serving individuals in the community who are identified as most in need of adult education and literacy services, including adults who have low literacy levels and adults who are English language learners. Adult education programmes should also demonstrate alignment between their proposed activities and services and the strategies and goals of the local WIOA plan, providing opportunities for cooperation between adult education, youth services, rehabilitation, and other workforce services (H.R. 803, 2014). 2.2. Components of an effective management system The second main component of programme quality support identified by Comings et al. (2006) is an effective management system. An effective management system includes governance of a programme, data collection and use, a planning process, regular evaluation, and financial management. Governance of a programme must include a governing body that is representative of the local community and meets regularly to oversee programme activities. A programme should have an advisory group and bylaws or a board of directors that ensures accountability, administration of programme activities, and stakeholder participation (TESOL, 2003). An advisory board can advocate for the programme and provide programme allies, inform programme directors regarding potential funding sources, advise on industry trends, offer expertise regarding professional credentials, and provide ideas for programme improvement (Hicks, Hancher-Rauch, Vansickle, & Satterblom, 2011). A programme should also gather and use stakeholder data for programme improvement and accountability (Comings et al., 2006). The final component of an effective management system identified by Comings et al. (2006) is financial management. A programme should have sound financial management procedures to collect and maintain fiscal information, guide programme budgeting, ensure continuity of funding, and meet reporting requirements (TESOL, 2003). Additionally, a programme should have sustainable funding and manage resources effectively by maintaining records, establishing and monitoring a budget, and engaging in fundraising (Comings et al., 2006). The Workforce Innovation Opportunity Act (WIOA) of 2014 requires states to implement fiscal management and accountability information systems based on guidelines to be established by the Secretary of Labor and Secretary of Education, in consultation with state governors, elected officials, and one-stop partners (H.R. 803, 2014). The guidelines promote “efficient collection and use of
108
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
fiscal and management information for reporting and monitoring use of funds authorized under the core programmes” (H.R. 803, 2014: 155). States are required to use some WIOA funds to align adult education activities with other core programmes, including training services and career pathways. States need to establish high quality professional development programmes to improve instruction related to the specific needs of adult learners, and to disseminate “information about models and promising practices related to such programmes” (H.R. 803, 2014,: 522). 2.3. Human resource management and professional development Human resource management is another area of programme quality improvement identified by Comings et al. (2006), encompassing staff selection, working conditions, and professional development. A clear process is needed for recruiting and hiring qualified instructional and support staff (Comings et al., 2006; TESOL, 2003). While qualifications may vary according to local educational agency requirements, staff should reflect the cultural diversity of the programme learners (TESOL, 2003). While many programmes rely on part-time staff, research suggests making adult education teaching positions full-time; this may be important to maximizing outcomes, such as educational functioning level gains or entrance into post-secondary training (Chisman, 2002). Danielson (2007) found that recruitment and hiring of K-12 teachers is facilitated by a coherent definition of good teaching, aligned with mentoring, professional development, and teacher evaluation. Bingham and Smith (2007) point to a need for more research in what contributes to teacher quality specifically in the field of adult education. Older research on adult education suggests that adult education teachers should have personal qualities that include general knowledge, sensitivity to people and good communication skills, as well as professional qualities that include a philosophy about adult education, the ability to translate that philosophy into practice, and organisational and managerial skills (Soifer et al., 1990). In terms of working conditions, programmes should provide a safe and clean working environment, with compensation and benefits commensurate with other professional staff at similar institutions (TESOL, 2003). Expectations need to be set that all teachers should continue to learn and be provided opportunities to interact and collaborate (Taylor, Smith, & Bingham, 2005). Research indicates that adult educators want to work in an environment that provides opportunities to dialogue with colleagues and directors, where they can exchange ideas and get feedback on their teaching (Marceau, 2003). One way adult education programmes can improve teachers’ working conditions is to pay teachers to attend professional development events (Smith, Hofer, Gillespie, Solomon, & Rowe, 2003). A professional development plan should meet the needs of the teachers and provide opportunities to receive training, practice new skills, and receive feedback (Comings et al., 2006). Such a plan should be created with input from staff and stakeholders and appropriate resources to implement the plan should be acquired (TESOL, 2003; Taylor et al., 2005). Much of the discussion of the effectiveness of professional development comes from K-12 research, but the need for meaningful professional development is widely accepted. Darling-Hammond (2000) suggests that policy investments in the quality of teachers, including teacher education, licensing, professional development, and hiring, may be related to improvements in student performance. Smith et al. (2003) found that quality professional development for adult education instructors is characterized by effective facilitation and group dynamics, and is designed to meet the needs of the participants. Bingham and Smith
(2007) found that adult education professional development must include activities that have been shown to increase teachers’ understanding, access, and use of research. Opportunities should be provided for teachers to examine current trends, best practices, and technology in adult education and English as a Second Language (TESOL, 2003). 2.4. The learning environment The final component of programme quality support identified by Comings et al. (2006) is a suitable environment for learning. Programmes must offer flexible hours and convenient locations, a physically safe and psychologically safe environment, and learning materials and resources relevant to adults. In his foundational work, Knowles (1973) presented a “process model” in which an adult education teacher establishes a climate for learning, establishes opportunities for mutual planning, and works with adult students to diagnose student needs, formulate objectives, design and conduct learning experiences with appropriate techniques and materials, and evaluate outcomes. To meet the needs of adult learners and the community, programmes should provide courses with flexible schedules in convenient locations (TESOL, 2003), offering classes both day and night to accommodate working adults, within the constraints of programme resources (Comings et al., 2006). According to most researchers, there are two main barriers to adult learning: internal and external. Internal barriers could include an adult student’s disposition and level of anxiety (Falasca, 2011). External barriers arise from life circumstances of adults, and could include home and job responsibilities (Cross, 1981). Programmes should make provisions for differences in style, time, place, and pace of learning (Knowles, 1973). Research demonstrates that learning materials should be up-to-date, culturally sensitive, and suitable for a variety of learning styles (TESOL, 2003). K-12 research suggests that resources may include texts, Supplementary materials, and print materials, but also may include human resources such as experts from the community (Danielson 2007). Another component of providing a suitable environment for learning is the physical environment of a programme. Adult education programmes should maintain a safe and comfortable physical environment that supports adult learning (Comings et al., 2006). The physical environment can include temperature, ventilation, access to bathrooms, adequate light, comfortable chairs, and access to refreshments (Knowles, 1973). A learning space that lacks a comfortable degree of personal space will detract from the learning process (Merriam & Brockett, 1997). Danielson (2007) stresses classroom safety and accessibility to learning. No exits should be blocked and traffic flow should be efficient. Students need access to learning resources and the teacher. Classroom furniture should be best arranged to suit the classroom activities, including group work, pair work, and class discussions. A programme should also be psychologically safe for adult learners by protecting student confidentiality and demonstrating respect for student, staff, and stakeholder cultures (Comings et al., 2006). Additionally, procedures should be established for ensuring internal and external stakeholder confidentiality (TESOL, 2003). 3. Methods Participants in this study included administrators of state funded adult education programmes in one southern state in the U. S. The adult education programmes varied in size and were located in both rural and urban areas of the state. Data were collected during the spring of 2015 using a modified version of NSCALL’s Evidence-based Program Self-assessment (2006). The original survey was designed as a self-assessment tool to be used by adult
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
education programmes to determine whether they are using principles of best practices based on research, theory, and professional wisdom. The original instrument included four components: programme quality support, entrance into a programme, participation in a programme, and reengagement in learning, and allowed the respondents to indicate whether or not they practice those principles. Respondents could also comment on each principle. The survey included multiple components in each survey item, and responses were limited to “we need help” or “we have or don’t need help.” For this study the modified survey instrument focused specifically on the first element, perceptions of programme quality support, and was revised to separate out individual principles that were combined in the original instrument. Additionally, the survey was modified to include a 5-point Likert-type scale. The instrument was pilot and field tested with eight adult education administrators who were asked to provide feedback via email regarding the clarity and readability of the questions and to make suggestions for improvement. Additional modifications were made based on their observations. The modified survey instrument was divided into five sections. In the first four sections, participants were asked to respond using a 5-point Likert-type scale, with 5 being strongly agree and 1 being strongly disagree. For the purposes of data analysis, the 5-point Likert-type scale questions were interpreted as: 1–1.49 equals “strongly disagree,” 1.5–2.49 equals “agree,” 2.5–3.49 is “neither agree or disagree,” 3.5–4.49 is “agree,” and 4.5–5.0 is “strongly agree.” Section one of the survey included four questions pertaining to the administrators’ perceptions of their centre’s role in the community. Section two included 13 questions pertaining to the participants’ perceptions of the effectiveness of the planning, evaluation, and governance of their programmes. Section three contained 12 questions related to how human resources are managed. Section four contained 11 questions related to the degree to which participants perceive they provide a suitable environment for learning for adult students in their programmes. The final section contained questions related to the demographics of individual programmes, such as whether the programme is urban or rural, and how many counties the programme serves (1–5 or more). The final question asked the participant to describe whether their Local Educational Administrator (LEA) is a public school district, a 2-year college, or other (part of a trade school). A Cronbach’s alpha performed on the completed survey resulted in a reliability of 0.883, which is above the 0.7 level of acceptability (Gliem & Gliem, 2003). All Likert-type items were included. A Cronbach’s Alpha is used to measure the internal consistency of the instrument (Cronbach, 1951). 4. Findings Of the 39 directors, 36 (92%) responded to the invitation to participate. As shown in Table 1, almost two-thirds (61.1%) described their adult education centre as rural, while the remaining described their programme as urban. One-half of the programmes served one county, while 19.4% served two counties, 13.9% served three counties, 2.8% served four counties, and 13.9% served five counties. Just over one-half (52.8) of local programmes in the state were administrated by 2-year colleges, and more than one-third (38.9%) by public school districts, with the remaining programmes part of trade or other types of schools. As shown in Table 2, responses on the first four statements on the survey related to a programme’s role in the community. The respondents agreed that their programmes are designed to meet community needs, and that they use their mission statements to help guide services and direction. The means on these four questions ranged from 4.08 to 4.36, indicating that all directors surveyed agreed or strongly agreed that they were implementing
109
Table 1 Demographic Information for Adult Education programs in Arkansas. Which best describes your program? Urban Rural Total
Frequency 14 22 36
How many counties does your program serve? 18 1 2 7 3 5 4 1 5 5 Total 36
Percent 38.9 61.1 100.0
50.0 19.4 13.9 2.8 13.9 100.0
Which best describes your Local Educational Administrator (LEA): Public School district 14 38.9 2-year College 19 52.8 Missing 3 8.3 Total 36 100.0
these practices. Standard deviation ranged from 0.639 to 0.770. The written responses suggested that directors use mission statements to help guide practice. Thirteen statements on the survey referred to a programme’s management system (Table 3). Results from the survey revealed that the respondents strongly agreed that their programmes monitor their budgets (M = 4.67, SD = 0.478), maintain accurate financial records (M = 4.64, SD = 0.639), and adhere to state and national policies (M = 4.61, SD = 0.494). They agreed that they have a planning process that facilitates programme development and accountability (M = 4.22, SD = 0.485). The survey results indicated that adult education directors perceive they have a regularly scheduled evaluation process (M = 4.19, SD = 0.749) and planning process (M = 4.06, SD = 0.674). While the programme directors surveyed perceive that they have advisory boards that are fairly representative of the local community (M = 4.08, SD = 0.604) and meet regularly (M = 4.08, SD = 0.841), responses indicated that directors had less agreement on whether the advisory boards provide direction to the programmes (M = 3.94, SD = 1.194). Respondents neither agreed or disagreed with the survey statement, “I think we have an adequate budget for our adult education programme.” This statement had the lowest mean of the 40 Likert-type questions (M = 2.58, SD = 1.402), with 31% of respondents choosing “strongly disagree,” indicating that almost a third of respondents perceive that their programmes are underfunded. Respondents also reported being less likely to seek funding for their programmes from multiple sources (M = 3.17, SD = 1.108). Twelve statements on the survey referred to human resources management, encompassing staff selection, working conditions, and professional development. The respondents agreed (M = 4.00, SD = 0.956) that they have a clear process for recruiting and hiring qualified instructional and support staff (Comings et al., 2006; TESOL, 2003). The respondents agreed that their programme’s hiring policy considers the skills, knowledge, and life experiences of applicants (M = 4.44, SD = 0.504). Results from the survey revealed that adult education directors agreed that they provide working conditions that support the work and growth of the staff (M = 4.44, SD = 0.735). They strongly agreed that they treat staff as professionals (M = 4.69, SD = 0.467), and agreed that the working conditions for faculty and staff contribute positively to student progress (M = 4.44, SD = 0.652). The survey results indicated that the directors agreed that they provide opportunities for training (M = 4.33, SD = 0.586), opportunities to practice new skills (M = 4.17, SD = 0.775), and provide staff opportunities to receive constructive feedback (M = 3.94,
110
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
Table 2 Descriptive Statistics for a Well-defined Role.
I think that our adult education program’s services are designed to meet community needs. I believe we have a clear mission statement that is used to provide direction for the adult education program. Our adult education program’s mission is used to guide most aspects of the program’s services. I believe our adult education program has a clear statement of goals that helps us meet our mission.
Mean
SD
4.36 4.25 4.14 4.08
0.639 0.692 0.683 0.770
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Table 3 Descriptive Statistics for Quality Management System.
I I I I I I I I I I I I I
feel we monitor our budget on a regular basis within our adult education program. am confident the financial records in our adult education program are accurately maintained. believe our program’s services adhere to policies set by state and national funding agencies. think our adult education program has adequate management of the financial resources. feel our program’s planning process facilitates program development and accountability. believe our program has a regularly scheduled evaluation process. believe our advisory board has regularly scheduled meetings. believe our advisory board is fairly representative of the local community. believe our program has a regularly scheduled planning process. think our adult education program uses its advisory board or board of directors to provide direction to the program. believe our advisory board properly guides the activities of the adult education program. believe we seek out fundraising opportunities from multiple sources for our adult education program. think we have an adequate budget for our adult education program.
Mean
SD
4.67 4.64 4.61 4.47 4.22 4.19 4.08 4.08 4.06 3.94 3.39 3.17 2.58
0.478 0.639 0.494 0.696 0.485 0.749 0.841 0.604 0.674 1.194 0.994 1.108 1.402
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
SD = 0.630). The directors surveyed agreed that their programmes’ professional development plans take into account the needs of the programme and staff (M = 4.19, SD = 0.624). There was a greater standard deviation related to staff salaries (M = 3.61, SD = 1.337) on the statement “Staff is provided with adequate pay and benefits,” with 11 respondents disagreeing or strongly disagreeing with that statement (Table 4). Ten statements on the survey related to providing a suitable environment for learning, which includes offering flexible hours and convenient locations, a both physically safe and psychologically safe environment, and learning materials and resources relevant to adults. Directors strongly agreed that their programmes make efforts to protect the confidentiality of students and staff (M = 4.83, SD = 0.878), and that they provide both physically safe (M = 4.56, SD = 0.652) and psychologically safe (M = 4.53, SD = 0.654) environments for students and staff, and that this supports student learning (M = 4.53, SD = 0.696). Directors were in agreement that their programmes offer flexible scheduling (M = 4.36, SD = 0.639) and convenient locations for students to learn (M = 4.36, SD = 0.639). Programme directors agreed that they maintain materials and resources appropriate for learning and for
adult students (M = 4.49, SD = 0.562), including software that was designed for use by adult students (M = 4.44, SD = 0.504) (Table 5). 5. Discussion, implications for practice, and recommendations for future research The survey results provide a current snapshot of the practices of local programmes in this southern state, and the challenges that the leaders in these programmes are facing. The single most important finding related to the funding of local programmes. Adequate funding for adult education programmes remains a concern for many directors, especially with the increased accountability measures coming from the new WIOA legislation. For example, in awarding grants to local programmes, federal WIOA legislation requires states to consider the degree to which the local programme would be responsive to regional needs; to serving individuals identified as most in need of adult education services (students with low literacy levels and English language learners); the ability of the programme to serve students with disabilities; the programme’s past effectiveness; the programme’s alignment activities; and the goals of local one-stop partners.
Table 4 Descriptive Statistics for Human Resource Management.
I I I I I I I I I I I I
believe staff within our program are treated as professionals. believe our program’s working conditions support staff improvement. believe working conditions for our faculty and staff contribute positively to student progress. think our adult education program’s hiring policy equally considers the skills, knowledge, and life experiences of applicants. believe our professional development plan provides staff opportunities to receive proper training. believe our adult education program has a clear process for hiring instructional and support staff. believe our staffs' professional development plan takes into account the needs of the program and staff. believe our professional development plan provides staff opportunities to practice new skills. feel our adult education program helps employees create a professional development plan that enables them to grow professionally. believe our adult education program has a clear process for recruiting instructional and support staff. believe our professional development plan provides staff opportunities to receive constructive feedback. believe our staff is provided with adequate pay and benefits.
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Mean
SD
4.69 4.44 4.44 4.44 4.33 4.22 4.19 4.17 4.17 4.00 3.94 3.61
0.467 0.735 0.652 0.504 0.586 0.898 0.624 0.775 0.609 0.956 0.630 1.337
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
111
Table 5 Descriptive Statistics for Adult Education Learning Environment.
I I I I I I I I I I
believe believe believe believe believe believe believe believe believe believe
our our our our our our our our our our
program makes appropriate efforts to protect the confidentiality of students and staff. program demonstrates proper respect for the cultures of students and staff. program’s physical environment is safe for staff and students. program’s physical environment supports adult learning. program environment is psychologically safe for staff and students. program uses appropriate/relevant written materials that were designed for use by adult students. program’s physical environment is comfortable for staff and students. program uses software that was designed for use by adult students. program provides instructional services at locations that are convenient for our students. program provides instructional services at hours convenient for students.
Mean
SD
4.83 4.78 4.56 4.53 4.53 4.49 4.47 4.44 4.36 4.36
0.878 0.866 0.652 0.696 0.654 0.562 0.696 0.504 0.639 0.639
Note. 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.
Other considerations impacted by funding include whether the local programme bases instruction on research-based best practices and whether technology is used to increase the amount and quality of learning. Programmes should also use well-trained instructors and administrators who have access to high quality professional development (H.R. 803, 2014). Hiring more full-time staff, as research suggests programmes should do (Chisman, 2002), is another substantial drain on already limited resources. With many states’ shrinking budgets for adult education services, meeting these considerations will prove difficult for local programmes. While this state’s Adult Education Division examines these considerations when awarding grants to local programmes, the amount available from the state budget for distribution to local programmes needs to increase to keep up with the pace of salary increases, costs, and the demand for adult basic education services. Further research could investigate how the lack of or decrease in adult basic education funding affects programme outcomes. The survey also indicated (M = 3.39, SD = 0.994) that directors are ambivalent about how advisory boards help guide the activities of their adult education programmes, indicating that they may not have a clear understanding of how boards can be useful in helping them reach their goals. An effective advisory group addresses issues related to programme administration and ensures accountability (TESOL, 2003). Board configuration may be different for every local programme and their influence and involvement of members likely varies significantly. Directors of adult education programmes might benefit from professional development focused on selecting suitable board members who are stakeholders in the programme or have some level of expertise in working with similar organisations, and effectively using the board to promote, guide, and improve their programme. Stakeholders might include business leaders, other service agency representatives, students, human resource managers, staff, and others. While this survey looks at the self-reported perceptions of directors of adult education programmes, future research might also look at the perceptions of programme quality of teachers and staff of these same programmes, and compare those perceptions to those of the administrators. While a local programme leader may feel he or she is using best practices efficiently, teachers, paraprofessionals, board members, and other stakeholders such as students and community members may see it differently. Exploring the perception of others who have a vested interest in the programme would provide a more accurate picture of areas that need attention and improvement. A limitation of this study was the small population. This study could be replicated and include adult education programme directors across the United States, yielding a broader perspective of what is occurring nationally. Additionally, a qualitative study might use focus groups and in-depth interviews to examine the perceptions of both directors and staff and the reasons behind their responses.
Quality programme support on a local level includes the existence of a well-defined role of the adult education programme, an effective local management system, a local human resource management system, and a suitable environment for learning (Comings et al., 2006). Many state assessment tools look at some or all of these areas, but without a standardized instrument, local programmes may not be judged using consistent criteria from state to state and programme to programme. There is a need for a research-based, standardized instrument that can be used as a guide for self-assessment by local programmes, and also for programme monitoring by the states, to include not only quality programme support, but also the components not addressed in this study: student entrance into a programme, participation in a programme, and reengagement in learning. A next step would be to modify and update the instrument developed by Comings et al. (2006), focusing on the three remaining areas, to create a comprehensive assessment tool that measures the four components identified in the original study.
References Bingham, M., & Smith, C. (2007). Professional development and evidence-based practice in adult education. In A. Belzer (Ed.), Toward defining and improving quality in adult basic education (pp. 69–84).New York: Routledge. Chisman, F. P. (2002). Leading from the middle: The state role in adult education & literacy. . . Retrieved from http://eric.ed.gov. Comings, J., Soricone, L., Santos, M., & National Centre for the Study of Adult Learning and Literacy (2006). An evidence-based adult education programme model appropriate for research. NCSALL occasional paper. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from http://www. ncsall.net. Cronbach, L. (1951). Cronbach alpha and the internal structure of tests. Psychmerika, 16, 297–334. Cross, K. P. (1981). Adults as learners increasing participation and facilitating learning. San Francisco: Jossey-Bass. Danielson, C. (2007). Enhancing professional practice: A framework for teaching, Alexandria, VA : Association for supervision & curriculum development. . Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives 8, 1. . Retrieved from http://www.politicalscience.uncc.edu. Falasca, M. (2011). Barriers to adult learning: Bridging the gap. Australian Journal of Adult Learning 51(3), 583–590. . Retrieved from http://files.eric.ed.gov/fulltext/ EJ954482.pdf. Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Midwest Research to Practice Conference in Adult, Continuing, and Community Education . . Retrieved December 13, 2016 https://www.google.com/webhp?sourceid=chromeinstant&ion=1&espv=2&ie=UTF-8#q=gliem%20and%20gliem%202003. Hicks, L., Hancher-Rauch, H., Vansickle, J., & Satterblom, A. (2011). Creating a programme advisory board: A key advocacy and marketing strategy for longevity in turbulent times. Journal of Physical Education Recreation & Dance, 82 (3), 46–52. Jones, D., & Kelly, P. (2007). Mounting pressures facing the U.S. workforce and the increasing need for adult education and literacy. New York, NY: National Commission on Adult Literacy. . Retrieved from http://www. nationalcommissiononadultliteracy.org/content/nchemspresentation.pdf. Knowles, M. (1973). The adult learner: A neglected species. Houston TX: Gulf 1990. Marceau, G. (2003). Professional development in adult basic education. New Directions for Adult and Continuing Education, 98, 67–74.
112
G. Udouj et al. / Evaluation and Program Planning 61 (2017) 106–112
McLendon, L., & Polis, K. (2009). A Resource packet for using stakeholder assessments to identify strengths and needs. Syracuse, NY: NAEPDC- Proliteracy America Joint Initiative. . Retrieved from http://lea.proliteracy.org/external/lpl/download/ Workshop%201%20Participant%20Pa cket%20Rev%207-12. pdf. Merriam, S. B., & Brockett, R. G. (1997). The profession and practice of adult education: An introduction. San Francisco: Jossey-Bass. Smith, C., Hofer, J., Gillespie, M., Solomon, M., & Rowe, K. (2003). How teachers change: A study of professional development in adult education. NCSALL reports number 25. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from www.proquest.com. Soifer, R., Irwin, M., Crumrine, B., Honzaki, E., Simmons, B., & Young, D. (1990). The complete theory-to-practice handbook of adult literacy: Curriculum design and teaching approaches. Columbia University, NY: Teachers College Press. Teachers of English to Speakers of Other Languages (TESOL) (2003). Standards for adult education ESOL programs. Alexandria, VA: Teachers of English to Speakers of other Languages, Inc.. Taylor, J., Smith, C., & Bingman, B. (2005). Programme administrators’ sourcebook. National Centre for the Study of Adult Learning and Literacy (NCSALL). . Retrieved from http://lea.proliteracy.org. U.S. Department of Education (2014). The workforce innovation and opportunity act, overview of title II: Adult education and literacy. . . Retrieved from http://www2. ed.gov/about/offices/list/ovae/pi/AdultEd/wioa-overview.pdf. Workforce Investment Act (WIA) (1998). Public law. 105–220. Workforce Investment, & Opportunity Act (WIOA) (2014). H.R. 803, 113th Cong. (2014). . Gary Udouj, Jr. is the Director of Adult Education for the Fort Smith Public Schools Fort Smith, Arkansas. He has served as chair of the Arkansas Adult Education Advisory Board and President of the Arkansas Association of Administrators of Adult
Education. He is the recipient of the 2015 Outstanding Doctoral Student in Adult and Lifelong Learning at the College of Education and Health Professions at the University of Arkansas.
Kenda Grover is an assistant professor in the Adult and Lifelong Learning Program at the University of Arkansas. She serves as the program coordinator for the master of education degree in ADLL. Her research interests include the professional development of faculty members who teach in online programs, leadership in adult education programs, self-directed learning and healthcare, and communities of practice in nonformal learning environments.
Greg Belcher is a professor in the Department of Technology and Workforce Learning and the director of the Kansas Center for Career and Technical Education at Pittsburg State University. His research interests includes the retention of new Career and Technical Education instructors, how to use student assessment outcomes in making instructional decisions and the evaluation of Career and Technical Education programs.
Kit Kacirek is an associate professor in the Adult and Lifelong Learning Program at the University of Arkansas. Her research agenda focuses on bridging the gap between theory and practice in learning theory and leadership development. Dr. Kacirek also serves as the program coordinator for the doctoral program in Adult and Lifelong Learning.