- Email: [email protected]
Participatory Evaluation: An Intended Catalyst to Learning within University
Available online at www.sciencedirect.com
ScienceDirect Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
CIEA 2014
Participatory evaluation: An intended catalyst to learning within university Lucia Raţiua*, Sofia Chiricăa, Claudia Lenuța Rusa a
Department of Psychology, Babes-Bolyai University, 1 Kogalniceanu, Cluj-Napoca, 400084, Romania
Abstract Based on the meaning of participative evaluation as a collaborative inquiry process, the purpose of this study was to investigate the evaluation-related communication among the internal stakeholders of higher education process. The study employed a quantitative design with 235 participants from a Romanian public university. Data about evaluation-related communication were collected by a survey consisted of three main dimensions: the communication initiative, communication partners and the status of the persons participating to such a communication. The findings indicate communication-initiating vs. communicationresponding role differences among participants, in the process of participatory evaluation of the quality of education programs, as characteristics of a collaborative inquiry process aiming to learning and change. © 2014 Elsevier Ltd. This is an open access article under the CC BY-NC-ND license © 2014 The Authors. Published by Elsevier Ltd. (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Alexandru Ioan Cuza University. Peer-review under responsibility of the Alexandru Ioan Cuza University. Keywords: participatory evaluation; organizational communication; organizational learning; stakeholders.
1. Introduction Participatory evaluation (PE) is a collaborative inquiry process among the emerging and innovative evaluation approaches such as collaborative, democratic-deliberative, empowerment, fourth-generation, inclusive and utilization-focused, explicitly endorsing the principle of stakeholder participation (Daigneault, Jacob, & Tremblay, 2012). Involving stakeholders in the evaluation process of planned change has become generally accepted within the evaluation community lately (Cousins 2003; Cullen, Coryn, & Rugh, 2011; Daigneault & Jacob, 2009; Mark 2001;
* Corresponding author. Tel.:+0-264-590197; fax.:+0-264-590197 E-mail address: [email protected]
1877-0428 © 2014 Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Alexandru Ioan Cuza University. doi:10.1016/j.sbspro.2014.07.626
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
141
Mathison, 2005; Poth & Shulha, 2008; Cousins & Whitmore, 1998). The benefits of stakeholder participation in evaluation are numerous, including creating opportunities for pooling resources, information, and data; anticipating reactions and problems; giving legitimacy and credibility to final decisions; and sharing risk and responsibility (Brandon, 1998; Campbell & Mark, 2006; Grant & Curtis, 2004; Orr, 2010; Posavac & Carey, 2007). Given the mentioned benefits, within the field of strategic change management, there is some evidence about a positive connection between participation and success of organizational change (Lines, 2005; Saksvik, Tvedt, Nytro, Andersen, Andersen, Buvik, & Torvatn, 2007). Although active client participation has been critical to organization development theory (Argyris, 1960; 1990; French & Bell, 1995) and process consulting (Schein, 1999), relatively little was studied how PE and the evaluation-related communication took place in the higher education organizations. The aim of the study was to investigate the evaluation-related communication among the main internal stakeholders of higher education process: students, academics and university administrators. 2. Theoretical framework The main distinctive feature of PE is the creation of a dynamic process by the stakeholders, through which the social production of knowledge occurs, contributing to a collective conception of learning about themselves, the organization in which they are involved and ultimately the essential characteristics underlying the phenomena being evaluated (Suárez-Herrera, Springett & Kagan, 2009). This dialectical process consists of not only the consensus between different and often conflicting stakeholder perspectives, but also the development of a set of ongoing practices based on mutual interaction, cooperation, dialogue and negotiation (Cousins & Whitmore, 1998; Garaway, 1995; Gregory, 2000; Rebien, 1996; Springett, 2001; Suárez-Herrera et al., 2009). Going beyond the reconciliation of differences, an interactive learning environment is created in order to provide a common perspective for all the stakeholders involved in the evaluative process (Billett, 2004; Garaway, 1995). Stakeholders are therefore continually engaged in a collective articulation of action working together, regardless of their organizational position, having creative capacities that allow them to get involved in a collective experience with all the senses, emotions and personal experiences that they bring with them (Garaway, 1995). Consequently, such an ongoing inquiry allows for both examination of underlying assumptions and dialogue (Garaway, 1995; Springett, 2001; Torres & Preskill, 2001) and has a profound impact on the ways in which stakeholders use their resources. This praxis implies a continual communication process integrated with the symbolic, physical and social infrastructures of the organization, through which its members negotiate their different values, attitudes and perceptions. Moreover, a sustainable network of stakeholders working together through communicative actions and supportive partnerships may come up and lead towards a political articulation of action (Suárez-Herrera et al., 2009). The emergence of sustainable networks together with the partnerships developed among stakeholders rely on communication processes through which participants learn to reflect on their own experiences, mutual interactions and shared information (Brisolara, 1998; Burke, 1998; Cousins & Whitmore, 1998; Garaway, 1995; Springett, 2001). Nonetheless, House and Howe’s (1999) critique regarding the power relations among the stakeholders moreover when these are internal stakeholders should be considered. Given the fact that the PE process is grounded in social and institutional authority structures within particular socio-political systems that inevitably influence the actors’ involvement and their practices, the power relations may jeopardize the inclusion of all contributions in a deliberative process (House, 2004; House & Howe, 1999). Internal stakeholders may have multiple value orientations depending on the position they have within organization. For example, program managers may have concerns about opening decision making to other members, including a general reluctance to work with others, fear of criticism or conflict, and reluctance to be transparent. On the other hand, other members within organization may have personal issues with other participants, may lack sufficient expertise, or may not have the time to commit to the PE process (Banta, 2005; Posavac & Carey, 2007). Nevertheless, the communicative dimension of the participatory evaluation which comes from the sustained engagement of stakeholders in the organizational context of the evaluation process could be considered a precursor of a desired change process (Lines, 2005). Considering the conceptualization of PE and the characteristics of the internal stakeholders as there were presented above, the present study investigates the evaluation-related communication within university as an
142
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
important component of the PE process of the professional learning and a precursor of planned change and development. 3. Method This study employed a quantitative design. Yet, the framework for the measurement instrument has not been empirically validated, being exploratory and specifically designed for this research. 3.1. Participants The research objective mentioned above was empirically studied using a sample of 235 participants, all members of a Romanian public university (N = 100 academics; N = 33 university administrators; N = 102 students) seen as main internal stakeholders of the higher education process. Both the students and the professors come from a wide range of faculties within university. 3.2. Instrument and procedure The measurement instrument was specifically developed for the purpose of this empirical research and consists of three main dimensions of the evaluation-related communication: the communication initiative, communication partners and the status of the persons participating to such a communication. The number of items was eight – the first two with two subdimensions and the other six with three subdimensions. The participants were asked to rate the frequency of the action the items refer to, on a 5-point Likert scale ranging from 1 = never to 5 = very often. In terms of content, the items addressed the quality of the professional learning within university, more specific the evaluation of the plan of a study program, the evaluation of learning methods and of study programs. The items were adapted to the participant’s status within university, resulting three forms of the scale. Example of an item for a student-participant: How often do you initiate discussions regarding the plan of the study program you are attending, with a) your colleagues (other students); b) academics; c) university administrators. The same item for an academicparticipant is: How often do you initiate discussions regarding the plan of the study program you are teaching, with a) your colleagues (other academics); b) students; c) university administrators. And, finally, for an university administrator-participant, the same item has the following form: How often do you initiate discussions regarding the plan of a study program, with a) your colleagues (other university administrators); b) academics; c) students. 4. Results and discussions Table 1 displays the means, standard deviations, and correlation among the main study variables. The data were simultaneously analyzed based on three main dimensions, namely the communication initiative, communication partners and the status of the persons participating to such a communication. Each of the dimensions was treated from the perspective of the participant’s status. The bivariate results indicate that response measures correlate positively and significantly with initiative measures among each of the three categories of participants. With regard to the deeper understanding of the process of participatory evaluation of professional learning within university, additional analyses were conducted. These additional analyses were guided by three important questions: 1) Who are the best initiators of the evaluation-related communication? 2) Who are the most preferred participants in the evaluation-related communication? 3) Who responds to the initiative of evaluation-related communication? Table 1. Means, standard deviations (SD), and bivariate correlations for the study variables Variable
Mean
SD
3.00
0.64
University administrators 1. Response
r
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145 2. Initiative
3.16
0.68
1. Response
2.81
0.79
2. Initiative
2.95
0.76
1. Response
2.41
0.61
2. Initiative
2.28
0.63
.775**
Academics .850**
Students .850**
N = 33 university administrators; N = 100 academics; N = 102 students. **p < .01
4.1 Who are the best initiators of the evaluation-related communication? The statistical analyses based on ANOVA and Hochberg’s post-hoc test show that university administrators and academics both respond to and initiate the evaluation-related communication at a larger extent than the students (F(2, 510) = 24.73, p < .001, η2 = .30; F(2, 505) = 65.94, p < .001, η2 = .45). Moreover, unlike the students, the academics and the university administrators see themselves more as initiators than receptors of the communication about evaluation (Figure 1).
Fig. 1. Mean distribution of the variables (response to and initiative in evaluation-related communication) among the participants
4.2. Who are the most preferred participants in the evaluation-related communication? The data indicate that the participants prefer to address the evaluation issues to persons of the same status (F(2, 510) = 6.30, p < .005, η2 = .15) and there is a difference statistically significant between students and academics. Further analyses indicate that the university administrators see their colleagues and the academics as partners of evaluation-related communication, but interact less with the students in this matter. Besides their colleagues, the academics report that they initiate discussion on evaluation topics more with students than with university administrators (F(2, 505) = 129.50, p < .001, η2 = .58).
143
144
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
4.3 Who respond to the initiative of evaluation-related communication? In terms of response, similar to initiative, the results show that the university administrators and the academics are more open to the evaluation-related communication initiated by persons of the same status. On the other hand, the students are more responsive to the evaluation-related communication coming from the academics comparatively to the university administrators (F(2, 505) = 90.83 p < .001, η2 = .50). However the university administrators respond more often than the academics to the discussions on evaluation initiated by the students. In summary, the data analyses outline directions of communication among the main internal stakeholders of the educational process and highlight the dynamics of the process focusing upon the evaluation of the professional learning within university. 5. Discussions and conclusions The main aim of the study was to investigate the participatory evaluation as a collaborative inquiry process among the internal stakeholders within university. The previous literature often asserts the benefits of the participative processes for the organization change (Lines, 2005; Saksvik et al., 2007) and for the stakeholders (Brandon, 1998; Campbell & Mark, 2006; Grant & Curtis, 2004; Orr, 2010; Posavac & Carey, 2007). However, there is limited empirical evidence of how these organizational outcomes are achieved. In this respect, the present study provides important contributions by describing the status differences (teacher, student, and administrator) in communication initiating vs. communication responding-role in participatory evaluation. First, the research underlines the communicative nature of the evaluative practice within university which is focused not only on the results, but also on the evaluation process itself as a component of the current professional practices. Second, the evaluation-related communication was explored and described from the perspective of the main internal stakeholders, all of them willing to investigate the quality of professional learning within university, despite the diversity in perceptions, values and interests given by their status. Consequently, stakeholders are not viewed exclusively as sources of evaluation data but also as important collaborators in the evaluation process. These results are encouraging insofar as they empirically offer suggestions about active involvement of the major stakeholders in a collaborative inquiry approach which can facilitate the development of a holistic process of intentional change. More specific, the study offers support for participatory evaluation as one of the challenges facing the universities in order to prepare the students for the unpredictability and complex environment outside the university. Higher education institutions need to be both stable and flexible in order to successfully adapt to these internal and external demands in a judicious way. It is critical for them to become learning organizations in which learning occurs in individuals, teams and throughout the organization (Watkins & Marsick, 1993). In the light of this perspective, participatory evaluation has gained not only popularity, but also it has been acknowledged as an organizational learning tool (Torres & Preskill, 2001). Finally, the present study indicates the quality of the organizational communication within the university as a fundamental process of the collaborative inquiry aiming to the organizational learning. This issue is particularly challenging in terms of a culture in which the evaluation is focused not only on the results, but also on the evaluation process itself as a component of the current professional practices oriented to learning at individual, group and organizational level. In these terms, participatory evaluation can facilitate teacher leadership, enhance professionalism and improve the quality of professional learning within university. Besides the contributions, the study has several limitations which should be approached and mitigated in the future study. First, the instrument was specifically designed for this study and it was not validated. Anyway, it provided a quite good description and insight of the evaluation-related communication among the stakeholders within university. The second limitation is the possibility of a common method bias due to the use of only selfreported scale items to measure the research constructs. In conclusions, notwithstanding the need for further research into participatory evaluation within university, the present research could be relevant both for theoretical and practical implications in the field.
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
145
References Argyris, C. (1960). Overcoming organizational defenses: Facilitating organizational learning. Needham, MA: Allyn & Bacon. Argyris, C. (1960). Understanding organizational behavior. Homewood, IL: Dorsey Press. Banta, T. W. (2005). Leaders’ views on engaging stakeholders in assessment. Assessment Update, 17, 3-16. Billett, S. (2004). Workplace Participatory Practices: Conceptualising Workplaces as Learning Environments. Journal of Workplace Learning 16(6), 312–24. Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325-337. Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19(3), 325. Brisolara, S. (1998). The history of participatory evaluation and current debates in the field. In E. Whitmore (Ed.), Understanding and practicing participatory evaluation (pp. 25–41). New Directions for Evaluation, 80. San Francisco, CA: Jossey-Bass. Brisolara, S. (1998). The History of Participatory Evaluation and Current Debates in the Field. New Directions for Evaluation 80, 25–41. Burke, B. (1998). Evaluating for a Change: Reflections on Participatory Methodology. New Directions for Evaluation 80, 43–56. Campbell, B., & Mark, M. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834-2863. Cousins, J. B. (2003). Utilization efforts of participatory evaluation. In T. Kellaghan & D. L. Stufflebeam (eds.), International handbook of educational evaluation (pp. 245-266). Dordrecht, UK: Kluwer. Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 5–23. doi: 10.1002/ev.1114. Cullen, A. E., Coryn, C. L. S., & Rugh, J. (2011). The politics and consequences of including stakeholders in international d evelopment evaluation. American Journal of Evaluation, 32(3), 345-361. Daigneault, P.-M., Jacob, S. & Tremblay, J. (2012). Measuring stakeholder participation in evaluation: An empirical validation of the Participatory Evaluation Measurement Instrument. Evaluation Review, 36(4), 243-270. Daigneault, S. M., & Jacob, S. (2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30, 330-348. French, W.L., & Bell C.H., Jr. (1995). Organization Development: Behavioral Science Interventions for Organization Improvement (5th ed.). Englewood Cliffs: Prentice Hall. Garaway, G. B. (1995). Participatory Evaluation. Studies in Educational Evaluation 21(1), 85–102. Grant, A., & Curtis, A. (2004). Refining evaluation criteria for public participation using stakeholder perspectives of process and outcomes. Rural Society, 14, 142-162. Gregory, A. (2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179-199. House, E.R. & Howe, K.R. (1999). Values in Evaluation and Social Research. Thousand Oaks, CA: SAGE. House, E.R. (2004). The role of the evaluator in a political world. The Canadian Journal of Program Evaluation 19(2), 1–16. Lines, R. (2005). How Social Accounts and Participation during Change Affect Organizational Learning. Journal of Workplace Learning 17(3), 157–77. Mark, M. (2001). Evaluation’s future: Furor, futile, or fertile? American Journal of Evaluation, 22, 457-479. Mathison, S. (2005). Encyclopedia of evaluation. Thousand Oaks, CA: Sage. Orr, S.K. (2010). Exploring Stakeholder Values and Interests in Evaluation. American Journal of Evaluation, 23, 209-218. Posavac, E. J., & Carey, R. G. (2007). Program evaluation: Methods and case studies. Upper Saddle River, NJ: Pearson Education, Inc. Poth, C., & Shulha, L. (2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34(4), 218–223. Rebien, C. C. (1996). Participatory evaluation of development assistance: Dealing with power and facilitative learning. Evaluation, 2, 151-171. Saksvik, P.O., Tvedt, S.D., Nytro, K., Andersen, G.R., Andersen, T.K., Buvik, M.P., & Torvatn, H. (2007) Developing criteria for healthy organizational change. Work Stress, 21(3), 243-263. Schein, E.H. (1999). Process consultation revisited: Building the helping relationship. Reading, MA: Addison-Wesley. Springett, J. (2001). Participatory Approaches to Evaluation in Health Promotion. In I. Rootman, M. Goodstadt, B. Hyndman, et al. (eds) Evaluation in Health Promotion: Principles and Perspectives (pp. 83–105). Copenhagen: World Health Organization. Suárez-Herrera, J., Springett, J., & Kagan, C. (2009). Critical connections between participatory evaluation, organizational learning and intentional change in pluralistic organizations. Evaluation 15(3), 321–42. Torres, R. T. & Preskill, H. (2001). Evaluation and Organizational Learning: Past, Present, and Future. American Journal of Evaluation 22(3), 387–95. Watkins, K. E., & Marsick, V. J. (1993). Sculpting the learning organization: Lessons in the art and science of systemic change. San Francisco: Jossey-Bass.
ScienceDirect Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
CIEA 2014
Participatory evaluation: An intended catalyst to learning within university Lucia Raţiua*, Sofia Chiricăa, Claudia Lenuța Rusa a
Department of Psychology, Babes-Bolyai University, 1 Kogalniceanu, Cluj-Napoca, 400084, Romania
Abstract Based on the meaning of participative evaluation as a collaborative inquiry process, the purpose of this study was to investigate the evaluation-related communication among the internal stakeholders of higher education process. The study employed a quantitative design with 235 participants from a Romanian public university. Data about evaluation-related communication were collected by a survey consisted of three main dimensions: the communication initiative, communication partners and the status of the persons participating to such a communication. The findings indicate communication-initiating vs. communicationresponding role differences among participants, in the process of participatory evaluation of the quality of education programs, as characteristics of a collaborative inquiry process aiming to learning and change. © 2014 Elsevier Ltd. This is an open access article under the CC BY-NC-ND license © 2014 The Authors. Published by Elsevier Ltd. (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Alexandru Ioan Cuza University. Peer-review under responsibility of the Alexandru Ioan Cuza University. Keywords: participatory evaluation; organizational communication; organizational learning; stakeholders.
1. Introduction Participatory evaluation (PE) is a collaborative inquiry process among the emerging and innovative evaluation approaches such as collaborative, democratic-deliberative, empowerment, fourth-generation, inclusive and utilization-focused, explicitly endorsing the principle of stakeholder participation (Daigneault, Jacob, & Tremblay, 2012). Involving stakeholders in the evaluation process of planned change has become generally accepted within the evaluation community lately (Cousins 2003; Cullen, Coryn, & Rugh, 2011; Daigneault & Jacob, 2009; Mark 2001;
* Corresponding author. Tel.:+0-264-590197; fax.:+0-264-590197 E-mail address: [email protected]
1877-0428 © 2014 Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer-review under responsibility of the Alexandru Ioan Cuza University. doi:10.1016/j.sbspro.2014.07.626
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
141
Mathison, 2005; Poth & Shulha, 2008; Cousins & Whitmore, 1998). The benefits of stakeholder participation in evaluation are numerous, including creating opportunities for pooling resources, information, and data; anticipating reactions and problems; giving legitimacy and credibility to final decisions; and sharing risk and responsibility (Brandon, 1998; Campbell & Mark, 2006; Grant & Curtis, 2004; Orr, 2010; Posavac & Carey, 2007). Given the mentioned benefits, within the field of strategic change management, there is some evidence about a positive connection between participation and success of organizational change (Lines, 2005; Saksvik, Tvedt, Nytro, Andersen, Andersen, Buvik, & Torvatn, 2007). Although active client participation has been critical to organization development theory (Argyris, 1960; 1990; French & Bell, 1995) and process consulting (Schein, 1999), relatively little was studied how PE and the evaluation-related communication took place in the higher education organizations. The aim of the study was to investigate the evaluation-related communication among the main internal stakeholders of higher education process: students, academics and university administrators. 2. Theoretical framework The main distinctive feature of PE is the creation of a dynamic process by the stakeholders, through which the social production of knowledge occurs, contributing to a collective conception of learning about themselves, the organization in which they are involved and ultimately the essential characteristics underlying the phenomena being evaluated (Suárez-Herrera, Springett & Kagan, 2009). This dialectical process consists of not only the consensus between different and often conflicting stakeholder perspectives, but also the development of a set of ongoing practices based on mutual interaction, cooperation, dialogue and negotiation (Cousins & Whitmore, 1998; Garaway, 1995; Gregory, 2000; Rebien, 1996; Springett, 2001; Suárez-Herrera et al., 2009). Going beyond the reconciliation of differences, an interactive learning environment is created in order to provide a common perspective for all the stakeholders involved in the evaluative process (Billett, 2004; Garaway, 1995). Stakeholders are therefore continually engaged in a collective articulation of action working together, regardless of their organizational position, having creative capacities that allow them to get involved in a collective experience with all the senses, emotions and personal experiences that they bring with them (Garaway, 1995). Consequently, such an ongoing inquiry allows for both examination of underlying assumptions and dialogue (Garaway, 1995; Springett, 2001; Torres & Preskill, 2001) and has a profound impact on the ways in which stakeholders use their resources. This praxis implies a continual communication process integrated with the symbolic, physical and social infrastructures of the organization, through which its members negotiate their different values, attitudes and perceptions. Moreover, a sustainable network of stakeholders working together through communicative actions and supportive partnerships may come up and lead towards a political articulation of action (Suárez-Herrera et al., 2009). The emergence of sustainable networks together with the partnerships developed among stakeholders rely on communication processes through which participants learn to reflect on their own experiences, mutual interactions and shared information (Brisolara, 1998; Burke, 1998; Cousins & Whitmore, 1998; Garaway, 1995; Springett, 2001). Nonetheless, House and Howe’s (1999) critique regarding the power relations among the stakeholders moreover when these are internal stakeholders should be considered. Given the fact that the PE process is grounded in social and institutional authority structures within particular socio-political systems that inevitably influence the actors’ involvement and their practices, the power relations may jeopardize the inclusion of all contributions in a deliberative process (House, 2004; House & Howe, 1999). Internal stakeholders may have multiple value orientations depending on the position they have within organization. For example, program managers may have concerns about opening decision making to other members, including a general reluctance to work with others, fear of criticism or conflict, and reluctance to be transparent. On the other hand, other members within organization may have personal issues with other participants, may lack sufficient expertise, or may not have the time to commit to the PE process (Banta, 2005; Posavac & Carey, 2007). Nevertheless, the communicative dimension of the participatory evaluation which comes from the sustained engagement of stakeholders in the organizational context of the evaluation process could be considered a precursor of a desired change process (Lines, 2005). Considering the conceptualization of PE and the characteristics of the internal stakeholders as there were presented above, the present study investigates the evaluation-related communication within university as an
142
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
important component of the PE process of the professional learning and a precursor of planned change and development. 3. Method This study employed a quantitative design. Yet, the framework for the measurement instrument has not been empirically validated, being exploratory and specifically designed for this research. 3.1. Participants The research objective mentioned above was empirically studied using a sample of 235 participants, all members of a Romanian public university (N = 100 academics; N = 33 university administrators; N = 102 students) seen as main internal stakeholders of the higher education process. Both the students and the professors come from a wide range of faculties within university. 3.2. Instrument and procedure The measurement instrument was specifically developed for the purpose of this empirical research and consists of three main dimensions of the evaluation-related communication: the communication initiative, communication partners and the status of the persons participating to such a communication. The number of items was eight – the first two with two subdimensions and the other six with three subdimensions. The participants were asked to rate the frequency of the action the items refer to, on a 5-point Likert scale ranging from 1 = never to 5 = very often. In terms of content, the items addressed the quality of the professional learning within university, more specific the evaluation of the plan of a study program, the evaluation of learning methods and of study programs. The items were adapted to the participant’s status within university, resulting three forms of the scale. Example of an item for a student-participant: How often do you initiate discussions regarding the plan of the study program you are attending, with a) your colleagues (other students); b) academics; c) university administrators. The same item for an academicparticipant is: How often do you initiate discussions regarding the plan of the study program you are teaching, with a) your colleagues (other academics); b) students; c) university administrators. And, finally, for an university administrator-participant, the same item has the following form: How often do you initiate discussions regarding the plan of a study program, with a) your colleagues (other university administrators); b) academics; c) students. 4. Results and discussions Table 1 displays the means, standard deviations, and correlation among the main study variables. The data were simultaneously analyzed based on three main dimensions, namely the communication initiative, communication partners and the status of the persons participating to such a communication. Each of the dimensions was treated from the perspective of the participant’s status. The bivariate results indicate that response measures correlate positively and significantly with initiative measures among each of the three categories of participants. With regard to the deeper understanding of the process of participatory evaluation of professional learning within university, additional analyses were conducted. These additional analyses were guided by three important questions: 1) Who are the best initiators of the evaluation-related communication? 2) Who are the most preferred participants in the evaluation-related communication? 3) Who responds to the initiative of evaluation-related communication? Table 1. Means, standard deviations (SD), and bivariate correlations for the study variables Variable
Mean
SD
3.00
0.64
University administrators 1. Response
r
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145 2. Initiative
3.16
0.68
1. Response
2.81
0.79
2. Initiative
2.95
0.76
1. Response
2.41
0.61
2. Initiative
2.28
0.63
.775**
Academics .850**
Students .850**
N = 33 university administrators; N = 100 academics; N = 102 students. **p < .01
4.1 Who are the best initiators of the evaluation-related communication? The statistical analyses based on ANOVA and Hochberg’s post-hoc test show that university administrators and academics both respond to and initiate the evaluation-related communication at a larger extent than the students (F(2, 510) = 24.73, p < .001, η2 = .30; F(2, 505) = 65.94, p < .001, η2 = .45). Moreover, unlike the students, the academics and the university administrators see themselves more as initiators than receptors of the communication about evaluation (Figure 1).
Fig. 1. Mean distribution of the variables (response to and initiative in evaluation-related communication) among the participants
4.2. Who are the most preferred participants in the evaluation-related communication? The data indicate that the participants prefer to address the evaluation issues to persons of the same status (F(2, 510) = 6.30, p < .005, η2 = .15) and there is a difference statistically significant between students and academics. Further analyses indicate that the university administrators see their colleagues and the academics as partners of evaluation-related communication, but interact less with the students in this matter. Besides their colleagues, the academics report that they initiate discussion on evaluation topics more with students than with university administrators (F(2, 505) = 129.50, p < .001, η2 = .58).
143
144
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
4.3 Who respond to the initiative of evaluation-related communication? In terms of response, similar to initiative, the results show that the university administrators and the academics are more open to the evaluation-related communication initiated by persons of the same status. On the other hand, the students are more responsive to the evaluation-related communication coming from the academics comparatively to the university administrators (F(2, 505) = 90.83 p < .001, η2 = .50). However the university administrators respond more often than the academics to the discussions on evaluation initiated by the students. In summary, the data analyses outline directions of communication among the main internal stakeholders of the educational process and highlight the dynamics of the process focusing upon the evaluation of the professional learning within university. 5. Discussions and conclusions The main aim of the study was to investigate the participatory evaluation as a collaborative inquiry process among the internal stakeholders within university. The previous literature often asserts the benefits of the participative processes for the organization change (Lines, 2005; Saksvik et al., 2007) and for the stakeholders (Brandon, 1998; Campbell & Mark, 2006; Grant & Curtis, 2004; Orr, 2010; Posavac & Carey, 2007). However, there is limited empirical evidence of how these organizational outcomes are achieved. In this respect, the present study provides important contributions by describing the status differences (teacher, student, and administrator) in communication initiating vs. communication responding-role in participatory evaluation. First, the research underlines the communicative nature of the evaluative practice within university which is focused not only on the results, but also on the evaluation process itself as a component of the current professional practices. Second, the evaluation-related communication was explored and described from the perspective of the main internal stakeholders, all of them willing to investigate the quality of professional learning within university, despite the diversity in perceptions, values and interests given by their status. Consequently, stakeholders are not viewed exclusively as sources of evaluation data but also as important collaborators in the evaluation process. These results are encouraging insofar as they empirically offer suggestions about active involvement of the major stakeholders in a collaborative inquiry approach which can facilitate the development of a holistic process of intentional change. More specific, the study offers support for participatory evaluation as one of the challenges facing the universities in order to prepare the students for the unpredictability and complex environment outside the university. Higher education institutions need to be both stable and flexible in order to successfully adapt to these internal and external demands in a judicious way. It is critical for them to become learning organizations in which learning occurs in individuals, teams and throughout the organization (Watkins & Marsick, 1993). In the light of this perspective, participatory evaluation has gained not only popularity, but also it has been acknowledged as an organizational learning tool (Torres & Preskill, 2001). Finally, the present study indicates the quality of the organizational communication within the university as a fundamental process of the collaborative inquiry aiming to the organizational learning. This issue is particularly challenging in terms of a culture in which the evaluation is focused not only on the results, but also on the evaluation process itself as a component of the current professional practices oriented to learning at individual, group and organizational level. In these terms, participatory evaluation can facilitate teacher leadership, enhance professionalism and improve the quality of professional learning within university. Besides the contributions, the study has several limitations which should be approached and mitigated in the future study. First, the instrument was specifically designed for this study and it was not validated. Anyway, it provided a quite good description and insight of the evaluation-related communication among the stakeholders within university. The second limitation is the possibility of a common method bias due to the use of only selfreported scale items to measure the research constructs. In conclusions, notwithstanding the need for further research into participatory evaluation within university, the present research could be relevant both for theoretical and practical implications in the field.
Lucia Raţiu et al. / Procedia - Social and Behavioral Sciences 142 (2014) 140 – 145
145
References Argyris, C. (1960). Overcoming organizational defenses: Facilitating organizational learning. Needham, MA: Allyn & Bacon. Argyris, C. (1960). Understanding organizational behavior. Homewood, IL: Dorsey Press. Banta, T. W. (2005). Leaders’ views on engaging stakeholders in assessment. Assessment Update, 17, 3-16. Billett, S. (2004). Workplace Participatory Practices: Conceptualising Workplaces as Learning Environments. Journal of Workplace Learning 16(6), 312–24. Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325-337. Brandon, P. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19(3), 325. Brisolara, S. (1998). The history of participatory evaluation and current debates in the field. In E. Whitmore (Ed.), Understanding and practicing participatory evaluation (pp. 25–41). New Directions for Evaluation, 80. San Francisco, CA: Jossey-Bass. Brisolara, S. (1998). The History of Participatory Evaluation and Current Debates in the Field. New Directions for Evaluation 80, 25–41. Burke, B. (1998). Evaluating for a Change: Reflections on Participatory Methodology. New Directions for Evaluation 80, 43–56. Campbell, B., & Mark, M. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834-2863. Cousins, J. B. (2003). Utilization efforts of participatory evaluation. In T. Kellaghan & D. L. Stufflebeam (eds.), International handbook of educational evaluation (pp. 245-266). Dordrecht, UK: Kluwer. Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 5–23. doi: 10.1002/ev.1114. Cullen, A. E., Coryn, C. L. S., & Rugh, J. (2011). The politics and consequences of including stakeholders in international d evelopment evaluation. American Journal of Evaluation, 32(3), 345-361. Daigneault, P.-M., Jacob, S. & Tremblay, J. (2012). Measuring stakeholder participation in evaluation: An empirical validation of the Participatory Evaluation Measurement Instrument. Evaluation Review, 36(4), 243-270. Daigneault, S. M., & Jacob, S. (2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30, 330-348. French, W.L., & Bell C.H., Jr. (1995). Organization Development: Behavioral Science Interventions for Organization Improvement (5th ed.). Englewood Cliffs: Prentice Hall. Garaway, G. B. (1995). Participatory Evaluation. Studies in Educational Evaluation 21(1), 85–102. Grant, A., & Curtis, A. (2004). Refining evaluation criteria for public participation using stakeholder perspectives of process and outcomes. Rural Society, 14, 142-162. Gregory, A. (2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179-199. House, E.R. & Howe, K.R. (1999). Values in Evaluation and Social Research. Thousand Oaks, CA: SAGE. House, E.R. (2004). The role of the evaluator in a political world. The Canadian Journal of Program Evaluation 19(2), 1–16. Lines, R. (2005). How Social Accounts and Participation during Change Affect Organizational Learning. Journal of Workplace Learning 17(3), 157–77. Mark, M. (2001). Evaluation’s future: Furor, futile, or fertile? American Journal of Evaluation, 22, 457-479. Mathison, S. (2005). Encyclopedia of evaluation. Thousand Oaks, CA: Sage. Orr, S.K. (2010). Exploring Stakeholder Values and Interests in Evaluation. American Journal of Evaluation, 23, 209-218. Posavac, E. J., & Carey, R. G. (2007). Program evaluation: Methods and case studies. Upper Saddle River, NJ: Pearson Education, Inc. Poth, C., & Shulha, L. (2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34(4), 218–223. Rebien, C. C. (1996). Participatory evaluation of development assistance: Dealing with power and facilitative learning. Evaluation, 2, 151-171. Saksvik, P.O., Tvedt, S.D., Nytro, K., Andersen, G.R., Andersen, T.K., Buvik, M.P., & Torvatn, H. (2007) Developing criteria for healthy organizational change. Work Stress, 21(3), 243-263. Schein, E.H. (1999). Process consultation revisited: Building the helping relationship. Reading, MA: Addison-Wesley. Springett, J. (2001). Participatory Approaches to Evaluation in Health Promotion. In I. Rootman, M. Goodstadt, B. Hyndman, et al. (eds) Evaluation in Health Promotion: Principles and Perspectives (pp. 83–105). Copenhagen: World Health Organization. Suárez-Herrera, J., Springett, J., & Kagan, C. (2009). Critical connections between participatory evaluation, organizational learning and intentional change in pluralistic organizations. Evaluation 15(3), 321–42. Torres, R. T. & Preskill, H. (2001). Evaluation and Organizational Learning: Past, Present, and Future. American Journal of Evaluation 22(3), 387–95. Watkins, K. E., & Marsick, V. J. (1993). Sculpting the learning organization: Lessons in the art and science of systemic change. San Francisco: Jossey-Bass.