Validation of a questionnaire to evaluate the impact of ISO 9001 Standards in schools with a Confirmatory Factor Analysis

Studies in Educational Evaluation 62 (2019) 37–48

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/locate/stueduc

Validation of a questionnaire to evaluate the impact of ISO 9001 Standards in schools with a Confirmatory Factor Analysis☆

T

Jesús Miguel Rodríguez-Mantilla , Mª José Fernández-Díaz, Víctor León Carrascosa ⁎

Complutense University of Madrid, Education Faculty, Department of Research Methods, Spain

ARTICLE INFO

ABSTRACT

Keywords: ISO 9001 Standards Impact evaluation Schools Measuring instrument Confirmatory Factor Analysis

Implementation of Quality Management Systems in educational organisations is a fact in many countries. Therefore, it is necessary to obtain evidence of the improvements and changes that the centres have because of the implementation. Thus, this paper presents the design of a solidly based questionnaire to evaluate the impact of ISO 9001 Standards in schools. Likewise, the analysis of the technical characteristics of the instrument is presented. We analysed the reliability, content and construct validity (the latter by means of Structural Equations Models implemented with Software AMOS 24). Results show that the overall reliability of the questionnaire is very good, with a Cronbach’s α of 0.985 and values higher than 0.93 in each of the six dimensions. The Confirmatory Factorial Analysis showed highly satisfactory results (IFI/ TLI/CF I > 0.90, RMSEA < 0.50, PRATIO > 0.85). The validity of the questionnaire is good, there is consistency between dimensions and subdimensions. Thus, the instrument presented combined the necessary technical characteristics for it to be considered a valid and reliable tool.

1. Introduction

1.1. Relevance of the Quality Management Systems

Implementation of Quality Management Systems (QMS) in organisations has been a fact for many decades now and gradually systems or systematized strategies have emerged to improve processes and evidence-based outcomes. It was in the business world, linked to production processes of goods or services, where implementation of these systems began, especially after the design of the ISO Standards in 1951. Education incorporated these systems several decades later, although it is important to note the difficulties their implementation in this field entails, derived, mostly, from the complexity of human nature (Fernández-Díaz, 2013). However, Total Quality Management models in their various forms (European Foundation for Quality Management, Colombian Model, of the Ibero-American Foundation, and others, as well as the ISO 9001 Standards), have gradually been included in the management education organisations at different levels (from Preschool to University) and across many different countries and continents. Some centres have adopted the Total Quality Management models and others the ISO 9001 Standards, but there are also many schools that have applied both, given their complementarity.

The aim of implementing QMS in organisations is to maintain and put in place mechanism to achieve continuous improvement, considering evaluation as a key tool that provides information on all of the processes and sub-processes conducted in an organisation as a basis for adopting sound measures. Although it seems evident that these systems should lead to improvements based on their approach and objectives, research on the changes they generate is limited and with mixed outcomes. Results are not univocal in relation to the school setting (De Vries, 2005; Gibb, 2003). Although some studies appear to show significant improvements achieved in education (Chen, Lyu, & Lin, 2004; Dobyns & Crawford-Mason, 1994; Kattman & Johnson, 2002; Stensaker, 2007), other studies indicate their effects are irrelevant or even detrimental (Hernández, Arcos, & Sevilla, 2013; López Alfaro, 2010; Malambo, 2015). This is one of the reasons why we decided to conduct this study: the need to obtain empirical evidence confirming the effectiveness of implementing these systems. Yet implementation of these systems must lead to substantial changes in the organisations, sustainable changes established over time that generate a different way of thinking or acting or even a different

☆ This study is part of a broader project: R&D+I EDU2013-44801-P, “Impact of the application of ISO 9001 Standards in schools and associated factors”, financed by the Ministry of Science and Innovation of Spain. ⁎ Corresponding author at: Complutense University of Madrid, Education Faculty, Department of Research Methods, C/Rector Royo Villanova, s/n, 28040, Madrid, Spain. E-mail addresses: [email protected] (J.M. Rodríguez-Mantilla), [email protected] (M.J. Fernández-Díaz), [email protected] (V. León Carrascosa).

https://doi.org/10.1016/j.stueduc.2019.03.013 Received 30 October 2018; Received in revised form 27 March 2019; Accepted 28 March 2019 0191-491X/ © 2019 Elsevier Ltd. All rights reserved.

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

culture that can affect the organisation's climate, the teachers' working procedures, the school's planning and management system, among other components of the organisation. In this line of research this is called "impact", which is related not to immediate results, such as those derived from improvement plans, but results that become established over time (Fernández-Díaz, 2013), leading to changes that make the institution generate a different organisation model (with higher levels of participation, more teamwork in all processes, more people-focused, with transformational and distributed leadership, among other aspects). And all this to contribute to achieve the organisation's objectives and, ultimately, generate better and more education for the students. However, to assess these effects, evaluation is an essential tool to obtain evidence of changes and specific improvements in the organisation and the impact caused by applying the system in the long-term. Likewise, this evaluation is useful to determine the factors that make some centres more effective than others, or for improvements or changes to affect some parts of the organisation and not others, for example, school climate, management system or digital skills.

entrepreneurship education programmes on competencies and motivation (Oosterbeek, VanPraag, & Ijsselstein, 2009), leadership and its impact on school performance and student learning (Hallinger & Heck, 2010), etc. Some of them refer specifically to QMS in non-university education centres, although there are not many (Egido, FernándezCruz, & Fernández-Díaz, 2016; Fernández-Cruz, Egido, & Carballo, 2016; Fernández-Díaz, Rodríguez-Mantilla, & Fontana-Abad, 2016) and a greater number refer to higher education (Chen, Chen, & Chen, 2013; De Miguel & Apodaca, 2009; Duque, 2013; Mehta, Verma, & Seth, 2013; Rodríguez-Ponce, Pedraja-Rejas, Araneda-Guirriman, GonzálezPlitt, & Rodríguez-Ponce, 2011; Stensaker, Langfeldt, Harvey, Huisman, & Westerheijden, 2011). 1.3. Measuring impact evaluation In this context, measuring impact evaluation is a key factor and it is obvious that if we do not measure it properly with suitable, reliable and valid techniques, the results and conclusions may be wrong. For Biesta (2009, 2015) the measurement culture has grown in education, both in education policies and in the practice of schools and teachers. This has made it possible to focus the discussions on factual data rather than only on assumptions or opinions, which is quite frequently the case. With these considerations, this study addresses the topic of using measurements for the evaluation of the impact after implementing a QMS in education organisations. So far, these systems have not included a plan to evaluate their impact and were not designed to facilitate the assessment of their effectiveness, nor are tools or techniques provided to collect information based on the goals of the implementation. Consequently, it has not been possible to conduct longitudinal studies with continuous follow-up, or initial studies to be able to collect benchmark data to compare with the end results (Fernández-Díaz, 2013). Given these limitations and the need to find evidence of the effectiveness of these plans, this study aimed to prepare an evaluation tool, in this case a questionnaire, with a strong theoretical base, that brings together the technical characteristics required for any good measuring tool, and that provides a quality assessment of the impact caused by one of the QMS applied in non-university schools, such as the ISO 9001 Standards. The focus of this tool was to evaluate the impact resulting from implementation of this system, based on the perception of the school staff who experienced these changes before the implementation began and who have followed the process over the years as direct agents of the changes that have been established and integrated into the organisation. This was the type of tool chosen in this case, but it should be noted that impact evaluation is assessed as a broad and comprehensive process, which should include both quantitative and qualitative techniques (Ávila-Fajardo & Riascos-Erazo, 2011). For Tejada and Ferrández (2007) questionnaires, interviews and focus groups are the most efficient and operative tools for gathering the necessary information and answering each one of the questions that can be drawn from the general objectives. Preparation of this questionnaire was based on a study of the schools, using a comprehensive and multidimensional approach, and the identification of the major dimensions where the impact may be found, with a thorough and in-depth analysis of the structure, operation and organisation of the schools and related literature (Antunez & Gairín, 1996; Cetzal, Delgado, & Reche, 2012; Lorenzo, 2011; Thurler & Maulini, 2010; Rodríguez, 2006; Trujillo, 2007). Based on the major dimensions, we defined sub-dimensions to be able to justify the contents to be assessed and with this disaggregation, we identified the indicators and items comprising the scale, with the idea of assessing the perception of the various stakeholders in the education community. It should be noted that the bibliographic analysis conducted showed there

1.2. Evaluation of the impact of Quality Management Systems We thus considered it necessary to include systems with techniques that make it possible to evaluate impact as described above. If the intervention processes do not have an impact or generate lasting changes, results will be poor. The paper by Tejada and Ferrández (2007) is along these lines, attempting to assess the improvements achieved in an organisation some time after delivering the training action and their influence on improved performance on the job, the organisation’s services and, consequently, organisational development. For these authors, impact evaluation refers to the external effects of training that are visible in the organisation and which operate some time after the training action, to verify the permanence and consistency of the changes occurring in subjects, improvement of professional practices, institutional changes, etc., according to the goals in the training plan (Fernández et al., 2008; Tejada & Ferrández, 2007). Likewise, Ferrández Lafuente (2006) defined impact evaluation as a "process aimed at measuring the results generated (changes and causes) by the training actions carried out in the original socio-professional setting over time" (Ferrández Lafuente, 2006, p. 20). Similarly, for Abdalá (2004) impact evaluation is understood as an evaluation process aimed at measuring the results of interventions in quantity, quality and scope, according to pre-established rules. Therefore, addressing impact evaluation is a significant aspect of improving organisations and the intervention processes. Not many studies can be found in the literature related to impact evaluation and, in some cases, they refer to impact in the sense of immediate results, another meaning of the word impact (Fernández-Díaz, 2013). As expressed by Perassi (2010) "one of the greatest gaps in the field of programme evaluation is the assessment of their impact. This makes it impossible to understand the effect the project may have had on the issues and causal factors related to what we aimed to change (collateral effects). In this sense, an opportunity is missed to exploit the positive impact on certain aspects in the scenario analysed, to be able to enhance their development, and other dimensions are neglected which were perhaps harmed" (Perassi, 2010, p. 14). Along these lines, there are in fact some impact evaluation studies in fields such as the environment (Glasson, Therivel, & Chadwick, 2013), where in recent years there has been a substantial increase, probably because of social concern over the negative impact observed on the planet, social impact (Maas & Liket, 2011), medicine, in-company training (Tejada & Ferrández, 2007), Information and Communications Technologies in education (ICT) (Ávila-Fajardo & Riascos-Erazo, 2011; Bilbao-Osorio, 2009; Marqués Graells, 2013), the impact of

38

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

are barely any tools to assess the impact of implementing a management system, or any other technique that could be used, save for a few exceptions (Fernández-Cruz et al., 2016; Egido et al., 2016; FernándezDíaz et al., 2016). Therefore, we identified the following dimensions on which to base the design of the questionnaire to evaluate the impact:

we intended to provide the scientific community with a tool that could be used to advance research and provide evidence on the impact caused by implementing ISO 9001 or other QMS standards in schools, as well as to identify the success factors that are aligned with results and that ensure the effectiveness of the implementation. 2. Method

1 Information and Communications Systems: Communication is a key aspect in schools. QMS consider communications to be one of the most important components in schools. They thus attempt to modify or generate a horizontal and vertical communication system that helps achieve the agreed objectives (Robles-García et al., 2005). The managers must interact with their teaching staff (Jackson & Marriott, 2012), their students, parents, etc. Communication among teachers is a key factor in planning and intervening in student teaching processes. 2 Management System: Related especially to the school’s Planning Culture, which is manifest in actions aimed at organising and structuring the activities carried out, whether related to academic aspects (teaching-learning process, services, counselling…) or Management (extra-curricular, complementary or any other type of activities) based on systematized design, work systems and information management generated by the QMS. This dimension also includes the Policy for Support and Recognition of the members involved in undertaking the school’s activities. 3 School Climate: Refers to the efficacy of the QMS to change and improve internal relations among all the members in the school, teachers, students, families, administration staff and managers, as well as to increase participation and engagement of everyone in the running of the school and improving quality. 4 Teaching-Learning Processes: Selection of this specific dimension is justified by the fact that teaching and learning processes are a crucial element in schools. Since learning is at the heart of schools, their essential purpose, the most important aspect of implementing a quality management system in a school is that it helps improve teaching and learning processes, thus leading to improved results (Goldberg & Cole, 2002). 5 Satisfaction of the Education Community: A clear reflection of the impact of a QMS is continuous improvement in the results of evaluations and the level of satisfaction of the various participants in the educational process: teachers, administration and support staff, students, families… 6 External Relations System and School Bonds with Society: QMSs are increasingly valuing the external promotion of institutions in their environment, through relationships with other schools, public affairs, mobility and exchange programmes, school image… Therefore, it is necessary to understand how this has been affected by implementation of the QMS. Furthermore, in the external relations system and the bond between the school and the local community, impact is operationalized in the following terms: a) the school has established sound relationships with centres in the area to conduct joint activities related to culture, sports, art, etc.; b) the management team has systematized a network of institutional relations considered important and beneficial for the school with partners, companies, educational institutions, banks, providers, etc.; and, c) the school has institutionalized interesting alliances that enrich its education efforts at a regional, national and international level.

2.1. Sample To conduct the study, we obtained a total sample of 2593 subjects (86.22% were teachers and 13.78% members of the management teams) from 85 schools in 4 Autonomous Communities of Spain (Madrid, Andalusia, Valencia and Castilla y León) that had been using the ISO 9001 Standards at the school for at least 3 years. The measurement instrument used in the study, as shown in the Appendix A, was made up of 94 items, so the observations/variables ratio obtained was 27.58. Hair, Black, Babin, and Anderson (2014) state that, as a general rule, it is advisable to have, as a minimum, a number of observations five times greater than the number of variables (p. 100); however, the generally accepted ratio to minimize problems with deviations from normality is fifteen to one (p. 573). This criteria, also established by other authors (such as Bentler & Chou, 1987; Bentler & Mooijaart, 1989; Catena, Ramos, & Trujillo, 2003; Comrey & Lee, 1992; Durán-Aponte, Elvira-Valdés, & Pujol, 2014; Everitt, 1975; Jackson, 2003; Kim, 2005; Kline, 2015; Lévy, Martín, & Román, 2006; Nunnally, Bernstein, & Berge, 1967; Pasquali, 2008; Tabachnick, Fidell, & Ullman, 2007; Wang & Wang, 2012; Worthington & Whittaker, 2006; among others), is used in a large number of research (as in Costa Ball, González Tornaría, del Arca, Masjuan, & Olson, 2013; Cupani, Vaiman, Font, Pizzichini, & Saretti, 2012; Demo, Rabelo, Nunes, & Rozzett, 2012; Durán-Aponte et al., 2014; Fernandes Malaquias & de Oliveira, 2014; Fernández-Cruz, Fernández-Díaz, & Rodríguez-Mantilla, 2018; FrancoParedes, Bautista-Díaz, Díaz-Reséndiz, & Arredondo-Urtiz, 2017; Gie Yong & Pearce, 2013; González-García, Añó-Sanz, Parra-Camacho, & Calabuig-Moreno, 2018; Holtzman & Vezzu, 2011; León & FernándezDíaz, 2017; Marsh, Kit-Tai, & Balla, 2010; Moreta-Herrera, López-Calle, Ramos-Ramírez, & López-Castro, 2018; Rodríguez-Mantilla & Fernández-Díaz, 2012; Rodríguez Mantilla & Fernández Díaz, 2015; Rositas, 2014; Tejero-González & Fernández-Díaz, 2009; Vargas & Mora-Esquivel, 2017; Wong, Wong, & Zhuang, 2017, among others). The subjects were selected through incidental sampling (voluntarily participating in the study those subjects who wanted to, MorenoBayardo, 1987, and Guilford & Fruchter, 1973), with 15% coming from public schools, 73.3% from private schools with state subsidies and 11.7% from private schools. 2.2. Instrument The impact evaluation of the implementation of ISO 9001 Standards in schools was conducted using an ad hoc questionnaire with six large dimensions, which were based on the specialised literature listed above in this paper: Communication System, Management System, Climate, Teaching-Learning Process, Satisfaction and External Relations. The list of sub-dimensions for each dimension is shown in Table 1. To obtain evidence assuring the instrument’s content validity, we selected, on the one hand, three education research experts to independently evaluate the items in the questionnaire; and, on the other, three education professionals. Each one of them was informed of the purpose of the test and the conceptualisation of the included universe, and they assessed each item in the questionnaire for relevance and clarity using a scale from 1 to 5. We considered removing those items that were below an average of 4 in both clarity and relevance, and those

With the above considerations, the aim of this study was to prepare a technically and theoretically sound questionnaire to evaluate the impact of applying ISO 9001 Standards in non-university education centres, which included the technical characteristics required for any good measurement tool, especially validation of the construct. With this

39

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

Table 1 List of dimensions, sub-dimensions and items in the final questionnaire to measure the impact of ISO 9001 Standards in schools. Dimension Communication System Management System

Climate Teaching-Learning Process

Satisfaction External Relations

Sub-dimensions Planning Effectiveness of meetings Review of school documents Annual Planning Support and Recognition Relations Conflict Resolution Teacher Coordination Plans of Action Family involvement Teaching Methods Evaluation Tutoring With other schools or institutions Exchange Programmes School Recognition Use of surrounding resources

Description

Ítems

Evaluation of communication between the school and teachers, families, departments, etc. Planning of schedules, meetings, personal guidance, etc. Meetings of management team with teachers, of coordinators, etc. Usefulness of document review: Education Project, Strategic Planning, etc. Consideration of student outcomes, teacher evaluations, etc., for annual planning Recognition of teachers by management team for their work Social interaction between teachers, students, families, etc. Conflicts with teachers, students, etc. Cooperation among teachers for work performance Plans of Action according to the results of student performance evaluations (internal and external) In their children's teaching process, involvement in school, etc. Evaluation of teaching methods Variety and appropriateness of evaluation techniques Usefulness of personal guidance / tutoring Satisfaction of teachers, students and families. Relations with other schools or institutions for joint projects, sports activities, etc. To learn languages and practice sports In the press, television, etc. Resources offered by the city council, corporations, finance companies, etc.

1–9 10–15 16–19 20–23 24–30 30–37 38–44 45–48 49–54 55–58 59–61 62–65 66–70 71–75 76–79 80–83 84–87 88–90 91–94

element-total corrected) of the items in order to determine the appropriateness of deleting any of them. Next, AMOS version 24 was used to determine the goodness of fit of the proposed theoretical factor model, by way of the dimensions and indicators described above, with a Confirmatory Factor Analysis following the criteria proposed by Byrne (2013) and Kline (2015) (CFI, TLI and IFI > 0.9, PRATIO, PNFI and PCFI > 0.9, RMSEA < 0.06 and HOELTER > 200) (see Table 3).

Table 2 Results of the reliability analyses of the overall questionnaire and by dimension.

Communication System Management System Climate Teaching-Learning Process Satisfaction External Relations TOTAL

Initial questionnaire (94 items)

Final questionnaire (83 items)

0.907 0.954 0.970 0.950

0.907 0.953 0.970 0.938

0.930 0.945 0.986

0.930 0.939 0.985

3. Results 3.1. Reliability After an initial, descriptive analysis of the answers, with the items showing no irregular behaviour for variability and central tendency (means between 2.84 and 4.41, with standard deviations of 1.13 and 0.77), we calculated the overall Cronbach’s alpha for the initial tool (0.986) for measuring the impact of ISO 9001 Standards. No unexpected values were found in the homogeneity indices (below 0.2 according to Hair et al., 2014). After making changes to the final tool, based on the results of the Confirmatory Factor Analysis, we calculated the overall alpha of the final instrument, which reached a level of 0.985. Likewise, we analysed the reliability of the tool by dimension (see Table 2), obtaining excellent results in all cases.

with a standard deviation greater than 1.5 (Cortada de Kohan, 1999). The experts' assessments showed the high relevance of all of the items proposed. The only changes and corrections made were small spelling or grammar corrections and a few changes in the phrasing. No expert considered it necessary to add or delete any of the items presented. Thus, the questionnaire was made up of 94 initial items (see Appendix A) (although after the Confirmatory Factor Analysis we removed some of them -marked with an asterisk in the Appendix A, leaving a total of 83 fin. l items) to which the teachers and members of the management team had to reply based on a Likert-type scale from 1 to 5, (where 1 indicates Nothing, never, and 5 indicates Very much, always) to assess the level of improvement, effectiveness or development of each aspect as a result of implementing ISO 9001 Standards at the school.

Table 3 Summary of fit indices of the three models to measure impact of ISO 9001 Standards in schools.

2.3. Procedure To achieve school participation in the study, we set up an initial meeting with the Principal and the Head of Quality, where we explained the purpose of the study, the procedure to be followed and set the dates to administer the questionnaire. To administer the questionnaires, members of the research team visited the schools on the appointed dates, ensuring complete anonymity and confidentiality of their assessments and results. Likewise, each school was assured delivery of a report on their results. 2.4. Data analysis First, the reliability was assessed by Cronbach's alpha using SPSS version 24, both for the whole questionnaire and for each one of its dimensions, and we analysed the homogeneity indices (correlation 40

Measure

Fit Level Recommended

Initial Model Value

Model 2 (Correlated Factors) Value

Final Model (Unidimensional) Value

IFI TLI CFI PRATIO PNFI PCFI RMSEA LO 90 HI 90 HOELTER .05 HOELTER .01

> 0.90

0.707 0.700 0.707 0.975 0.677 0.690 0.078 0.078 0.079 160 163

0.905 0.901 0.905 0.963 0.855 0.869 0.049 0.048 0.049 376 383

0.904 0.901 0.903 0.966 0.856 0.869 0.049 0.049 0.050 371 377

> 0.70 < 0.06 > 200

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

Fig. 1. Initial Structural Model to measure impact of ISO 9001 Standards.

3.2. Validity of the construct (Confirmatory Factor Analysis)

(2016), Duro, Simões, Ponciano, and Santana (2010), Kline (2015), Mantilla and Díaz (2012) and Schaufeli, Salanova, González-Romá, and Bakker (2002), among others, we started the estimation procedure of the model with the higher order factors and then tested the multidimensionality underlying the higher order factors (based on the theoretical model presented in Table 1 and according to the modification indices obtained). Similarly, the existence of a third order factor has been tested.

Having based the configuration of the tool's structure on the literature (Table 1), we conducted a Confirmatory Factor Analysis applying SEM (Structural Equation Modelling) to assess its construct validity. It is important to point out that, following the specifications established by the studies of authors such as Arias Martínez (2008), Byrne (2016), Costa, Marôco, Pinto‐Gouveia, Ferreira, and Castilho

41

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

Table 4 Modification Indices I. M.I. Tutoring

Evaluation Teaching Methods Family Involvement Plans of Action

Teacher Cooperation

Conflict Resolution

Relations

e73 e72 e71 e71 e69 e67 e64 e63 e62 e60 e59 e59 e57 e56 e55 e55 e51 e50 e50 e49 e47 e46 e45 e45 e45 e43 e42 e40 e40 e39

< < < < < < < < < < < < < < < < < < < < < < < < < < < < < <

> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

e74 e73 e74 e72 e70 e68 e65 e65 e63 e61 e61 e60 e58 e57 e57 e56 e52 e52 e51 e50 e48 e47 e48 e47 e46 e44 e44 e44 e41 e40

260.08 210.72 392.26 703.91 655.46 1957.1 325.42 233.39 238.68 50.52 122.87 582.95 156.04 244.36 280.60 135.60 422.99 413.83 889.20 381.28 136.20 363.67 177.69 200.86 230.45 173.67 235.91 104.22 263.30 951.54

M.I. Support and Recognition

Annual Planning Document Review Meetings Planning of personal guidance and schedules

Resources offered by the school School Recognition Exchange Programmes Relations with other schools or institutions

Thus, we specified the rules of correspondence and relationships between latent and manifest variables measured by the questionnaire. We proposed the Initial Measurement Model (Fig. 1) which included all the indicators contemplated in the theory in order to measure the six constructs. This model consisted of 6 latent variables: Communication (COM, defined by 9 variables), Management System (MS, defined by 28 variables), Climate (CLIMA, 17 variables), Teaching-Learning Process (TLP, 21 variables), Satisfaction (SAT, 4 variables) and External Relations (REL, 15 variables). Thus, the model consists of a total of 94 observed variables (from V01 to V94) and 94 error terms (from e01 to e94). Once the model was specified, and the multivariate normality assumed (Mardia coefficient = 1904.703 less than p·(p + 2), p being the number of variables observed, 94·(94 + 2) = 9024) (Bollen, 1989), we then estimated the model parameters with the Maximum Likelihood "ML" procedure - the most efficient and unbiased one when the multivariate normality assumptions are met (Hayduk, 1996)-. In relation to the adjustment indices, many authors have pointed out the inappropriateness of using Chi square with very large samples, since the index may not be reliable in those cases (Bagozzi, Yi, & Singh, 1991; Cupani, 2012; Mulaik et al., 1989). To align the Chi square sensitivity to the sample size, authors such as Lévy-Mangin and Varela (2006) have proposed analysing alternative absolute fit measures (see Table 3). Thus, among the results of this initial model, we found that the fit indices CFI = 0.707, TLI = 0.700 and IFI = 0.707 were below the 0.90 required, according to Kline (2015), due, partly, to unsatisfactory factor loadings in items 19 and 29 (below 0.50, the value indicated as necessary by Byrne, 2013), and therefore those items were removed. When checking the modification indices table, we found correlations between the error terms of several variables (Table 4, showing the most significant correlations). Accordingly, in order to confirm whether there

e36 e35 e34 e31 e30 e26 e25 e24 e21 e20 e20 e17 e16 e16 e14 e13 e12 e12 e92 e91 e91 e89 e88 e88 e86 e85 e84 e82 e81 e80

< < < < < < < < < < < < < < < < < < < < < < < < < < < < < <

> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

e37 e36 e37 e32 e31 e27 e26 e25 e22 e22 e21 e18 e18 e17 e15 e14 e14 e13 e93 e93 e92 e90 e90 e89 e87 e86 e85 e83 e83 e81

1.007.96 1.162.09 802.20 618.18 521.96 1.025.11 336.06 490.00 1.172.90 1.107.99 1.681.99 503.96 435.01 687.01 1.470.85 618.38 247.25 437.01 1104.8 99.43 439.32 314.56 473.87 316.42 584.18 794.44 391.46 538.61 531.31 149.86

was an improvement in the model fit, it seemed appropriate to include several latent variables or sub-factors (Kline, 2015), matching in all cases the sub-dimensions defined in theory:

• The • • •

sub-factors Tutoring, Evaluation, Teaching Methods, Family Involvement and Plans of Action, in the dimension Teaching-Learning Process. Relations, Conflict Resolution and Teacher Cooperation, in the dimension Climate. In the dimension Management System, the latent variables Support and Recognition, Annual Planning, Document Review, Meeting Management and Planning System. Sub-factors Relations with other schools or institutions, Exchange Programmes, School Recognition and Use of Resources (in the dimension External Relations).

Once the new sub-factors were included in the model, the modification indices showed the appropriateness of some covariates between error terms (Table 5). Some of them were justifiable from a theoretical standpoint, the correlations between the error terms e14–e15 were feasible (since they both alluded to personal guidance), e31–e32 (satisfaction and complaints), e35–e36 (incentives for excellent or innovative teachers), e39–40 (relations and climate), e06–e07, e77–e78 and e42–e53 (relationship and satisfaction of families and students with the school and family involvement), e02-e03 (communication of management team), e26–e27 (evaluation of extracurricular activities), e04e18 (related to counselling department) y e84–e87 (language immersion). Similarly, we found multiple saturation in nine items (v66, v10, v23, v83, v69, v70, v80, v58 and v73) on several dimensions and subdimensions. Given this multidimensionality, and according to Kline's

42

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

Table 5 Modification Indices II.

e14 e31 e35 e39 e06 e02 e26 e77 e04 e84 e42

< < < < < < < < < < <

> > > > > > > > > > >

e15 e32 e36 e40 e07 e03 e27 e78 e18 e87 e53

M.I.

Par Change

672.12 596.85 548.19 472.22 423.53 359.22 320.97 308.51 358.27 243.98 235.87

0.22 0.35 0.96 0.11 0.27 0.14 0.15 0.11 0.20 0.16 0.12

V66 V66 V10 V10 V23 V23 V83 V83 V69 V69 V70 V70 V80 V80 V58 V73

indications (2015), we decided to delete those items (Table 5). After conducting the modifications above, we obtained the model 2 (Fig. 2) estimated on a sample of 2593 subjects, with 206 variables: 83 observed variables (corresponding to the items) and 123 latent variables (23 are factors, 83 are error terms and 17 disturbance terms). Of these 206 variables, 106 were exogenous and 100 were endogenous. Furthermore, there were 291 parameters to be estimated and the model had 3278 degrees of freedom, thereby obtaining an overidentified model and with a possibility of being estimated. The parameters of the model 2 were, likewise, estimated with the Maximum Likelihood procedure (Mardia coefficient = 1576.486 less than 83·(83 + 2) = 7055) and we obtained satisfactory estimation results (Table 2), highlighting CFI = 0.905, TL = 0.901 and IFI = 0.905. In relation to the residuals, the RMSEA was 0.049 and an adequate sample size was achieved with a Hoelter index of 383. Also, the parsimony indices were high (PRATIO = 0.963, PNFI = 0.855 and PCFI = 0.869), therefore we can assert that, taking into account the parameters used, this measurement model is quite parsimonious. The modification indices showed no values worthy of note, so it was not necessary to include any more sub-factors in the model. Likewise, analysing the standardised values of the parameters, we confirmed the good quality of the indicators, since their factor loadings were higher than 0.50. Correlations between error terms acquired a substantial value in all cases (the lowest being 0.34), as well as the estimations of the error terms (the table has not been included due to space limitations). Finally, regarding the correlations between the model dimensions (not included in Fig. 2 to facilitate interpretation of the model charts) and based on the criteria by Hair et al. (2014), we found that all dimensions have significant and high or very high correlations (with values higher than 0.80), except the dimension Communication System which showed significant and moderate correlations (with values between 0.684 and 0.740) with the dimensions Teaching-Learning Process, External Relations, Climate and Satisfaction (Table 6). Based on these results, we wanted to analyse the unidimensionality of the instrument, so the correlations between the six large dimensions were deleted and a single factor was included (called CENT, when referring to the different dimensions of the centre evaluated). The results obtained in this final model showed satisfactory values (see Fig. 3 and Table 3), which demonstrated the unidimensionality of the instrument.

<— <— <— <— <— <— <— <— <— <— <— <— <— <— <— <—

Plans of Action Annual Planning Relations Climate Rel. with other inst Tutoring Rel. with other inst Relations Teaching Methods Annual Planning Teaching Methods Annual Planning Sup. & Recognition Teaching Methods Evaluation Teaching Methods

M.I.

Par Change

458.792 424.224 135.479 114.55 193.33 193.311 223.236 193.225 220.935 208.019 230.334 222.596 135.422 106.305 134.939 107.407

0.45 0.48 0.20 0.24 0.20 0.30 0.19 0.21 0.32 0.33 0.35 0.37 0.24 0.27 0.17 0.21

9001 Standards in schools. The analysis of the psychometric characteristics in the questionnaire to measure said construct have revealed excellent reliability overall and by dimension, thus showing adequate internal consistency. The theoretical basis used to set up the system of dimensions, subdimensions, indicators and items, together with the selection of research and education experts and professionals from the field of education, have ensured the tool’s high content validity. The appropriateness of this theoretical underpinning and the sound and robust configuration of the dimension system is evidenced in the Confirmatory Factor Analysis where, using modification indices, we found a match between the correlations of item error terms -indicating the suitability of grouping them into sub-factors- and the sub-dimension system proposed theoretically. The results show an existing covariate relationship between the various dimensions (high correlation between them all, except for the Communication System which presents a moderate correlation with the rest of the dimensions), which confirms the existing interdependence between the various factors. In this sense, there are many studies that confirm the relationships between these dimensions. For example, different theories and studies by many authors agree there is a high positive correlation between school Climate and Satisfaction of teachers and other staff (Fernández-Díaz, Fernández, y Herrero, & Toranzo, 2002; Rodríguez, 2006; Zubieta, Delfino, & Fernández, 2008), with External Relations established by the school (Fernández-Cruz, Rodríguez-Mantilla, & Fernández-Díaz, 2015) and with the TeachingLearning Process (Berger, Álamos, Milicic, & Alcalay, 2014; Giraldo & Mera, 2000). Likewise, there are studies supporting the existing relationship between Satisfaction and Teaching-Learning Process (Ramírez, Rojas, Cortés, Lozano, & Solís, 2013), External Relations and Management System (García, Brea, & Del Río, 2013), among others. In the study, undoubtedly, the sample size achieved is one of the key elements enabling this analysis (Hair et al., 2014). Nonetheless, we consider it would be appropriate to expand the study with a larger sample in order to increase its power of generalisation and to validate the instrument in other contexts (other Autonomous Communities, regions, countries, etc.). Along these lines, it should be noted that the type of incidental sampling, a common and justified procedure for this type of studies given the voluntary participation, could affect the external validity of the research, limiting the generalisation power of the results. In short, the results obtained and shown in this paper should be interpreted as an indicator of the adequate construct validity of the measurement instrument and the satisfactory dimension structure proposed (having found a uni-dimensional factor), and so, in conclusion and in line with the objective of this work, we can assert we have

4. Discussion and conclusions The results obtained have achieved the objective of preparing and validating a questionnaire to measure the impact of implementing ISO

43

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

Fig. 2. Structural Model 2 to measure impact of ISO 9001 Standards -the correlations between the six main dimensions have been hidden to facilitate interpretation of the model. These correlations are shown in Table 6.

44

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

To what extent do you believe the PLANNING SYSTEM has improved in your school regarding: 11. Subject scheduling. 12. Reviewing the learning process according to outcomes. 13. Systematic teacher meetings (coordination of subjects, evaluation boards…). 14. Student personal guidance. 15. Family guidance. Please assess the effectiveness of the following meetings: 16. Management team and teachers. 17. Area Coordinators or Department Heads. 18. Counselling Department and teachers. *19. Minutes are drafted for each meeting including the items covered and agreements made. Please indicate to what extent the ISO 9001 Standards have improved the USEFULNESS OF REVIEWING the following documents: 20. Education Project. 21. Syllabus Project. 22. General Annual Programme. *23. Strategic Planning. For ANNUAL PLANNING, more consideration is given to outcomes of: 24. Student academic performance. 25. Evaluation of teachers' work. 26. Evaluation of complementary or pedagogical activities. 27. Evaluation of extracurricular activities. 28. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on the school's Planning Culture. *29. Has a systematized intervention model been established to channel all actions of support, recognition and reward for the various members of the education community? Implementation of the ISO 9001 Standards has improved systematisation in your school of the following measures: 30. Study of staff expectations. 31. Analysis of complaints and suggestions received from staff. 32. Staff satisfaction evaluations. 33. Recognition of success of significant objectives achieved by staff. 34. Evaluation of support, recognition and reward policy. To what extent do you believe implementation of ISO 9001 Standards in your school has improved: 35. Recognition of teachers who systematically achieve excellent results with their students. 36. Incentives for teachers who carry out improvement or innovation projects. 37. Please assess, in general, the impact which implementation of the ISO 9001 Standards has had on the Management System for Support, Recognition and Reward of the staff in your school.

Table 6 Correlations between dimensions corresponding to the Model 2.

REL TLP CLIMA SAT QM COM

REL

TLP

CLIMA

SAT

QM

COM

1 0.896 0.836 0.823 0.806 0.684

1 0.906 0.851 0.881 0.742

1 0.861 0.836 0.702

1 0.823 0.691

1 0.838

1

Fig. 3. Final Unidimensional Structural Model to measure impact of ISO 9001 Standards (the rest of the variables and factors of the model have been hidden to facilitate interpretation of the model).

contributed to the field of science that studies Quality Management Systems in education with a valid and reliable questionnaire to measure the impact of implementing ISO 9001 Standards in schools. Appendix A 1. COMMUNICATION 1. As a result of implementing ISO 9001 Standards in your school, please assess the increase in the use of procedures to collect suggestions and complaints (suggestions box, email, etc.). Implementation of ISO 9001 Standards has helped improve the effectiveness of existing communication channels in the school to deliver and share information (notices, meetings, responsibilities, reminders, events, etc.) with members of the education community by: 2. Management Team. 3. Coordinators, Heads of Studies or Department Heads. 4. Counselling Department. 5. Teachers. 6. Families. 7. Students. 8. Others (secretary's office, medical service, etc.). 9. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on the school's Communication System.

3. CLIMATE 38. The ISO 9001 Standards have improved application of the Rules of Coexistence at the school. Implementation of ISO 9001 Standards has promoted: 39. A good school climate. 40. Better relationships between teachers. 41. Involvement of the management team to improve school climate. 42. Relationships between families and the school. 43. Teacher involvement in compliance with the Rules of Coexistence. 44. Better relationships between teachers and students. Thanks to implementation of ISO 9001 Standards: 45. There has been an improvement in the application of measures taken with students with disruptive behaviour.

2. MANAGEMENT SYSTEM *10. Implementation of ISO 9001 Standards has improved the school's general planning system. 45

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al.

46. There has been an improvement in the measures taken to resolve conflicts among teachers. 47. Teachers' conflict resolution skills have improved. 48. The Management Team is effective in resolving conflicts at the school. Implementation of ISO 9001 Standards in the school has increased: 49. Cooperation among teachers to organise events (Christmas, Science Week, etc.). 50. Teachers' interest in participating in innovation projects in the school. 51. Teachers' interest in learning new teaching methods, ICT training, etc. 52. Sharing of teaching experiences among teachers. 53. Because of implementing the QUALITY MODEL in accordance with ISO 9001 Standards, family involvement in the school has improved. 54. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on improving the school's Climate.

74. Counselling programmes external to the school are included in the Guidance Plan of Action. 75. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on the school's learning process. 5. SATISFACTION As a result of implementing ISO 9001 Standards, the level of satisfaction has increased over the last three years: 76. Among teachers. 77. Among students. 78. Among families. 79. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on the school's staff Satisfaction. 6. EXTERNAL RELATIONS As a result of applying ISO 9001 Standards: *80. Joint activities are being conducted with other schools (projects, sports activities, etc.). 81. There has been an increase in the relations the school has established with other bodies, such as banks, providers, etc. 82. There has been an increase in the benefits obtained by the school from these relations. *83. Actions are carried out to strengthen the network of relations with other institutions. Thanks to application of ISO 9001 Standards: 84. Exchange or language immersion programmes are systematically carried out to promote language learning. 85. Exchange programmes are systematically carried out to promote sports practice. 86. Exchange programmes are systematically carried out to promote participation in academic tournaments. 87. Specific actions are carried out to strengthen student exchange or language immersion agreements. Please assess the following aspects in relation to SCHOOL RECOGNITION: 88. The school has improved its image in its environment in the last three years. 89. The school has been recognised in the media (press, TV, etc.). 90. The school systematically conducts specific actions to improve its prestige. As a result of ISO 9001 Standards: 91. Resources offered in our environment (city hall, regional government, corporations, finance companies, etc.) are being used. 92. Specific actions are prepared to make better use of the resources offered in our environment. 93. As a result of this evaluation, specific plans and actions are prepared to make better use of the resources offered in our environment. 94. Please assess, in general, the impact you consider implementation of the ISO 9001 Standards has had on improving the school's Relations.

4. LEARNING PROCESS As a result of implementing the ISO 9001 Standards, PLANS OF ACTION are systematically being carried out: 55. Based on the results of student evaluations by the school. 56. Based on the results obtained in external tests (of the Autonomous Community, PISA, Cambridge, etc.). 57. For students with low performance. *58. For students with high performance. Thanks to implementation of the ISO 9001 Standards: 59. Family involvement in the learning process of their children has increased. 60. The system to inform families about their children's progress has improved. 61. Family involvement in the school is systematically evaluated. In relation to the TEACHING METHODS, thanks to implementation of the ISO 9001 Standards: 62. Compliance with the teaching methods included in the classroom programmes is systematically evaluated. 63. Teachers adapt teaching methods to student characteristics. 64. Flexible groups are arranged for core subjects according to students' learning pace. 65. Teachers have increased student motivation thanks to the methods used. In relation to EVALUATION and as a result of implementing the QUALITY MODEL in line with ISO 9001 Standards, please assess the following aspects: *66. Preparation of reinforcement plans and curriculum adaptation is based on evaluation. 67. Evaluation tools are used (such as grading guides, observation scales, etc.) to systematise evaluations. 68. Evaluation tools are used (such as grading guides, observation scales, etc.) to support student grades. *69. Evaluation criteria are clearly specified for the understanding of parents, students, etc. *70. Students may review the activities that have been evaluated in their learning. In relation to PERSONAL GUIDANCE and as a result of implementation of ISO 9001 Standards: 71. Head Teachers conduct personal guidance sessions with each student 72. There is a follow-up of the commitments made during guidance meetings with parents. *73. The Guidance Plan of Action is evaluated at the end of the school year to make improvements.

References Abdalá, E. (2004). Manual para la evaluación de impacto en programas de formación para jóvenes. Montevideo: CINTERFOR. Antunez, S., & Gairín, J. (1996). La organización escolar, Práctica y fundamentos. Barcelona: Grao. Arias Martínez, B. (2008). Desarrollo de un ejemplo de análisis factorial confirmatorio con LISREL, AMOS y SAS. Publicaciones del INICO, 75. Ávila-Fajardo, G. P., & Riascos-Erazo, S. C. (2011). Propuesta para la medición del impacto de las TIC en la enseñanza universitaria. Educación y educadores, 14(1), 169–188. Bagozzi, R. P., Yi, Y., & Singh, S. (1991). On the use of structural equation models in experimental designs: Two extensions. International Journal of Research in Marketing, 8(2), 125–140.

46

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al. Bentler, P. M., & Chou, C. P. (1987). Practical issues in structural modeling. Sociological Methods & Research, 16(1), 78–117. Bentler, P. M., & Mooijaart, A. B. (1989). Choice of structural model via parsimony: A rationale based on precision. Psychological Bulletin, 106(2), 315–317. Berger, C., Álamos, P., Milicic, N., & Alcalay, L. (2014). Rendimiento académico y las dimensiones personal y contextual del aprendizaje socioemocional: evidencias de su asociación en estudiantes chilenos. Universitas Psychologica, 13(2), 627–638. Biesta, G. (2009). Good education in an age of measurement: On the need to reconnect with the question of purpose in education. Journal of Personnel Evaluation in Education, 21(1), 33–46. Biesta, G. J. (2015). Good education in an age of measurement: Ethics, politics, democracy. London: Routledge. Bilbao-Osorio, B., & Pedró, F. (2009). A conceptual framework for benchmarking the use and assessing the impact of digital learning resources in school education. In F. Scheuermann, & F. Pedró (Eds.). Assessing the effects of ICT in education (pp. 107–118). Luxembourg: Publication Office of the European Union. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley. Byrne, B. M. (2013). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York: Routledge. Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Routledge. Catena, A., Ramos, M., & Trujillo, H. (2003). Análisis multivariado: un manual para investigadores. España: Biblioteca Nueva, S.L. Cetzal, R. S. P., Delgado, M. L., & Reche, P. C. (2012). El poder en las organizaciones escolares… todos lo buscan, pocos lo consiguen. Revista Educación y Ciencia, 2(40), 33–46. Chen, C., Chen, P. C., & Chen, P. Y. (2013). Teaching quality in higher education: An introductory review on a process-oriented teaching-quality model. Total Quality Management & Business Excellence, 25(1–2), 36–56. Chen, Y., Lyu, J., & Lin, Y. (2004). Education improvement through ISO 9000 implementation: Experiences in Taiwan. International Journal of Engineering Education, 20(1), 91–95. Comrey, L. A., & Lee, H. B. (1992). A first course in factor analysis (2nd Ed.). Hillside, NJ: Lawrence Erlbaum Associates. Cortada de Kohan, N. C. (1999). Teorías psicométricas y construcción de tests. Buenos Aires: Lugar editorial. Costa, J., Marôco, J., Pinto‐Gouveia, J., Ferreira, C., & Castilho, P. (2016). Validation of the psychometric properties of the Self‐Compassion Scale. Testing the factorial validity and factorial invariance of the measure among borderline personality disorder, anxiety disorder, eating disorder and general populations. Clinical Psychology & Psychotherapy, 23(5), 460–468. Costa Ball, D., González Tornaría, M., del Arca, D., Masjuan, N., & Olson, D. H. (2013). Propiedades Psicométricas del FACES IV: Estudio de validez en población uruguaya. Ciencias Psicológicas, 7(2), 119–132. Cupani, M. (2012). Análisis de Ecuaciones Estructurales: conceptos, etapas de desarrollo y un ejemplo de aplicación. REVISTA TESIS Facultad de Psicología, 2(1), 186–199. Cupani, M., Vaiman, M., Font, M. L., Pizzichini, F., & Saretti, B. (2012). Análisis factorial confirmatorio del neo-ffi utilizando parcelización de ítems y método bootstrap. Avaliação Psicológica, 11(2), 159–168. De Miguel, M., & Apodaca, P. (2009). Criteria for evaluating the impact of higher education quality assessment plans. Revista de Educación, 349, 295–310. De Vries, W. (2005). Calidad, eficiencia y evaluación de la educación superior. Madrid: Netbliblo. Demo, G., Rabelo, E., Nunes, I., & Rozzett, K. (2012). Human resources management policies and practices scale. (HRMPPS): Exploratory and confirmatory factor analysis. Brazilian administration review (BAR). 1–26. Dobyns, L., & Crawford-Mason, C. (1994). Thinking about quality: Progress, wisdom, and the Deming philosophy. New York: Random House. Duque, L. (2013). A framework for analysing higher education performance: students’ satisfaction, perceived learning outcomes, and dropout intentions. Total Quality Management & Business Excellence, 25(1–2), 1–21. Durán-Aponte, E., Elvira-Valdés, M. A., & Pujol, L. (2014). Validation of Self-efficacy Inventory for Multiple Intelligence revised (IAMI-R) in a sample of venezuelan university. Actualidades investigativas en Educación, 14(2), 1–23. Duro, D., Simões, M. R., Ponciano, E., & Santana, I. (2010). Validation studies of the Portuguese experimental version of the Montreal Cognitive Assessment (MoCA): Confirmatory factor analysis. Journal of Neurology, 257(5), 728–734. Egido, I., Fernández-Cruz, F. J., & Fernández-Díaz, M. J. (2016). Evaluation of the impact of quality management systems on school climate. IJEM, 30(4), 474–492. Everitt, S. (1975). Multivariate analysis: The need for data, and other problems. The British Journal of Psychiatry, 126, 237–240. Fernandes Malaquias, R., & de Oliveira, F. F. (2014). Project evaluation: Validation of a scale and analysis of its predictive capacity. Turkish Online Journal of Distance Education-TOJDE, 15(4), 90–111. Fernández, J. T., Lafuente, E. F., de los Santos, P. J., Torelló, Ó. M., Gámez, A. N., & Bueno, C. R. (2008). Implicaciones de la evaluación de impacto: Una experiencia en un programa de formación de formadores. Bordón. Revista de pedagogía, 60(1), 163–186. Fernández-Cruz, F. J., Fernández-Díaz, M. J., & Rodríguez-Mantilla, J. M. (2018). Diseño y validación de un instrumento de medida del perfil de formación docente en tecnologías de la información y comunicación. Revista española de pedagogía, 76(270), 247–270. Fernández-Cruz, F. J., Rodríguez-Mantilla, J. M., & Fernández-Díaz, M. J. (2015). Evaluación del impacto de la aplicación del modelo de excelencia europeo (EFQM) en el clima escolar y en la satisfacción de sus miembros. Investigar con y para la sociedad. Alicante: AIDIPE1179–1190.

Fernández-Cruz, F. J., Egido, I., & Carballo, R. (2016). Impact of quality management systems on teaching-learning processes. Quality Assurance in Education, 24(3), 394–415. Fernández-Díaz, M. J. (2013). Evaluación de impacto para un cambio sostenible en las organizaciones educativas. Revista Española de Pedagogía, 254, 119–138. Fernández-Díaz, M. J., Fernández, Álvarez, y Herrero, M., & Toranzo, E. (2002). La dirección escolar ante los retos del siglo XXI. Madrid: Síntesis. Fernández-Díaz, M. J., Rodríguez-Mantilla, J. M., & Fontana-Abad, M. (2016). Impact of implementation of quality management systems on internal communications and external relations at schools. Total Quality Management & Business Excellence, 27(1), 97–110. Ferrández Lafuente, E. (2006). La evaluación de impacto en el Master de Formación de Formadores. CIFO-FLC. Barcelona: Informes de Investigación Universidad de Barcelona. Franco-Paredes, K., Bautista-Díaz, M. L., Díaz-Reséndiz, F. J., & Arredondo-Urtiz, E. M. (2017). Propiedades psicométricas del Body Shape Questionnaire (BSQ-14) en mujeres universitarias mexicanas. Evaluar, 18(1), 1–11. García, J.Á., Brea, J. A. F., & Del Río, M. D. L. C. (2013). Implantación de un sistema de gestión de la calidad: Beneficios percibidos. Revista Venezolana de Gerencia, 18(63), 379–407. Gibb, J. (2003). What impact is implementing a quality system having on the vocational education and training classroom? Leabrook: National Centre for Vocational Education Research. Gie Yong, A., & Pearce, S. A. (2013). Beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9, 79–94. Giraldo, L. D., & Mera, R. (2000). Clima social escolar: percepción del estudiante. Colombia Médica, 31(1), 23–27. Glasson, J., Therivel, R., & Chadwick, A. (2013). Introduction to environmental impact assessment. London: Routledge. Goldberg, J. S., & Cole, B. R. (2002). Quality management in education: Building excellence and equity in student performance. Quality Management Journal, 9(4), 8. González-García, R. J., Añó-Sanz, V., Parra-Camacho, D., & Calabuig-Moreno, F. (2018). Perception of residents about the impact of sports tourism on the community: Analysis and scale-validation. Journal of Physical Education and Sport (JPES), 18(1), 149–156. Guilford, J. P., & Fruchter, B. (1973). Fundamental statistics in psychology and education (5th ed.). New York, NY, US: McGraw-Hill. Hair, J., Black, W., Babin, A., & Anderson, R. (2014). Multivariate data analysis: Pearson new international edition (7th ed.). New Jersey, Essex: Pearson. Hallinger, P., & Heck, R. H. (2010). Collaborative leadership and school improvement: Understanding the impact on school capacity and student learning. School Leadership and Management, 30(2), 95–110. Hayduk, L. A. (1996). LISREL issues, debates and strategies. London: JHU Press. Hernández, G., Arcos, J. L., & Sevilla, J. J. (2013). Gestión de la calidad bajo la Norma ISO 9001 en instituciones públicas de educación superior en México. Calidad en la educación, 39, 81–115. Holtzman, S., & Vezzu, S. (2011). Confirmatory factor analysis and structural equation modeling of noncognitive assessments using PROC CALIS. Statistics & Analysis, 1–9. Jackson, D. L. (2003). Revisiting sample size and number of parameter estimates: Some support for the N:q hypothesis. Structural Equation Modeling, 10(1), 128–141. Jackson, K. M., & Marriott, C. (2012). The interaction of principal and teacher instructional influence as a measure of leadership as an organizational quality. Educational Administration Quarterly, 48(2), 230–258. Kattman, R., & Johnson, F. G. (2002). Successful applications of quality systems in K-12 schools. Milwaukee: ASQ. Kim, K. H. (2005). The relation among fit indexes, power, and sample size in structural equation modeling. Structural Equation Modeling, 12(3), 368–390. Kline, R. B. (2015). Principles and practice of structural equation modeling. New York: Guilford publications. León, V., & Fernández-Díaz, M. J. (2017). Diseño y validación de un instrumento para evaluar la participación de las familias en los centros educativos. Revista Española de Orientación y Psicopedagogía (REOP), 28(3), 115–132. Lévy, J. P., Martín, M. T., & Román, M. V. (2006). Optimización según estructuras de covarianzas. Modelización con estructuras de covarianzas en ciencias sociales, 11–30. Lévy-Mangin, J. P., & Varela, J. (2006). Modelización con estructuras de covarianzas en ciencias sociales. A Coruña: Netbiblo. López Alfaro, P. (2010). El componente liderazgo en la validación de un modelo de gestión escolar hacia la calidad. Educação e Pesquisa, 36(3), 779–794. Lorenzo, M. (2011). Organización de centros educativos. Modelos emergentes. Madrid: La Muralla. Maas, K., & Liket, K. (2011). Social impact measurement: classification of methods. Environmental management accounting and supply chain management: 171-202. Netherlands: Springer. Malambo, D. E. (2015). Aplicación y seguimiento del modelo sistema gestión de calidad, desde el punto de vista de la eficiencia administrativa basada en la cultura organizacional en las instituciones de educación superior públicas en Colombia. Bogotá: Universidad Militar Nueva Granada. Mantilla, J. M. R., & Díaz, M. J. F. (2012). Development and validation of a measuring instrument for Burnout Syndrome in teachers. The Spanish Journal of Psychology, 15(3), 1456–1465. Marqués Graells, P. (2013). Impacto de las TIC en la educación: Funciones y limitaciones. 3c TIC: cuadernos de desarrollo aplicados a las TIC, 2(1), 1–15. Marsh, H. W., Kit-Tai, M., & Balla, J. R. (2010). Is more ever too much: The number of indicators per factor in confirmatory factor analysis. Journal Multivariate Behavioral Research, 33, 181–220. Mehta, N., Verma, P., & Seth, N. (2013). Total quality management implementation in

47

Studies in Educational Evaluation 62 (2019) 37–48

J.M. Rodríguez-Mantilla, et al. engineering education in India: An interpretive structural modelling approach. Total Quality Management & Business Excellence, 25(1–2), 1–21. Moreno-Bayardo, M. G. (1987). Introducción a la metodología de la investigación educativa. México: Editorial Progreso. Moreta-Herrera, R., López-Calle, C., Ramos-Ramírez, M., & López-Castro, J. (2018). Estructura factorial y fiabilidad del Cuestionario de Salud General de Goldberg (GHQ12) en universitarios ecuatorianos. Revista Argentina de Ciencias del Comportamiento, 10(3), 35–42. Mulaik, S. A., James, L. R., Van Alstine, J., Bennett, N., Lind, S., & Stilwell, C. D. (1989). Evaluation of goodness-of-fit indices for structural equation models. Psychological Bulletin, 105(3), 430–445. Nunnally, J. C., Bernstein, I. H., & Berge, J. M. T. (1967). Psychometric theory, Vol. 226. New York: McGraw-Hill. Oosterbeek, H., VanPraag, M., & Ijsselstein, A. (2009). The impact of entrepreneurship education on entrepreneurship skills and motivation. European Economic Review, 54, 442–454. Pasquali, L. (2008). Instrumentos psicológicos: manual prático de elaboração(Org.)(2nd ed.). Brasília: LabPAM-IBAPP. Perassi, Z. (2010). ¿En qué medida la evaluación colabora con la mejora escolar? Revista Iberoamericana de Educación, 54(4), 8–26. Ramírez, A. M., Rojas, L. E. A., Cortés, C. H., Lozano, R. Z. G., & Solís, J. E. M. (2013). Relación entre estilos de aprendizaje, rendimiento académico y otras variables relevantes de estudiantes universitarios. Revista de estilos de aprendizaje, 12(12), 151–166. Robles-García, M., Dierssen-Sotos, T., Martinez- Ochoa, E., Herrera-Carral, P., DíazMendi, A. R., & LLorca-Díaz, J. (2005). Variables relacionadas con la satisfacción laboral: un estudio transversal a partir del modelo EFQM. Gaceta Sanitaria, 19(2), 127–134. Rodríguez, D. (2006). Dimensión institucional, cultural y micropolítica: claves para entender las organizaciones educativas. Revista Electrónica Interuniversitaria de Formación del Profesorado, 9(1), 1–14. Rodríguez Mantilla, J. M., & Fernández Díaz, M. J. (2015). Diseño y validación de un instrumento de medida del clima en centros de educación secundaria. Educación XX1, 18(1), 71–98. Rodríguez-Mantilla, J. M., & Fernández-Díaz, M. J. (2012). Development and validation of a measuring instrument for burnout syndrome in teachers. The Spanish Journal of Psychology, 15(3), 1456–1465. Rodríguez-Ponce, E., Pedraja-Rejas, L., Araneda-Guirriman, C., González-Plitt, M., & Rodríguez-Ponce, J. (2011). El impacto del sistema de aseguramiento de la calidad en el servicio entregado por las universidades privadas en Chile. Ingeniare. Revista chilena

de ingeniería, 19(3), 409–419. Rositas, J. (2014). Los tamaños de las muestras en encuestas de las ciencias sociales y su repercusión en la generación del conocimiento. InnOvaciOnes de NegOciOs, 11(22), 235–268. Schaufeli, W. B., Salanova, M., González-Romá, V., & Bakker, A. B. (2002). The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness Studies, 3(1), 71–92. Stensaker, B., et al. (2007). Impact of quality processes. In L. Bollaert, S. Brus, B. Curvale, L. Harvey, E. Helle, & H. T. Jensen (Eds.). Embedding quality culture in higher education: a selection of papers from the 1st European forum for quality assurance (pp. 59–63). Brussels: European University Association. Stensaker, B., Langfeldt, L., Harvey, L., Huisman, J., & Westerheijden, D. F. (2011). An indepth study on the impact of external quality assurance. Assessment and Evaluation in Higher Education, 36(4), 465–478. Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2007). Using multivariate statistics, Vol. 5. Boston, MA: Pearson. Tejada, J., & Ferrández, E. (2007). La evaluación del impacto de la formación como estrategia de mejora en las organizaciones. Revista Electrónica de Investigación Educativa, 9(2), 1–16. Tejero-González, C., & Fernández-Díaz, M. J. (2009). Medición de la satisfacción laboral en la dirección escolar. RELIEVE, 15(2), 1–16. Thurler, M. G., & Maulini, O. (2010). La organización del trabajo escolar.: Una oportunidad para repensar la escuela. Barcelona: Graó. Trujillo, J. M. (2007). Análisis interno de la organización: liderazgo y micropolítica en un centro educativo. Innovación educativa, 17, 37–52. Vargas, T., & Mora-Esquivel, R. (2017). Tamaño de la muestra en modelos de ecuaciones estructurales con constructos latentes: Un método práctico. Revista Electrónica “Actualidades Investigativas en Educación”, 17(1), 1–34. Wang, J., & Wang, X. (2012). Structural equation modeling: applications using mplus. USA: John Wiley & Sons. Wong, P., Wong, D., & Zhuang, Y. (2017). Psychometric properties of the AIR self-determination scale: The Chinese version (AIR SDS-C) for Chinese people with intellectual disabilities. Journal of Intellectual Disability Research, 61(3), 233–244. Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. Zubieta, E. M., Delfino, G. I., & Fernández, O. D. (2008). Clima social emocional, confianza en las instituciones y percepción de problemas sociales: Un estudio con estudiantes universitarios urbanos argentinos. Psykhe (Santiago), 17(1), 5–16.

48