Modelling in evaluating a working life project in higher education

Studies in Educational Evaluation 38 (2012) 55–64

Contents lists available at SciVerse ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/stueduc

Modelling in evaluating a working life project in higher education Anneli Sarja a,*, Sirpa Janhonen b, Pirjo Havukainen c, Anne Vesterinen c a

Finnish Institute for Educational Research, University of Jyva¨skyla¨, Finland Institute of Health Sciences, University of Oulu, Finland c Laurea University of Applied Sciences, Finland b

A R T I C L E I N F O

A B S T R A C T

Article history: Received 11 July 2011 Received in revised form 5 June 2012 Accepted 6 June 2012 Available online 3 July 2012

This article describes an evaluation method based on collaboration between the higher education, a care home and university, in a R&D project. The aim of the project was to elaborate modelling as a tool of developmental evaluation for innovation and competence in project cooperation. The approach was based on activity theory. Modelling enabled a development of the curriculum and encouraged stakeholders to participate in the evaluation process. The results verified the features of the method: (1) the contradictions of the joint practice are a central source of evaluation; (2) comprehensive data collection methods are needed; (3) shared tools can be developed in interactive forums in ongoing evaluation, and (4) modelling makes professional expertise visible and brakes boundaries between different professions. ß 2012 Elsevier Ltd. All rights reserved.

Keywords: Program evaluation Evaluation methods Evaluation utilisation Systems approach Work-related learning

Introduction The purpose of this article is to describe the work-related learning process of higher education stakeholders in a joint research and development (R&D) project. In this project, called CaringTV1 higher education students were given the task of designing, implementing and evaluating interactive TV programmes as a service concept for the elderly. The programmes aimed to promote the health and well-being of elderly people by interactional involvement in the on-screen activities. The theoretical framework chosen for this study was developmental evaluation based on activity theory (Engestro¨m, 2001). In the organisational learning context, activity theory extends the unit of analysis beyond the individual learner. The present generation of activity theory focuses in particular on the challenges and possibilities present by inter-organisational learning. Moreover, the adoption of object-oriented and tool-mediated activity is one of the main contributions of activity theory (Engestro¨m & Kerosuo, 2007). Developmental evaluation is based on an interventionist approach developed from the basic ideas of activity theory. It is an approach to the study of transformations and learning in work and organisations where the conceptual models and tools of activity

* Corresponding author at: Finnish Institute for Educational Research, P.O. Box 35, FI-40014 University of Jyva¨skyla¨, Finland. Tel.: +358 40 4827496; fax: +358 14 617418. E-mail addresses: anneli.sarja@jyu.fi (A. Sarja), sirpa.janhonen@oulu.fi (S. Janhonen), pirjo.havukainen@laurea.fi (P. Havukainen), anne.vesterinen@laurea.fi (A. Vesterinen). 0191-491X/$ – see front matter ß 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.stueduc.2012.06.001

theory are donated to the participants in order to help them work on a problem of practice (Engestro¨m, 2001, 2005, 2007). Earlier research has indicated that the advantages of this approach include the development of the participants’ personal awareness and understanding of each other’s ways of thinking and acting, creating a shared understanding, and establishing natural collaboration between the participants (Edwards, Daniels, Gallagher, Leadbetter, & Warmington, 2009; Saari & Kallio, 2011). A weakness identified in most development projects has to do with the difficulty of changing participants’ ways of thinking and acting, which for some participants has proved too difficult (Engestro¨m, 2005). In the CaringTV project we wanted to test whether the method of modelling would elicit the desired developmental patterns in the various players’ ways of thinking and acting in the different stages of the project. It was expected that the models produced by the analyses in the different stages would lighten the burden on the participants. While revealing problems in their collaboration, the modelling would conceal the limitations of the participants’ personal approaches. In particular, it was hoped that the models would encourage participants to engage in an open joint dialogue in their project meetings. In this article, first, we present the context of the study, i.e., a collaborative R&D project conducted by vocational higher education students and a partnering working life organisation as part of a larger CaringTV scheme. Second, we compare the similarities and differences between program theory-driven evaluation (Donaldson & Gooler, 2003; Donaldson, 2007), complexity-sensitive developmental evaluation (Patton, 2008, 2011), and developmental evaluation based on activity theory (Engestro¨m, 2001, 2004).

56

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

In addition, we focus on the meaning-making process of modelling in each of these evaluation approaches. Third, we describe the joint research process in the CaringTV project in which the principal lecturers worked together with the stakeholders from the different participating organisations. The university researchers clarified and supported the activity theory-based modelling process with the principal lecturers from vocational higher education. Finally, the conclusion and discussion section considers the developmental evaluation process based on modelling and reviews its usefulness for the stakeholders as a tool for project co-operation. We also view the various phases of the modelling process in light of other research findings. The context of evaluation This study was conducted in the context of CaringTV, a collaborative development project between educational institutions and partnering working life organisations. More specifically, the CaringTV project was designed and executed in collaboration between a Finnish university of applied sciences, a care home, a company offering technical solutions, and university researchers. According to the ‘‘Learning-by-Developing’’ operational model (LbD model), which represents the curricular framework of the university of applied sciences, students’ learning is tied to development work with professionals. The LbD operation model combines higher education and working life development into a systematic R&D project. In the present instance the latter took the shape of the CaringTV project, which aimed at developing interactive television programmes for the elderly as a service concept. The university of applied sciences led the design, implementation and developmental evaluation of these programmes. The potential target users of the programmes were residents of the care home and elderly people living in its vicinity. The programmes were to contain health guidance and also operationally promote the old people’s health and well-being. A working life development project based on the LbD model brings students, teachers and working life representatives together to develop, examine, create, produce and share new kinds of innovative competencies. Co-operation in a working life project provides the opportunity to promote and develop interaction in authentic operational environments. The model combines individual learning with the learning and building of new knowledge of a community (Raij, 2007). Consequently, working as part of such a project not only produces new competencies, but also creates new kinds of networks through which R&D projects can be utilised to promote regional competitiveness and to respond to the challenges of the surrounding world. Different approaches to practice-based evaluation Many practice-based evaluation approaches typically seek to involve the stakeholders in the evaluation process by mapping their perspectives on the issues under construction. Moreover, many of these approaches underline the importance of mixed methods, i.e., use of various data collection methods in a range of relevant contexts. However, different approaches differ in the role assigned to stakeholders’ perspectives and the mixture of methods used. For example, Donaldson (2007) emphasises a concise three-step model for conducting the basic activities of program theory-driven evaluation science: (1) developing program impact theory, (2) formulating and prioritising evaluation questions, and (3) answering evaluation questions. According to Donaldson (2007), interactive, bi-directional program theory development is the most effective evaluation model in terms of reaching consensus between stakeholders and striving for greater precision in theorizing programs. Evaluation begins by designing a model through debates

and discussions in joint sessions between the participants in the program (Donaldson & Gooler, 2003; Donaldson, 2007). The model serves as a guide for formulating and prioritising evaluation questions, answering them, and assessing the outcomes. In this evaluation approach, both qualitative and quantitative methods are used in collecting and analysing the data (Chen, 1997; Donaldson & Gooler, 2003; Donaldson, 2007). The model’s ‘theory’ contributes to the attainment of the aims of the program. According to Patton (2011), developmental evaluation supports innovation development and guides adaptation to emergent and dynamic realities in complex environments. He summarises the main points of developmental evaluation, and emphasises that the mutual co-operation between stakeholders and evaluators has to be active and open. Moreover, Patton presents a ‘reflective practice cycle for ongoing developmental evaluation’. The phases of this reflective process are: (1) formulate question for focused reflection, (2) share stories, (3) analyse patterns and themes, and (4) identify implications: actions, lessons. The developmental evaluation process is closely linked to actual change in practices. Innovations can take the form of new projects, programs, products, organisational changes, policy reforms, and system interventions. In addition, rapid and timely feedback to stakeholders is essential. Dynamic and complexity-sensitive projects of this kind do not slow down or wait for evaluators’ reports. Consequently, the methods used in this approach are emergent and flexible. According to Patton (2008), this type of evaluation focuses on the interdependence of the relationships between the participants and the different kinds of interaction they engage in. It follows that this kind of continuous, dynamic evaluation has a selfcorrecting effect. As a matter of fact, also the designs can be dynamic. Dynamic co-operation will be successful only if the evaluators and stakeholders have regular contacts. It is also important that the evaluative conclusions are drawn collaboratively with the stakeholders. Consistently with the principles of the theory-driven type of evaluation, Patton does not exclude any evaluation method, design or tool but points out that a successful design calls for systems thinking to capture and map complex system dynamics and interdependencies, and track emergent interconnections (Patton, 2011). Activity theory offers a framework for studying R&D evaluation at the level of organisations, defined as activity systems, which are willing to collaborate on working life projects (Engestro¨m & Kerosuo, 2007; Engestro¨m, 1999, 2004). According to this framework, such project co-operation progresses through four critical phases: (1) planning of the developmental evaluation model and its tools, (2) analysis and modelling of current practices, (3) recognising the contradictions between the stakeholders’ perspectives and transforming the model, and (4) construction of a shared object through open dialogues. Continuing collaboration between the evaluators and the stakeholders is a central prerequisite for successful project development. In this approach, it is also important to bring out participants’ differing and even contradictory perspectives on the issues under construction. Different perspectives can be brought out by modelling the ongoing process of collaboration. The model is then used as a joint tool by the stakeholders when they meet to conceptualise, design and evaluate new practices and approaches. The participants’ various perspectives make up the shared object of activity for the meeting forums. The object embodies the meaning and the motive of the collective activity system. In developmental evaluation, the object is given to the stakeholders as a preliminary model (raw material). The model is then interpreted, constructed and changed by the evaluators in their meetings (Engestro¨m & Toiviainen, 2011). The meetings are based on open dialogue between the participants (Sarja & Janhonen, 2009) and negotiation about their common goals and tools as well as their mutual division of labour in

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

constructing the common outcome of their inter-professional collaboration. In open dialogue, the participants are willing to share their tacit knowledge. Open dialogue also involves an orientation towards collaborative practices, during the modelling process. On this view, dialogue is contextually embedded in current practical activities and challenges that connect the stakeholders with each other. The construction of different perspectives through practicebased open dialogue is the central feature of developmental evaluation. On the basis of these negotiations, a new operation model is designed allowing for regular revision of its correspondence with practice. The participants also review goal achievement in their joint meetings. In sum, developmental evaluation based on activity theory (Engestro¨m, 2004, 2005) uses modelling as a tool for change and innovation in bringing out the viewpoints of stakeholders coming from different organisations. Activity theory and developmental evaluation emphasise that the values, norms and working practices of professionals stem from historical development. In each collaborating organisation, or activity system, the stakeholders have over time adopted different ways of thinking and acting that are connected with their particular activities, including the tools used, the rules that direct their activities, different community cultures and the division of labour between them. In a developmental evaluation, the stakeholders become committed to actively participate in the process by transforming their earlier ways of thinking and acting. It follows that undertaking a developmental evaluation can be considered a learning process for them as well, since through the process they become conscious of their own contextual framework, and it is this consciousness that makes relational change possible. Furthermore, investigative procedures can be used to approach a development project, collecting data with flexible methods and detecting organisational contradictions in the development project by modelling the joint activities (Engestro¨m & Kerosuo, 2007; Hakkarainen & Janhonen, 1997). By holding regular forums between participants and enabling open interaction, it is possible to generate a partially shared object for an R&D project as well as new goals and organisational changes in working practices in order to achieve these (Ha¨ma¨la¨nen & Kauppi, 2000). In such forums, modelling is a tool commonly used to initiate focused dialogue. Interaction between professionals in an educational and those in a work organisation creates a potential for collective learning for all the stakeholders concerned, whereas the individual perspectives of professionals are often quite limited. Engestro¨m (2005), Patton (2011), and Donaldson (2007) have different views about the use of modelling in developmental evaluation (Table 1). According to Donaldson (2007), modelling can be defined as a ‘‘theory’’ designed jointly by researchers and participants as the operational model of the development project.

57

This operational model sets the focus for the research questions and determines the appropriate instruments for data collection. The researchers give the participants feedback on the basis of the results and the players involved then adjust their practices accordingly. Patton (2011) also stresses the importance of collaboration between the researchers and the actual players. This collaboration begins with a joint meeting, where the players share their views on their own developmental needs. Based on their stories and statements, relevant themes and dynamics within the complex system are mapped, and their mutual relationships tracked. The whole of this is then portrayed in the form of a model. The model will then direct the process of developmental evaluation (Patton, 2011). In developmental evaluation based on activity theory (Engestro¨m, 2001), the emphasis, once again, is on the continuing collaboration between the researchers and players throughout the development process. Modelling brings to light possible contradictions between the different players’ approaches and the targets of the activities in question. In joint meetings, which are based on open dialogue, the players seek solutions to these contradictions with the aim of developing new practices. According to Donaldson (2007) and Patton (2011), the process of developmental evaluation begins with a joint meeting. In contrast, in developmental evaluation based on activity theory (Engestro¨m, 2001) the first meeting is preceded by surveying participants’ personal points of view in individual interviews, as this best brings out their personal differences in thinking and acting (Table 1). Modelling as a tool of developmental evaluation in activity theory Modelling is a concept that is used to describe many different issues in different contexts. In the context of meaningful learning, modelling, as a metacognitive skill, has often been replaced by several other concepts. Metacognition refers to awareness of one’s own cognition, its control and regulation, i.e., skills of selfevaluation. It covers knowledge and beliefs about cognition (cognitive aspect) as well as regulation of cognition (skills and behaviour aspect). Metacognitive skills include, for example, selfreflection, utilisation of feedback, and collaboration (Flavell, 1979). Drawing on different sources, Irvine (1995) has found the following concepts describing the same model: concept trees, web teaching, knowledge mapping, semantic networking and cognitive mapping. All of these terms describe the same approach, that is to say, an individual’s diagrammatic interpretation of meaningful learning. Concept maps are used as tools when seeking to identify gaps and misconceptions in subject matter being taught or studied (Kinchin, Cabot, & Hay, 2008). Modelling is also a tool of metacognitive reflection. Raeithel (1983) proposed three types

Table 1 The focus of evaluation in practice-based theory-driven and different approaches of developmental evaluation. Issues

Programme theory driven evaluation (Donaldson, 2007)

Complexity-sensitive developmental evaluation (Patton, 2011)

Developmental evaluation as an activity system (Engestro¨m, 2001)

The collaboration between stakeholders and evaluators

The focus of collaboration is on developing a common understanding of the programme (‘‘theory’’) through debates and discussions The programme impact ‘‘theory’’ based on collaboration between stakeholders and evaluators is used as a guide for formulating and prioritising evaluation questions and also for answering them

Mutually respectful relationship during the reflective practice sessions. Stakeholders share stories for tracking programme development and document changes Patterns and themes based on stakeholders’ stories are used to map complex system dynamics and track emergent interconnections

Mixed methods are used in collecting and analysing data as well as in analysing the results

Mixed methods are emergent and flexible (only speed up matters)

The evaluators and stakeholders work together during the whole project in terms of open dialogues in regular and various sessions The evaluators use modelling to highlight the various viewpoints of the stakeholders as well as to point out the contradictions between different organisations. The model acts as the object of open dialogues producing iterative or structural change (new practices) Mixed methods as a support of developing ‘‘models’’. Semantic analysis and modelling are used for producing the ‘‘models’’

The idea of ‘‘modelling’’

The method

58

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

of reflection. In the metacognitive reflection, i.e., in centration, a participant/an actor concentrates on himself/herself and his/her performance in any given task. The concept map represents this first type of reflection as an evaluation method of individual learning, even though the acting process is not conscious at this stage. In this case, the participant/actor primarily seeks an answer to the question of how well the method he/she uses serves him/her with regard to his/her competencies. However, in directing outwards, i.e., in decentration, it is crucial to ask how the task could be arranged differently, in which case it is important to question the traditional individual learning method and its goal. Decentration helps clarify the working activities, the object of the work and the tools used for the work. Co-operation is important. As the relationship between the participant/actor and the community develops, a more collective way of working is achieved. As a result of reflection using decentration, the activity process develops into a social/collective process instead of an individual one. The work activities are not viewed from the perspective of what is best for the individual, but they are ‘our’ shared object (Raeithel, 1983). In the third type of reflection, i.e., in recentration, the participant/ actor considers himself/herself as part of a community, as he/she directs his/her attention not only to both a shared task and action and to the wider context of developing these, but also to the internal dynamics of the community. On this level, a shared development task could be expressed in the form of the following question: ‘‘How can we organise collaboration in as flexible a manner as possible?’’ The traditional understanding of the significance of metacognition in learning only answers the first question regarding the individual participant’s own competencies; this initial reflection (centration) is directed inwards to individual learning. However, the questions concerning the third level of reflection (recentration) are directed at the tensions connected with organising the collaboration between participants/actors. These tensions can best be made visible by modelling the perspectives of the different participants/actors, which can vary according to the particular situation. The central concepts are the processes of work activity, practice and collaboration that also function as the tool. According to Raeithel (1983), metacognitive reflection makes a person better aware of his/her own intellectual activities. Personal awareness develops gradually from the individual level to community level awareness. Centration (Urzentrierung) refers to the starting point of reflection, that is, the initial stage of individual awareness, where the person is becoming aware of the goal of activity, although at this point the activity is not yet conscious. In the next phase, decentration (Dezentrierung), the activity with its targets and necessary equipment are already outlined more clearly. At this stage collaboration is important. Through personal reflection, the activity develops into a social joint process, and is thus no longer an individual level process. Therefore, the activity is no longer considered from the perspective of individual performance but has become a shared object – ‘‘ours’’. Recentration is not always necessary but it becomes an essential prerequisite when people wish to find new perspectives for their collaboration. Recentration always yields a new perspective of some kind, which also increases shared awareness of the activity system as a whole. This is the phase for a transition from individual work tasks to professional collaboration (von ta¨tigkeit zur neuen Perspektive und zur Praxis). Central concepts here include activity processes, praxis, and collaboration, which can also be seen as devices of social interaction within the activity system. From the viewpoints of activity theory and developmental evaluation, modelling focuses on the entire process from the perspective of the project and the development of participants’/ actors’ competencies. With regard to modelling as an evaluation tool, the project can be seen as an element which increases cooperation through the modelling process. In our sample case, the

knowledge produced by the evaluation was instantly utilised in order to develop both the CaringTV project and the competencies of its collaborative parties. Modelling supports the development and direction of professional skills and competencies. The evaluation results are utilised throughout the process by making the collective development of all parties visible in the form of a developmental system model. In the context of activity theory, modelling helps reveal disturbances and conflicts between the perspectives of professionals in regard to their knowledge, the rules that guide their actions, community cultures, the division of labour and the object of their activities. This way, modelling acts as an inspirational starting point in the discussion between the participants in creating future goals, as well as in choosing and developing the methods to attain these goals (Engestro¨m, 1999). According to Engestro¨m, 1984, modelling also highlights the sophistication of the model that directs thoughts and activities. Thus, models form a hierarchical structure. In Engestro¨m’s (1984) pyramid, the bottom level is spontaneous orientation. Its formation is based on observation, immediate experiences, imitation, and trial and error. Spontaneous orientation can take any shape as a model. The next orientation level, the advanced organiser (e.g., the table of contents of a book), results from partitioning, comparing, organising, categorising and hierarchising. In addition, a successful and high quality performance can be separated into different phases and, following the logical sequence of solving a difficult problem, it is also possible to form an algorithm and a heuristic rule, both of which offer a clear directive. Creating a system model already requires system-analytical methods. For example, the activity system model can be formed with the help of an investigative procedure. Since the system model represents the inputs and outputs of activity and the factors connected with them, the model enables the collaborative designing, implementation and evaluation of joint actions. Approach and research task The first aim of our joint venture was to evaluate the development of professional collaboration in the CaringTV project. As already mentioned, the project was implemented in partnership with students and teachers as well as other project stakeholders. Our first evaluation question was:  How does stakeholder project collaboration develop during the CaringTV project when applying modelling as a tool for developmental evaluation? According to practice-based activity theory, the crucial issue of transformative learning is that it has to be constructed by cultural tools. Shared tools make the reconstruction of the object and motive for joint working possible. Therefore, inter-professional learning allows the creation of a new activity system (Engestro¨m, 2001). Based on this premise, the second research problem was formulated as follows:  How does the developmental evaluation of project stakeholder collaboration by means of modelling promote new kinds of innovative learning and the development of competencies in the collaborating organisations? Methodology Starting point and data collection In the CaringTV project, data were collected throughout the evaluation process in three different phases. The project commenced with discussions between the principal lecturers of the participating university of applied sciences and the university researchers. The

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

59

Table 2 Participants and the process of developmental evaluation. Participants

Phase 1 Modelling discussion

University researchers Principal lecturers Senior lecturers Students Elderly people Personnel of care home

2 2

Phase 2 Focus group

Round-table

2 2 3

2 3

Phase 3

Modelling discussion

Focus group

Round-table

2 2

1

1 2

1

resulting data were collected through focus group interviews with lecturers, students, a representative of the care home and the principal lecturers. Moreover, the lecturers and principal lecturers took part in round-table discussions. The first phase (phase 1) started with the analysis of current practices. The round-table discussions involved lecturers, elderly people, a representative of the care home and a principal lecturer. These discussions dealt with the interviews conducted with the elderly and the care home personnel at the beginning of the project, followed by the interviews with students and teachers in focus groups and round-table discussions. The data from the succeeding phases included student focus group interviews and round-table discussions with the lecturers, care home personnel and principal lecturers. Table 2 lists all the participants in the different forums (23 altogether) and the data collection phases of the developmental evaluation process. Within each modelling phase (1, 2 and 3), the data were analysed and a new model created and introduced in the

3

Modelling discussion

Focus group

Round-table

2 2

2

2 3

3 7 1

4 2

discussion forums where, on the basis of the discussion, the participants agreed upon the goals of the next phase and the methods for reaching them. After each phase, the lecturers and researchers met for theoretical discussions and to evaluate the results achieved in light of the data collection. They then integrated the results into the theory and planned the next data collection phase. Project meetings were held between every phase of the data collection process. The process of developmental evaluation for each phase is shown in Fig. 1. The data were collected by two principal lecturers from the university of applied sciences. The focus group interviews were conducted with representatives of the same group (Wibeck, ¨ berg, 2007). In the round-table discussions the Dahlgren, & O models created on the basis of the data analysis were evaluated. The model helped the participants bring their professional perspectives to bear regarding the implementation of their goals in the joint CaringTV project. During these round-table discussions,

Fig. 1. The research process and evolving model design in the CaringTV project.

60

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

the different project stakeholders could create joint concepts and develop a common language, and so learn to understand each other’s practices and the reasons behind them. When professionals commit themselves to development work practices, they also induce each other to express conflicting or contradictory as well as complementary points of view (Wibeck et al., 2007). Data analysis The data were analysed thematically by identifying discordances as well as innovations (expansions) between the participants’ perspectives within the modelling process. Constructing the shared object of joint activity through shared tools is a prerequisite for the evolution of inter-professional collaboration. Here, modelling in itself is an activity through which the contradictions preventing or restricting transformation of inter-professional collaboration at the different levels can be identified (Engestro¨m, 1999). Conceptual understanding of the different practices causing contradictions makes it possible to develop shared tools and instruments, i.e., a new model of joint activity (Engestro¨m, 1999). The main phases of modelling the developmental evaluation observed were: (1) identifying the perspectives and goals of key persons in the working life development project, CaringTV, (2) analysing the different perspectives in the framework of the activity system model, and (3) identifying the partially shared object of the CaringTV project from the viewpoint of the participating stakeholders. The results of this study are also illustrated by describing the progress of the co-operation processes in the CaringTV project and for each phase analysing the relevant orientation basis.

Fig. 2. Analysis of the individual stakeholders’ current practices.

Results

development. In their round-table discussion, the participants came to understand that their common goal in the CaringTV project was promotion of the well-being of elderly people. In the interviews, the project stakeholders focused on their own roles and tasks. The actions of the lecturers and students were guided by the curriculum of the university of applied sciences, whereas the actions of the nurses of the care home were guided by its working culture of effective care. The challenge of the modelling process was how to define the common object of collaboration connecting all the stakeholders in the CaringTV project. The ideal of the shared object is illustrated in the first model (Fig. 2). Because this article focuses on modelling as a tool for developmental evaluation, the figures show the development targets assigned to the participants at each stage of the development project. In other words, the model indicated the current stage of the process, and the participants always started their discussions inspired, in particular, by the model.

Starting the program: planning the developmental evaluation model and accompanying tools for the CaringTV project

Second phase: recognising the contradictions between the stakeholders’ perspectives

Choosing modelling as a tool for evaluating project co-operation The analysis and evaluation of the CaringTV project started with a discussion between the principal lecturers and the university researchers. The principal lecturers expressed the goals of their project stakeholders, connecting them to the different activities and societal functions of their background organisations. The researchers emphasised that it would be difficult to specify the precise form of the model at the outset of project co-operation; instead, it would be more important to define the stakeholders and the object of joint activities and the developmental transfers potentially catalysed by the project. The process continued with round-table discussions organised by the lecturers and involving the project participants.

During the interviews it became apparent that the division of labour between the stakeholders, as well as the perception of the outcome of the shared object for the R&D project, was still in want of clarification, as can be seen in Table 3. In the joint development process of the CaringTV project, the division of labour between the participants was created spontaneously, despite the fact that the project called for shared responsibility from all parties. Traditionally, the operational culture of the higher education institute has been based on autonomy, although an important element of the LbD model concerns partnership and a research-oriented approach. These contrasting perspectives became evident in the teacher interviews as well. For example, the lecturer of the course on ‘Care for the elderly’ considered teaching a task for the individual. However, the project manager/coordinator (also a lecturer), expected CaringTV to be clearly integrated into the other courses and studies provided in the higher education institute because she was both responsible for technological matters and also guiding student learning during this project. Meanwhile, a TV studio was being built especially for the project (Fig. 3). The second model (Fig. 3) continued to represent a spontaneous orientation base. The factors limiting the development of the project were identified in the next phase of the modelling process. The principal lecturers and researchers discussed how, and in what forums, the joint activities could be clarified through modelling in order to reach the shared outcome of promoting the well-being of the elderly. Contradictions were identified on the basis of the interviews and round-table discussions, and solutions were sought by evaluating the model in round-table discussions. The modelling revealed that the rules governing the joint

First phase: emergence of the first model—analysis of the individual stakeholders’ current practices The discussion expressed the different perspectives of the stakeholders of higher education and working life during the planning of the CaringTV project. In interviews the participants described their individual actions in their own organisations. The lecturers reported that their main task was to supervise the learning process of their students in caring for the elderly. Thus, the lecturers and students saw the CaringTV project as only one of the many learning methods available. In the beginning, the few elderly people who happened to be around at the time were told about the CaringTV project, and subsequently constituted the participants in the first programme. Later, with improved dissemination of information, the number of elderly participants increased. They also started to express their own wishes for

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

61

Table 3 Project-stakeholders’ perspectives on the CaringTV project. Participants

Knowledge

Rules

Community

Division of labour

Outcome

Higher education Coordinator

Linking project to other studies

Lecturers (teachers)

Utilising innovation competence

Implementing LbD model Curriculum

Co-operation in the learning network Higher education

Development of CaringTV as a service concept New evaluation methods

Students of social sciences and health care Care home Personnel

Interactive skills and competencies Content knowledge

Curriculum LbD model

Co-operation in the learning network

Technology, communication, study guidance Supervision of students learning processes Designing, implementing, evaluating the programmes

Practical arrangements

Effective care

Care home

Elderly people

Wishes concerning content of the programmes

–

Care home

activities, forms of co-operation and the division of labour had not been agreed upon, and that the stakeholders were not guided by the tasks of the project organisation in their efforts to reach the commonly set goals. Third phase: constructing a partially shared object of interprofessional collaboration In this phase, the division of labour and responsibilities between the participants in the project were defined. The model was shaped within the framework of activity theory (Engestro¨m, 2004) and revealed the practices constraining inter-professional collaboration. Moreover, the model served as a shared tool through which it was easy to externalise, compare and redefine the current and traditional ways of thinking and acting in each organisation Employing the model in round-table discussions activated the participants’ awareness and directed them to develop the CaringTV programme within their common development project. In the focus interviews, the shared activity system pertaining to the project was clarified as a result of developmental evaluation through which the responsibilities and the division of labour between the project manager, teachers and nurses of the care home became clearer. With the help of modelling, they were already able to solve some of the practical issues during the discussion phase, when different perspectives were openly dealt with. By exposing practices that limited interprofessional collaboration, it was possible to develop new, more effective joint practices. In fact, the contradictions, tensions and dilemmas of the activity system became the foundation for the

Technical execution of the programmes Participation in the programmes

Self-directedness, guidance and interactive skills Marketing of the programmes Well-being, health promotion

evaluation work, since these made innovations possible at the activity system level in this project, and ultimately helped clarify the benefits of CaringTV as a learning tool and instrument for joint development (Fig. 4). Fig. 4 shows what each group of participants learnt during the project and what they were ready to develop in the future in relation to the project. The elderly were activated to promote their own well-being. They appreciated the fact that their opinions made a difference in improving the programme. The nursing staff learnt to use the interactive CaringTV programme as a tool. They also learnt to value students as collaboration partners in promoting the well-being of the elderly. The students were enthusiastic about the joint planning and its realisation, which extended beyond the boundaries of the CaringTV programme to their studies in the field of social and health care more generally. The teachers realised the meaning of networking as well as the importance of development activities for supporting their own professional growth. The school staff, guided by the lecturers, learnt to apply modelling in evaluating learning and project activities. The jointly created model activated discussion also in the final stage of the project. Fourth phase: the distributed model of the CaringTV project as a tool of joint reflection The evolution of models proceeded from the spontaneous orientation base towards more elaborate models. Model 3 of the R&D project was based on interviews and the focus group and round-table discussions, and depicted the partially shared object of the project stakeholders on an ideal level (Fig. 4). Although the modelling process of project co-operation promoted the participant’s professional understanding, the modelling was not used as a tool for the joint planning, implementation and evaluation of interactive actions. In the round-table evaluation discussions, the model helped clarify the shortcomings in the mutual division of labour, both for each professional participating in the project and for the network of activity systems. The interactive practices of the collaborative stakeholders

Fig. 3. Recognising the contradictions between the stakeholders’ perspectives.

Along with the modelling, the project participants’ awareness of joint activities and their common goals was deepened, and the stakeholders’ skills and competencies progressed from the individual towards the collective level. The stakeholders gradually developed a shared vision regarding the object of the collaborative project and their joint responsibility. The division of labour was also clarified, and the stakeholders organised the project in mutual co-operation. In phase 4, another round-table discussion concerning the model was arranged. By the time this phase commenced, the potential of modelling as a shared cultural tool was already

62

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

Fig. 4. Constructing a partially shared object of inter-professional collaboration.

understood by all the participating organisations and their participants. The round-table discussions made the perspectives clearer with regard to innovations that were significant for the construction of both inter-professional collaboration and the working practices of the organisations in this R&D project. Stakeholders spontaneously evaluated the personal benefits they had gained from the project and the new opportunities it offered them (Table 3). From the perspective of the lecturers in higher education, this working life project created new ways of learning, and they reported that the documentation of the CaringTV programmes had, in particular, promoted the evaluation of co-operative learning. Other key issues pointed out by them included developing the curriculum and utilising health- and welfare-related technology in teaching and project work. They became aware of the strengths and advantages of having students from different subject areas in developing the content and interactivity of the CaringTV programmes. The strength of the health care students lay in their professional competence in matters related to elderly people’s health. The students of social services, in turn, concentrated on developing their interactive skills and competencies with a new tool, which in comparison with the health care students, enhanced their customer-orientation. By combining such complementary skills and perspectives, it was possible to create a new approach which took into account both content and interaction in the discussions with the elderly. The elderly considered the programmes produced by the students to be interactive, co-operative and entertaining. They presented some ideas for the further development of the content of the programmes. Through the programmes, the elderly also learned to evaluate their everyday practices from the perspective of promoting their well-being. Based on the assessment by the nurses of the care home, the R&D project’s main contribution was

an increase in the versatility of the services available to the elderly and the development of the staff’s skills in utilising technology. Moreover, the personnel expressed a willingness to invest in marketing the CaringTV project in the future in collaboration with the project participants and with the aid of written materials. Modelling transformed the R&D project into a new kind of learning environment and the project stakeholders were willing to comply with the resulting changes. It also brought about stakeholders’ willingness to not only analyse their own action models but to also evaluate the various working practices of the participating organisations and their joint development as part of the entire activity system. Meanwhile, throughout the collaboration process, the tool of modelling provided the participants with an opportunity to evaluate and transform the content of the CaringTV programmes. This shows that modelling, as a tool of learning and development, can be applied to different types of working life development projects. However, documenting the various phases of the project is necessary, as the different documents may contain evidence essential for the modelling of the project and/or for the developmental evaluation of the process. Conclusion and discussion This article has described a work-related learning process implemented among higher education stakeholders in a joint R&D project utilising a developmental evaluation approach based on activity theory and the tool of modelling. Models have two functions in the context of activity theory and transformative learning; they can serve as models of something and models for something (Mattila, 2006, 50). In particular, models act as tools enabling on object-oriented activities on the systemic level of joint working. In our sample case, the modelling process involved lecturers from higher education and personnel and clients of a

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

care home. During the process, in which principal lecturers (from a university of applied sciences) implemented the modelling in practice, drawing on the university researchers’ modelling expertise, each participant realised that successful collaboration would serve their common goal, viz. promoting the health and well-being of the elderly. In this article the main question has been how to implement a joint modelling process so that it can act not only as an investigative instrument for collaboration on a R&D project but also as a tool for improving the CaringTV project. In answer, we can state that the modelling process enabled the real co-operation to take place between the different parties, as shown in the reformed thinking and working patterns of individual players. The students were able to gain experience in multiprofessional collaboration starting from the fields of social and health care (degree programme in Nursing and degree programme in Social Services) and expanding to include a truly multidisciplinary student network (including degree programmes in beauty and cosmetics, and business management). Teachers’ activities also developed into broad network-based co-operation. They learnt that instruction can be designed in collaboration and draw on joint planning instead of solely individual efforts. The elderly took a keen interest in the development activities and acted very flexibly. The nurses learnt to take a new perspective on the students entering the nursing home for training. Earlier, the students were regarded as something of a nuisance, because they needed instruction and thus simply added to the staff’s workload. Along with the project, the nurses learnt to see students as innovative co-operation partners. In addition, the staff learnt to utilise TV equipment for information purposes in other contexts as well. The business partner whose equipment was used in the project did not want to be actually involved in the project. Yet, in the third phase of the project this partner realised that his organisation would need the feedback generated by the programme in order to use it for their own developmental purposes. The modelling process comprised four phases: (a) analysis of the project stakeholders’ current practices, (b) recognition of contradictory practices by transforming the model created in phase 1, (c) construction of the partially shared object of joint practice and, finally, (d) reflection on the distributed model of the CaringTV project. Moreover, the stakeholders responsible for the concrete setting up of the project acquainted themselves with the background theories of modelling as a tool of evaluation in project co-operation. In the course of the modelling process the stakeholders became aware of the partially shared object of their inter-professional collaboration and related factors. A visual model outline helped make visible the tensions in joint practices and activities and to see these as developmental challenges on the systemic level. This finding is congruent with earlier research where interorganisational practices have been organised and changed through conceptual tools or methods (e.g., Hakkarainen & Janhonen, 1997; Halonen, Kallio, & Saari, 2010; Miettinen & Virkkunen, 2005; Saari & Kallio, 2011). Next, we describe these four phases more thoroughly in the context of earlier research. In the initial phase of the developmental evaluation process, Analysis of the project stakeholders’ current practices, the evaluation discussion dealt with the model created by the principal lecturers based on the focus group interviews and round-table discussions with senior teachers, students and a person from the care home. Because the orientation activities of the stakeholders were partly based on resources provided in connection with their current practices (tools, collaborators, resources), the first model served as a spontaneous orientation basis for the project (Engestro¨m, 1984). As a result, every stakeholder focused on their own operations and the rules guiding them. The higher education lecturers acted according to the traditional curriculum while the coordinator of

63

the CaringTV project in higher education implemented the elements of authenticity and partnership of the LbD model (Table 3). According to these ideas, the R&D project should correspond with the areas in which the students wish to become experts. Moreover, partnership means multilevel collaboration and competence-sharing so that the students can work as researchers and developers as well as facilitators in the use of the modelling tool (Raij, 2007). On this level of a working life development project, the stakeholders’ reflection focused, in line with the idea of centration (Raeithel, 1983), on the co-ordination of their unconnected tasks. In the next phase of the developmental evaluation, Recognition of contradictions in the CaringTV project, a shared understanding of the joint project started to take form when the stakeholders showed readiness to use modelling as a tool in representing the rules governing their practices (Engestro¨m, 1999). Reflection on the discussions was identified according to decentration, in which the stakeholders questioned their operational tools and together sought to resolve the practical tensions arising out of co-operation in the project (see Raeithel, 1983). This finding can be explained by the practice-based theories of learning and working (Hakkarainen & Janhonen, 1997; Miettinen, 2000) and related open dialogue (Huotari, 2008; Sarja & Janhonen, 2009). In this view, dialogical interaction is contextually embedded in the shared object of activity and in the professionals’ willingness to expand and share their knowledge and expertise by going beyond the social context of their own reflections on their practices in relation to those of others. In dialogue-supporting innovations such as the one studied here the basis of coordination is sought in the life world of the elderly people instead of the stakeholders’ specialised tasks (Huotari, 2008). These results are in line with those of earlier research on inter-professional collaboration, in which the organisational contexts and constraints within which the professionals work are redefined (Edwards et al., 2009) so as to enable the formation of a partially shared object between the collaborating activity systems (Konkola, Tuomi-Gro¨hn, Lambert, & Ludvigsen, 2007). In the final phase of the evaluation process, the distributed model of CaringTV was utilised as a tool of joint reflection. As a result, recentration was identified in the reflection that took place in the discussions, such as when the stakeholders questioned and redefined the rules that governed the ways they worked together on the CaringTV project (Raeithel, 1983). The first model (Fig. 2) represented a spontaneous orientation base (Engestro¨m, 1984). It was built on the participants’ experiences and impressions of the project. A spontaneous orientation base can manifest itself in any form. In this phase the stakeholders’ discussion illustrated reflection in accordance with centration, as described by Raeithel (1983). Planning, implementing and evaluating of the CaringTV programmes engaged the students in authentic working life problems. In order to achieve their learning goals, the students were required to work in partnership with the other stakeholders in this working life project. Accordingly, the experiential and interactive nature of the students’ learning was highlighted when they received direct feedback on the interactive and participatory potential of the television programmes even as they were implementing them. According to the students’ assessment, they had learnt to support client involvement as well as to design and evaluate the programmes they produced for the elderly in order to promote their well-being. In summary, it is worth noting that in this project, modelling was not, in the first instance, very likely to yield direct and concrete changes. In fact, in this study the joint process of developmental evaluation generated an ideal model for the activity system towards which the CaringTV development project should be steered in the future. Thus, a major issue and challenge for future research is how the distributed model of CaringTV will motivate the participating

64

A. Sarja et al. / Studies in Educational Evaluation 38 (2012) 55–64

professionals whose tasks and duties regarding the developmental evaluation of their object-oriented work practices are very different. Carrying out a developmental evaluation means continuous interaction between a practice and its evaluation (Halonen et al., 2010; Saari & Kallio, 2011). Nevertheless, modelling makes it possible to incorporate such an evaluation in the various processes of steering co-operation so that it becomes an organic part of the developing of the joint activity. When modelling is integrated in the planning, implementation and transformation of the joint activities, it can be seen as a continuous process of collecting evaluation knowledge as well as interpretative reflection on the findings of the evaluation and what they mean (Ha¨ma¨la¨nen & Kauppi, 2000). Moreover, a joint model provides a new cognitive tool for students, teachers and workplace representatives to apply in developing their professional competencies, skills and expertise in everyday practice. On the basis of our empirical analysis and in light of Mattilas’ (2006) study of interdisciplinary models and their functions in scientific practice, we argue that through modelling, development projects can be better bound to their objects and outcomes, as in the present CaringTV programme, and thus be reviewed and evaluated for further actions. Acknowledgements The authors would like to express their sincere gratitude to the anonymous reviewers who offered them insightful advice on improving the quality of the manuscript. References Chen, H. (1997). Applying mixed methods under the framework of theory-driven evaluations. New Directions for Evaluation, 74(2), 61–72. Donaldson, S. (2007). Program theory-driven evaluation science: Strategies and applications. New York: Psychology Press. Donaldson, S. I., & Gooler, L. (2003). Theory-driven evaluation in action: Lessons from a $20 million state wide work and health initiative. Evaluation and Program Planning, 26(4), 355–366. Edwards, A., Daniels, H., Gallagher, T., Leadbetter, J., & Warmington, P. (2009). Improving inter-professional collaborations: Multi-agency working for children’s wellbeing. London: Routledge. Engestro¨m, Y. (1984). Orientointi opetuksessa. [Orientation in instruction]. Helsinki: Valtion painatuskeskus. [In Finnish]. Engestro¨m, Y. (1999). Innovative learning in work teams: Analyzing cycles of knowledge creation in practice. In Y. Engestro¨m, R. Miettinen, R.-L. Punama¨ki (Eds.), Perspectives on activity theory (pp. 377–404). Cambridge: Cambridge University Press.

Engestro¨m, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of Education and Work, 14(1), 133–156. Engestro¨m, Y. (2004). New forms of learning in co-configuration work. Journal of Workplace Learning, 16(1/2), 11–21. Engestro¨m, Y. (2005). Developmental work research: Expanding activity theory in practice. Berlin: Lehmanns Media. Engestro¨m, Y. (2007). Putting activity theory to work: The change laboratory as an application of double stimulation. In H. Daniels, M. Cole, & J. V. Wertsch (Eds.), The Cambridge companion to Vygotsky (pp. 363–382). Cambridge: Cambridge University Press. Engestro¨m, Y., & Kerosuo, H. (2007). From workplace learning to inter-organizational learning and back: The contribution of activity theory. Journal of Workplace Learning, 19(6), 336–342. Engestro¨m, Y., & Toiviainen, H. (2011). Co-configurational design of learning instrumentalities: An activity-theoretical perspective. In S. R. Ludvigsen, A. Lund, I. Rasmussen, & R. Sa¨ljo¨ (Eds.), Learning across sites: New tools, infrastructures and practices (pp. 33–52). London: Routledge. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitivedevelopment inquiry. American Psychologist, 34(10), 906–911. Hakkarainen, P. E., & Janhonen, S. (1997). Teaching practice as a testbench of learning in Master’s degree education for nurse teachers in Finland. Nurse Education Today, 17(6), 454–462. Halonen, M., Kallio, K., & Saari, E. (2010). Towards co-creation of service research projects: A method for learning in networks. International Journal of Quality and Service Sciences, 2(1), 128–145. Huotari, R. (2008). Development of collaboration in multiproblem cases: Some possibilities and challenges. Journal of Social Work, 8(1), 83–98. Ha¨ma¨la¨nen, K., & Kauppi, A. (2000). Evaluation in today’s education: From control to empowerment. Lifelong Learning in Europe, 5(2), 68–75. Irvine, L. M. C. (1995). Can concept mapping be used to promote meaningful learning in nurse education? Journal of Advanced Nursing, 21(6), 1175–1179. Kinchin, I. M., Cabot, L. B., & Hay, D. B. (2008). Visualising expertise: Towards an authentic pedagogy for higher education. Teaching in Higher Education, 13(3), 315–326. Konkola, R., Tuomi-Gro¨hn, T., Lambert, P., & Ludvigsen, S. (2007). Promoting learning and transfer between school and workplace. Journal of Education and Work, 20(3), 211–228. Mattila, E. (2006). Questions to artificial nature: A philosophical study of interdisciplinary models and their functions in scientific practice. University of Helsinki: Philosophical Studies from the University of Helsinki 14. Miettinen, R. (2000). Varieties of constructivism in education: Where do we stand? Lifelong Learning in Europe, 7(1), 41–48. Miettinen, R., & Virkkunen, J. (2005). Epistemic objects, artefacts and organizational change. Organization, 12(3), 437–456. Patton, M. Q. (2008). Utilization-focused evaluation. Los Angeles: Sage. Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press. Raeithel, A. (1983). Ta¨tigkeit, Arbeit und Praxis. Campus: Frankfurt am Main. Raij, K. (2007). Learning by developing. Laurea Publications A 58. Helsinki: Edita Prima. Saari, E., & Kallio, K. (2011). Developmental impact evaluation for facilitating learning in innovation networks. American Journal of Evaluation, 32(2), 227–245. Sarja, A., & Janhonen, S. (2009). Methodological reflections: Supervisory discourses and practice-based learning. Teaching in Higher Education, 14(6), 619–630. ¨ berg, G. (2007). Learning in focus groups: An analytical Wibeck, V., Dahlgren, M. A., & O dimension for enhancing focus group research. Qualitative Research, 7(2), 249–267.