- Email: [email protected]
Evaluation of a problem-based curriculum in midwifery
Evaluation of a problem-based curriculum in midwifery Christine McCourt and B. Gail Thomas
Objective: to evaluate the implementation of a ‘problem-based’ learning (PBL) curriculum within midwifery. Design: a ‘realistic evaluation’ model based on cost-e¡ectiveness models of economics, and incorporating both qualitative and quantitative methodology.The design used historical comparison, comparing students following the new curriculum with a baseline of previous student cohorts and exploring sta¡ and student experiences. Setting: a ‘new’ university, providing a degree-level 18 month pre-registration midwifery programme for an intake of about 40 students per cohort. Participants: all students in the cohort qualifying immediately before the implementation and all those in the ¢rst cohort following the new curriculum were invited to participate fully. Assessment outcomes for three cohorts of students qualifying before and the ¢rst three cohorts qualifying following the change were analysed. All clinical sta¡ directly involved in mentoring the relevant student groups, all academic sta¡ involved in delivering the new curriculum and the course external examiner were invited to participate. Intervention: the evaluation studied a major change in the way the overall midwifery curriculum was delivered, widely described as PBL. Measurements: students’experiences and perspectives were sought through review of routine evaluation documents, nominal group technique exercises and focus group discussions at about six months following quali¢cation. Clinical sta¡ experiences and perspectives were sought through written structured questionnaires. Academic sta¡ views were sought through personal semi-structured interviews. Participant observation of the process, review of course documents and of sta¡ re£ective commentaries were also conducted. Data on student completion rates and ¢nal grade were analysed.
Christine McCourt, BA, PhD Senior lecturer in health services research Wolfson Institute of Health and Human Sciences Thames Valley University 32-38 Uxbridge Road London W5 2BS E-mail: [email protected] B. Gail Thomas, RGN, RM, ADM, PGCEA, MSc Dean, Wolfson Institute of Health and Human Sciences Thames Valley University 32-38 Uxbridge Road London W5 2BS (Correspondence to CM) Received 23 July 2000 Revised 23 November 2000 Accepted 5 April 2001
Findings: although students’ views of the curriculum were generally positive, they experienced some discomfort and di⁄culty in adjustment to a new style of learning. Small group learning, independence and skills in using and conveying information and research evidence were valued aspects of the new approach.Quality of clinical placement experience and university-practice links were an important area of positive experience or concern. Clinical sta¡ views of the curriculum showed considerable stability, with major concerns being the balance of theory and practice time and skills which were not in£uenced by the curriculum change. At both points, ability to critically evaluate and use evidence in practice was regarded highly. Student outcomes showed some evidence of possible divergence of grades in the new curriculum. Conclusions and implications for practice: the manner and context of implementation of educational change may have important implications for student experiences and outcomes and the experiences of educators. Adequate preparation at all levels for a di¡erent, more independent style of learning is important and students are likely to need clear guidance and feedback on their progress in the early stages of the curriculum, to monitor progress and to provide reassurance. Longer-term research is needed to assess the impact of PBL on theory-practice links and on the midwives as practitioners. & 2001 Harcourt Publishers Ltd Midwifery (2001) 17, 323^331 & 2001 Harcourt Publishers Ltd doi:10.1054/midw.2001.0276, available online at http://www.idealibrary.com on
324 Midwifery
INTRODUCTION In 1997, a new curriculum for pre-registration (short) midwifery (an 18 month programme for registered general nurses) was implemented at Thames Valley University’s Wolfson Institute of Health Sciences (Thomas et al. 1998) modelled on a ‘problem-based learning’ (PBL) approach (Barrows & Tamblyn 1980, Barrows 1986). Briefly, a traditional class format was replaced with small group ‘learning sets’ around a series of ‘triggers’ conducted over three-week cycles. In each cycle, the students were introduced to a real-life scenario or ‘trigger’ based on midwifery practice, intended to stimulate students to question and to want to learn. The group planned what areas of learning needed to be covered, who would lead each, undertook this work with support from a range of resources and presented their investigative work at the end of the trigger cycle. The term ‘lecturer’ was replaced with ‘learning set leader’ to reflect change to a more facilitative role: while the approach to education had previously been fairly evidencebased and student centred, students were now expected to adopt a more independent and group-centred approach. The evaluation that was attached to the development and the key themes and issues arising at an early stage are presented in this article. As such, the issues discussed relate particularly to a change process and responses to change, rather than the impact of an established educational programme or approach.
Background PBL has been adopted in a number of medical schools, particularly in North America and Australia (Schilling et al. 1995, Margetson 1996) and for other health-related curricula (Creedy & Hand 1994, Sadlo 1994). This was the first course in midwifery in the UK to adopt a PBL approach to the whole curriculum, although such principles have been piloted for midwifery modules elsewhere (Pansini-Murrell 1996, Wise 1996, 2000) and interest in this educational approach is growing. The aims and principles of PBL are generally seen as to promote learning which is relevant to real life experience and which can be retained and used well by the student. They draw on broad educational principles and theories set out by writers such as Freire (1972) and applied in concepts such as ‘adult learning’, (Knowles 1984, Rogers 1994), ‘lifelong learning’ (Dearing 1997), ‘deep versus surface learning’ (Entwistle 1987) and of the ‘reflective practitioner’ (Schon 1983, 1987). The aims were interpreted locally in terms of enabling students to understand the relevance of theoretical issues to practice, to apply ideas
and knowledge to real life situations, to continue learning once qualified and to take a reflective, questioning and woman-centred approach in their work (Quant 1999).
The existing evidence on Problem Based Learning A number of descriptive and evaluative studies of this approach have been conducted, primarily in medicine, which generally indicate positive outcomes although, some were part-programmes or only included volunteer students. A metaanalysis of the research (Vernon & Blake 1993) found that PBL curricula and courses were generally popular with students and well evaluated by staff. However, students appeared to achieve lower scores on traditional examinations that tested factual knowledge. They showed better performance where ‘process’ measures were tested. Various forms of comparison were used in the studies reviewed, including pure experimentation, own-control and static-control. No differences were apparent across the different study designs. The reviewers concluded that ‘PBL’ educated students may perform better in the long term, due to improved motivation and understanding of learning processes and argued that formal examination scores may not be a good guide to future clinical performance. Evaluating the occupational therapy degree programme in the UK Sadlo (1994) also found good student motivation and response but slightly poorer examination scores. In an experimental study of PBL in nursing, Lewis and Tamblyn (1987) found that traditional students performed slightly better in independent observation of clinical placement performance. A study of nursing student experiences identified differences in students’ approaches towards their work, in particular a greater sense of personal responsibility for learning and for practice among PBL educated students (Biley & Smith 1998). Such studies reflect the arguments of the theoretical literature that PBL may change students’ attitudes and approaches to learning and thereby, their approaches to practice (Benner 1984, Margetson 1991, Schilling et al. 1995). Both the theoretical and research literature indicate that possible benefits of this approach may be subtle and long term and so may not be easily captured by conventional forms of assessment, such as written examinations. The research also suggests that PBL educated students may show some gaps or weaknesses in factual knowledge on assessment, which may be addressed over time by a more effective development of learning skills, sense of responsibility and enquiring approach.
Evaluation of a problem-based curriculum in midwifery
Design of the evaluation Our approach was based on the classical economic model of cost effectiveness evaluation which has been widely used in programme evaluation in the UK and north America, looking at inputs, process and outcomes of a programme to understand its effectiveness (Fig. 1) (Donabedian 1966). The adapted model is often called ‘realistic evaluation’ and usually incorporates both qualitative and quantitative research methods (Youll & McCourt-Perring 1993, Buxton & Hanney 1994, Pawson & Tilley 1997). Unlike the more positivist traditions of economics, this approach does not assume that all the issues to be taken into account are predictable, controllable or easily measurable. It allows for researchers to get involved by providing feedback during the evaluation, and for those directly involved (stakeholders) to have a say in how things are done (Quinn-Paton 1980, Guba & Lincoln 1989,Youll & McCourt-Perring 1993). Therefore, evaluation was conducted alongside and in close relation to curriculum development with the aim of providing feedback to that process, but led by researchers within the Institute who were not directly involved in planning or delivery of the new curriculum. Historical comparison was used to identify and explore possible effects of the change: experiences and outcomes for students following the new curriculum were compared with a baseline of students qualifying under the previous degree curriculum. Such a model needs to be treated with caution and cannot identify causal relations in the manner of pure experimental research (Oakley 1998) but can take account of the complexity and unpredictability of complex organisational changes and the wide range of variables that may influence process and outcomes.
325
authority in relation to their education. Participants were given detailed written and verbal information on the evaluation methods and aims and were assured that participation in interviews and focus groups was voluntary and confidential. All data gathered were anonymised to protect individual confidentiality and former students taking part in focus groups were asked, verbally and in written information sheets, to respect the confidentiality of their peers’ contributions. All routine student evaluation records were anonymous. A steering group involving researchers and curriculum managers was convened to coordinate and monitor both the implementation and the evaluation. Data were gathered relevant to input, process and outcomes of the programme. A range of methods was used reflecting the complexity of the situation under analysis, the need to gain a range of perspectives and desire to triangulate sources where possible. Choice of methods was also pragmatic, since a very short time span was available for collection of baseline data before the new programme commenced. Hence a more structured approach was used with clinical mentors. The specific methods used are described briefly for each of the three stages in turn:
Inputs: the baseline . Survey of clinical mentors’ views of the existing curriculum (n=38) A brief questionnaire including 5-point scales on seven statements relating to curriculum quality and two open questions on the strengths and weaknesses of the course was distributed to all clinical mentors via students and academic staff (See Table 1). As there were no records of the number of staff acting as clinical assessors/ mentors we do not know the proportion of mentors who responded. . Interview with the external examiner
METHODS
A semi-structured interview was conducted to elicit views of the strengths and weakness of the current curriculum and its development over time.
Ethical considerations At the time this evaluation commenced there was no Research Ethics Committee in the University. The work would be examining sensitive issues for those undergoing change and involved students who might feel reluctant to criticise those with
. Focus group discussions with qualified midwives (n=11/35):
recently
All midwives qualifying in the last cohort before implementation (n=35) were invited by letter
Fig. 1 Process model for programme evaluation INPUT ? context/history existing service model resources (human and material) programme aims programme design
PROCESS ? context ^ other changes or developments ways change is implemented new model: organisation and practice resources (human and material) operation of programme any changes to design/practice
OUTCOME informal and formal assessments e.g. audit data, exam results views of stakeholders any change in use assessment of e¡ects in practice description of model in practice
326 Midwifery
Table 1 Clinical mentors survey responses ^ previous curriculum n=38 Question Score ? Do students/are students
1
2
3
4
5
Missing
mode
Mean
Know how to judge the quality of research Know how to use research in practice Able to apply what they learn in the classroom to what they see in practice Question things they hear about in practice Able to identify and negotiate own learning Able to relate well in a small group Seem keen to carry on learning when they f|nish Total (mean)
0
4
13
15
4
2
4
3.53
0
2
11
21
3
1
4
3.68
0
4
18
14
0
1
3
3.19
0
2
7
16
13
0
4
4.05
0
1
11
20
5
1
4
3.78
0
0
7
19
9
3
4
4.06
0
3
11
14
9
1
4
3.78
0
16
78
119
43
9
and follow-up telephone calls to take part at about 3–6 months following qualification. The midwives were asked, using a semi-structured topic guide, to discuss their views of the course and how it had prepared them for practice. The discussions were taped and transcribed in full. Unstructured telephone interviews were also conducted to avoid excluding the views of those who did not wish to participate in a group discussion. There was considerable difficulty in organising these groups but 11 from the pre PBL and 12 from the post PBL courses participated. Many had moved elsewhere or were travelling (13 and 12 respectively could not be contacted) and 3 from each cohort were unable to attend on the day due to duty rota problems. Those who declined (4 and 2 respectively) gave practical reasons and or were not practising as midwives. Telephone interviews were conducted with those (2 pre and 2 post PBL) who did not wish to take part in a group discussion, primarily as they were not practising or were considering leaving midwifery. . Observation of key planning and development meetings . Review of course documents, including routine written student evaluations (n=32/35)
The Process . Interviews with new ‘learning set leaders’ (n=10) and review of reflective commentaries. The experiences of academic staff will be reported elsewhere. . Student evaluation using Nominal Group Technique exercises (Reed & Proctor 1993) These were conducted in the learning sets, at the review of each trigger to provide an analysis of students’ responses to the curriculum, as it unfolded (n=48 exercises per cohort across all
3.72
learning sets, with all students present for the session involved). Students identified issues in small groups as strengths/positives or weaknesses/problems and then voted, casting three votes each, on a pooled list of issues, with their learning set leader acting as facilitator. This was continued for the first three cohorts of students, to allow time for settling in and review of the curriculum and development over time. . Review of course documents and routine student evaluations. . Participant observation of curriculum meetings, staff development and staff/student evaluation sessions to gain a fuller view of the change process and the responses to this, and to provide regular informal feedback to participants.
Outcomes Data gathered at the baseline were repeated for the first cohort of ‘PBL’ students as follows: . Survey of clinical mentors’ views of the new curriculum (n=38) The survey was administered as before . Interview with the external examiner A semi-structured follow-up interview to reflect on the curriculum change. . Focus group discussions with recently qualified midwives (n=12/35) Discussions and telephone interviews were conducted, following the same processes as before, with the first cohort of students qualifying under the new curriculum (n=35). . Analysis of data on formal educational outcomes Formal outcomes were assessed for six intakes of students–three before and three after the introduction of the new curriculum to avoid the possibility of normal variation from one year to another skewing the apparent outcomes.
Evaluation of a problem-based curriculum in midwifery
327
Outcomes analysed included final grades, pass and completion rates.
researcher’s understanding of the data (Webb 1989, Reason 1994).
Analysis Structured data–mentor questionnaires and student grades–were analysed using descriptive statistics and content analysis for the open questions. Inferential statistics were not used as they were not felt to be appropriate in a study where only one small aspect of the work used structured measures and the numbers were small. We were concerned not to attach undue significance to such measures in what was clearly a very complex changing situation. Qualitative data were analysed using a textual approach, assisted by use of the NUD*IST software programme (Richards & Richards 1994). In this method, the texts of transcripts are coded in detail, using open codes generated from the data. Codes are then indexed to identify key themes. The use of such an approach facilitates the grounding of theory in the data by facilitating constant, iterative movement between a coded passage of text and its context as well as between different areas of data generating the same or linked codes (Strauss & Corbin 1990). However, the use of such a package, which utilises a relatively rigid ‘tree’ structure for generation of themes, was found to be limited for a complex study with multiple sources of data, particularly that generated by participant observation. The nominal group technique data were prestructured by the process as students themselves undertook the first phase of analysis by identifying and voting on key issues. Subsequent analysis involved reviewing the reports of the exercises and the issues identified in each cross-sectionally to draw out key recurring themes. A further important area of analysis in evaluative research of this type is the process of feedback and discussion with the different participants, which may stimulate both change in practice and in the
FINDINGS The findings are presented from three main perspectives: clinical mentors’, students’, and formal assessment. Those of academic staff will be reported elsewhere. Findings from the new curriculum are set alongside those of the previous curriculum to facilitate comparison.
Clinical mentors’ perspectives The structured findings of the pre- and postimplementation surveys are given in Tables 1 and 2. At both time points, 38 forms were returned and 33 and 32 of these respectively included some open comments. Overall, views of both curricula were positive, the most frequent score being 4, although the mean scores for PBL students were marginally lower than those before the change. The question focused on the relationship between theory and practice scored a more neutral 3 in the pre-implementation survey, suggesting that mentors were a little less confident about students’ ability to link theory and practice than they were about other aspects of education. No clear changes were found with the new approach. In their open comments, a set of themes emerged which remained fairly stable over time. Several themes had both positive and negative aspects. For example, the students’ independence and self-motivation was seen as positive but some mentors also felt that students needed more guidance and direction. The key areas of strength in both curricula identified were: . students’ confidence and ability to use research evidence and their enhanced access to research resources, literature and a good knowledge base;
Table 2 Clinical mentor responses ^ 1st cohort of new curriculum, n=38 Question: /score ?do students
1
2
3
4
5
Missing
Mode
Mean
Know how to judge the quality of research Know how to use research in practice Able to apply what they learn in the classroom to what they see in practice Question things which they hear about in practice Are able to identify and negotiate own learning Relate well in a small group Seem keen to carry on learning when they f|nish Total (mean overall)
1
2
18
11
5
1
3
3.46
1
3
15
15
4
0
3/4
3.47
0
5
15
17
1
0
4
3.37
0
4
8
18
8
0
4
3.79
0
5
12
14
7
0
4
3.61
0 1
1 0
6 13
22 21
8 3
1 0
4 4
4 3.66
3
20
87
118
36
2
3.62
328
Midwifery
. their independence and self-direction; . students taking a critical thinking questioning approach.
and
The key themes emerging as weaknesses in both curricula were: . time pressures leading to conflicts between academic and practice work; . lack of clinical practice time and experience (affecting clinical skills and confidence); . a theory/practice skills gap. Mentors’ comments about weaknesses were more concerned with limits on clinical time and experience than the nature of the universitybased programme. Knowledge and awareness of the new curriculum was high and they were conscious of some of the anxieties being expressed by students during their studies. However, analysis of the questionnaire responses indicated few differences and, as before, mentors’ critical comments focused mainly on conflicts between time given to theory and practice which have not been affected by the change in educational approach.
Students’ perspectives Student Nominal Group Technique evaluations The key issues raised in student evaluations, using the nominal group technique process, are summarised in Figure 2. Overall, they suggest that students saw both strengths and weaknesses in aspects of the curriculum such as small group and self-directed learning, where there was much of value but also much to learn from experience.
They valued the principles of the approach but expressed anxieties about levels of guidance and direction and the coverage and depth of their knowledge. Their comments also hint at parallel uncertainties amongst academic staff. During the course, students’ comments shifted to reveal adaptation and greater comfort with the style of learning over time, but concerns about the need for more input from lecturers, guidance and feedback on their progress remained.
Former students’ views – group and individual interviews Students qualifying just before the introduction of PBL were generally positive about their experience. Although some students experienced initial difficulties in understanding the relevance of some course material, they felt they had been able to retain academic knowledge and it had ‘come together’ or ‘clicked in’ once they were practising. They expressed confidence in dealing with new situations and uncertainty, with finding and using or critiquing research evidence and established practice. They also showed interest in further learning, at this stage mainly via gaining practice experience. The main problem areas identified were linked to poor or inconsistent input in clinical placements, particularly due to fragmentation and staff shortages within the service. They were also critical of some mentors’ didactic approach to education and to practice – one example given was of being shown precisely how to lay out a delivery trolley without any explanation of why. Several also felt that the links between college and practice placements
Fig. 2 Students’ NGTresponses summary Advantages Self directed learning: Encourages self-discipline and management, motivation, conf|dence, sense of own direction, independence
Disadvantages Self-directed learning: Uncertainty and anxiety ^ depth and scope, need more guidance & sense of direction, fears about knowing enough, fears about imbalance, gaps in knowledge
Information Developing skills in using information and research evidence, searching and researching, analysis
Information Tutors holding back, not giving us their knowledge, need more feedback and informal assessment Problems accessing resources in time Lack of conf|dence in presenting
Developing presentation and communication skills Lateral thinking and problem solving
Lack of focus, unsure where to begin
Group learning: Group dynamics, working together Co-operation and interaction skills, sharing, gives more time to cover topics Discussion, active participative style Role plays
Group learning: Have to rely on and trust others, people not pulling weight Sharing means only covering part of topic in depth Gaps if others’ work not adequate Reluctance to criticise peers
The triggers: Learning from real life situations Relevance to practice, linking to practice Inherent interest Learning the process
The triggers Easier once have practice experience T|metabling problems, unfamiliarity with style Need to learn the process
Evaluation of a problem-based curriculum in midwifery
329
had not been adequate, leaving them feeling unsupported if they did experience problems. Their main concerns about continuing to learn, or to use research evidence in practice, focused on access to facilities, time or structures to facilitate this. The responses of the first group qualifying under the new curriculum echoed those of the previous cohort of students in many ways. They were generally positive about their universitybased education but some expressed concerns about the quality and coherence of their clinical placement experience. They were equally confident in using research evidence and felt they had developed valuable skills in seeking out and using new information. However, they expressed more anxiety about possible gaps in their skills and knowledge on qualifying than the previous group, who were aware of the their limits but confident that they would continue to learn. An important benefit of the new curriculum was in developing their skills and confidence in putting across information to others. For example, they felt more prepared and willing to take on the challenge of leading antenatal classes. They also reflected on the experience of being in some ways an ‘experimental’ group. They described the approach as quite different from their previous education, including their nursing curricula, particularly in requiring a high degree of personal independence and inter-dependence within the group. Their accounts of the trigger process also suggest that considerable skills in facilitation were important to the approach. While some lecturers used critical questioning to stimulate discussion and provide feedback to students on the depth of their work, others were described as ‘sitting back’ and feeling ‘unable to teach’. The issues for academic staff, on which they touched will be discussed elsewhere, but what is notable from the students’ viewpoint is the degree to which uncertainty about the curriculum change might be conveyed by the particular approaches taken.
at the beginning, more structured teaching early in the course, a shift to focusing on the whole trigger topic rather than dividing the work into particular areas and clearer guidance and feedback on clinical and theory assessments. Routine evaluations by the following cohort of students (n=25) showed similar themes but there had been a shift in the comments related to the PBL approach, with more students stating that they liked the approach and noting the relevance of triggers to practice. However, there was still a demand for more structured teaching early in the programme and more practical teaching sessions. Some students continued to express concern about their coverage of topics and whether they had missed any of the ‘basics’. Their comments suggested that subsequent groups were less anxious about the programme change, but still held some concerns about this more independent style of learning.
Routine student evaluations Thirty-two students in the last cohort of students qualifying before the new curriculum completed routine evaluation forms that showed similar views to those given in the group discussions. Their suggestions for improvement included better preparation for clinical mentors, better university/practice links, a stronger role for the link tutor and more practical teaching sessions. A few also requested more small group and case study-based teaching. Twenty-three students in the first PBL cohort completed routine evaluation forms and the majority participated in a verbal end-of-course evaluation that also gave similar views to those in our group discussions. Suggestions for improvement included better preparation for the approach
Outcomes External examiner’s perspective The examiner was able to comment on development of the degree-level course since its inception and felt it had been refined and improved over time. Standards overall, and teacher expectations, were high, although there was a problem of large variations of ability and performance between students. A follow-up interview and review of external examiner reports indicated that there had been little change in students’ performance in formal assessments: levels of midwifery knowledge and overall standards remained good. At both points, student difficulties or failures were viewed as mainly linked to difficulties in expressing ideas in written form and with critical analysis, rather than lack of substantive knowledge. She felt this reflected the wide intake on such courses and the need for structured learning support before or early in the curriculum for some students. Formal outcomes assessment Analysis of formal assessment of students indicated limited change in students’ overall performance, but some evidence of divergence of grades (see Table 3). In the three cohorts examined following the curriculum change, the percentage of students with 1st class and 3rd class degrees or failure increased. Rates of student withdrawal or deferrals, which may reflect personal or academic difficulties, or a decision that this was not the right course for them, remained stable. This analysis, therefore, does not indicate clear changes, but the possibility that the pattern of allocation of grades may be widening under the new approach and the need to keep this under review.
330 Midwifery Table 3 Analysis of degree classif|cations Degree classif|cation
n
Pre PBL Post PBL
6 16
1st % 5 12
2.1
3rd
2.2
n
%
n
%
n
%
61 40
50 30
39 46
32 35
3 9
2 7
not completed or failed n % 13 20
10 15
Pre-pbl=3 intakes, March 1996 ^March 1997 n=122 Post-pbl=3 intakes, Sept.1997^Sept.1998 n=131 (plus 2 still deferred)
DISCUSSION Strengths and limitations of the research There is an inherent problem in investigating cause and effect relationships in educational research where controlled experimentation is not possible and the range of potential factors is complex and hard to predict. Triangulation of methods, and inclusion of a range of perspectives, allowed for the identification of possible biases or inconsistencies in the findings. Overall the levels of agreement between the different groups was high, suggesting that the patterns of response obtained were unlikely to be biased towards the views of particular groups and that respondents felt able to be reasonably frank and open with the researchers. Additionally, the process of feedback to participants enabled researchers to compare the analysis with the understandings of the different groups involved in the process. Such factors provide increased confidence in the validity and reliability of the research findings. It is important to remember that this article examines something new and consequently, people’s responses to change. Longer-term study is planned to enable an assessment of experiences and outcomes once the course is more established and to explore the longer-term impact of the students’ education. Also, the change we have focused on is necessarily complex and cannot be described and understood entirely or simply in terms of a model that is likely to be implemented differently in different situations. The context and baseline for change are also important. The evidence gathered shows considerable stability of responses and outcomes, as compared with baseline data before implementation of the PBL approach. We did not find any clear evidence of enhanced formal outcomes or improvements in students’ ability to link theory to practice. However, stability at such an early stage of a fairly radical change to teaching and learning method might be considered a positive outcome. We did note a possible trend towards greater divergence of students’ final grades. The reasons for this are unclear and it is not possible, on the basis of this study, to comment on whether this should be interpreted as a negative or positive outcome, or whether it is likely to be
enduring. It could be argued, for example, that a PBL curriculum is better able to distinguish students who are not well suited to a career in midwifery and facilitates those who are well prepared and motivated to achieve. Conversely, it could be suggested that either the approach itself, or the newness of the curriculum led to a less directive type of guidance, increasing the possibility of lower results for less confident students. Such a possibility was reflected in the higher levels of anxiety expressed by the first cohort of students following the PBL approach, as compared to previous students. Their anxieties about gaps in skills and knowledge were not, however, reflected in the data gathered on outcomes from mentors and examiners who regarded the group as having similar levels of midwifery knowledge to previous students. It was, therefore, largely a selfperception that may have been influenced by several factors. In this first cohort, students had observed raised levels of uncertainty amongst teaching staff, who were themselves adjusting to considerable change, which undermined their own sense of confidence. They expressed concerns about gaps since they felt more competent in those topics they had focused on personally in the triggers than those covered by others in their group. This was not only a matter of needing to develop trust in other group members and learning to work effectively as a ‘team’ but also a recognition that information is, indeed, retained better when you have sought it out and studied it in depth yourself. Students’ anxieties about reliance on the work of others in the group, and their initial reluctance to take a critical, questioning approach, also suggested that they were themselves bringing particular, more didactic, models and expectations of teaching to their work. They felt ‘teaching’ by the lecturer was being replaced with ‘teaching’ by their peers. Initial anxieties about presentation, likewise, encouraged them to focus on their own or their peers’ skills in presentation as a performance, rather than as a means to discussion and mutual learning. A third possible interpretation of their expressed anxiety is that, given greater responsibility for seeking out knowledge, these students developed a heightened awareness of the scope and complexity of the information ‘out there’ and the limits of
Evaluation of a problem-based curriculum in midwifery
331
their own knowledge on qualifying. This need not be a negative finding if those students have sufficient confidence, motivation and opportunities to continue to learn once in practice. Midwives who, in their education, learn (or at least do not ‘unlearn’) how to seek out and use information, to manage uncertainty as part of a learning process may be better placed to become truly reflective practitioners. A number of adjustments were made to the developing approach in response to this evaluation, including more initial preparation for the students for this style of learning, providing workshops on practice related topics and providing feedback and guidance to students in a way that is timely and appropriate to the PBL approach. Further attention will be needed to clinical learning and to university/placement links, including preparation of mentors to work within this framework. Longer-term research is planned to assess the impact on the students as practitioners.
NUD*IST: Non-nUmerical Data Indexing, Searching and Theory building. Sage, Scolari, London. www.scolari.co.uk Oakley A 1998 Experimentation in social science: the case of health promotion. Social Sciences in Health 4(2): 73–89 Pansini-Murrell J 1996 Incorporating problem-based learning: striving towards women-centred care. British Journal of Midwifery 4(9): 479–482 Pawson R, Tilley M 1997 Realistic evaluation. Sage, London Quant V 1999 How do student midwives perceive their problem-based learning course and does it prepare them to provide woman-focused care? Unpublished MA dissertation, University of Reading, UK Quinn-Paton M 1980 Qualitative evaluation methods. Sage, Beverley Hills Reason P 1994 Three approaches to participative inquiry. In: Denzin NK, Lincoln YS (eds) Handbook of qualitative research. Sage, London Reed J, Procter S 1993 Nurse Education: a reflective approach. Edward Arnold, London Richards TJ, Richards L 1994 Using computers in qualitative research. In: Denzin NK, Lincoln YS (eds) Handbook of qualitative research. Sage, London Rogers C 1994 (3rd ed.) Freedom to learn for the 1990s. Merrill, New York Sadlo G 1994 Problem Based Learning in the development of an Occupational Therapy Curriculum – Part 2. The BSc at the London School of Occupational Therapy. British Journal of Occupational Therapy 57(3): 79–84 Schilling K, Ginn DS, Mickelson P et al. 1995 Integration of information-seeking skills and activities into a problem based curriculum. Bulletin of the Medical Library Association 83(2): 176–182 Schon DA 1983 The reflective practitioner. Basic Books, New York Schon DA 1987 Educating the reflective practitioner. Jossey-Bass, San Francisco Silverman D (ed.) 1997 Qualitative research, theory method and practice. Sage, London Strauss A, Corbin J 1990 Basics of qualitative research. Grounded theory procedures and techniques. Sage, London Thomas BG, Quant V, Cooke P 1998 The development of a problem based curriculum in midwifery. Midwifery 14: 261–265 Vernon DTA, Blake RL 1993 Does problem based learning work? A meta analysis of evaluative research. Academic Medicine 68(7): 550–563 Webb C 1989 Action research: philosophy, methods and personal experiences. Journal of Advanced Nursing 14: 403–410 Wise J 1996 Preparation for caseload management. Modern Midwife 6(1): 15–17 Wise J 2000 Problem-based learning in midwifery. In: Glen S, Wilkie K (eds) Problem-based learning in nursing. A new model for a new context? Macmillan, Basingstoke Youll P, McCourt-Perring C 1993 Raising voices: evaluation of the Caring in Homes Initiative. HMSO, London Youll P, McCourt-Perring C 1993 Ensuring quality in residential care: the work of the Caring in Homes Initiative. Evaluation Report to the Department of Health. Centre for the Evaluation of Public Policy and Practice, Brunel University, Uxbridge, Middlesex
REFERENCES Barrows H, Tamblyn R 1980 Problem based learning: an approach to medical education. Springer, New York Barrows HS 1986 A taxonomy of problem based learning methods. Medical Education 20: 481–486 Benner P 1984 From novice to expert: excellence and power in clinical nursing practice. Addison-Wesley, Mento Park, Canada Biley F, Smith K 1998 ‘The buck stops here’: accepting responsibility for learning and actions after graduation from a problem based learning nursing education curriculum. Journal of Advanced Nursing 27: 1021– 1029 Buxton M, Hanney S 1994 Assessing payback from Department of Health Research and Development: preliminary report. HERG Research Report No. 19. Brunel Univeristy, Uxbridge, Middlesex Creedy D, Hand B 1994 The implementation of problem based learning: changing pedagogy in nurse education. Journal of Advanced Nursing 20: 696–702 Dearing R 1997 Higher education in the learning society. National Committee of Enquiry into Higher Education. Main Report. The Stationery Office, London. Donabedian A 1966 Evaluating the quality of medical care. Millbank Memorial Quarterly 44 (July 1966 pt. 2.): 166–203 Entwistle N, Ramsden H 1983 Understanding student learning. Croom Helm, London Freire P 1972 Pedagogy of the oppressed. Penguin Books, London Guba E, Lincoln Y 1989 Fourth generation evaluation. Sage, London Knowles M 1984 The adult learner: a neglected species. Gulf Publishing Company, London Lewis KT, Tamblyn RM 1987 The problem based learning approach in baccalaureate nursing education: how effective is it? Nursing Papers 2: 17–26 Margetson D 1991 Why is problem based learning a challenge? In: Boud D, Felliti G (eds) The challenge of problem based learning. Kogan Page, London
Objective: to evaluate the implementation of a ‘problem-based’ learning (PBL) curriculum within midwifery. Design: a ‘realistic evaluation’ model based on cost-e¡ectiveness models of economics, and incorporating both qualitative and quantitative methodology.The design used historical comparison, comparing students following the new curriculum with a baseline of previous student cohorts and exploring sta¡ and student experiences. Setting: a ‘new’ university, providing a degree-level 18 month pre-registration midwifery programme for an intake of about 40 students per cohort. Participants: all students in the cohort qualifying immediately before the implementation and all those in the ¢rst cohort following the new curriculum were invited to participate fully. Assessment outcomes for three cohorts of students qualifying before and the ¢rst three cohorts qualifying following the change were analysed. All clinical sta¡ directly involved in mentoring the relevant student groups, all academic sta¡ involved in delivering the new curriculum and the course external examiner were invited to participate. Intervention: the evaluation studied a major change in the way the overall midwifery curriculum was delivered, widely described as PBL. Measurements: students’experiences and perspectives were sought through review of routine evaluation documents, nominal group technique exercises and focus group discussions at about six months following quali¢cation. Clinical sta¡ experiences and perspectives were sought through written structured questionnaires. Academic sta¡ views were sought through personal semi-structured interviews. Participant observation of the process, review of course documents and of sta¡ re£ective commentaries were also conducted. Data on student completion rates and ¢nal grade were analysed.
Christine McCourt, BA, PhD Senior lecturer in health services research Wolfson Institute of Health and Human Sciences Thames Valley University 32-38 Uxbridge Road London W5 2BS E-mail: [email protected] B. Gail Thomas, RGN, RM, ADM, PGCEA, MSc Dean, Wolfson Institute of Health and Human Sciences Thames Valley University 32-38 Uxbridge Road London W5 2BS (Correspondence to CM) Received 23 July 2000 Revised 23 November 2000 Accepted 5 April 2001
Findings: although students’ views of the curriculum were generally positive, they experienced some discomfort and di⁄culty in adjustment to a new style of learning. Small group learning, independence and skills in using and conveying information and research evidence were valued aspects of the new approach.Quality of clinical placement experience and university-practice links were an important area of positive experience or concern. Clinical sta¡ views of the curriculum showed considerable stability, with major concerns being the balance of theory and practice time and skills which were not in£uenced by the curriculum change. At both points, ability to critically evaluate and use evidence in practice was regarded highly. Student outcomes showed some evidence of possible divergence of grades in the new curriculum. Conclusions and implications for practice: the manner and context of implementation of educational change may have important implications for student experiences and outcomes and the experiences of educators. Adequate preparation at all levels for a di¡erent, more independent style of learning is important and students are likely to need clear guidance and feedback on their progress in the early stages of the curriculum, to monitor progress and to provide reassurance. Longer-term research is needed to assess the impact of PBL on theory-practice links and on the midwives as practitioners. & 2001 Harcourt Publishers Ltd Midwifery (2001) 17, 323^331 & 2001 Harcourt Publishers Ltd doi:10.1054/midw.2001.0276, available online at http://www.idealibrary.com on
324 Midwifery
INTRODUCTION In 1997, a new curriculum for pre-registration (short) midwifery (an 18 month programme for registered general nurses) was implemented at Thames Valley University’s Wolfson Institute of Health Sciences (Thomas et al. 1998) modelled on a ‘problem-based learning’ (PBL) approach (Barrows & Tamblyn 1980, Barrows 1986). Briefly, a traditional class format was replaced with small group ‘learning sets’ around a series of ‘triggers’ conducted over three-week cycles. In each cycle, the students were introduced to a real-life scenario or ‘trigger’ based on midwifery practice, intended to stimulate students to question and to want to learn. The group planned what areas of learning needed to be covered, who would lead each, undertook this work with support from a range of resources and presented their investigative work at the end of the trigger cycle. The term ‘lecturer’ was replaced with ‘learning set leader’ to reflect change to a more facilitative role: while the approach to education had previously been fairly evidencebased and student centred, students were now expected to adopt a more independent and group-centred approach. The evaluation that was attached to the development and the key themes and issues arising at an early stage are presented in this article. As such, the issues discussed relate particularly to a change process and responses to change, rather than the impact of an established educational programme or approach.
Background PBL has been adopted in a number of medical schools, particularly in North America and Australia (Schilling et al. 1995, Margetson 1996) and for other health-related curricula (Creedy & Hand 1994, Sadlo 1994). This was the first course in midwifery in the UK to adopt a PBL approach to the whole curriculum, although such principles have been piloted for midwifery modules elsewhere (Pansini-Murrell 1996, Wise 1996, 2000) and interest in this educational approach is growing. The aims and principles of PBL are generally seen as to promote learning which is relevant to real life experience and which can be retained and used well by the student. They draw on broad educational principles and theories set out by writers such as Freire (1972) and applied in concepts such as ‘adult learning’, (Knowles 1984, Rogers 1994), ‘lifelong learning’ (Dearing 1997), ‘deep versus surface learning’ (Entwistle 1987) and of the ‘reflective practitioner’ (Schon 1983, 1987). The aims were interpreted locally in terms of enabling students to understand the relevance of theoretical issues to practice, to apply ideas
and knowledge to real life situations, to continue learning once qualified and to take a reflective, questioning and woman-centred approach in their work (Quant 1999).
The existing evidence on Problem Based Learning A number of descriptive and evaluative studies of this approach have been conducted, primarily in medicine, which generally indicate positive outcomes although, some were part-programmes or only included volunteer students. A metaanalysis of the research (Vernon & Blake 1993) found that PBL curricula and courses were generally popular with students and well evaluated by staff. However, students appeared to achieve lower scores on traditional examinations that tested factual knowledge. They showed better performance where ‘process’ measures were tested. Various forms of comparison were used in the studies reviewed, including pure experimentation, own-control and static-control. No differences were apparent across the different study designs. The reviewers concluded that ‘PBL’ educated students may perform better in the long term, due to improved motivation and understanding of learning processes and argued that formal examination scores may not be a good guide to future clinical performance. Evaluating the occupational therapy degree programme in the UK Sadlo (1994) also found good student motivation and response but slightly poorer examination scores. In an experimental study of PBL in nursing, Lewis and Tamblyn (1987) found that traditional students performed slightly better in independent observation of clinical placement performance. A study of nursing student experiences identified differences in students’ approaches towards their work, in particular a greater sense of personal responsibility for learning and for practice among PBL educated students (Biley & Smith 1998). Such studies reflect the arguments of the theoretical literature that PBL may change students’ attitudes and approaches to learning and thereby, their approaches to practice (Benner 1984, Margetson 1991, Schilling et al. 1995). Both the theoretical and research literature indicate that possible benefits of this approach may be subtle and long term and so may not be easily captured by conventional forms of assessment, such as written examinations. The research also suggests that PBL educated students may show some gaps or weaknesses in factual knowledge on assessment, which may be addressed over time by a more effective development of learning skills, sense of responsibility and enquiring approach.
Evaluation of a problem-based curriculum in midwifery
Design of the evaluation Our approach was based on the classical economic model of cost effectiveness evaluation which has been widely used in programme evaluation in the UK and north America, looking at inputs, process and outcomes of a programme to understand its effectiveness (Fig. 1) (Donabedian 1966). The adapted model is often called ‘realistic evaluation’ and usually incorporates both qualitative and quantitative research methods (Youll & McCourt-Perring 1993, Buxton & Hanney 1994, Pawson & Tilley 1997). Unlike the more positivist traditions of economics, this approach does not assume that all the issues to be taken into account are predictable, controllable or easily measurable. It allows for researchers to get involved by providing feedback during the evaluation, and for those directly involved (stakeholders) to have a say in how things are done (Quinn-Paton 1980, Guba & Lincoln 1989,Youll & McCourt-Perring 1993). Therefore, evaluation was conducted alongside and in close relation to curriculum development with the aim of providing feedback to that process, but led by researchers within the Institute who were not directly involved in planning or delivery of the new curriculum. Historical comparison was used to identify and explore possible effects of the change: experiences and outcomes for students following the new curriculum were compared with a baseline of students qualifying under the previous degree curriculum. Such a model needs to be treated with caution and cannot identify causal relations in the manner of pure experimental research (Oakley 1998) but can take account of the complexity and unpredictability of complex organisational changes and the wide range of variables that may influence process and outcomes.
325
authority in relation to their education. Participants were given detailed written and verbal information on the evaluation methods and aims and were assured that participation in interviews and focus groups was voluntary and confidential. All data gathered were anonymised to protect individual confidentiality and former students taking part in focus groups were asked, verbally and in written information sheets, to respect the confidentiality of their peers’ contributions. All routine student evaluation records were anonymous. A steering group involving researchers and curriculum managers was convened to coordinate and monitor both the implementation and the evaluation. Data were gathered relevant to input, process and outcomes of the programme. A range of methods was used reflecting the complexity of the situation under analysis, the need to gain a range of perspectives and desire to triangulate sources where possible. Choice of methods was also pragmatic, since a very short time span was available for collection of baseline data before the new programme commenced. Hence a more structured approach was used with clinical mentors. The specific methods used are described briefly for each of the three stages in turn:
Inputs: the baseline . Survey of clinical mentors’ views of the existing curriculum (n=38) A brief questionnaire including 5-point scales on seven statements relating to curriculum quality and two open questions on the strengths and weaknesses of the course was distributed to all clinical mentors via students and academic staff (See Table 1). As there were no records of the number of staff acting as clinical assessors/ mentors we do not know the proportion of mentors who responded. . Interview with the external examiner
METHODS
A semi-structured interview was conducted to elicit views of the strengths and weakness of the current curriculum and its development over time.
Ethical considerations At the time this evaluation commenced there was no Research Ethics Committee in the University. The work would be examining sensitive issues for those undergoing change and involved students who might feel reluctant to criticise those with
. Focus group discussions with qualified midwives (n=11/35):
recently
All midwives qualifying in the last cohort before implementation (n=35) were invited by letter
Fig. 1 Process model for programme evaluation INPUT ? context/history existing service model resources (human and material) programme aims programme design
PROCESS ? context ^ other changes or developments ways change is implemented new model: organisation and practice resources (human and material) operation of programme any changes to design/practice
OUTCOME informal and formal assessments e.g. audit data, exam results views of stakeholders any change in use assessment of e¡ects in practice description of model in practice
326 Midwifery
Table 1 Clinical mentors survey responses ^ previous curriculum n=38 Question Score ? Do students/are students
1
2
3
4
5
Missing
mode
Mean
Know how to judge the quality of research Know how to use research in practice Able to apply what they learn in the classroom to what they see in practice Question things they hear about in practice Able to identify and negotiate own learning Able to relate well in a small group Seem keen to carry on learning when they f|nish Total (mean)
0
4
13
15
4
2
4
3.53
0
2
11
21
3
1
4
3.68
0
4
18
14
0
1
3
3.19
0
2
7
16
13
0
4
4.05
0
1
11
20
5
1
4
3.78
0
0
7
19
9
3
4
4.06
0
3
11
14
9
1
4
3.78
0
16
78
119
43
9
and follow-up telephone calls to take part at about 3–6 months following qualification. The midwives were asked, using a semi-structured topic guide, to discuss their views of the course and how it had prepared them for practice. The discussions were taped and transcribed in full. Unstructured telephone interviews were also conducted to avoid excluding the views of those who did not wish to participate in a group discussion. There was considerable difficulty in organising these groups but 11 from the pre PBL and 12 from the post PBL courses participated. Many had moved elsewhere or were travelling (13 and 12 respectively could not be contacted) and 3 from each cohort were unable to attend on the day due to duty rota problems. Those who declined (4 and 2 respectively) gave practical reasons and or were not practising as midwives. Telephone interviews were conducted with those (2 pre and 2 post PBL) who did not wish to take part in a group discussion, primarily as they were not practising or were considering leaving midwifery. . Observation of key planning and development meetings . Review of course documents, including routine written student evaluations (n=32/35)
The Process . Interviews with new ‘learning set leaders’ (n=10) and review of reflective commentaries. The experiences of academic staff will be reported elsewhere. . Student evaluation using Nominal Group Technique exercises (Reed & Proctor 1993) These were conducted in the learning sets, at the review of each trigger to provide an analysis of students’ responses to the curriculum, as it unfolded (n=48 exercises per cohort across all
3.72
learning sets, with all students present for the session involved). Students identified issues in small groups as strengths/positives or weaknesses/problems and then voted, casting three votes each, on a pooled list of issues, with their learning set leader acting as facilitator. This was continued for the first three cohorts of students, to allow time for settling in and review of the curriculum and development over time. . Review of course documents and routine student evaluations. . Participant observation of curriculum meetings, staff development and staff/student evaluation sessions to gain a fuller view of the change process and the responses to this, and to provide regular informal feedback to participants.
Outcomes Data gathered at the baseline were repeated for the first cohort of ‘PBL’ students as follows: . Survey of clinical mentors’ views of the new curriculum (n=38) The survey was administered as before . Interview with the external examiner A semi-structured follow-up interview to reflect on the curriculum change. . Focus group discussions with recently qualified midwives (n=12/35) Discussions and telephone interviews were conducted, following the same processes as before, with the first cohort of students qualifying under the new curriculum (n=35). . Analysis of data on formal educational outcomes Formal outcomes were assessed for six intakes of students–three before and three after the introduction of the new curriculum to avoid the possibility of normal variation from one year to another skewing the apparent outcomes.
Evaluation of a problem-based curriculum in midwifery
327
Outcomes analysed included final grades, pass and completion rates.
researcher’s understanding of the data (Webb 1989, Reason 1994).
Analysis Structured data–mentor questionnaires and student grades–were analysed using descriptive statistics and content analysis for the open questions. Inferential statistics were not used as they were not felt to be appropriate in a study where only one small aspect of the work used structured measures and the numbers were small. We were concerned not to attach undue significance to such measures in what was clearly a very complex changing situation. Qualitative data were analysed using a textual approach, assisted by use of the NUD*IST software programme (Richards & Richards 1994). In this method, the texts of transcripts are coded in detail, using open codes generated from the data. Codes are then indexed to identify key themes. The use of such an approach facilitates the grounding of theory in the data by facilitating constant, iterative movement between a coded passage of text and its context as well as between different areas of data generating the same or linked codes (Strauss & Corbin 1990). However, the use of such a package, which utilises a relatively rigid ‘tree’ structure for generation of themes, was found to be limited for a complex study with multiple sources of data, particularly that generated by participant observation. The nominal group technique data were prestructured by the process as students themselves undertook the first phase of analysis by identifying and voting on key issues. Subsequent analysis involved reviewing the reports of the exercises and the issues identified in each cross-sectionally to draw out key recurring themes. A further important area of analysis in evaluative research of this type is the process of feedback and discussion with the different participants, which may stimulate both change in practice and in the
FINDINGS The findings are presented from three main perspectives: clinical mentors’, students’, and formal assessment. Those of academic staff will be reported elsewhere. Findings from the new curriculum are set alongside those of the previous curriculum to facilitate comparison.
Clinical mentors’ perspectives The structured findings of the pre- and postimplementation surveys are given in Tables 1 and 2. At both time points, 38 forms were returned and 33 and 32 of these respectively included some open comments. Overall, views of both curricula were positive, the most frequent score being 4, although the mean scores for PBL students were marginally lower than those before the change. The question focused on the relationship between theory and practice scored a more neutral 3 in the pre-implementation survey, suggesting that mentors were a little less confident about students’ ability to link theory and practice than they were about other aspects of education. No clear changes were found with the new approach. In their open comments, a set of themes emerged which remained fairly stable over time. Several themes had both positive and negative aspects. For example, the students’ independence and self-motivation was seen as positive but some mentors also felt that students needed more guidance and direction. The key areas of strength in both curricula identified were: . students’ confidence and ability to use research evidence and their enhanced access to research resources, literature and a good knowledge base;
Table 2 Clinical mentor responses ^ 1st cohort of new curriculum, n=38 Question: /score ?do students
1
2
3
4
5
Missing
Mode
Mean
Know how to judge the quality of research Know how to use research in practice Able to apply what they learn in the classroom to what they see in practice Question things which they hear about in practice Are able to identify and negotiate own learning Relate well in a small group Seem keen to carry on learning when they f|nish Total (mean overall)
1
2
18
11
5
1
3
3.46
1
3
15
15
4
0
3/4
3.47
0
5
15
17
1
0
4
3.37
0
4
8
18
8
0
4
3.79
0
5
12
14
7
0
4
3.61
0 1
1 0
6 13
22 21
8 3
1 0
4 4
4 3.66
3
20
87
118
36
2
3.62
328
Midwifery
. their independence and self-direction; . students taking a critical thinking questioning approach.
and
The key themes emerging as weaknesses in both curricula were: . time pressures leading to conflicts between academic and practice work; . lack of clinical practice time and experience (affecting clinical skills and confidence); . a theory/practice skills gap. Mentors’ comments about weaknesses were more concerned with limits on clinical time and experience than the nature of the universitybased programme. Knowledge and awareness of the new curriculum was high and they were conscious of some of the anxieties being expressed by students during their studies. However, analysis of the questionnaire responses indicated few differences and, as before, mentors’ critical comments focused mainly on conflicts between time given to theory and practice which have not been affected by the change in educational approach.
Students’ perspectives Student Nominal Group Technique evaluations The key issues raised in student evaluations, using the nominal group technique process, are summarised in Figure 2. Overall, they suggest that students saw both strengths and weaknesses in aspects of the curriculum such as small group and self-directed learning, where there was much of value but also much to learn from experience.
They valued the principles of the approach but expressed anxieties about levels of guidance and direction and the coverage and depth of their knowledge. Their comments also hint at parallel uncertainties amongst academic staff. During the course, students’ comments shifted to reveal adaptation and greater comfort with the style of learning over time, but concerns about the need for more input from lecturers, guidance and feedback on their progress remained.
Former students’ views – group and individual interviews Students qualifying just before the introduction of PBL were generally positive about their experience. Although some students experienced initial difficulties in understanding the relevance of some course material, they felt they had been able to retain academic knowledge and it had ‘come together’ or ‘clicked in’ once they were practising. They expressed confidence in dealing with new situations and uncertainty, with finding and using or critiquing research evidence and established practice. They also showed interest in further learning, at this stage mainly via gaining practice experience. The main problem areas identified were linked to poor or inconsistent input in clinical placements, particularly due to fragmentation and staff shortages within the service. They were also critical of some mentors’ didactic approach to education and to practice – one example given was of being shown precisely how to lay out a delivery trolley without any explanation of why. Several also felt that the links between college and practice placements
Fig. 2 Students’ NGTresponses summary Advantages Self directed learning: Encourages self-discipline and management, motivation, conf|dence, sense of own direction, independence
Disadvantages Self-directed learning: Uncertainty and anxiety ^ depth and scope, need more guidance & sense of direction, fears about knowing enough, fears about imbalance, gaps in knowledge
Information Developing skills in using information and research evidence, searching and researching, analysis
Information Tutors holding back, not giving us their knowledge, need more feedback and informal assessment Problems accessing resources in time Lack of conf|dence in presenting
Developing presentation and communication skills Lateral thinking and problem solving
Lack of focus, unsure where to begin
Group learning: Group dynamics, working together Co-operation and interaction skills, sharing, gives more time to cover topics Discussion, active participative style Role plays
Group learning: Have to rely on and trust others, people not pulling weight Sharing means only covering part of topic in depth Gaps if others’ work not adequate Reluctance to criticise peers
The triggers: Learning from real life situations Relevance to practice, linking to practice Inherent interest Learning the process
The triggers Easier once have practice experience T|metabling problems, unfamiliarity with style Need to learn the process
Evaluation of a problem-based curriculum in midwifery
329
had not been adequate, leaving them feeling unsupported if they did experience problems. Their main concerns about continuing to learn, or to use research evidence in practice, focused on access to facilities, time or structures to facilitate this. The responses of the first group qualifying under the new curriculum echoed those of the previous cohort of students in many ways. They were generally positive about their universitybased education but some expressed concerns about the quality and coherence of their clinical placement experience. They were equally confident in using research evidence and felt they had developed valuable skills in seeking out and using new information. However, they expressed more anxiety about possible gaps in their skills and knowledge on qualifying than the previous group, who were aware of the their limits but confident that they would continue to learn. An important benefit of the new curriculum was in developing their skills and confidence in putting across information to others. For example, they felt more prepared and willing to take on the challenge of leading antenatal classes. They also reflected on the experience of being in some ways an ‘experimental’ group. They described the approach as quite different from their previous education, including their nursing curricula, particularly in requiring a high degree of personal independence and inter-dependence within the group. Their accounts of the trigger process also suggest that considerable skills in facilitation were important to the approach. While some lecturers used critical questioning to stimulate discussion and provide feedback to students on the depth of their work, others were described as ‘sitting back’ and feeling ‘unable to teach’. The issues for academic staff, on which they touched will be discussed elsewhere, but what is notable from the students’ viewpoint is the degree to which uncertainty about the curriculum change might be conveyed by the particular approaches taken.
at the beginning, more structured teaching early in the course, a shift to focusing on the whole trigger topic rather than dividing the work into particular areas and clearer guidance and feedback on clinical and theory assessments. Routine evaluations by the following cohort of students (n=25) showed similar themes but there had been a shift in the comments related to the PBL approach, with more students stating that they liked the approach and noting the relevance of triggers to practice. However, there was still a demand for more structured teaching early in the programme and more practical teaching sessions. Some students continued to express concern about their coverage of topics and whether they had missed any of the ‘basics’. Their comments suggested that subsequent groups were less anxious about the programme change, but still held some concerns about this more independent style of learning.
Routine student evaluations Thirty-two students in the last cohort of students qualifying before the new curriculum completed routine evaluation forms that showed similar views to those given in the group discussions. Their suggestions for improvement included better preparation for clinical mentors, better university/practice links, a stronger role for the link tutor and more practical teaching sessions. A few also requested more small group and case study-based teaching. Twenty-three students in the first PBL cohort completed routine evaluation forms and the majority participated in a verbal end-of-course evaluation that also gave similar views to those in our group discussions. Suggestions for improvement included better preparation for the approach
Outcomes External examiner’s perspective The examiner was able to comment on development of the degree-level course since its inception and felt it had been refined and improved over time. Standards overall, and teacher expectations, were high, although there was a problem of large variations of ability and performance between students. A follow-up interview and review of external examiner reports indicated that there had been little change in students’ performance in formal assessments: levels of midwifery knowledge and overall standards remained good. At both points, student difficulties or failures were viewed as mainly linked to difficulties in expressing ideas in written form and with critical analysis, rather than lack of substantive knowledge. She felt this reflected the wide intake on such courses and the need for structured learning support before or early in the curriculum for some students. Formal outcomes assessment Analysis of formal assessment of students indicated limited change in students’ overall performance, but some evidence of divergence of grades (see Table 3). In the three cohorts examined following the curriculum change, the percentage of students with 1st class and 3rd class degrees or failure increased. Rates of student withdrawal or deferrals, which may reflect personal or academic difficulties, or a decision that this was not the right course for them, remained stable. This analysis, therefore, does not indicate clear changes, but the possibility that the pattern of allocation of grades may be widening under the new approach and the need to keep this under review.
330 Midwifery Table 3 Analysis of degree classif|cations Degree classif|cation
n
Pre PBL Post PBL
6 16
1st % 5 12
2.1
3rd
2.2
n
%
n
%
n
%
61 40
50 30
39 46
32 35
3 9
2 7
not completed or failed n % 13 20
10 15
Pre-pbl=3 intakes, March 1996 ^March 1997 n=122 Post-pbl=3 intakes, Sept.1997^Sept.1998 n=131 (plus 2 still deferred)
DISCUSSION Strengths and limitations of the research There is an inherent problem in investigating cause and effect relationships in educational research where controlled experimentation is not possible and the range of potential factors is complex and hard to predict. Triangulation of methods, and inclusion of a range of perspectives, allowed for the identification of possible biases or inconsistencies in the findings. Overall the levels of agreement between the different groups was high, suggesting that the patterns of response obtained were unlikely to be biased towards the views of particular groups and that respondents felt able to be reasonably frank and open with the researchers. Additionally, the process of feedback to participants enabled researchers to compare the analysis with the understandings of the different groups involved in the process. Such factors provide increased confidence in the validity and reliability of the research findings. It is important to remember that this article examines something new and consequently, people’s responses to change. Longer-term study is planned to enable an assessment of experiences and outcomes once the course is more established and to explore the longer-term impact of the students’ education. Also, the change we have focused on is necessarily complex and cannot be described and understood entirely or simply in terms of a model that is likely to be implemented differently in different situations. The context and baseline for change are also important. The evidence gathered shows considerable stability of responses and outcomes, as compared with baseline data before implementation of the PBL approach. We did not find any clear evidence of enhanced formal outcomes or improvements in students’ ability to link theory to practice. However, stability at such an early stage of a fairly radical change to teaching and learning method might be considered a positive outcome. We did note a possible trend towards greater divergence of students’ final grades. The reasons for this are unclear and it is not possible, on the basis of this study, to comment on whether this should be interpreted as a negative or positive outcome, or whether it is likely to be
enduring. It could be argued, for example, that a PBL curriculum is better able to distinguish students who are not well suited to a career in midwifery and facilitates those who are well prepared and motivated to achieve. Conversely, it could be suggested that either the approach itself, or the newness of the curriculum led to a less directive type of guidance, increasing the possibility of lower results for less confident students. Such a possibility was reflected in the higher levels of anxiety expressed by the first cohort of students following the PBL approach, as compared to previous students. Their anxieties about gaps in skills and knowledge were not, however, reflected in the data gathered on outcomes from mentors and examiners who regarded the group as having similar levels of midwifery knowledge to previous students. It was, therefore, largely a selfperception that may have been influenced by several factors. In this first cohort, students had observed raised levels of uncertainty amongst teaching staff, who were themselves adjusting to considerable change, which undermined their own sense of confidence. They expressed concerns about gaps since they felt more competent in those topics they had focused on personally in the triggers than those covered by others in their group. This was not only a matter of needing to develop trust in other group members and learning to work effectively as a ‘team’ but also a recognition that information is, indeed, retained better when you have sought it out and studied it in depth yourself. Students’ anxieties about reliance on the work of others in the group, and their initial reluctance to take a critical, questioning approach, also suggested that they were themselves bringing particular, more didactic, models and expectations of teaching to their work. They felt ‘teaching’ by the lecturer was being replaced with ‘teaching’ by their peers. Initial anxieties about presentation, likewise, encouraged them to focus on their own or their peers’ skills in presentation as a performance, rather than as a means to discussion and mutual learning. A third possible interpretation of their expressed anxiety is that, given greater responsibility for seeking out knowledge, these students developed a heightened awareness of the scope and complexity of the information ‘out there’ and the limits of
Evaluation of a problem-based curriculum in midwifery
331
their own knowledge on qualifying. This need not be a negative finding if those students have sufficient confidence, motivation and opportunities to continue to learn once in practice. Midwives who, in their education, learn (or at least do not ‘unlearn’) how to seek out and use information, to manage uncertainty as part of a learning process may be better placed to become truly reflective practitioners. A number of adjustments were made to the developing approach in response to this evaluation, including more initial preparation for the students for this style of learning, providing workshops on practice related topics and providing feedback and guidance to students in a way that is timely and appropriate to the PBL approach. Further attention will be needed to clinical learning and to university/placement links, including preparation of mentors to work within this framework. Longer-term research is planned to assess the impact on the students as practitioners.
NUD*IST: Non-nUmerical Data Indexing, Searching and Theory building. Sage, Scolari, London. www.scolari.co.uk Oakley A 1998 Experimentation in social science: the case of health promotion. Social Sciences in Health 4(2): 73–89 Pansini-Murrell J 1996 Incorporating problem-based learning: striving towards women-centred care. British Journal of Midwifery 4(9): 479–482 Pawson R, Tilley M 1997 Realistic evaluation. Sage, London Quant V 1999 How do student midwives perceive their problem-based learning course and does it prepare them to provide woman-focused care? Unpublished MA dissertation, University of Reading, UK Quinn-Paton M 1980 Qualitative evaluation methods. Sage, Beverley Hills Reason P 1994 Three approaches to participative inquiry. In: Denzin NK, Lincoln YS (eds) Handbook of qualitative research. Sage, London Reed J, Procter S 1993 Nurse Education: a reflective approach. Edward Arnold, London Richards TJ, Richards L 1994 Using computers in qualitative research. In: Denzin NK, Lincoln YS (eds) Handbook of qualitative research. Sage, London Rogers C 1994 (3rd ed.) Freedom to learn for the 1990s. Merrill, New York Sadlo G 1994 Problem Based Learning in the development of an Occupational Therapy Curriculum – Part 2. The BSc at the London School of Occupational Therapy. British Journal of Occupational Therapy 57(3): 79–84 Schilling K, Ginn DS, Mickelson P et al. 1995 Integration of information-seeking skills and activities into a problem based curriculum. Bulletin of the Medical Library Association 83(2): 176–182 Schon DA 1983 The reflective practitioner. Basic Books, New York Schon DA 1987 Educating the reflective practitioner. Jossey-Bass, San Francisco Silverman D (ed.) 1997 Qualitative research, theory method and practice. Sage, London Strauss A, Corbin J 1990 Basics of qualitative research. Grounded theory procedures and techniques. Sage, London Thomas BG, Quant V, Cooke P 1998 The development of a problem based curriculum in midwifery. Midwifery 14: 261–265 Vernon DTA, Blake RL 1993 Does problem based learning work? A meta analysis of evaluative research. Academic Medicine 68(7): 550–563 Webb C 1989 Action research: philosophy, methods and personal experiences. Journal of Advanced Nursing 14: 403–410 Wise J 1996 Preparation for caseload management. Modern Midwife 6(1): 15–17 Wise J 2000 Problem-based learning in midwifery. In: Glen S, Wilkie K (eds) Problem-based learning in nursing. A new model for a new context? Macmillan, Basingstoke Youll P, McCourt-Perring C 1993 Raising voices: evaluation of the Caring in Homes Initiative. HMSO, London Youll P, McCourt-Perring C 1993 Ensuring quality in residential care: the work of the Caring in Homes Initiative. Evaluation Report to the Department of Health. Centre for the Evaluation of Public Policy and Practice, Brunel University, Uxbridge, Middlesex
REFERENCES Barrows H, Tamblyn R 1980 Problem based learning: an approach to medical education. Springer, New York Barrows HS 1986 A taxonomy of problem based learning methods. Medical Education 20: 481–486 Benner P 1984 From novice to expert: excellence and power in clinical nursing practice. Addison-Wesley, Mento Park, Canada Biley F, Smith K 1998 ‘The buck stops here’: accepting responsibility for learning and actions after graduation from a problem based learning nursing education curriculum. Journal of Advanced Nursing 27: 1021– 1029 Buxton M, Hanney S 1994 Assessing payback from Department of Health Research and Development: preliminary report. HERG Research Report No. 19. Brunel Univeristy, Uxbridge, Middlesex Creedy D, Hand B 1994 The implementation of problem based learning: changing pedagogy in nurse education. Journal of Advanced Nursing 20: 696–702 Dearing R 1997 Higher education in the learning society. National Committee of Enquiry into Higher Education. Main Report. The Stationery Office, London. Donabedian A 1966 Evaluating the quality of medical care. Millbank Memorial Quarterly 44 (July 1966 pt. 2.): 166–203 Entwistle N, Ramsden H 1983 Understanding student learning. Croom Helm, London Freire P 1972 Pedagogy of the oppressed. Penguin Books, London Guba E, Lincoln Y 1989 Fourth generation evaluation. Sage, London Knowles M 1984 The adult learner: a neglected species. Gulf Publishing Company, London Lewis KT, Tamblyn RM 1987 The problem based learning approach in baccalaureate nursing education: how effective is it? Nursing Papers 2: 17–26 Margetson D 1991 Why is problem based learning a challenge? In: Boud D, Felliti G (eds) The challenge of problem based learning. Kogan Page, London