- Email: [email protected]
Evaluating Portfolio Use as a Tool for Assessment and Professional Development in Graduate Nursing Education
EVALUATING PORTFOLIO USE AS A TOOL FOR ASSESSMENT AND PROFESSIONAL DEVELOPMENT IN GRADUATE NURSING EDUCATION MARYBETH RYAN, PHD, RN⁎ The portfolio is emerging as an efficient and effective method for evaluating program outcomes and professional development in nursing education. Although there is a host of literature about the use of portfolios in undergraduate nursing programs, fewer reports exist about their use in graduate nursing education. This article presents the results of a formative evaluation process, using student and faculty focus groups, conducted at a midsized university's graduate nursing education program to determine the effectiveness of portfolio use. Content analysis of the focus group data yielded three student themes and two faculty themes with associated theme clusters that revealed similarities and unique perceptions of students and faculty regarding the portfolio process. The information gleaned will provide direction to faculty as they make decisions about the use of this evaluation method in the graduate program. (Index words: Portfolio; Assessment; Program outcomes; Professional development; Graduate nursing education) J Prof Nurs 27:84–91, 2011. © 2011 Published by Elsevier Inc.
I
DENTIFYING GRADUATE NURSING program outcomes to be attained by students is a critical but not difficult task. The challenge for faculty is to actually determine what students have learned; how they have changed academically, professionally, and personally; and whether program outcomes were met at the end of their journey through the curriculum. There is a growing national and international trend in nursing education programs to use portfolios to assess learning and competence (Harris, Dolan, & Fairbairn, 2001; Karlowicz, 2000). They have become a valuable alternative/ adjunct method for assessing student performance (McCready, 2006; McMullan, 2008). McMullen et al. (2003) define a portfolio as “a collection of evidence, usually in written form, of both the products and processes of learning. It attests to achievement and professional development by providing critical analysis of its contents” (p. 283). Numerous accounts of portfolio use in undergraduate nursing programs have been described (Gallagher, 2001; Harris ∗Adelphi University School of Nursing, Garden City, NY. Address correspondence to Dr. Ryan: Adelphi University School of Nursing, South Avenue, Garden City, New York 11530. 8755-7223/10/$ - see front matter 84 doi:10.1016/j.profnurs.2010.09.008
et al., 2001; Karlowicz, 2000; Kear & Bear, 2007; Ramey & Hay, 2003; Tracy, Marino, Richo, & Daly, 2000), but fewer reports have been provided about their use in graduate nursing education (Cangelosi, 2008; Coffey, 2005; Hayes, Chandler, Merriam, & King, 2002; Jasper & Fulton, 2005; Joyce, 2005). A newly revised curriculum was the impetus for implementing the use of portfolios in a graduate nursing program at a midsized private university. The portfolio was viewed as a vehicle for self-reflection and a way for students to document academic and professional accomplishments as they progressed through the master's program. It would allow students to demonstrate their grasp of advanced practice standards, document attainment of the master's program objectives, develop their ability to effectively critique their work, convey their values and view of themselves as a professional to others, showcase professional practice and career accomplishments, and, finally, assist them with career development and planning. Two years after the initiative was introduced, the author undertook a formative evaluation of the portfolio process by conducting focus groups to explore students' and faculty's experiences with using this assessment method. The article elaborates on the themes that emerged, the similarities and differences
Journal of Professional Nursing, Vol 27, No. 2 (March–April), 2011: pp 84–91 © 2011 Published by Elsevier Inc.
EVALUATING PORTFOLIO USE
between student and faculty views, and future directions necessary in the master's program for this method of assessing students' learning to be successful.
Literature Review Foremost among the many benefits for students is that portfolio use allows them to be accountable and autonomous learners responsible for the direction and quality of their learning and professional development progress (Cangelosi, 2008; Harris et al., 2001). Selfconfidence is built as students, receiving feedback about portfolio work, recognize and appreciate their ability, skills, and future potential (Kelly, 1995). Student growth is promoted in the areas of reflective thinking and practice, critical incident analysis work, and writing skills (Harris et al., 2001; Karlowicz, 2000; Schaffer, Nelson, & Litt, 2005; Wenzel, Briggs, & Puryear, 1998). Also beneficial for students and faculty is the creation of strong relationships between them as a result of continued portfolio-related dialogue. Ongoing discussion about the student's portfolio work establishes a two-way learning process from which both parties can benefit. For example, a faculty member might reexamine and change a teaching strategy or course content as a result of reviewing students' reflections on course material. It is important that faculty members play a key role in fostering this relationship and support students as they reflect on their learning experiences and build their portfolios (Harris et al., 2001; Karlowicz, 2000). Ryan and Carlton (1997) posit that students can use their portfolio when seeking future employment or other career opportunities. Of course, as Jasper (1995) indicates, “the potential of the portfolio depends upon the individual valuing its contents and regarding it as an essential part of his/her development” (p. 253). For students to be successful in using portfolios, it is important that they are given guidelines that clearly explain the portfolio's purpose, structure, and content (McCready, 2006; McMullen et al., 2003). Redman (1994) asserts that regardless of its purpose, a portfolio must be organized from the outset; the inclusion of specific portfolio sections/components is advocated (Karlowicz, 2000; Price, 1994; Ryan & Carlton, 1997). Problems related to using portfolios in nursing education have been documented (Harris et al., 2001; Jasper, 1995; Karlowicz, 2000). The first issue relates to time; students may view the portfolio as time consuming and postpone developing it as they focus on other course work. Faculty also may view the work entailed as time consuming. As a result, commitment on their part may vary leading to inequalities in student support, guidance, and assessment (Harris et al., 2001). Students may be reluctant to engage in self-reflection and not value or have developed the knowledge or skills to do it. As well, faculty may need assistance in developing and fostering students' self-reflection and self-assessment abilities (Karlowicz, 2000). Finally, a major issue involves the assessment/evaluation process. When students' portfolios are complete,
85
evaluation can provide information about the integrity of the academic program and whether students have achieved the program's outcomes (Ramey & Hay, 2003). Of concern are the methods used to measure students' competence and professional development. Reliability and validity are aspects that must be addressed when using portfolios, particularly in view of the primarily qualitative data included (Gannon, Draper, Watson, Proctor, & Norman, 2001; McCready, 2006). Some authors report the use of scoring rubrics as part of the evaluation process (Jasper & Fulton, 2005; Karlowicz, 2000; Ramey & Hay, 2003). Several authors advocate the need for further research in this area (Gannon et al., 2001; McCready, 2006; Pitts, Coles, & Thomas, 1999; Snadden, 1999).
The Professional Portfolio Initially, faculty teaching in the various master's programs in the School of Nursing met to develop the process for using the portfolio. The literature and extensive discussion concerning the rationale, objectives for, and content to be included in the portfolio served to shape the Professional Portfolio Document (PPD; Ackerhalt, Ryan, & Bogard, 2005) that students receive in the first required master's core course. The PPD conveys the portfolio's purpose, structure, and expectations of the student as the portfolio builder. The required portfolio components include six sections: (a) an introduction where the student introduces herself or himself and her or his ideas about advanced practice; (b) a resume; (c) documentation of academic achievements; (d) documentation of professional or work-related projects, publications, and so on; (e) the Capstone Project, a scholarly written project that is required for graduation; and (f) an essay summarizing growth as an advanced practice nurse. When including various exhibits/artifacts, students are expected to clearly explain the exhibit's importance, why it was selected, the learning that occurred, and how the exhibit illustrates academic or professional achievement, advanced practice standards, or attainment of master's program objectives. In the last part of the PPD, two tools are included that are used for formative assessment: student and academic faculty advisor evaluation tools. Each academic year, students are expected to complete a self-evaluation using the form and meet with a faculty advisor to discuss their portfolio work. The faculty advisor reviews the portfolio and also completes an evaluation tool. These forms yield nominal level and qualitative data. At program completion, a summative assessment of students' work is done. Many authors discuss the continued use of portfolios after formal nursing education is completed and their potential to create professional nursing knowledge (Billings & Kowalski, 2008; Jasper, 1995; Mullen, 2007; Willians & Jordan, 2007). Jasper (1995) writes, “… portfolios are likely to record the knowledge embedded in practice, which is often hard to describe, yet represents nursing's expertise” (p. 249). Portfolios, then, maintained after graduate nursing education could provide a fertile
86
MARYBETH RYAN
area for qualitative research exploring the knowledge to be gleaned from advanced nursing practice.
Method of Study A focus group research design was used in this study. Focus groups are used to gather information from a small number of individuals, usually six to eight, regarding their views on a specific area of interest. A critical component in the process of conducting a focus group interview occurs in the planning phase when a series of sequential questions is constructed that will engage participants and provide the most useful information (Krueger & Casey, 2000).
Participants Using purposive sampling, the researcher recruited by telephone students who were currently enrolled in the master's program. Faculty was recruited during a master's curriculum committee meeting and by e-mail. In focus group research, members who “have certain characteristics in common that relate to the topic” are selected (Krueger & Casey, 2000, p. 4). Of the eight female student participants, four were in nursing administration, three in nursing education, and one in the adult nurse practitioner program. The number of credits completed ranged from 9 to 42 with a mean of 23 credits earned. All six faculty participants, seven females and one male, were full-time faculty teaching in the master's program and responsible for advising students who were compiling their portfolios. Approval to conduct the study was obtained from the university's institutional review board. Since the study received exempt status, an informed consent was not required. To ensure participant anonymity, however, a code number was used on the demographic information form, and only first names were used during the interviews. In terms of confidentiality, participants were asked not to disclose outside the group what was shared during the session.
Data Collection Separate focus group discussions were held on campus for students and faculty. Each session was approximately 1 hour in length, and discussion centered on the openended questions constructed for each group. The sessions were audio recorded and transcribed verbatim.
Data Analysis Content analysis was used to analyze the focus group interviews; the group was the unit of analysis. Content analysis is a systematic, replicable process in which content from narrative data is broken into smaller units that are coded and categorized. From these categories, prominent themes emerge (Polit & Beck, 2008; Stemler, 2001). The conventional approach to content analysis as presented by Hsieh and Shannon (2005) was used; this method is applied in studies where a phenomenon is to be described. The advantage of this approach is that direct information is obtained from study participants “without
imposing preconceived categories or theoretical perspectives” (Hsieh & Shannon, 2005, p. 1279–1280). The researcher read the transcripts several times and listened to the audiotapes to become further immersed in the data. Participants' words that captured key thoughts, feelings, and concepts were underlined, and notes were made by the researcher about her thoughts and first impressions. Initial labels for codes emerging from this process were organized, and emergent categories for related codes were gleaned. These were further organized as clusters, and prominent themes were then identified which gave meaning to the participants' experiences. Thus, an inductive coding process was used (McNeeseSmith, 1999). Several measures were taken, beginning with the focus groups and throughout the analytical process, to ensure methodological rigor. At the conclusion of each group session, the researcher summarized key points that had been shared by participants. She listened to the audiotapes to verify accuracy of the transcripts and verified the coding scheme with a colleague expert in content analysis. The transcripts and the researcher's written notes allow other investigators to follow the decisionmaking process and strengthen the auditability of the study. Finally, each participant was sent a brief written description of the themes and theme clusters to validate the material that was shared by group members and requested to provide any feedback as desired.
Themes From Student Data Three prominent themes with accompanying clusters emerged from the analysis of the student data, namely, Theme I—Dealing With Uncertainty, Theme II—Clarifying the Process, and Theme III—Valuing Outcomes. Table 1 presents each theme with related clusters.
Theme I—Dealing With Uncertainty This theme was composed of two clusters that reflected the uncertainty students felt about the process of compiling a portfolio and the expectations of themselves and faculty regarding this process. The first cluster, Understanding the Process, emerged from student concerns regarding what the portfolio was and the rationale for it, and how to proceed with compiling the specific exhibits/artifacts that should be included. One student commented, I wasn't sure exactly what this was about, it seemed foreign to me. I thought well a portfolio everyone [is] supposed to have one at this stage in your life but I didn't understand the need to have it so to speak. Another student said, “I was taking the portfolio as an academic professional portfolio not a career professional portfolio. I don't know if I am mistaken in that.” Many concerns surfaced regarding the content to be included. A student said, “I had no idea how it was to be presented,” whereas another added, “We have talked to a
EVALUATING PORTFOLIO USE
87
Table 1. Student and Faculty Focus Group Themes and Related Clusters Themes Student themes I—Dealing With Uncertainty II—Clarifying the Process III—Valuing Outcomes
Faculty themes I—Taking Stock II—Reworking the Process
Related Clusters Understanding the Process Coping With Advisement Knowing What and How Adding Other Resources Appreciating Self and Academic Development Realizing Professional Benefits
Giving Meaning to the Process Reflecting on Advisement Making Needed Changes Seeing the Portfolio as a Springboard
couple of the students and each person's portfolio is completely different from the next because they are not really sure what to do.” Two students shared that they included work that they were proud of. One student expressed a concern echoed by others about adding group projects to the portfolio, I felt the same way you did in terms of the group work that we did since a lot of the projects we do are in group work, how much of the group work could I put in, you know, am I able to put things in that have other people's name[s] on it…so there are some questions there like I wasn't exactly sure which of the projects we worked on actually were appropriate to put in. The second cluster, Coping With Advisement, was composed of phrases regarding thoughts about portfolio advisement and the expectations and lack of understanding of students and faculty about the advisement process. Students varied in the number of times they met with their academic advisor to review their portfolio work from having had no meeting to meeting every semester. A student shared, “I have taken three classes and I haven't met with anyone regarding my portfolio. I just have everything on my little zip drive, so obviously I need to make a meeting….” Another participant stated, “I never thought that I should go to a professor and sit down because it wasn't told to me that these were the expectations.” One student explained, I don't think that the advisors were informed enough. I think my advisor was asking me to do additional stuff and wasn't sure if it was supposed to be graded, how it was to be presented if I was presenting it to her, so she really doesn't know what the expectations for this is.
Theme II—Clarifying the Process The second theme with its two clusters addressed changes that could be implemented to smooth the way for students. The first cluster, Knowing What and How,
focused primarily on rectifying concerns expressed in the first theme related to students' introduction to the portfolio and advisement. One student recalling when the portfolio was first presented said, “…they did discuss the portfolio and that was on the very first day, so I thought that it was a good timing to present it on that first day of the very first class of the graduate program.” A student disagreed, feeling that the presentation in class was “a little overwhelming.” She asserted, I think it should be the first semester when you go for advisement it's mentioned by the advisor so you have some private time with the advisor to go through it and then it's followed up instead of presented in another class where it's not really that teacher's goal to instruct you; he is just providing you a simple piece of paper. Another student stated, I think also if it's presented in a way rather than being a school of nursing project that it's presented in a way that this is an actual tool that you're going to be using as a professional, so people feel more of a connection to it rather that it be something over here that they have to get to at some point to fulfill some requirement…. In terms of the frequency of advisement meetings, students agreed that meeting once a semester was important, and a student shared, “So, I think meeting more frequently and once a semester. I don't know if it is overwhelming for the advisor but from the student point of view, it is beneficial.” Another expressed, “I think it wouldn't be as scary as it is now if there was a little bit more guidance and just to break it down.” Finally, one participant added, ...I think it would be a good idea to have the advisor be consistent because if that is the person who is guiding you and showing you direction through this entire journey of self-discovery with the professional portfolio as a guide, you know because you do develop a sort of personal connection with your advisor and then boom you have someone else…. In the cluster, Adding Other Resources, several salient suggestions were offered to further facilitate the process of portfolio compilation. One student expressed that information about the portfolio could be “on the Web page somewhere under the school of nursing.” She also said that attaching an article with the outline about the portfolio information would be helpful “to make the students understand that this is not something that you are just going to be graded on but it's something to your benefit to have it and work on it while you are here.” Having samples available for students was also thought to be important, as well as the use of student peers. As a student shared, “…I don't think it is really such a bad idea to just connect a student that is just starting with someone who is toward the end, just to, you know, make it a little
88
MARYBETH RYAN
easier….” This individual also suggested having a student farther along in the program attend the class when the portfolio is first discussed and explain that, I was scared to death at first, too, but I just want to make sure that if anybody has any questions or any [need for] clarity, like for the first fifteen minutes of that class, that might take away a lot of anxiety away from the students. That would have helped me.
Theme III—Valuing Outcomes
Commenting on an actual experience, a participant shared, …I found that recently, having the portfolio, even in its infancy stage, was very helpful because on a resume you just can talk about some of the projects you have done, but with the portfolio, I could actually open it up and show some of the things, so it became very helpful in a way that I hadn't even really thought of.
The third theme emerged from two clusters and best explained the positive aspects of the portfolio. Students valued its use in various ways, including providing direction for their learning and helping them visualize their progress in the master's program. Despite the confusion about the process and content required for completing the portfolio, the cluster “Appreciating Self” and “Academic Development” revealed benefits that students became aware of through their portfolio work. A student said, “…it sort of focused me and basically built up my efficacy, my self-efficacy…so it has helped me to start thinking as opposed to just doing.” Another shared,
In addition to future professional benefits stemming from the portfolio, another student added how she drew from course work she had stored in her portfolio to effect a practice change in her present work setting.
I took the projects that we have done in our previous classes as far as any papers that were written and any corrections that I got back. I would put them in the portfolio and then it would help me as far as to develop better writing skills and to see how much I grew into a better writer. One participant expressed,
Two prominent themes with accompanying clusters emerged from analysis of the faculty audiotape and transcription, namely, Theme I—Taking Stock and Theme II—Reworking the Process. Table 1 presents each theme with related clusters.
Academically, I thought it was great for the Capstone because it helped me to pick out where I was the strongest on papers and I was able to choose a topic that I got excellent feedback from to work on my Capstone. Another student elaborated,
This theme was composed of two clusters that revealed faculty's reflection on the process of compiling the portfolio. The first cluster, Giving Meaning to the Process, emerged from faculty's thoughts about the purpose and process of portfolio work and how students viewed the portfolio, especially its value. One faculty member expressed that,
…having the portfolio gave you a sense of focus and direction as to where you are going because when you are going to classes you are just learning for the sake of learning but having the professional portfolio gives you a certain set direction and keeps you on track as to whether you are meeting the learning objectives I have set for myself. The second cluster in this theme, Realizing Professional Benefits, captured students' realization that the portfolio would be useful for future employment. It was seen as a medium for presenting one's academic and professional abilities and strengths to a prospective employer. A student in the administration track said, For me, it would be something useful to bring to an interview because it has my resume in it already. It would be able to show my part in group projects and my growth and things that I have done that are skills that a manager is going to need, like doing PowerPoints and research and different charts and things like that, so I thought, I think it is going to be useful for me later.
In the technology class I had, we had a project. I had to make an evaluation tool for work which now everybody in my unit uses to do their evaluation with. So, I never thought about it before, but I actually took it off my portfolio zip drive thing and brought it to work and everybody uses it now.
Themes From Faculty Data
Theme I—Taking Stock
The portfolio reflects our own interpretation of what curriculum is and I think that amongst us we have very different ways of defining the concern of the curriculum or what the curriculum itself is and I think that that really is reflected in what we want from the portfolio and what we look for. She stated, too, that “the portfolio is really a student's story of a transition.” Another member said that for students, the portfolio “should be representative of who they are and what they can present as they finish up their study…something that they can carry with them and reflect on and it also should show growth, the process of growth.” Concerns surfaced about how students viewed the portfolio and what they were compiling. One faculty member said, “…the biggest problem I see is this being seen as sort of an add on make work project.” This member added, “I think that for both students and faculty, there is a need to be clear about what it is, why it's there.” Another offered that,
EVALUATING PORTFOLIO USE
They are seeing it as a summary almost as to a lot of the work they have done and they are producing their best papers and sticking everything in but I don't know that they see that as something they are going to take forward. Participants acknowledged that students varied in their approaches to portfolio work. They also expressed that students often had limited experience regarding material that was to be included. “Some of them barely know what a resume is.” The second cluster, Reflecting on Advisement, was composed of faculty's thoughts about their involvement in and frustrations with the advisement process. Guiding students as they compiled the portfolio occurred in and outside the classroom; time spent working with students varied. One faculty member offered, I have primarily worked with them in the classroom setting where I went over the whole portfolio with them and as they developed the projects for the class, I would again remind them that it was a great project for them to put in their portfolio. Another expressed, “I make recommendations also in class that they include certain papers…then of course, I ask students to make an appointment with me to bring in their portfolio so I can look at it….” In contrast, one group member said that she had not worked with any students because she needed to be more informed about the process in order to guide them. Supporting this, a faculty member added, “…if we had a little more help just from the beginning, I know we went over it but it might have helped just [if] we first grouped together to discuss it somewhere along the line.” Frustration was apparent as the group members discussed getting students to meet with them to review their portfolio. One member captured the thoughts of others stating, “…when we first started this I had students come by because they were probably hearing about it in the courses…but then it seems like it has really tapered off and the students are not coming in.” It was agreed, however, that the portfolio was not required for students who were in the master's program before the portfolio was implemented; this was one reason some student advisees were not presenting portfolios to their faculty advisors. Offsetting these advisement-related concerns, one faculty member offered her positive experience advising students, “I was impressed that they arrived and came and that they were ready to go for me to review it. So that was a good thing.”
Theme II—Reworking the Process The second theme's two clusters emerged from ideas regarding improving the portfolio process and creating other strategies that would further enhance students' academic and professional development and support portfolio work. In the first cluster, Making Needed Changes, the need to secure faculty buy-in, engage students, and make changes in the process was clearly
89
indicated. A member said, “I think more faculty buy-in so that faculty don't just see this as one more thing on the list we've got to deal with.” Another expressed, “there was a lot of buy-in when we developed this but we are a new faculty…I think there is [sic] just maybe three of us or five of us who were really around when we developed it and it needs to be reworked by a new faculty. I think maybe we have to take it apart and put it together in a new way.” The moderator asked the group members where in the curriculum they thought the portfolio should be introduced. One member shared, “Maybe it needs to come right in the beginning. Well, it should be in orientation for the new students and then we could emphasize it again in the other courses.” Another said, “I think it would be good if we introduce it in the orientation and reinforce it in the core courses.” Also considered was whether developing a script or specific formats for using the portfolio in the core courses was needed. The participants also discussed providing additional approaches and resources to facilitate students' portfolio compilation. Having a “mock portfolio” was suggested as a tangible thing “…like an example of a portfolio to show and share with students, a visual, this is what a portfolio looks like.” Supporting this, a member added, “It would be helpful for us as advisees. It would be helpful for the students.” Helping students develop a better resume also was addressed. A member stated, “I want to make sure that they have guidelines to develop it that are appropriate.” Finally, the suggestion to have students compile their portfolio using a CD format received enthusiastic support from group members; this format was seen as efficient and effective. Regarding its use by students with a prospective employer, a member contributed, “You know it is true the disk is very impressive when you just hand somebody a disk.” In the second cluster, Seeing the Portfolio as a Springboard, ideas were generated by faculty regarding other strategies that would enhance students' academic and professional development and serve as adjuncts to portfolio work. Observing that one of the aims of the portfolio was emphasis on professional development at the advanced practice level, a member said, “We could also look at an ongoing online discussion or message board…a forum for looking at some of the issues in their own professional development and preparing for practice at an advanced level.” Adding to that was the suggestion for on campus “chats and forums and just roundtables or whatever…it gives more opportunity for exchanges and also helps to inform our own practice as to who we are.” Commenting on the need for more faculty discussion about advanced practice and other curriculum issues, a member envisioned a fishbowl where the faculty sat and had a discussion and the students were first observing and listening and then everybody discussed with each other and commented
90
MARYBETH RYAN
so we would get a chance to talk and hear what we think and how we see things.
Discussion The themes and related clusters revealed that students and faculty identified many parallel ideas, issues, and concerns regarding the portfolio. Both indicated that the intent of the process, guidelines for portfolio compilation, and roles to be played by both students and faculty need to be clarified. Portfolio advisement sessions varied among the students, and participants in both group expressed concerns related to them. In most instances, the existing process of advisor–advisee review of portfolio work did not permit the creation of a strong student– faculty relationship, which would be beneficial to both. Each group offered creative approaches to strengthening the portfolio process. Both advocated the need for a template/exemplar to facilitate students' work. It was suggested in the student group to use the student Web site to provide information about the portfolio, and faculty proposed having an electronic version of the portfolio. Although this approach to portfolio management can be an efficient, effective, and a more contemporary method for assessing students' learning, it is not without its challenges (Villano, 2006). Divergent viewpoints were apparent regarding students' perception of the value of completing the portfolio. Faculty perceived students to be focused only on compiling their portfolios and not seeing the benefits of doing so. Students, on the other hand, elaborated on the sense of focus and direction and other personal, academic, and professional benefits resulting from their work; this echoes Cangelosi's (2008) student evaluations. In fact, despite the many suggestions students offered to refine and strengthen the portfolio process, they were positive and very satisfied with the apparent and anticipated outcomes to be garnered from their portfolio experience. Faculty, thus, seemed to underestimate the value that students derived from their portfolio work. Notable for its absence was a theme regarding selfreflection. Faculty intended the portfolio to promote students' use of reflection as they compiled their work. Presently, students are expected to clearly explain the exhibit's importance, why it was selected, the learning that occurred, and how the exhibit illustrates academic or professional achievement, advanced practice standards, or attainment of master's program objectives. These annotations help students to develop critical incident work, self-critique, and writing skills (Jasper, 1995; Schaffer et al., 2005). When queried during the focus group sessions about their exhibit annotations, however, students reported that they were not annotating their work; faculty substantiated that students were not engaging in this reflective work. Further discussion about this critical component, methods used to annotate, and benefits derived did not occur. It is evident, therefore, that students need more direction about self-reflecting on and annotating their learning experiences. Specific strategies need to be
implemented to develop these skills, such as Ramey and Hay's (2003) method of identifying key courses in the curriculum, in which students are required to make portfolio entries and reflect on how the course outcomes support program outcomes. This approach might help students attend to reflection and annotation more systematically and foster the development of these skills. Faculty also can use the opportunity that advisement provides to dialogue more with students' about their selfreflection work.
Conclusions Faculty believe that using the portfolio provides an alternative/adjunct method for assessing student performance while fostering advanced nursing practice role development and attainment of established graduate nursing program outcomes. Students believe that using the portfolio keeps them focused and on track academically. They view it as a vehicle that will be useful when seeking future employment. Although faculty and students identified these benefits, primary issues and concerns discussed during the focus group interviews related to further clarifying the procedure for portfolio compilation and advisement and student and faculty roles critical to the process. An important aspect of portfolio development not adequately addressed by students or faculty, namely, helping students develop self-reflection skills and annotation habits to further strengthen their portfolio experience, requires further attention. It behooves faculty now to revamp the portfolio using the information gleaned from participants to guide them and improve the process. This is especially important at a time when several new faculty members who were not present when the original initiative began are teaching in the master's program. Rethinking the purpose, process, and expected outcomes with the faculty now teaching in the master's program will engage them and promote a renewed commitment to this valuable method of student assessment.
Acknowledgments The author thanks Dr. Jane White for the guidance she provided regarding focus group methodology and content analysis.
References Ackerhalt, E. J., Ryan, M., & Bogard, S. (2005). Professional portfolio document. Unpublished manuscript. Adelphi University, Garden City, New York. Billings, D. M. & Kowalski, K. (2008). Developing your career as a nurse educator: The professional portfolio. Journal of Continuing Education in Nursing, 39, 532–533. Cangelosi, P. R. (2008). Learning portfolios: Giving meaning to practice. Nurse Educator, 33, 125–127. Coffey, A. (2005). The clinical learning portfolio: A practice development experience in gerontological nursing. International Journal of Older People Nursing in association with Journal of Clinical Nursing, 14, 75–83.
EVALUATING PORTFOLIO USE
Gallagher, P. (2001). An evaluation of a standards based portfolio. Nurse Education Today, 21, 409–416. Gannon, F. T., Draper, P. R., Watson, R., Proctor, S. & Norman, I. J (2001). Putting portfolios in their place. Nurse Education Today, 21, 534–540. Harris, S., Dolan, G. & Fairbairn, G. (2001). Reflecting on the use of student portfolios. Nurse Education Today, 21, 278–286. Hayes, E., Chandler, G., Merriam, D. & King, M. C. (2002). The master's portfolio: Validating a career in advanced practice nursing. Journal of the American Academy of Nurse Practitioners, 14, 119–125. Hsieh, H. -F. & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. Jasper, M. A. (1995). The potential of the professional portfolio for nursing. Journal of Clinical Nursing, 4, 249–255. Jasper, M. A. & Fulton, J. (2005). Marking criteria for assessing practice-based portfolios at master's level. Nurse Education Today, 25, 377–389. Joyce, P. (2005). A framework for portfolio development in postgraduate nursing practice. Journal of Clinical Nursing, 14, 456–463. Karlowicz, K. A. (2000). The value of student portfolios to evaluate undergraduate nursing programs. Nurse Educator, 25, 82–87. Kear, M. E. & Bear, M. (2007). Using portfolio evaluation for program outcome assessment. Journal of Nursing Education, 46, 109–114. Kelly, J (1995). The really useful guide to portfolios and profiles. Nursing Standard, 9, 3–8. Krueger, R. A. & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage. McCready, T. (2006). Portfolios and the assessment of competence in nursing: A literature review. International Journal of Nursing Studies, 1–9. McMullan, M. (2008). Using portfolios for clinical practice learning and assessment: The pre-registration nursing student's perspective. Nurse Education Today, 28, 873–879. McMullen, M., Endacott, R., Gray, M. A., Jasper, M., Miller, C. L., Scholes, J., et al. (2003). Portfolios and assessment of
91
competence: A review of the literature. Journal of Advanced Nursing, 41, 283–294. McNeese-Smith, D. K. (1999). A content analysis of staff nurse descriptions of job satisfaction and dissatisfaction. Journal of Advanced Nursing, 29, 1332–1341. Mullen, P. (2007). The good portfolio. Nursing Standard, 21, 62–63. Pitts, J., Coles, C. & Thomas, P. (1999). Educational portfolios in the assessment of general practice trainers: Reliability of assessors. Medical Education, 33, 515–520. Polit, D. F. & Beck, C. T. (2008). Nursing research: Generating and assessing evidence for nursing practice (8th ed.). Philadelphia: Lippincott Williams & Wilkins. Price, A. (1994). Midwifery portfolios: Making reflective records. Modern Midwife, 35–38. Ramey, S. L. & Hay, M. L. (2003). Using electronic portfolios to measure student achievement and assess curricular integrity. Nurse Educator, 28, 31–36. Redman, W. (1994). Portfolios for development: A guide for trainers and managers. New Jersey: Nichols. Ryan, M. & Carlton, K. H. (1997). Portfolio applications in a school of nursing. Nurse Educator, 22, 35–39. Schaffer, M. A., Nelson, P. & Litt, E. (2005). Using portfolios to evaluate achievement of population-based public health nursing competencies in baccalaureate nursing students. Nursing Education Perspectives, 26, 104–112. Snadden, D. (1999). Portfolios—Attempting to measure the unmeasurable? Medical Education, 33, 478–479. Stemler, S. (2001). An overview of content analysis. [Electronic version]. Practical Assessment, Research & Evaluation, 7, 1–10. Tracy, S. M., Marino, G. J., Richo, K. M. & Daly, E. M. (2000). The clinical achievement portfolio. Nurse Educator, 25, 241–246. Villano, M. (2006). Electronic student assessment. Campus Technology, 21–27. Wenzel, L. S., Briggs, K. L. & Puryear, B. L. (1998). Portfolios: Authentic assessment in the age of the curriculum revolution [Electronic version]. Journal of Nursing Education, 37, 208–213. Willians, M. & Jordan, K. (2007). The nursing professional portfolio. Journal for Nurses in Staff Development, 23, 125–131.
I
DENTIFYING GRADUATE NURSING program outcomes to be attained by students is a critical but not difficult task. The challenge for faculty is to actually determine what students have learned; how they have changed academically, professionally, and personally; and whether program outcomes were met at the end of their journey through the curriculum. There is a growing national and international trend in nursing education programs to use portfolios to assess learning and competence (Harris, Dolan, & Fairbairn, 2001; Karlowicz, 2000). They have become a valuable alternative/ adjunct method for assessing student performance (McCready, 2006; McMullan, 2008). McMullen et al. (2003) define a portfolio as “a collection of evidence, usually in written form, of both the products and processes of learning. It attests to achievement and professional development by providing critical analysis of its contents” (p. 283). Numerous accounts of portfolio use in undergraduate nursing programs have been described (Gallagher, 2001; Harris ∗Adelphi University School of Nursing, Garden City, NY. Address correspondence to Dr. Ryan: Adelphi University School of Nursing, South Avenue, Garden City, New York 11530. 8755-7223/10/$ - see front matter 84 doi:10.1016/j.profnurs.2010.09.008
et al., 2001; Karlowicz, 2000; Kear & Bear, 2007; Ramey & Hay, 2003; Tracy, Marino, Richo, & Daly, 2000), but fewer reports have been provided about their use in graduate nursing education (Cangelosi, 2008; Coffey, 2005; Hayes, Chandler, Merriam, & King, 2002; Jasper & Fulton, 2005; Joyce, 2005). A newly revised curriculum was the impetus for implementing the use of portfolios in a graduate nursing program at a midsized private university. The portfolio was viewed as a vehicle for self-reflection and a way for students to document academic and professional accomplishments as they progressed through the master's program. It would allow students to demonstrate their grasp of advanced practice standards, document attainment of the master's program objectives, develop their ability to effectively critique their work, convey their values and view of themselves as a professional to others, showcase professional practice and career accomplishments, and, finally, assist them with career development and planning. Two years after the initiative was introduced, the author undertook a formative evaluation of the portfolio process by conducting focus groups to explore students' and faculty's experiences with using this assessment method. The article elaborates on the themes that emerged, the similarities and differences
Journal of Professional Nursing, Vol 27, No. 2 (March–April), 2011: pp 84–91 © 2011 Published by Elsevier Inc.
EVALUATING PORTFOLIO USE
between student and faculty views, and future directions necessary in the master's program for this method of assessing students' learning to be successful.
Literature Review Foremost among the many benefits for students is that portfolio use allows them to be accountable and autonomous learners responsible for the direction and quality of their learning and professional development progress (Cangelosi, 2008; Harris et al., 2001). Selfconfidence is built as students, receiving feedback about portfolio work, recognize and appreciate their ability, skills, and future potential (Kelly, 1995). Student growth is promoted in the areas of reflective thinking and practice, critical incident analysis work, and writing skills (Harris et al., 2001; Karlowicz, 2000; Schaffer, Nelson, & Litt, 2005; Wenzel, Briggs, & Puryear, 1998). Also beneficial for students and faculty is the creation of strong relationships between them as a result of continued portfolio-related dialogue. Ongoing discussion about the student's portfolio work establishes a two-way learning process from which both parties can benefit. For example, a faculty member might reexamine and change a teaching strategy or course content as a result of reviewing students' reflections on course material. It is important that faculty members play a key role in fostering this relationship and support students as they reflect on their learning experiences and build their portfolios (Harris et al., 2001; Karlowicz, 2000). Ryan and Carlton (1997) posit that students can use their portfolio when seeking future employment or other career opportunities. Of course, as Jasper (1995) indicates, “the potential of the portfolio depends upon the individual valuing its contents and regarding it as an essential part of his/her development” (p. 253). For students to be successful in using portfolios, it is important that they are given guidelines that clearly explain the portfolio's purpose, structure, and content (McCready, 2006; McMullen et al., 2003). Redman (1994) asserts that regardless of its purpose, a portfolio must be organized from the outset; the inclusion of specific portfolio sections/components is advocated (Karlowicz, 2000; Price, 1994; Ryan & Carlton, 1997). Problems related to using portfolios in nursing education have been documented (Harris et al., 2001; Jasper, 1995; Karlowicz, 2000). The first issue relates to time; students may view the portfolio as time consuming and postpone developing it as they focus on other course work. Faculty also may view the work entailed as time consuming. As a result, commitment on their part may vary leading to inequalities in student support, guidance, and assessment (Harris et al., 2001). Students may be reluctant to engage in self-reflection and not value or have developed the knowledge or skills to do it. As well, faculty may need assistance in developing and fostering students' self-reflection and self-assessment abilities (Karlowicz, 2000). Finally, a major issue involves the assessment/evaluation process. When students' portfolios are complete,
85
evaluation can provide information about the integrity of the academic program and whether students have achieved the program's outcomes (Ramey & Hay, 2003). Of concern are the methods used to measure students' competence and professional development. Reliability and validity are aspects that must be addressed when using portfolios, particularly in view of the primarily qualitative data included (Gannon, Draper, Watson, Proctor, & Norman, 2001; McCready, 2006). Some authors report the use of scoring rubrics as part of the evaluation process (Jasper & Fulton, 2005; Karlowicz, 2000; Ramey & Hay, 2003). Several authors advocate the need for further research in this area (Gannon et al., 2001; McCready, 2006; Pitts, Coles, & Thomas, 1999; Snadden, 1999).
The Professional Portfolio Initially, faculty teaching in the various master's programs in the School of Nursing met to develop the process for using the portfolio. The literature and extensive discussion concerning the rationale, objectives for, and content to be included in the portfolio served to shape the Professional Portfolio Document (PPD; Ackerhalt, Ryan, & Bogard, 2005) that students receive in the first required master's core course. The PPD conveys the portfolio's purpose, structure, and expectations of the student as the portfolio builder. The required portfolio components include six sections: (a) an introduction where the student introduces herself or himself and her or his ideas about advanced practice; (b) a resume; (c) documentation of academic achievements; (d) documentation of professional or work-related projects, publications, and so on; (e) the Capstone Project, a scholarly written project that is required for graduation; and (f) an essay summarizing growth as an advanced practice nurse. When including various exhibits/artifacts, students are expected to clearly explain the exhibit's importance, why it was selected, the learning that occurred, and how the exhibit illustrates academic or professional achievement, advanced practice standards, or attainment of master's program objectives. In the last part of the PPD, two tools are included that are used for formative assessment: student and academic faculty advisor evaluation tools. Each academic year, students are expected to complete a self-evaluation using the form and meet with a faculty advisor to discuss their portfolio work. The faculty advisor reviews the portfolio and also completes an evaluation tool. These forms yield nominal level and qualitative data. At program completion, a summative assessment of students' work is done. Many authors discuss the continued use of portfolios after formal nursing education is completed and their potential to create professional nursing knowledge (Billings & Kowalski, 2008; Jasper, 1995; Mullen, 2007; Willians & Jordan, 2007). Jasper (1995) writes, “… portfolios are likely to record the knowledge embedded in practice, which is often hard to describe, yet represents nursing's expertise” (p. 249). Portfolios, then, maintained after graduate nursing education could provide a fertile
86
MARYBETH RYAN
area for qualitative research exploring the knowledge to be gleaned from advanced nursing practice.
Method of Study A focus group research design was used in this study. Focus groups are used to gather information from a small number of individuals, usually six to eight, regarding their views on a specific area of interest. A critical component in the process of conducting a focus group interview occurs in the planning phase when a series of sequential questions is constructed that will engage participants and provide the most useful information (Krueger & Casey, 2000).
Participants Using purposive sampling, the researcher recruited by telephone students who were currently enrolled in the master's program. Faculty was recruited during a master's curriculum committee meeting and by e-mail. In focus group research, members who “have certain characteristics in common that relate to the topic” are selected (Krueger & Casey, 2000, p. 4). Of the eight female student participants, four were in nursing administration, three in nursing education, and one in the adult nurse practitioner program. The number of credits completed ranged from 9 to 42 with a mean of 23 credits earned. All six faculty participants, seven females and one male, were full-time faculty teaching in the master's program and responsible for advising students who were compiling their portfolios. Approval to conduct the study was obtained from the university's institutional review board. Since the study received exempt status, an informed consent was not required. To ensure participant anonymity, however, a code number was used on the demographic information form, and only first names were used during the interviews. In terms of confidentiality, participants were asked not to disclose outside the group what was shared during the session.
Data Collection Separate focus group discussions were held on campus for students and faculty. Each session was approximately 1 hour in length, and discussion centered on the openended questions constructed for each group. The sessions were audio recorded and transcribed verbatim.
Data Analysis Content analysis was used to analyze the focus group interviews; the group was the unit of analysis. Content analysis is a systematic, replicable process in which content from narrative data is broken into smaller units that are coded and categorized. From these categories, prominent themes emerge (Polit & Beck, 2008; Stemler, 2001). The conventional approach to content analysis as presented by Hsieh and Shannon (2005) was used; this method is applied in studies where a phenomenon is to be described. The advantage of this approach is that direct information is obtained from study participants “without
imposing preconceived categories or theoretical perspectives” (Hsieh & Shannon, 2005, p. 1279–1280). The researcher read the transcripts several times and listened to the audiotapes to become further immersed in the data. Participants' words that captured key thoughts, feelings, and concepts were underlined, and notes were made by the researcher about her thoughts and first impressions. Initial labels for codes emerging from this process were organized, and emergent categories for related codes were gleaned. These were further organized as clusters, and prominent themes were then identified which gave meaning to the participants' experiences. Thus, an inductive coding process was used (McNeeseSmith, 1999). Several measures were taken, beginning with the focus groups and throughout the analytical process, to ensure methodological rigor. At the conclusion of each group session, the researcher summarized key points that had been shared by participants. She listened to the audiotapes to verify accuracy of the transcripts and verified the coding scheme with a colleague expert in content analysis. The transcripts and the researcher's written notes allow other investigators to follow the decisionmaking process and strengthen the auditability of the study. Finally, each participant was sent a brief written description of the themes and theme clusters to validate the material that was shared by group members and requested to provide any feedback as desired.
Themes From Student Data Three prominent themes with accompanying clusters emerged from the analysis of the student data, namely, Theme I—Dealing With Uncertainty, Theme II—Clarifying the Process, and Theme III—Valuing Outcomes. Table 1 presents each theme with related clusters.
Theme I—Dealing With Uncertainty This theme was composed of two clusters that reflected the uncertainty students felt about the process of compiling a portfolio and the expectations of themselves and faculty regarding this process. The first cluster, Understanding the Process, emerged from student concerns regarding what the portfolio was and the rationale for it, and how to proceed with compiling the specific exhibits/artifacts that should be included. One student commented, I wasn't sure exactly what this was about, it seemed foreign to me. I thought well a portfolio everyone [is] supposed to have one at this stage in your life but I didn't understand the need to have it so to speak. Another student said, “I was taking the portfolio as an academic professional portfolio not a career professional portfolio. I don't know if I am mistaken in that.” Many concerns surfaced regarding the content to be included. A student said, “I had no idea how it was to be presented,” whereas another added, “We have talked to a
EVALUATING PORTFOLIO USE
87
Table 1. Student and Faculty Focus Group Themes and Related Clusters Themes Student themes I—Dealing With Uncertainty II—Clarifying the Process III—Valuing Outcomes
Faculty themes I—Taking Stock II—Reworking the Process
Related Clusters Understanding the Process Coping With Advisement Knowing What and How Adding Other Resources Appreciating Self and Academic Development Realizing Professional Benefits
Giving Meaning to the Process Reflecting on Advisement Making Needed Changes Seeing the Portfolio as a Springboard
couple of the students and each person's portfolio is completely different from the next because they are not really sure what to do.” Two students shared that they included work that they were proud of. One student expressed a concern echoed by others about adding group projects to the portfolio, I felt the same way you did in terms of the group work that we did since a lot of the projects we do are in group work, how much of the group work could I put in, you know, am I able to put things in that have other people's name[s] on it…so there are some questions there like I wasn't exactly sure which of the projects we worked on actually were appropriate to put in. The second cluster, Coping With Advisement, was composed of phrases regarding thoughts about portfolio advisement and the expectations and lack of understanding of students and faculty about the advisement process. Students varied in the number of times they met with their academic advisor to review their portfolio work from having had no meeting to meeting every semester. A student shared, “I have taken three classes and I haven't met with anyone regarding my portfolio. I just have everything on my little zip drive, so obviously I need to make a meeting….” Another participant stated, “I never thought that I should go to a professor and sit down because it wasn't told to me that these were the expectations.” One student explained, I don't think that the advisors were informed enough. I think my advisor was asking me to do additional stuff and wasn't sure if it was supposed to be graded, how it was to be presented if I was presenting it to her, so she really doesn't know what the expectations for this is.
Theme II—Clarifying the Process The second theme with its two clusters addressed changes that could be implemented to smooth the way for students. The first cluster, Knowing What and How,
focused primarily on rectifying concerns expressed in the first theme related to students' introduction to the portfolio and advisement. One student recalling when the portfolio was first presented said, “…they did discuss the portfolio and that was on the very first day, so I thought that it was a good timing to present it on that first day of the very first class of the graduate program.” A student disagreed, feeling that the presentation in class was “a little overwhelming.” She asserted, I think it should be the first semester when you go for advisement it's mentioned by the advisor so you have some private time with the advisor to go through it and then it's followed up instead of presented in another class where it's not really that teacher's goal to instruct you; he is just providing you a simple piece of paper. Another student stated, I think also if it's presented in a way rather than being a school of nursing project that it's presented in a way that this is an actual tool that you're going to be using as a professional, so people feel more of a connection to it rather that it be something over here that they have to get to at some point to fulfill some requirement…. In terms of the frequency of advisement meetings, students agreed that meeting once a semester was important, and a student shared, “So, I think meeting more frequently and once a semester. I don't know if it is overwhelming for the advisor but from the student point of view, it is beneficial.” Another expressed, “I think it wouldn't be as scary as it is now if there was a little bit more guidance and just to break it down.” Finally, one participant added, ...I think it would be a good idea to have the advisor be consistent because if that is the person who is guiding you and showing you direction through this entire journey of self-discovery with the professional portfolio as a guide, you know because you do develop a sort of personal connection with your advisor and then boom you have someone else…. In the cluster, Adding Other Resources, several salient suggestions were offered to further facilitate the process of portfolio compilation. One student expressed that information about the portfolio could be “on the Web page somewhere under the school of nursing.” She also said that attaching an article with the outline about the portfolio information would be helpful “to make the students understand that this is not something that you are just going to be graded on but it's something to your benefit to have it and work on it while you are here.” Having samples available for students was also thought to be important, as well as the use of student peers. As a student shared, “…I don't think it is really such a bad idea to just connect a student that is just starting with someone who is toward the end, just to, you know, make it a little
88
MARYBETH RYAN
easier….” This individual also suggested having a student farther along in the program attend the class when the portfolio is first discussed and explain that, I was scared to death at first, too, but I just want to make sure that if anybody has any questions or any [need for] clarity, like for the first fifteen minutes of that class, that might take away a lot of anxiety away from the students. That would have helped me.
Theme III—Valuing Outcomes
Commenting on an actual experience, a participant shared, …I found that recently, having the portfolio, even in its infancy stage, was very helpful because on a resume you just can talk about some of the projects you have done, but with the portfolio, I could actually open it up and show some of the things, so it became very helpful in a way that I hadn't even really thought of.
The third theme emerged from two clusters and best explained the positive aspects of the portfolio. Students valued its use in various ways, including providing direction for their learning and helping them visualize their progress in the master's program. Despite the confusion about the process and content required for completing the portfolio, the cluster “Appreciating Self” and “Academic Development” revealed benefits that students became aware of through their portfolio work. A student said, “…it sort of focused me and basically built up my efficacy, my self-efficacy…so it has helped me to start thinking as opposed to just doing.” Another shared,
In addition to future professional benefits stemming from the portfolio, another student added how she drew from course work she had stored in her portfolio to effect a practice change in her present work setting.
I took the projects that we have done in our previous classes as far as any papers that were written and any corrections that I got back. I would put them in the portfolio and then it would help me as far as to develop better writing skills and to see how much I grew into a better writer. One participant expressed,
Two prominent themes with accompanying clusters emerged from analysis of the faculty audiotape and transcription, namely, Theme I—Taking Stock and Theme II—Reworking the Process. Table 1 presents each theme with related clusters.
Academically, I thought it was great for the Capstone because it helped me to pick out where I was the strongest on papers and I was able to choose a topic that I got excellent feedback from to work on my Capstone. Another student elaborated,
This theme was composed of two clusters that revealed faculty's reflection on the process of compiling the portfolio. The first cluster, Giving Meaning to the Process, emerged from faculty's thoughts about the purpose and process of portfolio work and how students viewed the portfolio, especially its value. One faculty member expressed that,
…having the portfolio gave you a sense of focus and direction as to where you are going because when you are going to classes you are just learning for the sake of learning but having the professional portfolio gives you a certain set direction and keeps you on track as to whether you are meeting the learning objectives I have set for myself. The second cluster in this theme, Realizing Professional Benefits, captured students' realization that the portfolio would be useful for future employment. It was seen as a medium for presenting one's academic and professional abilities and strengths to a prospective employer. A student in the administration track said, For me, it would be something useful to bring to an interview because it has my resume in it already. It would be able to show my part in group projects and my growth and things that I have done that are skills that a manager is going to need, like doing PowerPoints and research and different charts and things like that, so I thought, I think it is going to be useful for me later.
In the technology class I had, we had a project. I had to make an evaluation tool for work which now everybody in my unit uses to do their evaluation with. So, I never thought about it before, but I actually took it off my portfolio zip drive thing and brought it to work and everybody uses it now.
Themes From Faculty Data
Theme I—Taking Stock
The portfolio reflects our own interpretation of what curriculum is and I think that amongst us we have very different ways of defining the concern of the curriculum or what the curriculum itself is and I think that that really is reflected in what we want from the portfolio and what we look for. She stated, too, that “the portfolio is really a student's story of a transition.” Another member said that for students, the portfolio “should be representative of who they are and what they can present as they finish up their study…something that they can carry with them and reflect on and it also should show growth, the process of growth.” Concerns surfaced about how students viewed the portfolio and what they were compiling. One faculty member said, “…the biggest problem I see is this being seen as sort of an add on make work project.” This member added, “I think that for both students and faculty, there is a need to be clear about what it is, why it's there.” Another offered that,
EVALUATING PORTFOLIO USE
They are seeing it as a summary almost as to a lot of the work they have done and they are producing their best papers and sticking everything in but I don't know that they see that as something they are going to take forward. Participants acknowledged that students varied in their approaches to portfolio work. They also expressed that students often had limited experience regarding material that was to be included. “Some of them barely know what a resume is.” The second cluster, Reflecting on Advisement, was composed of faculty's thoughts about their involvement in and frustrations with the advisement process. Guiding students as they compiled the portfolio occurred in and outside the classroom; time spent working with students varied. One faculty member offered, I have primarily worked with them in the classroom setting where I went over the whole portfolio with them and as they developed the projects for the class, I would again remind them that it was a great project for them to put in their portfolio. Another expressed, “I make recommendations also in class that they include certain papers…then of course, I ask students to make an appointment with me to bring in their portfolio so I can look at it….” In contrast, one group member said that she had not worked with any students because she needed to be more informed about the process in order to guide them. Supporting this, a faculty member added, “…if we had a little more help just from the beginning, I know we went over it but it might have helped just [if] we first grouped together to discuss it somewhere along the line.” Frustration was apparent as the group members discussed getting students to meet with them to review their portfolio. One member captured the thoughts of others stating, “…when we first started this I had students come by because they were probably hearing about it in the courses…but then it seems like it has really tapered off and the students are not coming in.” It was agreed, however, that the portfolio was not required for students who were in the master's program before the portfolio was implemented; this was one reason some student advisees were not presenting portfolios to their faculty advisors. Offsetting these advisement-related concerns, one faculty member offered her positive experience advising students, “I was impressed that they arrived and came and that they were ready to go for me to review it. So that was a good thing.”
Theme II—Reworking the Process The second theme's two clusters emerged from ideas regarding improving the portfolio process and creating other strategies that would further enhance students' academic and professional development and support portfolio work. In the first cluster, Making Needed Changes, the need to secure faculty buy-in, engage students, and make changes in the process was clearly
89
indicated. A member said, “I think more faculty buy-in so that faculty don't just see this as one more thing on the list we've got to deal with.” Another expressed, “there was a lot of buy-in when we developed this but we are a new faculty…I think there is [sic] just maybe three of us or five of us who were really around when we developed it and it needs to be reworked by a new faculty. I think maybe we have to take it apart and put it together in a new way.” The moderator asked the group members where in the curriculum they thought the portfolio should be introduced. One member shared, “Maybe it needs to come right in the beginning. Well, it should be in orientation for the new students and then we could emphasize it again in the other courses.” Another said, “I think it would be good if we introduce it in the orientation and reinforce it in the core courses.” Also considered was whether developing a script or specific formats for using the portfolio in the core courses was needed. The participants also discussed providing additional approaches and resources to facilitate students' portfolio compilation. Having a “mock portfolio” was suggested as a tangible thing “…like an example of a portfolio to show and share with students, a visual, this is what a portfolio looks like.” Supporting this, a member added, “It would be helpful for us as advisees. It would be helpful for the students.” Helping students develop a better resume also was addressed. A member stated, “I want to make sure that they have guidelines to develop it that are appropriate.” Finally, the suggestion to have students compile their portfolio using a CD format received enthusiastic support from group members; this format was seen as efficient and effective. Regarding its use by students with a prospective employer, a member contributed, “You know it is true the disk is very impressive when you just hand somebody a disk.” In the second cluster, Seeing the Portfolio as a Springboard, ideas were generated by faculty regarding other strategies that would enhance students' academic and professional development and serve as adjuncts to portfolio work. Observing that one of the aims of the portfolio was emphasis on professional development at the advanced practice level, a member said, “We could also look at an ongoing online discussion or message board…a forum for looking at some of the issues in their own professional development and preparing for practice at an advanced level.” Adding to that was the suggestion for on campus “chats and forums and just roundtables or whatever…it gives more opportunity for exchanges and also helps to inform our own practice as to who we are.” Commenting on the need for more faculty discussion about advanced practice and other curriculum issues, a member envisioned a fishbowl where the faculty sat and had a discussion and the students were first observing and listening and then everybody discussed with each other and commented
90
MARYBETH RYAN
so we would get a chance to talk and hear what we think and how we see things.
Discussion The themes and related clusters revealed that students and faculty identified many parallel ideas, issues, and concerns regarding the portfolio. Both indicated that the intent of the process, guidelines for portfolio compilation, and roles to be played by both students and faculty need to be clarified. Portfolio advisement sessions varied among the students, and participants in both group expressed concerns related to them. In most instances, the existing process of advisor–advisee review of portfolio work did not permit the creation of a strong student– faculty relationship, which would be beneficial to both. Each group offered creative approaches to strengthening the portfolio process. Both advocated the need for a template/exemplar to facilitate students' work. It was suggested in the student group to use the student Web site to provide information about the portfolio, and faculty proposed having an electronic version of the portfolio. Although this approach to portfolio management can be an efficient, effective, and a more contemporary method for assessing students' learning, it is not without its challenges (Villano, 2006). Divergent viewpoints were apparent regarding students' perception of the value of completing the portfolio. Faculty perceived students to be focused only on compiling their portfolios and not seeing the benefits of doing so. Students, on the other hand, elaborated on the sense of focus and direction and other personal, academic, and professional benefits resulting from their work; this echoes Cangelosi's (2008) student evaluations. In fact, despite the many suggestions students offered to refine and strengthen the portfolio process, they were positive and very satisfied with the apparent and anticipated outcomes to be garnered from their portfolio experience. Faculty, thus, seemed to underestimate the value that students derived from their portfolio work. Notable for its absence was a theme regarding selfreflection. Faculty intended the portfolio to promote students' use of reflection as they compiled their work. Presently, students are expected to clearly explain the exhibit's importance, why it was selected, the learning that occurred, and how the exhibit illustrates academic or professional achievement, advanced practice standards, or attainment of master's program objectives. These annotations help students to develop critical incident work, self-critique, and writing skills (Jasper, 1995; Schaffer et al., 2005). When queried during the focus group sessions about their exhibit annotations, however, students reported that they were not annotating their work; faculty substantiated that students were not engaging in this reflective work. Further discussion about this critical component, methods used to annotate, and benefits derived did not occur. It is evident, therefore, that students need more direction about self-reflecting on and annotating their learning experiences. Specific strategies need to be
implemented to develop these skills, such as Ramey and Hay's (2003) method of identifying key courses in the curriculum, in which students are required to make portfolio entries and reflect on how the course outcomes support program outcomes. This approach might help students attend to reflection and annotation more systematically and foster the development of these skills. Faculty also can use the opportunity that advisement provides to dialogue more with students' about their selfreflection work.
Conclusions Faculty believe that using the portfolio provides an alternative/adjunct method for assessing student performance while fostering advanced nursing practice role development and attainment of established graduate nursing program outcomes. Students believe that using the portfolio keeps them focused and on track academically. They view it as a vehicle that will be useful when seeking future employment. Although faculty and students identified these benefits, primary issues and concerns discussed during the focus group interviews related to further clarifying the procedure for portfolio compilation and advisement and student and faculty roles critical to the process. An important aspect of portfolio development not adequately addressed by students or faculty, namely, helping students develop self-reflection skills and annotation habits to further strengthen their portfolio experience, requires further attention. It behooves faculty now to revamp the portfolio using the information gleaned from participants to guide them and improve the process. This is especially important at a time when several new faculty members who were not present when the original initiative began are teaching in the master's program. Rethinking the purpose, process, and expected outcomes with the faculty now teaching in the master's program will engage them and promote a renewed commitment to this valuable method of student assessment.
Acknowledgments The author thanks Dr. Jane White for the guidance she provided regarding focus group methodology and content analysis.
References Ackerhalt, E. J., Ryan, M., & Bogard, S. (2005). Professional portfolio document. Unpublished manuscript. Adelphi University, Garden City, New York. Billings, D. M. & Kowalski, K. (2008). Developing your career as a nurse educator: The professional portfolio. Journal of Continuing Education in Nursing, 39, 532–533. Cangelosi, P. R. (2008). Learning portfolios: Giving meaning to practice. Nurse Educator, 33, 125–127. Coffey, A. (2005). The clinical learning portfolio: A practice development experience in gerontological nursing. International Journal of Older People Nursing in association with Journal of Clinical Nursing, 14, 75–83.
EVALUATING PORTFOLIO USE
Gallagher, P. (2001). An evaluation of a standards based portfolio. Nurse Education Today, 21, 409–416. Gannon, F. T., Draper, P. R., Watson, R., Proctor, S. & Norman, I. J (2001). Putting portfolios in their place. Nurse Education Today, 21, 534–540. Harris, S., Dolan, G. & Fairbairn, G. (2001). Reflecting on the use of student portfolios. Nurse Education Today, 21, 278–286. Hayes, E., Chandler, G., Merriam, D. & King, M. C. (2002). The master's portfolio: Validating a career in advanced practice nursing. Journal of the American Academy of Nurse Practitioners, 14, 119–125. Hsieh, H. -F. & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. Jasper, M. A. (1995). The potential of the professional portfolio for nursing. Journal of Clinical Nursing, 4, 249–255. Jasper, M. A. & Fulton, J. (2005). Marking criteria for assessing practice-based portfolios at master's level. Nurse Education Today, 25, 377–389. Joyce, P. (2005). A framework for portfolio development in postgraduate nursing practice. Journal of Clinical Nursing, 14, 456–463. Karlowicz, K. A. (2000). The value of student portfolios to evaluate undergraduate nursing programs. Nurse Educator, 25, 82–87. Kear, M. E. & Bear, M. (2007). Using portfolio evaluation for program outcome assessment. Journal of Nursing Education, 46, 109–114. Kelly, J (1995). The really useful guide to portfolios and profiles. Nursing Standard, 9, 3–8. Krueger, R. A. & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage. McCready, T. (2006). Portfolios and the assessment of competence in nursing: A literature review. International Journal of Nursing Studies, 1–9. McMullan, M. (2008). Using portfolios for clinical practice learning and assessment: The pre-registration nursing student's perspective. Nurse Education Today, 28, 873–879. McMullen, M., Endacott, R., Gray, M. A., Jasper, M., Miller, C. L., Scholes, J., et al. (2003). Portfolios and assessment of
91
competence: A review of the literature. Journal of Advanced Nursing, 41, 283–294. McNeese-Smith, D. K. (1999). A content analysis of staff nurse descriptions of job satisfaction and dissatisfaction. Journal of Advanced Nursing, 29, 1332–1341. Mullen, P. (2007). The good portfolio. Nursing Standard, 21, 62–63. Pitts, J., Coles, C. & Thomas, P. (1999). Educational portfolios in the assessment of general practice trainers: Reliability of assessors. Medical Education, 33, 515–520. Polit, D. F. & Beck, C. T. (2008). Nursing research: Generating and assessing evidence for nursing practice (8th ed.). Philadelphia: Lippincott Williams & Wilkins. Price, A. (1994). Midwifery portfolios: Making reflective records. Modern Midwife, 35–38. Ramey, S. L. & Hay, M. L. (2003). Using electronic portfolios to measure student achievement and assess curricular integrity. Nurse Educator, 28, 31–36. Redman, W. (1994). Portfolios for development: A guide for trainers and managers. New Jersey: Nichols. Ryan, M. & Carlton, K. H. (1997). Portfolio applications in a school of nursing. Nurse Educator, 22, 35–39. Schaffer, M. A., Nelson, P. & Litt, E. (2005). Using portfolios to evaluate achievement of population-based public health nursing competencies in baccalaureate nursing students. Nursing Education Perspectives, 26, 104–112. Snadden, D. (1999). Portfolios—Attempting to measure the unmeasurable? Medical Education, 33, 478–479. Stemler, S. (2001). An overview of content analysis. [Electronic version]. Practical Assessment, Research & Evaluation, 7, 1–10. Tracy, S. M., Marino, G. J., Richo, K. M. & Daly, E. M. (2000). The clinical achievement portfolio. Nurse Educator, 25, 241–246. Villano, M. (2006). Electronic student assessment. Campus Technology, 21–27. Wenzel, L. S., Briggs, K. L. & Puryear, B. L. (1998). Portfolios: Authentic assessment in the age of the curriculum revolution [Electronic version]. Journal of Nursing Education, 37, 208–213. Willians, M. & Jordan, K. (2007). The nursing professional portfolio. Journal for Nurses in Staff Development, 23, 125–131.