Undergraduate business students’ perceptions of learning outcomes in problem based and faculty centered courses

The International Journal of Management Education 16 (2018) 121–130

Contents lists available at ScienceDirect

The International Journal of Management Education journal homepage: www.elsevier.com/locate/ijme

Research Notes

Undergraduate business students’ perceptions of learning outcomes in problem based and faculty centered courses

T

Petra Garnjosta,∗, Stephen M. Brownb a b

htw saar, Business School, Hochschule für Technik und Wirtschaft des Saarlandes, Saarbrücken, Germany Jack Welch College of Business, Sacred Heart University, Fairfield, CT, USA

AR TI CLE I NF O

AB S T R A CT

Keywords: Problem based learning Learning outcomes Knowledge acquisition Problem solving Critical thinking Teamwork Self-directed learning

This article compares undergraduate business students' perception of the effectiveness of faculty centric pedagogy and problem based learning (PBL) pedagogy. The 303 participating students had experienced both methodologies. The survey measured the students' perceptions of five learning outcomes: knowledge acquisition, problem solving, critical thinking, teamwork, and self-directed learning. Three areas were measured asking the students to assess their experiences using the rubric's designed by the American Association of Colleges and Universities' Values Project (Sullivan, 2015). The content knowledge gain was measured based on Blooms Taxonomy (Krathwohl, 2002), and the self-directed learning by Knowles definition 1975. The scales were tested for validity and reliability in a pre-test with a different student group. No significant difference based on students' perception of learning outcomes in undergraduate business courses is found between PBL taught classes and faculty lead classes across all five perceptions of learning outcomes. There is also no significant relationship between demographic characteristics of the subjects and learning outcomes. The research results opens the questions the effects of individual course based implementations of PBL, differences in the effects of PBL for undergraduate and graduate students, and the effectiveness of PBL pedagogy in a single course in a degree program.

1. Introduction Management education is often criticized for being disconnected from practice (Smith, 2005). An online survey conducted in 2013 on behalf of the Association of American Colleges and Universities provided a detailed insight in priorities of college graduates from the perspective of employer (Hart Research Associates, 2013). Ninety-three percent of the 318 participating employer agreed, “a candidate's demonstrated capacity to think critically, and solve complex problems is more important than their undergraduate major” (Hart Research Associates, 2013, p. 22). Employees also endorse practices that require students to demonstrate both acquisition of knowledge and its application (Hart Research Associates, 2013). Based on their recommendation, universities should increase emphasis on critical thinking (82% agreement), complex problem solving (81%), written and oral communication (80%), and on the application of knowledge and skills in real world settings (78%) (Hart Research Associates, 2013). Business schools have responded to these criticisms by implementing more application based pedagogical methods, such as problem based learning (PBL), as a teaching method in addition to faculty centered approaches. According to Smith (2005), PBL is a learner centered pedagogical method which basic process is relatively simple. Teams of students solve realistic unstructured problems from their field of professional practice by defining problems, identifying gaps in their knowledge, collecting relevant information,

∗

Corresponding author. E-mail addresses: [email protected] (P. Garnjost), [email protected] (S.M. Brown).

https://doi.org/10.1016/j.ijme.2017.12.004 Received 1 September 2016; Received in revised form 10 October 2017; Accepted 4 December 2017 1472-8117/ © 2018 Elsevier Ltd. All rights reserved.

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

and proposing solutions. PBL's goal is long-term learning that results in behavioral change, not just conceptual mastery (Brownell & Jameson, 2004). The approach emphasizes the practice “of problem solving skills, usually not acquired in traditional college coursework” (Willis, 2002, p. 282), and can “bridge the gap between theory and practice” (Hsieh & Knight, 2008, p. 29). More than one decade ago, Sherwood (2004) noted, “problem-based learning has great potential for management education” (Sherwood, 2004, p. 536). The professional gains include: the need to have team skills and good team member attitudes, discipline specific knowledge that has outstripped the ability to know it all, requirements of efficiency of organizations within which graduates practice, the presentation of unique and unfamiliar problems, and pressure to solve problems with incomplete or contradictory data (Brownell & Jameson, 2004). Previous research has contributed to understand the positive effects of PBL on these cognitive processes (Hartman, Moberg, & Lambert, 2013). The literature supports that PBL enhances deep learning (Dods, 1997; Gijbels, Dochy, Van den Bossche, & Segers, 2005; Ryan, Dolling, & Barnet, 2004), increases critical thinking and ethical reasoning (Becker, Viljoen, Botma, & Bester, 2003; Kong, Qin, Zhou, Mou, & Gao, 2014), improves self-directed and independent learning (Kong et al., 2014), improves teamwork, collaboration, understanding of own and other's professional roles (Cook & Moyle, 2002; Cusack and O'Donoghue, 2012; Kong et al., 2014; Sharp & Primrose, 2003; Williams, 1999) and even increases attendance (Creghan & Adair-Creghan, 2015; Lieux, 1996). However, a crucial question is if students actually perceive these gains and agree that PBL has a better way to enhance their professional skills as compared to lecture based courses (Walker & Leary, 2009) that is the core question of this research study. The current study investigates whether students' perceptions of their learning outcomes are more positive in a problem-based learning format class than a traditional faculty lead lecture-based format class. Our findings contribute to the existing knowledge of studentbased pedagogy by increasing our understanding of how students perceive the results of their own learning experiences in problembased learning classes. 2. Literature review The genesis of PBL was famously in the medical field at McMaster University in Canada in the 1960's and spread from there to other medical schools (Klegeris & Hurren, 2011). PBL later was adopted by other professional disciplines such as allied health, engineering, the sciences, and business (Strobel & van Barneveld, 2009). There is a major difference in the implementation of PBL pedagogy in medicine and health related disciplines versus implementation in other disciplines. The health related programs tend to use PBL as the methodology for the entire program as surveys represented in meta-reviews show (e.g. Albanese & Mitchell, 1993; Vernon & Blake, 1993). The other disciplines tend to implement PBL in a course or courses but not usually the entire program, especially not for undergraduates (Chang, 2001; Matthews, 2004; Mitchell, Canavan, & Smith, 2010; Willis, 2002). Most literature on PBL learning outcomes is available from business programs based on PBL pedagogy mainly at graduate level (e.g. Stinson & Milter, 1996). For undergraduate business programs, research is more limited where we find evaluations of PBL primarily at the single course level (e.g. Bamford, Karjalainen, & Jenavs, 2012; Heagy & Lehmann, 2005; Sherwood, 2004; Stanley & Marsden, 2012). Only one study explicitly compares business programs with one of them entirely PBL taught (Dochy, Segers, Van Den Bossche, & Struyven, 2005). The current review of the extant literature on the effectiveness of PBL focuses on the five learning outcomes employers have identified as expecting from business students –skill sets: problem solving, critical thinking, knowledge acquisition, teamwork and self-directed learning. The review relied heavily on published meta-analysis of the research (Albanese & Mitchell, 1993; Berkson, 1993; Vernon & Blake, 1993; Colliver, 2000; Dochy, Segers, Van den Bossche, & Gijbels, 2003; Gijbels et al., 2005; Kalaian, Mullan, & Kasim, 1999; Newman, 2003), and on a meta-synthesis by including most of the mentioned meta-analysis (Strobel & van Barneveld, 2009). These meta-analysis are based on research material covering PBL in medical or health education, expect for Dochy et al., 2005, who included a survey in the field of education. None of the meta-analyses report on business program results, as little research has been conducted outside of the medical and education fields (Hmelo-Silver, 2004). The current research contributes to the existing literature by increasing our understanding of how PBL learning goals are achieved across the spectrum of learners in a business program. 2.1. Knowledge acquisition Albanese and Mitchell’s (1993) meta-review on the outcomes and implementation issues of PBL included 10 studies in the medical field, published between 1972 and 1992, eight of which examined full-time PBL curricula. These studies all used the National Board Medical Exams 1 (NBME 1, a standardized exam measuring the acquisition of scientific knowledge in the medical field) to measure the effects of knowledge acquisition of PBL compared to lecture based teaching methods. “PBL students scored lower on basic science examinations and viewed themselves less well prepared in the basic sciences in comparison to their conventionally trained counterparts” (Albanese & Mitchell, 1993, p. 58). They concluded that “PBL students might have deficits developing adequate cognitive scaffolding” (Albanese & Mitchell, 1993, p. 61). Vernon and Blake (1993) looked at 22 studies in the field of medical education. Knowledge acquisition was also measured with NBME 1, and all their data confirm a significant trend favoring traditional teaching methods over PBL methods. However, when other outcome measures were used to test the knowledge acquisition, the trend in favor of traditional teaching was not significant (Vernon & Blake, 1993, p. 556). Kalaian et al. (1999) also focused on 22 medical education studies from 1970 to 1997 and confirmed, that traditional learning approaches tend to produce better results for basic science knowledge than PBL. Also Colliver's review on the medical field was based on the three meta-analyses of 1993 and other surveys published between 1992 and 1998 (Colliver, 2000, pp. 259–266) indicated that 122

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

there was no convincing evidence that PBL improved the knowledge base of medical students. Dochy et al. (2003) investigated students' perception of a problem-based learning environment and if students perceive that PBL enhances learning. Thirty-three of the 43 studies explicitly measured knowledge and knowledge application, and their results suggest, that PBL falls short compared to traditional approaches when the assessment method chosen is focusing on recognition, such as the NBME 1 exam. Newman (2003) reviewed twelve studies also in the medical education domain and found similar results. Gijbels et al. (2005) reviewed 40 studies published between 1976 and 2000, all of them in the medical field, except for one. Their focus was to investigate the influence of assessment methods on the results of effectiveness as the main independent variable. According to their results, traditionally taught students performed better at knowledge levels that emphasized concepts, but not principles (understanding the link between concepts) or application knowledge structures (Gijbels et al., 2005, p. 43). Strobel and van Barneveld (2009) concluded looking at eight meta synthesis, all cited above, that for “knowledge assessment category, measure of short-term knowledge acquisition and retention, assess with NBME1, or multiple choice questions, we see mixed results, but tended to favor traditional learning approaches “ (Strobel & van Barneveld, 2009, p. 53.) However, the authors stated “that the better the capacity of an instrument to evaluate the application of knowledge by the student the greater the ascertained effect of PBL” (Strobel & van Barneveld, 2009, p. 53). In a single business course, Carriger (2016) found that content learning and knowledge acquisition was higher in the lecture based approach as compared to a PBL based approach when measured by course grades, exam grades or writing assignment grades. Bamford et al. (2012) found students in a PBL Operations Management class perceived a higher level of skills and knowledge from the PBL pedagogy. Further, significant improvements with regard to knowledge and problem-solving skills, could be found when including a game as part of the PBL method (Kanet & Barut, 2003). In Accounting the results are mixed. Stanley and Marsden (2012) did not find evidence that a PBL approach in Accounting improved the acquisition of knowledge for undergraduates, which is partly in contrast to the findings of Breton (1999), and Heagy and Lehmann (2005). Breton (1999), Heagy and Lehmann (2005) and Phillips and Vaidyanathan (2004) found that students exposed to PBL performed better in accounting theory and introductory accounting courses on case questions. Overall, the research on PBL taught courses in business programs leans toward there not being an increase in knowledge acquisition through PBL over more traditional faculty centric learning environments. 2.2. Problem solving Albanese and Mitchell (1993) reviewed seven studies that reported learning outcomes applying the medical knowledge, skills and understanding of clinical science essential for the provision of patient care of clinical knowledge and performance. According to their results PBL students scored higher on clinical examinations (NBME 2) in five of seven studies compared to students taught with conventional methods. However, the results were statistically significant only for Moore, Block, and Mitchell (1990) on the ethics problem-solving tasks (cited in Albanese & Mitchell, 1993, p.58). Using the assessment method of rating graduates and undergraduates by clinical supervisors regarding their problem solving skills in a clinical setting, the results were either more positive for PBL educated students or not significantly different (Albanese & Mitchell, 1993). Vernon and Blake (1993) also found in clinical functioning, the ability to solve problems in clinical environments and for performance outcomes measured with NBME 2, a slight but nonsignificant trend in favor of PBL programs. However, their findings showed significantly better results for PBL student outcomes of clinical performance based on observation of supervisors (Vernon & Blake, 1993, p. 557). Kalaian et al. (1999) reviewed 9 studies measuring NBME 2 outcomes and compared them with traditional curricular. They found positive results for NBME 2 which can be interpreted to show PBL tends to produce better results for clinical knowledge and skills. Dochy et al. confirmed, that assessment methods that focus more on application of knowledge such as NBME 2 showed larger effects of PBL versus traditional learning environments (Dochy et al., 2003). However, Newman, who selected 12 studies in the medical education that looked at applied practice found mixed results for problem-solving skills (Newman, 2003). Colliver (2000, pp. 259–266) found no convincing evidence that PBL improved the clinical performance. As mentioned in the section on knowledge acquisition, Gijbels et al. (2005) demonstrated that PBL students perform better when assessments look at knowledge from the angle of principles (understanding the link between concepts) and application knowledge structures (crucial for problem solving) (Gijbels et al., 2005). From their perspective, “the implications of assessment and the levels in the knowledge structure being measured must be considered when one examines the effects of problem-based learning” (Gijbels et al., 2005, p. 47). The discrepancy in reported results on the effectiveness of PBL for knowledge retention seemed to stem particularly from the differences in defining learning as long-term (PBL favorable) or short term (traditional teaching methods favorable) retention of knowledge (Strobel & van Barneveld, 2009, p. 54). The limited data on problem solving in PBL taught business classes point towards favoring PBL over lecture based approaches. Students perceived the PBL method as improving their ability to deal with unstructured problems and problems based on incomplete facts (Stanley & Marsden, 2012). They also found that “PBL results in better problem solving skills, as students agreed that they were now more comfortable dealing with unstructured problems” (Stanley & Marsden, 2012, p. 286). Kanet and Barut (2003) found support in their longitudinal study for their hypotheses of increasing problem solving skills in Production and Operations Management when taught in a PBL format. Daly, White, Zisk, and Cavazos (2012) found anecdotal evidence for a very positive reception of PBL section in an International Management class. Mainly based on faculty observation they state that “PBL format increased students’ problem solving skills, as they were able to acquire better understanding of the complexity of the trade-offs between risks and rewards in strategic decision making in International Management”(Daly et al., 2012, p. 271). 123

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

2.3. Critical thinking The results measuring the effect of PBL on critical thinking are mixed. Cook and Moyle (2002), Morales-Mann and Kaitell (2001), and Joe and Elizabeth (1999) found that PBL produces clear benefits for students' critical thinking. However, Choi, 2004 and Lyons, 2008 found evidence that PBL courses do not improve critical thinking compared to tradtional methods. A quasi-experimental study of undergraduate nursing students in 2008 by Yuan and colleagues found tremendous improvement in students' critical thinking skills through PBL instruction (Yuan, Williams, & Fan, 2008). Students felt that they analyze situations in different ways (91%), think in different ways (82%), and feel stimulated to think critically (74%) (Yuan et al., 2008). Kong et al. (2014) conducted a systematic review and meta-analysis of the development of nursing students' critical thinking based on nine studies. These studies measured critical thinking abilities of nursing students’ based on different assessment tools (Blooms' taxonomy, CCTS test Form 2000, ATI, CCTDI in the US, Turkish, Korean or Chinese versions). In only a small number of these studies, PBL was superior to traditional lectures in the training of critical thinking (Kong et al., 2014, p. 467). In the business related field of Hotel and Hospitality Management, Lieux found evidence that students perceived that PBL helped them develop their ability to think critically (Lieux, 1996). Pennell and Miles (2009) had evidence based on student feedback that PBL actually made them think. Nargundkar et al. compared exam results of a business analysis course before and after using PBL. His results on exam questions requiring critical thinking skills showed that student scored significantly higher when using guided PBL as a teaching method. Carriger's results based on HR Management suggest that it is important to consider the GPA of students and the year of college when looking at PBL and critical thinking. According to his findings, problem-based learning seems to promote more critical thinking when it is implemented earlier in a student's career in college and when that student has a higher than class average academic standing. Students further along in their college careers, regardless of academic standing, seem to respond less positively to the problem-based learning (Carriger, 2016). Overall there is mixed results of the impact of PBL on critical thinking. It appears that the research has a slight favor for PBL producing higher critical thinking skills, especially in business programs.

2.4. Teamwork The skill set of teamwork has not been stressed in PBL research. Albanese and Mitchell (1993) found out that PBL students learned to work in a group based environment and to handle group dynamics which is in line with the results of Lieux (1996) who found that PBL has helped students to develop their team skills. Stanley and Marsden (2012) found that students perceived an increase in their teamwork skills through participation in PBL. Bamford et al., 2012 state that for their Operations Management students “that the group work PBA (Problem Based Assessment) provides a better learning experience for students, but it is a worse classificatory of student result than the conventional exam” (Bamford et al., 2012, p. 1508). Based on this limited number of studies lookingat acquiring teamwork skills, there seems to be support from a perceptional point of view that PBL increases teamwork skills among students.

2.5. Self-directed learning The concept of PBL allows the students to play an active role in the learning process, and to be accountable for their own learning finds support (Chagas, Faria, Mourato, Pereira, & Santos, 2012). As there is more than one right answer, it encourages also to explore different solutions and to use the learned knowledge throughout the process of finding the most adequate solution (Karantzas et al., 2013; Mykytyn, Pearson, Paul, & Mykytyn, 2008). This theoretical approach of PBL and self-directed learning is supported by the empirical results of Albanese and Mitchell (1993). Their results suggest that PBL students tend to study differently compared to students in conventional courses, as they control a substantially greater degree of their learning efforts (Albanese & Mitchell, 1993). Students in traditional courses “are less likely to study for understanding or to analyze what they need to know for a given task and study accordingly” (Albanese & Mitchell, 1993, p. 62). Along the same lines are the findings from Vernon and Blake (1993), where PBL students place more emphasis on meaning and understanding than on recall. The findings suggest that there is a greater degree of independent study in PBL programs than in traditional programs (Vernon & Blake, 1993, p. 557). Berkson (1993) also questions if PBL promotes self-directed learning better than traditional approaches. Berkson observes that students place more emphasis on understanding rather than reproducing, which was the opposite pattern from students engaged in traditional learning methods. More recently Ali, Gameel, and El Sebai (2010) investigated the effect of PBL on nursing students' learning approach and their self-directed learning abilities. They found that PBL increases self-directed learning abilities among nurse students based on a Self-Directed Instrument Tool, which is in line with findings based on the perception of students by Srinivasan, Wilkes, Stevenson, Nguyen, and Slavin (2007). Stanley and Marsden (2012) found evidence for Accounting that PBL creates a learning environment where “inquiry activates selfdirected learning, information mining, dialogue and collaborative problem-solving can be incorporated in the design to enrich the student learning experience” (Stanley & Marsden, 2012, p. 286). Daly et al. (2012) reported anecdotal evidence for a very positive reception of PBL in an International Management Session, as students took ownership of the material incorporated in the exercise, were motivated and active learners and more engaged in the whole process. There seems to be support for the idea that the PBL method increases self-directed learning and inquiry. 124

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

3. Summary The literature review on the effectiveness of PBL regarding the five skill sets as intended learning outcomes reveal a mixed results (for details see Appendix 1). Depending on the assessment tools used in the study and depending on PBL as main instructing method or implemented on a course base in a mainly lecture based curriculum, the results are more or less in favor of PBL. Overall, there seems to be more evidence for the effectiveness for PBL when it is the main teaching method for a program as we see in medical and health related programs compare to business programs. We also see different assessment methods used to investigate effectiveness of PBL. Knowledge based assessment tools are in favor of lectures based classes. Knowledge application assessed with perception of supervisors, lecturer or students seems to find evidence for PBL. For the other skill sets investigated in this literature review, questionnaires were used to measure students' perception based on different items and scales with positive but not significant results for critical thinking, teamwork and self-directed learning. This supports Bamford et al. suggestion that “it is advisable to create targeted questionnaires to capture the student opinion more accurately” (Bamford et al., 2012, p. 1509). 4. Hypotheses This research investigates the students' perception of learning outcomes testing the following hypotheses. H1: Undergraduate business students perceive no difference in knowledge acquisition when courses are taught in a PBL or lecture based pedagogy. H2: Business students perceive no difference in problem solving when taught in a PBL or to lecture based pedagogy. H3: Business students perceive no difference in critical thinking skill when taught in a PBL or lecture based pedagogy. H4: Business students perceive no difference in teamwork skills when taught in a PBL or lecture based pedagogy. H5: Business students perceive no difference in self-directed learning skills when taught in a PBL or lecture based pedagogy. 5. Methodology/research methods This study was conducted to compare Business students' perceived learning outcomes for a problem-based approach to a lecturebased approach. The focus of this survey was only the students’ perception of their learning outcomes for the skill set: knowledge acquisition, problem solving, critical thinking teamwork, and self-directed learning. In addition to this scale items testing, the survey was tested by a peer group to evaluate if the questions were understood and to monitor the length of the survey. All data of the online-survey were analyzed using paired sample t tests to compare the means of learning outcomes depending on the pedagogy chosen. A series of multivariate analysis of variances was used to see if there were differences among demographic variables. We further used Bonferroni correction to account for multiple comparisons. 5.1. Survey construction The online survey consisted of different sections. The first section collected demographic data from the participants such as gender, age, year of college, work experience, GPA, and Business major. The second section of the survey asked for a rating of statements reflecting learning outcomes of five skills sets on a five point Likert scale. The five skill sets are knowledge acquisition, problem solving, critical thinking, teamwork, and self-directed learning (for scales, reliability and validity see measure). The first learning environment was traditionally faculty centered, labeled as “lecture based instruction” reflecting Barrows definition as a large-class, instructor-driven, lecture-based deliveries within a curriculum, which compartmentalized the content (Barrows, 2002). In such a learning environment, there is no need for free inquiry as the idea is to diagnose the problem and generate a solution based on the relevant information already provided (Walker & Leary, 2009, p. 17). It also requires students to demonstrate understanding by replicating materials provided by the faculty member on exams (Kuruganti, Needham, & Zundel, 2012). This led to the definition of lecture based classes used in the survey and explained to the participants as: “In a lecture based class, the content is mainly presented by the instructor. Students work on exercises or cases with clear defined problems and questions related to the content presented in class”. The key components of PBL defined by Barrows (2002) led to the definition of PBL. According to him, the components of PBL comprise an ill-defined problem given by the faculty member before coverage of specific applicable content. The problem is messy and designed to facilitate the accomplishment of specific learning goals. The problems have more than one possible solution, usually require interdisciplinary knowledge, and are realistic applications of the content. The student teams direct their inquiry into the content needed to solve the presented problem. Their inquiry is unencumbered, but faculty facilitated. Student often take the problem in different and unexpected directions. Free inquiry and dialogue are imperative to the PBL process. Faculty members are facilitators, tutors, experienced professionals, but not the source of all knowledge or the organizer of the learning experience which led to the following definition for the survey: “In problem based learning classes, small student teams have to define and solve unstructured real world business problems of nonprofit or for-profit organizations. Instructors facilitate the groups by asking questions, but not by defining knowledge gaps, learning goals, or by providing lectures.” Students were only able to answer questions related to pedagogical environments which they had experienced. The survey closed with questions addressing the respondents’ satisfaction with each learning environment. Only students that experienced both pedagogies were analyzed in the final analysis. Thus a matched pairs sample was used. 125

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

5.2. Measures All variables were measured on one survey that was administered to the subjects at one point in time. The variables used to measure problem solving skills, critical thinking skills and teamwork skills were the equivalent VALUE Rubrics from the Association of American Colleges and Universities (Sullivan, 2015). The VALUE rubrics provide an approach to assess learning outcomes of the skill sets mentioned above. These rubrics have been tested for reliability and validity prior to this research (Rhodes, 2010). The reliability measures for the scales to measure knowledge acquisition in the survey were developed based on the cognitive domain of Bloom's Taxonomy (Krathwohl, 2002). The criteria of being able to recall data, understand, apply, analyze, synthesize and evaluate were reformulated into survey items and tested for validity and reliability in a pre-test with a different student group with a Cronbach's alpha of 0.922 in the study and 0.922 in the pre-test. The scale to measure self-directed learning has been developed based on Knowles definition in 1975: “(…)’self-directed learning’ describes a process by which individuals take the initiative, with or without the assistance of others, in diagnosing their learning needs, formulating learning goals, identify human and material resources for learning, choosing and implement appropriate learning strategies, and evaluating learning outcomes.” (Knowles, 1975, p. 18) His definition is still guiding the discussion of self-directed learning and PBL (Zimmerman & Lebeau, 2000) therefore its elements were transformed to the items of the self-directed learning scale: recognize learning needs, formulate learning goals, identify relevant resources, shoes strategy, and evaluate outcomes. The reliability of these scales was 0.908 in the study and 0.802 in the pre-test. As can be seen in chart 1, reliability test of all five scales based on Cronbach's Alpha showed adequate results. A confirmatory factor analysis for all multi-item scales shows that 5 factors were extracted, and all items load on the skill set they are supposed to measure based on the rotated component matrix with the extraction method of principal component analysis, multivariate analysis with Kaiser Normalization and rotation converged in 7 iterations. 5.3. Sample All participants were business students in a university in the northeastern United States. The classes and sections of classes participating in the survey were stratified into types of pedagogy (faculty centered approach, project based learning, flipped classrooms, problem based learning, and service learning) and then sampled based in that classification. In total, 405 undergraduate business students were asked to participate in the online survey in their last week of spring term. The link of the survey was emailed to them prior to a brief introduction by the researcher in one of the selected classes. Students had the opportunity to answer the questionnaire during class time, but also had the choice not to do so. It was explained to the students, that the survey is not an evaluation of the class they were currently attending, that their participation is voluntary and that the data collected are anonymous. The online-survey was approved by the IRB of the university, and each student signed a consent form by actively clicking on the agree button to start the online survey. In total 316 students answered the questionnaire, and after data cleaning based on incomplete questionnaires, 303 questionnaires remained. Out of these 303 participants, all of them have experienced lecture based classes and 104 classes they perceived as taught in a PBL format. 5.4. Participants and demographics From the 303 participants of the business school students, 44.2% were female. The largest group represented in the survey is Freshman with 44.5%, followed by nearly the same percentage of Sophomore (19.3%), Junior (18.6%), and Senior (17.3%). This corresponds with the percentages regarding the represented age groups. 71.2% of all participants were under 21; 26.5% between 21 and 25, and 2.3% over 26 years of age. Most of the participants (34.4%) were Marketing and Sports Management major, followed by 23.4% Management and Business major, 19.7% Finance and Economics major, 13% Accounting major, and 8.4% undecided Business. Self-reported students’ performance level is also very diverse in the survey. Only 3% are weak students with a GPA under 2.5, and 15.6% between 2.5 and 2.8. More than one third (33.2%) has a GPA between 2.9 and 3.2 and one third between 3.3 and 3.6 (30.6%). 17.6% of the sample are strong students with a GPA between 3.7 and 4.0. Nearly 36% of the survey participants had no work experience, 20% up to three months and 16% six months or less. 20.8% had one year or more work experience including internships. 6. Results and analysis Paired samples t-tests were conducted comparing the perceived learning outcomes of lecture based classes and PBL based means for all five learning outcomes to identify significant differences (see Table 1). Hypothesis 1 postulated that undergraduate business students perceive no difference in knowledge acquisition when courses are taught in a PBL or lecture based pedagogy. As can be seen in Table 2, there is no significant difference between the two treatments and the null hypothesis is accepted. Students perceived no difference in knowledge acquisition. Hypothesis 2 formulated that business students perceive no difference in problem solving when taught in a PBL or to lecture based pedagogy. The results of the paired sample t-tests illustrated in Table 2 show no significant difference between the two treatments. Students perceived no difference in problem solving when taught is a PBL or lecture based class. The null hypothesis is accepted. Hypothesis 3 posited that business students perceive no difference in critical thinking skill when taught in a PBL or lecture based pedagogy. The data provided in Table 2 show no significant difference between the two treatments. Students perceived no difference in critical thinking when taught is a PBL or lecture based class. The null hypothesis is accepted. 126

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

Table 1 Paired samples statistics and correlations. Perceived Learning outcome

N

Correlation

Sig

Knowledge Acquisition Problem solving Critical Thinking Team Work Self-directed learning

104 105 104 104 104

.289 0.133 .309 0.352 0.348

0.003 .176 0.001 0.000 0.000

Table 2 Results of hypothesis testing. Perceived learning outcomes

Mean Lecture based

Mean Problem based

t-value

Significance (2 tailed)

Knowledge acquisition Problem Solving Critical Thinking Teamwork Self-directed learning

3.9327 3.9762 3.9340 4.0453 3.9750

4.0115 4.0952 4.0353 4.1316 4.0769

−1.029 −1.642 −1.736 −1.266 −1.477

0.306 0.104 0.086 0.208 0.143

Chart 1 Reliability of survey scales for all 5 skill sets. Scale

Cronbach's Alpha

Number of items

Problem solving Critical Thinking Teamwork Self-directed learning Knowledge acquisition

0.896 0.892 0.918 0.908 0.922

6 6 7 5 5

Hypothesis 4 stated that business students perceive no difference in teamwork skills when taught in a PBL or lecture based pedagogy. There is no significant difference between the two treatments. Students perceived no difference in teamwork when taught is a PBL or lecture based class. The null hypothesis is accepted (see Table 2). Hypothesis 5 theorized that business students perceive no difference in self-directed learning skills when taught in a PBL or lecture based pedagogy. As documented in Table 2, no significant difference between the two treatments could be identified. Students perceived no difference in self-directed learning when taught is a PBL or Lecture based class. The null hypothesis is accepted. A multivariate analysis of variances was conducted to determine if there were differences among demographic variables such as age, major and GPA, but no significant relationship could be identified. 7. Discussion and recommendations PBL tends to be seen as a pedagogy which will solve a lot of issues in management education of undergraduates, as from a theoretical perspective can close the gap between knowledge and skills. As employers more and more require certain skills from business students such as team working, problem solving and critical thinking the implementation of PBL can be an adequate solution. However, looking at empirical research measuring the learning outcomes of PBL based on this skill set, the proclaimed strengths of this pedagogy can be questioned. Most of the previous research on PBL in business programs was based on single PBL course evaluations compared with the same content taught in a lecture based format (e.g. Carriger, 2016; Bamford et al., 2012; Stanley & Marsden, 2012; Heagy & Lehmann, 2005; Phillips and Vaidyanathan, 2004; Breton, 1999). PBL research in business programs did look at assessments measuring outcomes based on test results (Carriger, 2016; Bamford et al., 2012; Heagy & Lehmann, 2005; Phillips and Vaidyanathan, 2004; Breton, 1999; Stanley & Marsden, 2012) or perception of the instructors (e.g. Daly et al., 2012). Some looked at students’ perceptions of learning outcomes using open comments on course evaluation sheets or by rating one or two questions referring to learning outcomes (Bamford et al., 2012; Lieux, 1996; Stanley & Marsden, 2012). These empirical data on undergraduate business courses delivered with PBL pedagogy show, that there are mixed results, partly based on what learning assessment method has been chosen. This study adds the perspectives of students on PBL pedagogy based on their perceived learning outcomes of PBL pedagogy and investigated students’ perception of the effectiveness of PBL in comparison with lecture based classes in undergraduate business curriculum. The current research used tested scales that were developed from the literature for each of the five cognitive skill sets in business curriculum. There are two results in this study that deserve further discussion. The first is that students perceive lecture based classes in a business curriculum as leading to a high level of outcomes in the five skill areas investigated. In fact, there was no 127

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

significant difference in students' perception of gains in the five skill areas based on pedagogy. These results may be surprising to supporters of the PBL pedagogy, but are somewhat consistent with the extant literature. What we see is that there is no significant result which support the superiority of PBL pedagogy based on students' perception of learning outcomes in undergraduate business courses. The superiority of PBL pedagogy compared to lecture based classes does not exist per se. It is important to reiterate, most of the literature that reported positive results when PBL pedagogy is used were implemented in graduate level programs in which the entire program uses PBL pedagogy. This provides many areas for further discussion and research. There have been researchers who have said that an entire program needs to use the PBL pedagogy in order for it to be fully effective (Dochy et al., 2003). The explanation seems to be that students need time to be acclimated to this method of learning, and become familiar with the expectations and process (Chye, Neo, & Da Silva, 2000). Most college students are familiar with a teacher focused pedagogy which assigns them a more passive role within the learning process. Students' perception of the learning process is mainly based on this traditional role of learner and teacher. Switching roles and as a consequence behavior and expectations required from a learner centered pedagogy such as PBL, needs time for adjustments. This view proposes that a new learning curve for students starts with being exposed to PBL pedagogy. Students have to get familiar with new structures and new processes of learning. The positive results of curricular entirely taught with PBL pedagogy support this idea. At the same time, the results can be interpreted as advice to not just implementing one single course taught with PBL format, but rather integrate PBL pedagogy on a broader scale throughout the business curriculum. Further, the results might also be interpreted as supporting the idea that the pedagogy is not the most important determinant of student learning, at least as long as they have been taught in a traditional learning environment during High School. This, of course, is a very controversial view. Another fact based on the empirical results of PBL pedagogy is that it seems to work better in graduate programs (Brownell & Jameson, 2004). This provides us with another set of questions. Can this be traced back to a maturation threshold for efficacy of PBL? As age had no significance within the current sample, we cannot find any approval of this assumption. The result might be based on the fundamental differences between undergraduate and graduate programs. PBL probably has a prerequisite set of discipline knowledge. Graduate students generally have that basic discipline knowledge. They thus can apply that knowledge to presented problems. The PBL pedagogy is probably not the most efficient way to present basic concepts. In business programs that introduce many disciplines, there is a lot of introductory concepts that are part of the curriculum. A pedagogy where at least partly the teacher is determining the knowledge gap might support the students' learning process. Also, graduate program problems tend to be more interdisciplinary, and thus more in line with PBL methods. The findings of this research are at least in parts contrary to expectations coming from experience and existing research. The theory of the difference of outcomes in a single course of PBL and a total PBL based curriculum or program. This is a difficult theory to test in practice. Most schools would not have two similar programs with these two different pedagogical designs. However, similar programs could be compared. The main area for further research is this question: Do undergraduate business PBL programs further promote knowledge acquisition, problem solving, critical thinking, teamwork, and self-directed learning? Can individual course implementations of PBL have the positive effects of PBL? Is there a difference in the effects of PBL for undergraduate and graduate students? It should also be considered to what extent a more qualitative approach based on the analysis of students feedback would enhance the results about the specific contribution of PBL to students learning in comparison to lecture style. Appendix Chart 1 Summary of literature review Skill set

Discipline

Knowledge acquisition

medicine Standardized test NBME I Other knowledge tests business Knowledge tests

Problemsolving

Assessment tools

medicine NBME 2 Rating by clinical supervisor business

Empirical studies

Effect of PBL

Albanese and Mitchell’s (1993), Vernon and Blake (1993), Berkson No (1993), Kalaian et al. (1999), Colliver (2000, pp. 259–266), Dochy partly significant et al. (2003) Vernon and Blake (1993) No, but not significant Carriger (2016), Bamford et al. (2012), Kanet and Barut (2003), Heagy and Lehmann (2005), Phillips and Vaidyanathan (2004), Breton (1999), Stanley and Marsden (2012) Albanese and Mitchell (1993), Vernon and Blake et al. (1993), Dochy et al., 2003) Albanese and Mitchell (1993), Kalaian et al. (1999), Gijbels et al. (2005) Stanley and Marsden (2012) 128

Yes No

Yes, but not significantly Yes, but not significant or no effect

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

Critical thinking

Students' perception Faculty observation/ students' perception medicine Standardized tests

business

Teamwork

Students' perception Standardized tool

medicine business Students' perception Self-directed medicine Standardized learning tool Students' perception business Students' perception Faculty observation/ students' perception

Kanet and Barut (2003) Daly et al. (2012)

Choi (2004); Lyons (2008) Kong et al. (2014)

Lieux (1996) Pennell and Miles (2009) Nargundkar, Samaddar, and Mukhopadhyay (2014) Carriger (2016) Albanese and Mitchell (1993), Lieux (1996), Stanley and Marsden (2012), Bamford et al. (2012)

Positive, not significant Positive, not significant

No Yes but only in small number of studies Yes Yes Mixed Yes Yes

Ali et al. (2010)

Yes

Srinivasan et al. (2007)

Yes

Stanley and Marsden (2012)

Yes

Daly et al. (2012)

Yes

References Albanese, M., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52–81. Ali, M., Gameel, W., & El Sebai, N. A. M. (2010). Effects of Problem-based learning on Nursing students' approaches to learning and their elf directed learning abilities. International Journal of Academic Research, 2(4). Bamford, D., Karjalainen, K., & Jenavs, E. (2012). An evaluation of problem-based assessment in teaching Operations management. International Journal of Operations & Production Management, 32(12), 1493–1514. Barrows, H. (2002). Is it truly possible to have such a thing as PBL? Distance Education, 23(1), 119–122. Becker, S., Viljoen, M. J., Botma, Y., & Bester, I. J. (2003). Integration of study material in the problem-based learning method. Curationis, 26(1), 57–61. Berkson, L. (1993). Problem-based learning: Have the expectations been met? Academic Medicine, 68(10), 79–88. Breton, G. (1999). Some empirical evidence on the superiority of the problem-based learning (PBL) method. Accounting Education, 8(1), 1–12. Brownell, J., & Jameson, D. A. (2004). Problem-based learning in graduate management Education: An integrative model and interdisciplinary application. Journal of Management Education, 28(5), 558–577. Carriger, M. S. (2016). What is the best way to develop new Managers? Problem-Based learning vs. Lecture-Based instruction. International Journal of Management in Education, 14(2), 92–101. Chagas, I., Faria, C., Mourato, D., Pereira, G., & Santos, A. (2012). Problem-based learning in an online course of health education. European Journal of Open, Distance and E-Learning, 15(1). Chang, C. Y. (2001). Comparing the impacts of a problem-based computer-assisted instruction and the direct-interactive teaching method on student science achievement. Journal of Science Education and Technology, 10(2), 147–153. Choi, H. (2004). The effects of problem-based learning on the metacognition, critical thinking, and problem solving process of nursing students. Taehan Kanho Hakhoe Chi, 34(5), 712–721. Chye, M., Neo, L., & Da Silva, G. (2000). Student evaluation of the subject under problem-based learning Mode: The case of practice of entrepreneurship. Second asiapacific conference on PBL. Colliver, J. A. (2000). Effectiveness of problem-based learning Curricular: Research and theory. Academic medicine, Vol. 75(3), 259–266. Cook, M., & Moyle, K. (2002). Students' evaluation of problem-based learning. Nurse Education Today, 22, 330–339. Creghan, C., & Adair-Creghan, K. (2015). The positive impact of project-based learning on attendance of an economically disadvantaged student population: A multiyear study. Interdisciplinary Journal of Problem-Based Learning, 9(2), 7. Cusack, T., & O'Donoghue, G. (2012). The introduction of an interprofessional education Module: Students' perceptions. Quality in Primary Care, 20(3), 231–238. Daly, P. S., White, M. M., Zisk, D. S., & Cavazos, D. E. (2012). Problem-based teaching in international management: A political/economic risk assessment exercise. Journal of Teaching in International Business, 23(4), 260–276. Dochy, F., Segers, M., Van Den Bossche, P., & Struyven, K. (2005). Students' perceptions of a problem-based learning environment. Learning Environments Research, 8(1), 41–66. Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A meta-analysis. Learning and Instruction, 13(5), 533–568. Dods, R. F. (1997). An action research study of the effectiveness of problem-based learning in promoting the acquisition and retention of knowledge. Journal for the Education of the Gifted, 20(4), 423–437. Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75(1), 27–61. Hart Research Associates (2013). It takes more than a major. Employer priorities for college learning and student success, Washington. 2013. http://www.aacu.org/sites/ default/files/files/LEAP/2013_EmployerSurvey.pdf. Hartman, K. B., Moberg, C. R., & Lambert, J. M. (2013). Effectiveness of problem-based learning in introductory business courses. Journal of Instructional Psychology,

129

The International Journal of Management Education 16 (2018) 121–130

P. Garnjost, S.M. Brown

12, 1–13. Heagy, C., & Lehmann, C. (2005). Is PBL an improved delivery method for the accounting curriculum. Advances in Accounting Education, 7, 221–251. Hmelo-Silver, C. E. (2004). Problem-based Learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. Hsieh, C., & Knight, L. (2008). Problem-based learning for engineering Students: An evidence-based comparative study. The Journal of Academic Librarianship, 34(1), 25–30. Joe, W. M., & Elizabeth, A. (1999). Problem-based Learning: An outcomes study. Nurse Education, 24(20), 33–36. Kalaian, H. A., Mullan, P. B., & Kasim, R. A. (1999). What can studies of problem-based learning tell Us? Synthesizing and modeling PBL effects on national board of medical examination Performance: Hierarchical linear modeling meta-analytic approach. Advances in Health Sciences Education, 4, 2009–2221. Kanet, J. J., & Barut, M. (2003). Problem-based learning for production and Operations management. Decision Sciences Journal of Innovative Education, 1(1), 99–118. Karantzas, G. C., Avery, M. R., Macfarlane, S., Mussap, A., Tooley, G., Hazelwood, Z., et al. (2013). Enhancing critical analysis and problem-solving skills in undergraduate Psychology: An evaluation of a collaborative learning and problem-based learning approach. Australian Journal of Psychology, 65(1), 38–45. Klegeris, A., & Hurren, H. (2011). Impact of problem-based learning in a large classroom Setting: Student perception and problem-solving skills. Advances in Physiology Education, 35, 408–415. Knowles, M. S. (1975). Self-directed learning: A guide for learners and teachers. New York: Association Press. Kong, L. N., Qin, B., Zhou, Y. Q., Mou, S. Y., & Gao, H. M. (2014). The effectiveness of problem-based learning on development of nursing students' critical thinking: A systematic review and meta-analysis. International Journal of Nursing Studies, 51(3), 458–469. Krathwohl, D. R. (2002). A revision of Bloom's Taxonomy: An overview. Theory into Practice, 41(4), 212–218. Kuruganti, U., Needham, T., & Zundel, P. (2012). Patterns and rates of learning in two problem-based learning courses using outcome based assessment and elaboration theory. The Canadian Journal for the Scholarship of Teaching and Learning, 3(1), 4. Lieux, E. M. (1996). A comparative study of learning in lecture vs. Problem-Based format. About teaching-# 50. A newsletter of the center for teaching effectiveness. University of Delaware. Lyons, E. M. (2008). Examining the effects of problem-based learning and NCLEX-RN scores on the critical thinking skills of associate degree nursing students in a southeastern community college. International Journal of Nursing Education Scholarship, 5(1), 1–17. Matthews, B. (2004). The effects of direct and problem-based learning instruction in an undergraduate introductory engineering graphics course. https://repository.lib.ncsu. edu/handle/1840.16/4584. Mitchell, J. E., Canavan, B., & Smith, J. (2010). Problem-based learning in communication Systems: Student perceptions and achievement. IEEE Transactions on Education, 53(4), 587–594. Moore, G. T., Block, S., & Mitchell, R. (1990). A randomized controlled trial evaluating the impact of the new pathway curriculum at harvard medical school. Report to the fund for the improvement of post-secondary educationHarvard Med. School. (Unpublished report) https://eric.ed.gov/?id=ED359866. Morales-Mann, E., & Kaitell, C. A. (2001). Problem-based learning in a new canadian curriculum. Journal of Advanced Nursing, 33(1), 13–19. Mykytyn, K., Pearson, A., Paul, S., & Mykytyn, P. P. (2008). The use of problem-based learning to enhance MIS education. Decision Sciences Journal of Innovative Education, 6(1), 89–113. Nargundkar, S., Samaddar, S., & Mukhopadhyay, S. (2014). A guided problem-based learning (PBL) Approach: Impact on critical thinking. Decision Sciences Journal of Innovative Education, 12(2), 91–108. Newman, M. J. (2003). Special report 2: A pilot systematic review and meta-analysis on the effectiveness of problem based learning. Learning and teaching support networkUK: Middlesex Universityhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.6561&rep=rep1&type=pdf. Pennell, M., & Miles, L. (2009). “It actually made me think”: Problem-based learning in the business communications classroom. Business Communication Quarterly, 72(4), 377–394. Phillips, F., & Vaidyanathan, G. (2004). Should case materials precede or follow lectures? Issues in Accounting Education, 19(3), 305–319. Rhodes, T. L. (2010). Assessing outcomes and improving achievement: Tips and tools for using rubrics. Association of American Colleges and Universitieshttps://www.aacu. org/publications-research/publications/assessing-outcomes-and-improving-achievement-tips-and-tools-using. Ryan, G., Dolling, T., & Barnet, S. (2004). Supporting the problem-based learning process in the clinical Years: Evaluation of an online clinical reasoning guide. Medical Education, 38(6), 638–645. Sharp, D. M., & Primrose, C. S. (2003). The “virtual family”: An evaluation of an innovative approach using problem-based learning to integrate curriculum themes in a nursing undergraduate programe. Nurse Education Today, 23(3), 219–225. Sherwood, A. L. (2004). Problem-based learning in management education: A framework for designing context. Journal of Management Education, 28(5), 536–557. Smith, G. F. (2005). Problem-based Learning: Can it improve managerial thinking? Journal of Management Education, 29(2), 357–378. Srinivasan, M., Wilkes, M., Stevenson, F., Nguyen, T., & Slavin, S. (2007). Comparing problem-based learning with case-based Learning: Effects of a major curricular shift at two institutions. Academic Medicine, 82(1), 74–82. Stanley, T., & Marsden, S. (2012). Problem-based Learning: Does accounting education need it? Journal of Accounting Education, 30, 267–289. Stinson, J. E., & Milter, R. G. (1996). Problem-based learning in business Education: Curriculum design and implementation issues. New Directions for Teaching and Learning, 68, 33–42. Strobel, J., & van Barneveld, A. (2009). When is PBL more Effective? A meta-synthesis of meta-analyses comparing PBL to conventional classroom. Interdisciplinary Journal of Problem-Based Learning, 3(1), 44–58. Sullivan, D. F. (2015). The VALUE Breakthrough: Getting the assessment of student learning in college right. AAC&U Publication. Vernon, D. T. A., & Blake, R. L. (1993). Does problem-based learning Work? A meta-analysis of evaluation research. Academic Medicine, 68(7), 550–563. Walker, A., & Leary, H. (2009). A problem based learning meta Analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 6. Williams, A. F. (1999). An antipodean evaluation of problem-based learning by clinical educators. Nurse Education Today, 19(8), 659–667. Willis, A. S. (2002). Problem-based learning in a general psychology course. The Journal of General Education, 51(4), 282–292. Yuan, H., Williams, B. A., & Fan, L. (2008). A systematic review of selected evidence on developing nursing students’ critical thinking through problem-based learning. Nurse Education Today, 28(6), 657–663. Zimmerman, B. J., & Lebeau, R. B. (2000). A commentary on self-directed learning. Problem-based learning: A Research Perspective on Learning Interactions, 299–313.

130