Peer evaluations within experiential pedagogy: Fairness, objectivity, retaliation safeguarding, constructive feedback, and experiential learning as part of peer assessment

The International Journal of Management Education 16 (2018) 92–104

Contents lists available at ScienceDirect

The International Journal of Management Education journal homepage: www.elsevier.com/locate/ijme

Peer evaluations within experiential pedagogy: Fairness, objectivity, retaliation safeguarding, constructive feedback, and experiential learning as part of peer assessment

T

Matthew VanSchenkhofa,∗, Matthew Houseworthb, Mary McCordc, Jessica Lannind a

Montana State University, 341 Reid Hall, Bozeman, MT 59717, United States University of Central Missouri, 400E Dockery, Warrensburg, MO 64093, United States c University of Central Missouri, 405G Dockery, Warrensburg, MO 64093, United States d University of Central Missouri, 2100 Lovinger, Warrensburg, MO 64093, United States b

AR TI CLE I NF O

AB S T R A CT

Keywords: Peer assessment Fairness Objectivity Retaliation safeguarding Constructive feedback Social protection Experiential learning

This paper describes the analysis of peer assessment processes in experiential learning courses at a Midwestern public university. Factor analysis was used in the creation of constructs measuring perceptions of fairness, objectivity, retaliation safeguarding, and constructive feedback. Constructs were based on student fears that peer assessment is unfair, peers measure based on emotions (objectivity), giving a bad score will hurt me in later courses (retaliation safeguarding), and feedback is not developmental. As well, it was determined whether there was a belief, by end users, that peer assessment is a critical component of experiential learning. The authors surveyed 477 students within semester courses practicing experiential learning pedagogies and developed constructs for fairness (α = 0.785), objectivity (α = 0.774), retaliation safeguarding (α = 0.862), constructive feedback (α = 0.692) and experiential learning (α = 0.743). No within-factor multicollinearity was found in any construct.

1. Introduction Peer feedback is a critical component of experiential learning and is directly associated with problem based, team based, and classroom as organization based instructional methods. Peer assessment or evaluation has been afforded serious attention in several learning environments in higher education, especially with regard to classrooms that utilize experiential learning and active learning as the primary pedagogical methods (Carvalho, 2013). Topping, Smith, Swanson, and Elliott (2000) defines peer assessment as an “arrangement in which individuals consider the amount, level, value, worth, quality, or success of the products or outcomes of learning of peers of similar status” (p. 150). Peer assessment is facilitated with each individual team member rating their peer's performance, especially when learning teams are employed in the classroom (Carvalho, 2013). The scope and frequency of the peer feedback depends on the specific learning model being utilized by the instructor. There are several desired outcomes and goals associated with the peer assessment process. McGarr and Clifford (2012) assert that the goal of peer assessment is to “provide opportunities for students to assess each other's work and contribute to a community of practice where critically supportive dialogue leads to deeper student learning.” Additional researchers such as Ohland et al. (2012) state that peer assessment offers an opportunity for students to learn about expectations, accountability, and the true purpose of

∗

Corresponding author. Tel.: +1 406 994 2883; fax: +1 406 994 2013. E-mail addresses: [email protected] (M. VanSchenkhof), [email protected] (M. Houseworth), [email protected] (M. McCord), [email protected] (J. Lannin). https://doi.org/10.1016/j.ijme.2017.12.003 Received 12 August 2016; Received in revised form 26 October 2017; Accepted 4 December 2017 1472-8117/ © 2017 Published by Elsevier Ltd.

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

giving and receiving developmental feedback. Further, while the goals of peer assessment are diverse, instructors using experiential learning methods may utilize peer assessment as the reflective portion of the pedagogy and to develop team skills. These assessments not only increase accountability of teammates, but also compel engagement and effort associated with team activities (Ohland et al., 2012). Peer assessment requires self and peer contribution to instructional requirements. As such, it requires reflection of personal and peer contribution and weighting of responsibilities in a scoring and feedback format. In addition, peer assessment creates solutions for team member disengagement, increases the degree to which team members' grades reflect their actual contribution to outcomes, and urges students to be responsible for demonstration of course content (Ohland et al., 2012). 1.1. Need for solving the problem While literature and instructional experiences find peer assessment to be a critical component of experiential learning pedagogy (Michaelsen, Sweet, & Parmalee, 2009), there is, at present, no analysis of peer assessment components, or whether they are believed to impact the scoring and feedback associated with peer assessment within classroom settings. To determine how students perceive which components are critical and directly associated with experiential learning, students were asked whether peer assessment is relevant to this pedagogy. The authors discussed peer assessment with ten faculty at two Midwestern universities totaling more than 120 years utilizing experiential methods including problem-based, team-based and classroom-as-organization pedagogies. Ten elements were discussed with four critical components found comprising an effective peer assessment process. All ten experiential faculty agreed that students must feel the peer assessment is fair, objective, and contains relevant constructive feedback. Eight concurred that peer assessment should control for potential future retaliation (retaliation safeguarding). While research has documented several benefits associated with the peer assessment and evaluation process in teaching and learning, resistance and questions associated with its effectiveness exist. According to Carvalho (2013), one significant concern is associated with a student's capacity to fairly and objectively assess his or her teammates' performance. These discrepancies occur either for a lack of student expertise in assessment or for a variety of subjective predispositions including “friendship marking” in which individual students provide higher grades to friends, or “decibel marking” in which the most prevailing team members receive the highest grades (Carvalho, 2013). This type of experience in particular calls into question the objectivity, fairness, and the appropriateness of grade weight of the peer assessment and evaluation process. For this reason the authors have chosen to analyze both “Fair” and “Objectivity” as separate constructs. Another major concern with the peer assessment is related to the consequences stemming from direct social contact among teammates (Carvalho, 2013). For example, Vu and Dall’Alba (2007) note that “peer assessment may cause friction among peers, including feelings of hurt or betrayal resulting from comments or unexpected marks.” When one considers the frequency of teamoriented activities included in an experiential learning environment, and the fact that in many business school's students will most likely have teammates in many future courses, the idea of social protection or retaliation safeguarding is called into question. During the peer assessment and evaluation process, students may decline to provide feedback or a corresponding grade that is representative of their teammates' true performance in the course for fear of retaliation (Carvalho, 2013). Ideally, students and instructors require a peer assessment instrument that not only minimizes resistance associated with the process, but also has the capacity to add value through feedback to the involved individuals. According to Ohland et al. (2012), “if a student peer evaluation instrument is well-designed, any feedback that students receive from it is likely to have more value for student learning.” Ohland et al. (2012) continue by asserting that peer assessment offers an opportunity for students to learn about expectations, accountability, and the true purpose of giving and receiving developmental feedback. 1.2. Statement of the problem The authors believe that student perceptions of peer evaluation process and the instruments being utilized positively or negatively impact their experiential learning courses. Measuring the extent and strength of student perceptions is not possible at present due to lack of any scale or instrument on the subject of peer assessment. Therefore, the central problem discussed within this research study is the creation, validity and reliability of four constructs that can eventually be used to ascertain how students perceive peer assessment within their courses taught using an experiential learning pedagogy. The instrument will include the construct for experiential learning pedagogy as a control as it applies to the pedagogy that employs peer assessment. This new instrument will measure five concepts: objectivity, fairness, retaliation safeguarding, constructive feedback and experiential learning. 1.3. Research questions This study attempts to answer the following research questions: QUESTION 1: Does the instrument measure five separate parsimonious constructs? (Objectivity”, Fairness”, Experiential Learning”, “Constructive Feedback” and “Retaliation Safeguarding”). QUESTION 2: Do items selected to measure a concept actually do so? It is expected that all subject responses are based on specific observable data experienced by students in their courses. Additionally, three of the five constructs (fair, objective, experiential learning) are adequately understood and able to be defined by the students. Two construct terms (retaliation safeguarding and constructive feedback) are discussed within the instrument via proxies created based on the literature review. 93

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

2. Literature 2.1. Experiential learning This study analyzes concepts that occur based on the interactions between peers within the classroom versus understanding and/ or application a specific concept such as mathematics, strategy, or another course subject. Kolb (1984) suggests that experiential learning theory is “a holistic integrative perspective on learning that combines experience, perception, cognition, and behavior” (p. 21). Wolfe and Byrne (1975) suggest that the opportunities presented within the peer evaluation process could be those that occur outside of the specified content that is related to the course. Wolfe and Byrne (1975) and Gentry (1990) associate this with behavioral or process learning within an instructor controlled environment. Gentry (1990), Kolb (1984), and Wolfe and Byrne (1975) assert that learning, when the process is emphasized, has an opportunity to use experiences to understand, apply, and modify concepts. Within the experiential learning umbrella, the expectations of the peer evaluations are outlined to involve students with the expertise to complete them based on the extensive interactions that occur throughout the semester. Kolb (1984) discusses experiential learning as participants having to deal with conflict resolution and adapting concepts to their own views. This includes reflection in the form of peer evaluation. Our investigation is whether there are opportunities to determine the value of peer evaluation within an experiential learning situation based on specific concepts that are relevant to the students and the process. Wolfe and Byrne (1975) proposed that experiential procedures include four specific phases: Design, conduct, evaluation, and feedback. These allow the instructor to maintain control throughout the design allowing for a controlled amount of learning to occur via experiential means. These four phases have been continually supported in their scope and design and applied to different academic areas including medical, business, and hospitality (Benson, Provident, & Szucs, 2013; Graf, 2001; Greene, 2011; Mollaei & Rahnama, 2012). Procedures may include practicing specific skills, i.e. business writing, presenting, or include a range of skillsets such as interpersonal interactions. This would predominantly occur within Wolfe and Byrne's design and conduct phases via the design and maintenance of the design (1975). An additional key component of their experiential approach is for there to be an evaluation phase (Wolfe & Byrne, 1975). While course grade assessment typically occurs via the instructor, class peers are expected to be involved in determining the effectiveness of an assignment. The final component associated with Wolfe and Byrne's (1975) design is the inclusion of feedback. This may be in the format of a numerical scale but should include a response as to why a peer or peers earned a specific numerical response. In the case of peer evaluations, the evaluation and feedback phases represent application of interpersonal skills that are a major aspect of the items which make up each concept. Gentry, in his 1990 chapter discussing experiential learning, focuses on the instructor viewpoint and defines experiential learning as appropriate learning from a teacher structured experience. This includes students actively participating in practicing of concepts, ideas and theories in an active mistake environment and providing a reflection or feedback on the experience. Further, Gentry (1990) delineates the capacity of experiential methods to include practicing business related activities and/or the practice of those that are behavioral. (Wolfe and Byrne (1975) may associate these more closely with “process activities” (p. 326).) Gentry (1990) goes further suggesting that this experiential environment is one that has opportunities for variability that allows for the development and solving of conflict via interactive means within a specific peer group. We believe that this experience relates mostly toward Gentry's (1990) “live case” (p. 19) due to the large amount of variability, participation, and the requirement to work within the same peer environment for an extended time. Additional support for this is found in the amount of informal and required formal feedback that occurs over this time. Industry expectations of the skills required by today's college graduates have created a significant shift in higher education about the delivery of content in the classroom. According to the Association to Advance Collegiate Schools of Business (AACSBInternational Accreditation Standards 2016), students pursing a degree in business or general management discipline must experience learning in eight specific skill areas including “written and oral communication, ethical understanding and reasoning, analytical thinking, information technology, interpersonal relations and teamwork, diverse and multicultural work environments, reflective thinking, and application of knowledge” (p. 33). While the development of competence in each of these skill areas is vital to the overall success of the student, skills associated with “interpersonal relations and teamwork” in particular have been identified among others as “critical” by business executives (Robles, 2012, p. 463). AACSB-Association to Advance Collegiate Schools of Business International (2016) emphasizes the use of both active learning and experiential teaching methods such as Problem Based Learning (PBL). Problem Based Learning is defined as “an instructional learnercentered approach that empowers learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem” (Savery, 2015, p. 5). Within the PBL umbrella of experiential learning methods resides Team-Based Learning (TBL) and Classroom-As-Organization (CAO), as well as other models for both the teaching and learning of these skills (p. 30). From a learning experience perspective, it is expected that each of these teaching models utilize a peer evaluation or assessment process as a means of not only disseminating content knowledge, but also to support the development of industry-required skills. We discuss Team-Based Learning in this review because approximately 25% of this study's responses occur within this pedagogical method. Although Michaelsen and Sweet (2012) never clearly define experiential learning as a concept, they instead define TeamBased Learning (TBL) as an instructional strategy that includes “linking each learning activity to the next and explicitly designing assignments” (p. 9) that promote the development of teams. This supports Savery's (2015) definition of PBL due to the focus on the development of skills to solve problems within a team framework. Further, part of the foundation of TBL includes participants being held accountable for a team's work, that there is appropriate feedback, and assignments are appropriate for team interaction and 94

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

development. These concepts marry with Gentry's (1990) definition for experiential learning and TBL utilizes a model that incorporates Wolfe and Byrne's (1975) phases (Michaelsen & Sweet, 2012). 2.2. Fairness Fairness, as a concept, is often discussed but rarely measured throughout academic literature. It is often associated with justice (Colquitt, 2001; Gross, 2008; Moorman, 1991) and trust (Butler, 1991). Gross (2008) asserts that how decisions are made, procedural justice, combines the perceptions of an ability to be involved alongside the belief the process must be fair. Colquitt (2001) breaks procedural justice further into perceptions of interactional justice, focusing on the interpersonal relationships between parties and procedural justice, focusing on the method or process associated with the assessment. Moorman (1991) centers questions surrounding procedural and distributive justice as it applies to fairness within organizational citizenship behavior. Butler (1991) stresses that fairness is directly associated with trust and found fairness discernible within his analysis of trust within organizations. Procedural justice was indicated by Colquitt (2001) as dependent upon individuals providing accurate assessments and understanding the points of view of their peers. Additionally, it was thought by the authors that although the peer assessment occurs at a distinct point in time, individuals would be given opportunities to provide viewpoints throughout the assessed period. This would then add to the procedural justice component. Moorman's (1991) discussion of potential causal relationships between procedural and distributive justice add credence that fairness may be tied to how the peer assessment method is carried out and the interactional perceptions of the parties involved. Interactional justice is discerned based on experiential learning pedagogy. Magin (2001) and Montgomery (1986) state collusion, inter-member cohesiveness, and gender bias were not significantly present in groups or teams that worked together for a considerable period. The common use of team-based applications, peer evaluation expectations, and the student centric mode of this research requires student interaction through interpersonal means. Hence interactional justice is possible within the peer review process. This paper's authors selected a similar interactional and procedural justice and trust lens in the development of a scale to measure the construct of fairness within a peer evaluation process. The fairness questions within the instrument centered on personal and interactional associations between the team peer assessments. Byrne and Miller (2009) analyzed justice and fairness within the PROFILOR® instrument. Within the 196-item instrument, ten subject matter experts found that a fairness dimension was supported based on questions developed by Colquitt (2001) and Moorman (1991). Byrne and Miller (2009) found variability in measuring for fairness based on participant self-reporting. This was believed to be created by the differing perspectives based on individual circumstance versus a consistency across a procedure, organization, or like-employee population. Byrne and Miller's (2009) study in particular, analyzed fairness and justice across peers, reports, and superiors. This paper's authors contend that this variability is less likely within peer-reviewed teams due to uniformity across a samelevel peer group. As such, questions developed by Colquitt (2001) and Moorman (1991) were utilized as the fairness portion of the instrument. The fairness construct items are available in Appendix A. 2.3. Objectivity Objectivity within peer assessment appears to be primarily based upon three factors, two of which are controllable by the instructor and one that is possible within only larger teams. Both anonymity and clear methodology for completing the peer assessment were considered directly relevant to objectivity within peer assessment (Brindley & Scoffield, 1998; Carvalho, 2013; Chen, 2010; Papinczak, Young, & Groves, 2007; and Patri, 2002). Additionally, Brindley and Scoffield (1998), Chen (2010), Papinczak et al. (2007), and Patri (2002) discuss the impact of students' beliefs that they are personally competent to deliver feedback influences their perception that peer assessment may not be objective. The format of the peer evaluation and whether rules and what role the assessors played appeared to be the most important part of the objectivity discussion. A key function of objectivity was found by Brindley and Scoffield (1998) to be addressed when students were able to develop or have some leverage over the assessment criteria. This “ownership” enhanced the learning outcomes associated with peer assessment and increased objectivity due to the process being participative. Chen (2010), too, discusses the impact that students may have as “co-deciders” of assessment criteria. He states that less than 9% of study participants doubted the objectivity of the assessment that included this participative process. This is the first numerical designation of potential objectivity belief within peer assessment and must be tempered with the fact that Chen's study focused on the use of online peer assessment methods. Papinczak et al. (2007) and Carvalho's (2013) studies specifically discuss the dangers of friendship marking and dishonesty to a peer assessment instrument's objectivity. Both discuss this in terms of being a belief going into the pedagogical arena and at the start of courses requiring peer assessment. Neither study discusses having the same issue at its conclusion. Hence, This paper's authors believe that this objectivity issue decreases throughout the session or is primarily a fear at the beginning of the process. Similar fears were discussed by Patri (2002) but the fears were related to self-doubt versus doubt about peers. Her study focused more on the validity of peer assessment from a self-analysis as well as peer review. It found self-assessment results often did not mirror peer assessment results. But, again, the belief that peers would not be objective is not discussed beyond being a fear at the beginning of the course. This paper's authors are in agreement with the literature. They believe that clear instructions and peer assessment methodology are critically important to developing objectivity within peer assessment (Brindley & Scoffield, 1998; Carvalho, 2013; Chen, 2010; Papinczak et al., 2007; Patri, 2002). Development of the instrument with the instructor and peers increases the likelihood that 95

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

methodology is understood. Chen (2010) states: “Explicitly explaining and defining assessment criteria should be a top priority for teachers to conduct meaningful self or peer assessment.” Brindley and Scoffield (1998) discuss controlling for student perceptions positively impacts objectivity: “The main pressure appears to stem from the assessment process as a whole and from their peers.” The literature discussed lack of anonymity as a threat to objectivity and the researchers agree with this belief. This paper's researchers concede that anonymity is relatively not possible within the pedagogical models used throughout the surveyed classes, i.e. within teams of seven or fewer peers. This paper's authors' have attempted anonymity through multiple methods from standardized forms to online peer assessment to the instructor creating documents. In all cases students were able to, based on conversations, determine which comments and scores came from each peer, mostly through the process of elimination. Even with this evidence, the authors believe that objectivity is possible and that anonymity may play a lesser role than previously thought. The concept of objectivity is believed to be important based on discussions within the literature. However, there appears to be no test for determining whether a peer assessment has been developed to control for objectivity. Additionally, literature and the authors believe the ability of students to assess peers objectively is of prime significance within the work-world. Papinczak et al. (2007) state: “A key aspect of this [peer assessment] is developing the ability to judge objectively the quality of their own and others' work.” Five items were developed to measure the construct ‘Objectivity’. Those items are available in Appendix A.

2.4. Retaliation safeguarding While a study from Meyers, Ohland, Silliman, McWilliams, and Kijewski-Correa (2008) addressed the measurement of constructive feedback in peer assessment, a clear definition of the concept of retaliation safeguarding; the likelihood a peer believes their assessment of another will negatively harm themselves in the future, is not explored in today's relevant literature. In fact, when reviewing literature for information associated with retaliation safeguarding or social protection, only research in criminology discussed the concept, and no instrument for its measurement was available. The authors consider a key issue in peer-scored feedback is whether the peers completing the evaluation believe critical or negative scores will harm their scores in future feedback iterations or classes. Hence, retaliation safeguarding is an important component of a peer assessment instrument. Through this study, the authors attempt to enhance this concept to create a retaliation safeguarding construct within the peer assessment. Multiple papers centered on “retaliation” and “social justice” were reviewed. Literature surrounding this principle is mostly qualitative in nature. Three principles were thought to be relevant and were investigated to determine whether the concept of retaliation safeguarding could be measured: culture of and relationships within the team (Davis & Henderson, 2003; Jacobs, 2004); qualifications of their peer set (Wu, Davison, & Sheehan, 2012); and perception of being singled out (DeRidder, Schruijer, & Rijsman, 1999). Defensive silence, consciously withholding information or actions as a form of self-protection, was investigated but found to not pertain to this study. Perkins (2014) asserts that methods to control for defensive silence include openness within a team to specifically include communication. Dedahanov, Kim, and Rhee (2015) and Rhee, Dedahanov, and Lee (2014) maintain that adequate communication minimizes inter-member threats. Students completing the instrument were in teams for an entire semester hence, there communication and openness opportunities were abundant. Additionally, mutuality, the level of engagement between peers, is reduced via longer-term team interactions. Members have had the opportunity to engage in many interactions that allow each member of the team to act from a position of power (Clark & Dumas, 2015). The culture within the team and the culture supported by the instructor is thought, by the authors, to impact the likelihood of retaliation safeguarding. Jacobs (2004) asserts that determining whether the culture encompasses mutual deterrence or mutual escalation would explain the likelihood of future retaliation. Additionally, Jacobs writes that informal social control assists with maintaining and regulating deterrence and maximizing social order. Magin (2001) also found that social control minimizes collusion as it pertains to inflating peer assessment scores. Davis and Henderson (2003) in their study found that smaller ethnic communities wield local power to solve community issues. This paper's authors expect a like-minded culture to be present within the team-based pedagogy as well. To test informal social control, two questions were asked: “I believe the culture within my team to be one that operates in a transparent manner” and “I believe the culture within my team to be one in which we value and support each other.” Wu et al. (2012), in assessing pharmacy student perceptions toward peer assessment, found that students' feeling they were qualified to provide quality scored feedback was an important determinant. The question “I am qualified to provide scored feedback for my peers for this course” was added to determine value. Additionally, the authors investigated whether peers' beliefs that their teammates were qualified provided more support to the retaliation safeguarding concept. The question “My peers are qualified to provide scored feedback for this course” was asked. The authors wanted to determine whether peers believed they were being singled out in the peer review process and if it would impact retaliation safeguarding perceptions. DeRidder et al. (1999) investigated personal attacks and maintain the victim needs to have some belief that they alone were targeted versus the team. DeRidder et al. assert the likelihood of retaliation increases when the victim believes they were singled out and treated harshly or improperly. To determine whether this would support the retaliation safeguarding concept the question “A peer seriously harmed my final course grade based on their peer evaluation score” was included in the instrument. Additionally, “I provided appropriate feedback scores across the entire team” was included in order to determine whether retaliation safeguarding is a perception of ones-self or whether it resides as a peer or group mentality. Retaliation safeguarding is a concept in which the authors could not find an available and valid instrument. The authors will use the seven questions to determine whether a retaliation safeguarding concept can be measured based on suppositions generated from available criminal justice qualitative literature. These seven items are available in Appendix A. 96

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

2.5. Constructive feedback Standard thinking about peer assessment techniques tends to focus on student apprehension to scoring their peers. Throughout the literature, the discussion of the feedback aspect of assessment has not been discussed as much as peer scoring. Investigating the instrument and procedures associated with feedback, absent the scoring piece, is relevant and important to the experiential learning pedagogy. McGarr and Clifford (2012) discuss this aspect thoroughly and a dichotomy exists associated with providing feedback. They find students believe providing feedback is an important skill yet those students that are low performers or unfamiliar with the technique resist giving it feedback. Instead they prefer instructor scored assessment. Much of the peer feedback literature supports inclusion of a peer assessment process within a pedagogy as a method for students to manage their own learning in an active manner (Dolmans & Gijbels, 2013; Liu & Carless, 2006; McGarr & Clifford, 2012). Liu and Carless (2006) believe that peer feedback is more important than the assessment aspect of the peer review process. Their 2006 article also observed, on average, a higher percentage of academics resist peer assessment than students. There is an inherent belief by faculty that peer assessment is the same thing as peer scoring. Interestingly, academics are expected to be experts at assessment but that aspect does not fall readily to peer interactions within a shared work circumstance. Liu and Carless (2006) assert that reliable relationship assessment is difficult to achieve by faculty unless they are directly associated with all aspects of team dynamics. Understanding work dynamics is not realistic for instructors that have multiple teams or groups within a class. Dolmans and Gijbels (2013) discuss only homogeneous groups within a problem-based pedagogy. They found that peer feedback supported student openness within their teams and they believed that was associated with better learning. The authors argue that within the team-based, problem-based and classroom-as-organization pedagogy this occurs as much or more within diverse groups and teams. Furthermore, the challenge of initial team formation based on heterogeneous backgrounds and understanding peer review responsibilities, as a standard, develops more open dialogue earlier in the session. McGarr and Clifford (2012) contend that pedagogical expectations along with future work-related skills are highly associated with student acceptance of peer review and assessment. They state that student exposure to formative assessment feedback is important to understanding expectations and engaging within the review process. This leads to questions of whether much of peer review and assessment transpire utilizing appropriate methods for formative feedback to happen. McGarr and Clifford (2012) similarly discuss that the more academically mature students (Junior and above), are less likely to have a negative view of peer review. The authors concur with that sentiment. Further, they believe the culture of peer review within courses and the department increase student acceptance of the use of peer feedback. The three items used to measure ‘constructive feedback’ are available in Appendix A. 3. Methodology The overarching goal of this stream of research is to understand student perceptions of peer assessment within experiential learning pedagogy and their sensitivities toward the concepts of constructive feedback, fairness, objectivity, and retaliation safeguarding as they apply to peer evaluations. The first step was to explore the literature and determine the key concepts. These key concepts were explored and disentangled, leading to individual instruments that operationalized each concept. Finally, the concepts of constructive feedback, fairness, objectivity, retaliation safeguarding and experiential learning are tested for construct validity. The authors intend to present parsimonious instruments for each concept. The purpose of the proposed instrument is to provide additional insight pertinent to the peer assessment process in order to develop universal peer-assessment instruments in the future. The research was approved by the university's Institutional Review Board. Ten instructors with sixteen different undergraduate class populations allowed students the opportunity to complete a four-page paper survey. All instructors utilized experiential learning pedagogy and administered a mid and end of semester peer assessment valued at ≥ 20% of the total course grade. All instructors were within the Department of Management at a Midwestern public university and taught in the areas of management, entrepreneurship and human resources. Class sizes ranged from 16 to 75 students and all but one were held on the main campus. The majority of the students were studying management, entrepreneurship, and hospitality. There were 441 complete surveys submitted (n = 441). The data collection instrument was quantitative, and consisted of a 32 question survey. The survey instrument includes eight demographic questions, 23 construct questions utilizing a Likert 7-point scale measuring perceptions of a peer evaluation process, and one short-answer question asking participants to define experiential learning. The instrument questions are shown in Appendix A. All surveys were entered into SurveyMonkey (SurveyMonkey Inc., 2016) for ease of data transcription and management. Descriptive statistics were determined utilizing Statistical Package for the Social Sciences (SPSS) IBM version 22 (IBM Corp., 2013) statistical software for the quantitative questions associated with the study. Content analysis was performed on the individual definitions for experiential learning utilizing SurveyMonkey text analysis of the frequency of specific phrases and words. 3.1. Limitations Limitations applied to the study. The authors acknowledge that with the construction of five new concepts studied that levels of cross-loading will occur. Multidimensional questions across constructs were expected, but that was not the central purpose or focus of this paper. Instead, the focus is the development of five parsimonious constructs. Beyond that issue, other limitations exist. All participants were students in the same institution and predominantly from the same college. Each class participating in the study was autonomous and independent of each other, so the results of the survey may vary. The subject matter within each participating course was different, so the results of the survey were dependent on experiential 97

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

learning pedagogy and the peer assessment process rather than on specific subject matter. This study utilized convenience sampling. The sample was a group of students exposed to the experiential learning pedagogy within the college. As such, the results are not considered generalizable across institutions. 3.2. Instrument development The 32 question instrument was designed based on previous research on each individual concept. Deductive scale development occurred based on a review of previous literature for each concept. A total of 48 questions were created based on the literature review. Due to a lack of existing theory or known instruments about three of the concepts (objectivity, retaliation safeguarding, and constructive feedback) potential questions for each concept were reviewed by five experiential learning professionals for face and content validity. Based on the experts' feedback the instrument was reduced to 24 questions: Five questions measured objectivity; three measured fairness, seven measured retaliation safeguarding, three measured constructive feedback, and six measured experiential learning as it pertains to peer assessment. The final questions utilized are available in appendix A. 4. Results This study tested the validity and reliability of five instruments that measured student perceptions of peer assessment objectivity, fairness, retaliation safeguarding, constructive feedback, and experiential learning. The authors developed four concepts of experiential learning; constructive feedback, objectivity, and retaliation safeguarding for testing and utilized Byrne and Miller's (2009) instrument to test for fairness. Factor analysis was used to reduce each concept's original items to parsimonious concepts. Additionally, factor analysis was also used to determine the validity of each concept, and Cronbach's alpha and Tukey's HSD were used as a measure of concept reliability. To address within-concept validity, each concept was tested using the full population of 447 observations. Confirmatory factor analysis for each concept indicated that not all were an acceptable fit to the data. Before addressing the research questions, each scale was re-specified so that it better represented the data. Appendix A shows the original scale items with each concept and item specified. Five concepts were expected: “Objectivity,” “Fairness,” “Retaliation Safeguarding,” “Experiential Learning,” and “Constructive Feedback.” Each concept's factor analysis, with item number, item text, and component significance are provided below. The authors chose a very high level of reliability, > = 0.70, for retaining items in the individual concepts. Question 1: Does the instrument measure five separate parsimonious constructs? (Objectivity”, Fairness”, Experiential Learning”, “Constructive Feedback” and “Retaliation Safeguarding”) (within-concept validity). – SUPPORTED. Table 1 illustrates the Principle Component Analysis for “Fairness.” As expected, the previously validated construct for “Fair” loaded on one factor, with all item significance levels over 0.70. This concept was developed based on a previous instrument and was expected to respond in this way. One item was worded as a negative (“The peer evaluation was subjective.”), so the data scores were reversed, creating a similar and relatable indicator versus the opposite. Two items were below the author's 0.70 level of significance, including the negatively worded item and its mirror (“The peer evaluation was objective.”). Studies show that negatively worded items can create response problems. Many times both positive and negative items are included for one component in the attempt to remove acquiescence bias or response bias. The problem with negatively worded questions is confusion for the respondent on reversing the scale, as shown by inconsistent responses. In a study of this issue, positively worded items were inconsistent 2.59% of the time, versus 8% for negatively worded items, and the difference between positively worded vs. negatively worded questions produced significantly different results (Colosi, 2005). Principle Component Analysis for “Objectivity” is displayed on Table 2. The concept was refactored using only three factors. Upon analysis it was found only two factors loaded above 0.70. The final item for “Objectivity” was loaded with these two factors. Two-item measures are of concern when discussing reliability of the instrument (Hinkin, 1995). The authors are aware of this and will address the inclusion of a two-item instrument in future models. Retaliation Safeguarding was over-defined, with 8 items. There were no previous quantitative studies indicating a measurement instrument. Hence, the authors measured the concept from two viewpoints of culture and qualification. We see from Table 3 that “Retaliation Safeguarding” has two factors. Items that are below 0.70 for Factor 1 are rejected as part of the concept “Retaliation Safeguarding”. Without items these three items, “Retaliation Safeguarding” becomes a one factor concept. The concept “Constructive Feedback” yielded one factor. All items had significance scores above 0.70. Table 4 illustrates this Table 1 Principle component analysis for “fairness”. Fairness

Final Factor

The peer evaluation justly measured my actual class performance I was able to express my views and feelings during the peer evaluation process The peer evaluation procedures were based on accurate information Principle Component Analysis with Varimax with Kaiser Normalization

0.870 0.781 0.862

98

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Table 2 Principle component analysis for “objectivity”. Objectivity

Initial Factor

Factor 2

Final Factor

The peer evaluation accurately graded my peers' performance Instructions for correctly completing the peer evaluation were clear The peer evaluation accurately graded my performance The peer evaluation was subjective The peer evaluation was objective Principle Component Analysis with Varimax with Kaiser Normalization

0.841 0.709 0.832 0.109 0.561

0.739 0.536 0.730

0.904 0.904

Table 3 Principle component analysis for “retaliation safeguarding”. Retaliation Safeguarding

Factor 1

Factor 2

Final Factor

My peers provided appropriate feedback scores across the entire team I believe the culture within my team to be one that operates in a transparent manner I am qualified to provide scored feedback for my peers for this course I believe the culture within my team to be one in which we value and support each other My peers are qualified to provide scored feedback for this course I provided appropriate feedback scores across the entire team A peer seriously harmed my final course grade based on their peer evaluation score I am able to determine which of my peers gave each score and feedback Principle Component Analysis with Varimax with Kaiser Normalization

0.801 0.812 0.737 0.806 0.802 0.560 −0.218 0.284

−0.062 0.024 0.077 −0.015 −0.051 0.151 0.803 0.751

0.812 0.831 0.726 0.827 0.814

Table 4 Principle component analysis for “constructive feedback”. Constructive Feedback

Final Factor

I am qualified to provide practical feedback utilizing the peer evaluation The peer assessment process provided practical feedback to me My peers are qualified to provide practical feedback utilizing the peer evaluation Principle Component Analysis with Varimax with Kaiser Normalization

0.778 0.774 0.834

factor. “Experiential Learning” is illustrated in Table 5. Our final concept had 5 items and two factors. One item was a negatively worded question, so the authors reversed scoring before analyzing the data. The item that was reversed, “The grade weight associated with peer assessment does not add value to experiential learning.” is below the 0.70 threshold and will be eliminated. As discussed above, negatively worded questions can give significantly different results. A final, one factor analysis of “Experiential Learning” is in Table 5. QUESTION 2: Do items selected to measure a concept actually do so? (within-concept reliability) – SUPPORTED. Hinkin (1995) discusses methods for determining content validity. For research in which instruments are being developed, he suggests multiple stages that support a final determination for content validity. The stages include scale development based on literature review; questions analyzed by content experts; upon a completed survey, alphas of an acceptable level; and construct itemtotal correlations for internal consistency. We assessed scale reliability with coefficient alpha (Hatcher, 1994, p. 140). If the scales being used were shown to be unreliable we would know to abandon analysis of concepts as they relate to our research questions. Tukey's test for nonadditivity was determined for each of the parsimonious constructs. Significant interactions between the measures of each construct were found. Tukey's test for additivity does not explain what the effect sizes are, nor does it explain between what construct questions contain significant interactions. To test for collinearity between factors within each construct

Table 5 Principle component analysis for “experiential learning”. Experiential Learning

Factor 1

Factor 2

Final Factor

The peer evaluation's impact on my final grade is comparable to the other grades in this course Evaluating my peers allows me to practice experiential learning in this course the instructor used a high level of experiential learning in this course The grade weight associated with peer assessment is important when using experiential learning The grade weight associated with peer assessment does not add value to experiential learning Principle Component Analysis with Varimax with Kaiser Normalization

0.701 0.831 0.739 0.742 0.125

−0.077 −0.143 −0.072 0.140 0.979

0.703 0.835 0.741 0.737

99

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Table 6 Differences between constructs and instructor. ANOVA Sum of Squares

df

Mean Square

F

Sig.

Fairness

Between Groups Within Groups Total

60.726 527.763 588.489

9 430 439

6.747 1.227

5.497

0.000

Objectivity

Between Groups Within Groups Total

42.150 459.558 501.708

9 430 439

4.683 1.069

4.382

0.000

Retaliation Safeguarding

Between Groups Within Groups Total

26.080 360.679 386.759

9 427 436

2.898 0.845

3.431

0.000

Constructive Feedback

Between Groups Within Groups Total

25.368 333.167 358.535

9 431 440

2.819 0.773

3.646

0.000

Experiential Learning

Between Groups Within Groups Total

49.679 337.262 386.941

9 429 438

5.520 0.786

7.021

.000

correlations were examined as well as Variance Inflation Factors (VIF) and its inverse, tolerance. O'Brien (2007) avows, when discussing VIFs, that when tolerance is greater than 0.10 and VIF is below 10 that there are strong indicators of no multicollinearity. Furthermore, if there are no correlations greater than 0.90 then there is no signal of multicollinearity. This paper's authors determined whether statistically significant interactions were present between a specific instructor and each of the constructs. An analysis of variance was the test to determine whether differences occurred between the two groups. Table 6 illustrates that significant differences (p = < 0.00) were found for each of the constructs developed in this research. This same dependent variable of instructor was utilized in determining correlations, tolerance and VIF for each of the constructs. This is shown in Table 7. The factor Objectivity was determined to be comprised of two factors. The greatest correlation is 0.102 hence there is no signal of multicollinearity. Moreover the tolerance factors are greater than 0.10 and VIF is not greater than ten. Even

Table 7 Multicollinearity within Factors for each Construct when Interacting with Instructor. Model

Correlations

Objectivity The peer evaluation accurately graded my peer's (team members')performance The peer evaluation accurately graded my performance Fairness The peer evaluation justly measured my actual class performance I was able to express my views and feelings during the peer evaluation process The peer evaluation procedures were based on accurate information Constructive Feedback I am qualified To provide practical feedback utilized the peer evaluation The peer assessment process provided practical feedback to me My peers are qualified To provide practical feedback utilized the peer evaluation Retaliation My peers provided appropriate feedback scores across the entire team I believe the culture within my team to be one that operation in a transparent manner I am qualified to provide scored feedback for my peers for this course I believe the culture within my team to be one in which we value and support each other My peers are qualified to provide scored feedback for this course Experiential Learning The peer evaluation's impact on my final grade is comparable to the other grades in this course Evaluating my peers allows me to practice experiential learning in this course Based on your definition of experiential learning, the instructor used a high level of experiential learning, the instructor used a high level of experiential learning in this course The grade weight associated with peer assessment is important when using experiential learning

100

Collinearity Statistics

Zero-Order

Psrtial

0.102 0.054

0.088 −0.014

0.088 0.070 0.094

Part

Tolerance

VIF

0.088 −0.014

0.599 0.599

1.668 1.668

0.028 0.019 0.043

0.028 0.018 0.043

0.522 0.698 0.535

1.915 1.432 1.870

0.038

0.002

0.002

0.736

1.359

0.046 0.069

0.014 0.049

0.014 0.049

0.741 0.662

1.350 1.511

0.044 0.090 0.027 −0.004 0.061

−0.006 0.116 −0.016 −0.103 0.045

−0.006 0.116 −0.016 −0.102 0.045

−0.494 0.408 0.546 0.432 0.447

2.026 2.450 1.833 2.316 2.239

0.054 0.006 0.057

0.068 −0.031 0.076

0.068 −0.031 0.075

0.731 0.571 0.688

1.396 1.752 1.453

−0.040

−0.075

−0.075

0.725

1.379

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Table 8 Cronbach Coefficient Alphas & Tukey's Test for Additivity for each Construct.

Objectivity Fairness Retaliation Safeguarding Constructive Feedback Experiential Learning

Cronbach Coefficient α

N

Tukey's Test for Additivity

0.774 0.785 0.862 0.692 0.743

443 440 437 441 439

3.39 5.207 6.695 6.248 2.717

though Tukey's test for nonadditivity found significant interactions between the factors comprising the construct Objectivity we determined that multicollinearity is not present. The authors measured within-concept validity using the full sample population. All 447 observations were used, but with missing records the N for each concept varied from 437 to 443. Coefficient alpha reliability estimates (Cronbach, 1951) all exceeded Nunnally's (1978) suggested cut-off of 0.70, and are reported in Tables 1 through 5. Additionally, it was determined there was no multicollinearity within each parsimonious construct. Therefore, within-concept validity is supported for “Objectivity,” “Fairness,” “Retaliation Safeguarding,” and “Experiential Learning.” “Constructive Feedback” falls just below the 0.70 threshold for Nunnally's (1978) recommended cut-off yet the authors believe the construct is justifiably measured. Table 8 contains the Cronbach Coefficient Alphas for each of the five parsimonious constructs discussed within this research.

5. Discussion Based on the authors experience with peer assessments in experiential learning pedagogy, a priori decisions were carefully made. Five constructs were selected to be used when measuring peer assessments; retaliation safeguarding, fair, objective, constructive feedback and experiential learning. One construct, ‘fair’ had an existing validated scale. Using past (published) research from experiential learning and other disciplines, the authors wrote multiple questionnaire items to assess ‘objective,’ ‘retaliation safeguarding,’ ‘experiential learning,’ and ‘constructive feedback’. These constructs, or concepts like them, are also heavily mentioned in the literature. Only one concept, Fairness, had been previously operationalized. The three items comprising the concept of Fairness were adopted from Colquitt (2001) and Moorman (1991). This concept was previously validated in Byrne and Miller's study (2009). It was not surprising that no items dropped out, and all items were significant well above the 0.70 cutoff. For the other constructs: objectivity, retaliation safeguarding and constructive feedback, this paper's authors created new items based on literature review, authors' experiences, and intensive discussion. Constructive Feedback was the only other concept to yield one factor with all items above the 0.70 significance on the first pass. With only three items, Constructive Feedback was not over defined, the usual practice when developing new instruments. The items had such high face validity the authors believed additional items would not contribute much more variance to the overall concept. According to the literature review, students involved in peer evaluations know how to measure feedback given and received. Our experience with the three items of the Constructive Feedback concept supported this belief. Items were dropped in analysis of concepts Objectivity, Retaliation Safeguarding, and Experiential Learning. In some instances, items dropped did not contribute to the meaning of the construct measured. In other cases the items dropped due to questionnaire development issues, such as respondent confusion. Examples of questionnaire and item development problems were found in Objectivity and Experiential Learning. The two items that dropped from Objectivity probably had more to do with questionnaire development than the meaning of the items. These two items were identical except for the flipped meaning of the words “objective” and “subjective” (The peer evaluation was subjective/… was objective). As mentioned in the results of Objectivity, negatively worded items are confusing for respondents and can lead to inconsistent responses. The concept Experiential Learning had a similar problem with a negatively worded item. The reverse question “The grade weight associated with peer assessment does not add value to experiential learning” dropped out of this concept. Perhaps its negative wording confused respondents. Another explanation is that its opposite item “The grade weight associated with peer assessment is important when using experiential learning” added more variance to the concept than its redundant and negative worded doppelganger. Only retaliation safeguarding had items dropped because they didn't actually measure the concept's meaning. The literature on retaliation safeguarding suggested elements of 1) Culture of and relationships within the team (Davis & Henderson, 2003; Jacobs, 2004); 2) Qualifications of their peer set (Wu et al., 2012), and 3) Perception of being singled out (DeRidder et al., 1999). We see from Table 3 that culture is a strong underpinning of retaliation safeguarding and remained part of the construct while 2 items measuring students' perceptions of their peers' qualifications and 1 item measuring the fear of being singled out dropped out. The dropped question that was designed to measure ‘perception of being singled out’ was “I am able to determine which of my peers gave each score and feedback.” In this case, the direction of the perception was reversed – rather than fear of being singled out or identified, the student was asked if they could single out and identify others. In hindsight, this item doesn't measure the issue correctly. So, retaliation safeguarding is measuring attitudes concerning culture and the fear of being singled out. 101

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Factor analysis of each concept eliminated items that were constructed poorly and items that didn't actually measure the concepts meaning. Our reduction of the items left focused indicators of concepts using a few meaningful questions. The pedagogies, classroom-as-organization, team-based and problem-based learning; utilize peer evaluation as a central component of experiential learning. As a singular component peer evaluation has primarily been studied in qualitative format to determine whether either the process or the evaluation provides support to the pedagogy. This paper's authors believe that four new constructs can be measured that determine whether a peer assessment instrument is considered fair, objective, provides for retaliation safeguarding and constructive for the end user. Further, that peer assessment directly and significantly supports experiential learning pedagogies. There were some expectations that due to intercorrelations the scale as a whole would not have adequate construct validity. There are possible explanations for these results. First, the concepts within the instrument could be broadly defined and overlapping, making them difficult to operationalize. Second, items within the new instrument may not represent their underlying concepts. Finally, our population of students engaged in peer evaluation during experiential learning courses may not have provided an adequate test sample. Since the sample did survey a large number (n = 477) of students enrolled in experiential learning courses and directly involved in peer assessment, an inadequate test sample or non-representative population is not the likely culprit. Factor analysis was used to reduce the items underlying each construct, and statistical tests for fit and significance showed items are attributed to separate construct. Therefore, the problem with validity was not because underlying items do not fit the constructs measured. The most likely explanation for the high level of intercorrelation is that the concepts were overlapping. As the authors developed these constructs, some items were found to be redundant. An example was found in the ‘Experiential Learning’ items “the grade weight associated with peer assessment is important when using experiential learning” and its reverse “The grade weight associated with peer assessment does not add value to experiential learning”. It makes sense that when the negative item was reverse scored, it basically measured the same thing and was conceptually so similar that one of the items didn't add unique variance. Our reduction of the items left focused indicators of a concept using a few meaningful questions. Although the instrument as whole has not been tested using exploratory factor analysis, intercorrelation implies that many items will measure more than one concept. For example, the concept ‘Fair’ and ‘Objective’ may show constructs with different names but similar meanings. If the authors do not separate the concepts clearly in future models, the indiscrete constructs will be difficult to interpret and theoretically meaningless. Because we know there is intercorrelation, we need to find the latent constructs underlying the concepts defined in this paper. Because we wish to find the nature and number of underlying factors that are responsible for covariation, EFA was appropriate. In summary, a principal component analysis shows that these components did emerge, but there was inter-correlation between components. This paper was not intended to present a model with a final instrument to measure that model, but to suggest separate concepts that might be used in a future model. 5.1. Future research In future research the authors will create a model of noncorrelated constructs using structural equation modeling. The authors will assess the instrument with confirmatory factor analysis (CFA). The current population will be used to develop methods moving forward based on an exploratory factor analysis. The current population represents a sample at least 5 times the number of times being tested, or at least 150 subjects, whichever is largest (Hatcher, 1994). The original PPA survey is of 24 items, therefore the lower bound of exploratory analysis is 120 responses. To be more robust, the authors plan to have a sample size at least 10 times the number of items being tested, or 240 responses. A random sample of 240 observations will be used for model development, and the remaining 180 + observations used for model confirmation. We will evaluate model fit using Chi-Square, Probability of Chi-Square, and Tucker and Lewis's Reliability Coefficient. Items will not be allowed to load on more than one construct, nor their error terms allowed to correlate. Further, the authors will seek best practices among the peer evaluation processes used within the experiential learning courses. Determining specific functions based on end users input that are best practices toward the new constructs may help create a benchmark for peer assessment practice and procedure. 6. Conclusion The Perception of Peer Assessment (PPA) concepts were designed specifically for experiential learning students, and grounded in previous studies of student assessments. Before conceptual testing of research questions using the PPA item questions occurred, we first determined the degree to which chosen items (e.g. “The peer evaluation instrument was objective”) measured the construct they were supposed to measure (e.g. objectivity). Each concept was tested for ‘construct validity’ to measure the strength of within-construct (Hoyle & Smith, 1994) item relationships. There are a number of pedagogical and conceptual reasons why such validity is important. First, when an instrument says that it measures something such as objectivity vs. fairness, academics and teachers using that measure will assume that it measures what it says it measures. If the construct is not truly valid, that assumption is not justified. Secondly, authors often study the impact of various pedagogy methods and report scores as a way to compare between them (e.g. experiential learning vs. other teaching methods). These comparisons are then used to suggest prescriptive practices for teaching and learning. However, if the scale does not have adequate concept validity, the author's findings would not be valid. 102

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Finally, the validity of PPA's outcome measures is dependent on the processes linking experiential learning pedagogies to student assessment outcomes. Presently there are no instruments that can be used to test the importance of peer assessment evaluations. This study is the first to introduce using these concepts together for this purpose. Appendix A. Construct research questions Fairness: The peer evaluation justly measured my actual class performance. I was able to express my views and feelings during the peer evaluation process. The peer evaluation procedures were based on accurate information. Objectivity: The peer evaluation was subjective. The peer evaluation accurately graded my peers' (team members') performance. The instructions for correctly completing the peer evaluation was clear. The peer evaluation was objective. The peer evaluation accurately graded my performance. Constructive Feedback: I am qualified to provide practical feedback utilizing the peer evaluation. The peer assessment process provided practical feedback to me. My peers are qualified to provide practical feedback utilizing the peer evaluation. Retaliation Safeguarding: I provided appropriate feedback scores across the entire team. My peers provided appropriate feedback scores across the entire team. I believe the culture within my team to be one that operates in a transparent manner. I am qualified to provide scored feedback for my peers for this course. I believe the culture within my team to be one in which we value and support each other. My peers are qualified to provide scored feedback for this course. A peer seriously harmed my final course grade based on their peer evaluation score. I am able to determine which of my peers gave each score and feedback. Experiential Learning: The peer evaluation's impact on my final grade is comparable to the other grades in this course. Define experiential learning in your own words. (Fill in the blank – this question only). Evaluating my peers allows me to practice experiential learning in this course. Based on your definition of experiential learning, the instructor used a high level of experiential learning in this course. The grade weight associated with peer assessment is important when using experiential learning. The grade weight associated with peer assessment does not add value to experiential learning. References Association to Advance Collegiate Schools of Business International (2016). Eligibility procedures and accreditation standards for business accreditation. Tampa, FL: Author. Benson, J. D., Provident, I., & Szucs, K. A. (2013). An experiential learning lab embedded in a didactic course: Outcomes from a pediatric intervention course. Occupational Therapy in Health Care, 27(1), 46–57. Brindley, C., & Scoffield, S. (1998). Peer assessment in undergraduate programmes. Teaching in Higher Education, 3(1), 79–90. Butler, J. K. (1991). Toward understanding and measuring conditions of trust: Evolution of conditions of trust inventory. Journal of Management, 17(3), 643–663. Byrne, Z. S., & Miller, B. K. (2009). Is justice the same for everyone? Examining fairness items using multiple-group analysis. Journal of Business and Psychology, 24, 51–64. http://dx.doi.org/10.1007/s10869-009-9091-7. Carvalho, A. (2013). Students' perceptions of fairness in peer assessment: Evidence from a problem-based learning course. Teaching in Higher Education, 18(5), 491–505. Chen, C. H. (2010). The implementation and evaluation of a mobile self-and peer-assessment system. Computers & Education, 55(1), 229–236. Clark, I., & Dumas, G. (2015). Toward a neural basis for peer-interaction: What makes peer-learning tick? Frontiers in Psychology, 6(28), 1–12. http://dx.doi.org/10. 3389/fpsyg.2015.00028. Colosi, R. (2005). Negatively worded questions cause respondent confusion. Proceedings of the survey research methods section (pp. 2896–2903). American Statistical Association. Colquitt, J. A. (2001). On the dimensionality of organizational justice: A construct validation of a measure. Journal of Applied Psychology, 86(3), 386. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. Davis, R. C., & Henderson, N. J. (2003). Willingness to report crimes: The role of ethnic group membership and community efficacy. Crime & Delinquency, 49(4), 564–580. Dedahanov, A. T., Kim, C., & Rhee, J. (2015). Centralization and communication opportunities as predictors of acquiescent or prosocial science. Social Behavior & Personality, 43(3), 481–492. DeRidder, R., Schruijer, S. G., & Rijsman, J. B. (1999). Retaliation to personalistic attack. Aggressive Behavior, 25(2), 91–96. Dolmans, D., & Gijbels, D. (2013). Research on problem-based learning: Future challenges. Medical Education, 47(2), 214–218. Gentry, J. W. (1990). What is experiential learning. Guide to business gaming and experiential learning, 9, 20. Graf, L. A. (2001). ABSEL's contributions to experiential learning/experiential exercises: The decade of the 1970s. Simulation & Gaming, 32(1), 40–65. Greene, H. (2011). Freshmen marketing: A first-year experience with experiential learning. Marketing Education Review, 21(1), 79–88. Gross, C. (2008). A measure of fairness: An investigative framework to explore perceptions of fairness and justice in real-life social conflict. Human Ecology Review, 15(2), 130.

103

The International Journal of Management Education 16 (2018) 92–104

M. VanSchenkhof et al.

Hatcher, L. (1994). Step by step approach to using the SAS system for factorial analysis and structural equation modeling. NC, USA: SAS Institute Inc. Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967–988. Hoyle, R. H., & Smith, G. T. (1994). Formulating clinical research hypotheses as structural equation models: A conceptual overview. Journal of Consulting and Clinical Psychology, 62(3), 429. IBM Corp (Released 2013). IBM SPSS statistics for windows, version 22.0. Armonk, NY: IBM Corp. Jacobs, B. A. (2004). A typology of street criminal retaliation. Journal of Research in Crime and Delinquency, 41(3), 295–323. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. New Jersey: Prentice-Hall. Liu, N. F., & Carless, D. (2006). Peer feedback: The learning element of peer assessment. Teaching in Higher Education, 11(3), 279–290. Magin, D. (2001). Reciprocity as a source of bias in multiple peer assessment of group work. Studies in Higher Education, 26(1), 53–63. McGarr, O., & Clifford, A. M. (2012). ‘Just enough to make you take it seriously’: Exploring students' attitudes towards peer assessment. Higher Education, 65(6), 677–693. Meyers, K., Ohland, M., Silliman, S., McWilliams, L., & Kijewski-Correa, T. (2008). Comparison of two peer evaluation instruments for project teams. Proceedings of ASEE 2008 annual conference. Michaelsen, L., & Sweet, M. (2012). Team-based learning in the social sciences and humanities: Group work that works to generate critical thinking and engagement. Sterling, VA: Stylus Publishing. Michaelsen, L., Sweet, M., & Parmalee, D. (2009). Team-based learning: Small group learning's next big step: New directions for teaching and learning. San Francisco, CA: Jossey-Bass. Mollaei, F., & Rahnama, H. (2012). Experiential education contributing to language learning. International Journal of Humanities and Social Science, 2(21), 268–279. Montgomery, B. (1986). An interactionist analysis of small group peer assessment. Small Group Behavior, 17, 19–37. Moorman, R. H. (1991). Relationship between organizational justice and organizational citizenship behaviors: Do fairness perceptions influence employee citizenship? Journal of Applied Psychology, 76(6), 845. Nunnally, J. (1978). Psychometric theory. New York, NY: McGraw-Hill. Ohland, M. W., Loughry, M. L., Woehr, D. J., Bullard, L. G., Felder, R. M., Finelli, C. J., ... Schmucker, D. G. (2012). The comprehensive assessment of team member effectiveness: Development of a behaviorally anchored rating scale for self- and peer evaluation. The Academy of Management Learning and Education, 11(4), 609–630. O’brien, R. M. (2007). A caution regarding rules of thumb for variance inflation factors. Quality and Quantity, 41(5), 673–690. Papinczak, T., Young, L., & Groves, M. (2007). Peer assessment in problem-based learning: A qualitative study. Advances in Health Sciences Education, 12(2), 169–186. Patri, M. (2002). The influence of peer feedback on self-and peer-assessment of oral skills. Language Testing, 19(2), 109–131. Perkins, D. (2014). Conceptualizing defensive silence in project-manager-to-project-sponsor communication. The Leadership & Organization Development Journal, 35(1), 2–19. Rhee, J., Dedahanov, A., & Lee, D. (2014). Relationships among power distance, collectivism, punishment, and acquiescent, defensive or prosocial silence. Social Behavior & Personality, 42(5), 705–720. Robles, M. M. (2012). Executive perceptions of the top 10 soft skills needed in today's workplace. Business Communication Quarterly, 75(4), 453–465. Savery, J. R. (2015). Overview of problem-based learning: Definitions and distinctions. In A. Walker, H. Leary, C. E. Hmelo-Silver, & P. A. Ertmer (Eds.). Essential readings in problem-based learning (pp. 5–16). West Lafayette: IN: Purdue University Press. SurveyMonkey Inc (Released 2016). Surveymonkey. Palo Alto, CA: SurveyMonkey Inc. Topping, K. J., Smith, E. F., Swanson, I., & Elliott, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149–169. Vu, T. T., & Dall'Alba, G. (2007). Students' experience of peer assessment in a professional course. Assessment & Evaluation in Higher Education, 32(5), 541–556. Wolfe, D. E., & Byrne, E. T. (1975). Research on experiential learning: Enhancing the process. Developments in business simulation and experiential learning, 2. Wu, K., Davison, L., & Sheehan, A. H. (2012). Pharmacy students' perceptions of and attitudes towards peer assessment within a drug literature evaluation course. American Journal of Pharmaceutical Education, 76(4), 62.

104