Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

International Journal of Intercultural Relations xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

International Journal of Intercultural Relations journal homepage: www.elsevier.com/locate/ijintrel

Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders John W. Miller a,∗ , Jennifer S. Tucker b a b

Air Command & Staff College, Gunnison Court Pike Road, AL 36064, United States US Army Research Institute, United States

a r t i c l e

i n f o

Keywords: Critical thinking Intercultural competence Self-Regulation Situational judgment test Cultural dimensions DMIS Military leaders

a b s t r a c t Intercultural competence is a goal of many educational and training programs for military leaders who, when deployed overseas, are required to think critically and make strategic decisions in culturally complex environments. Critical thinking skills are not only essential leadership tools, but are also the keys to development of intercultural competence. The primary objective of this study is to gain a better understanding of the relationship between critical thinking and intercultural competence. Another objective is to determine which intercultural competencies learners find most challenging. Research methodology included a thorough examination of the curriculum, content, testing data, and end-of-course survey results of a non-credit, self-paced, instructorless online course for 2241 mid-career Air Force officers. Assessing critical thinking and intercultural competence development is a challenge in any educational context, but especially in online, self-paced courses without the benefit of teacher or student interaction. Situational judgment tests (SJTs) are a practical technique for assessing progress. Results revealed a significant relationship between critical thinking and intercultural competence SJT scores. Item analysis indicated that some scenarios were more challenging than others from an intercultural perspective. According to the findings, certain cultural differences challenge the decision-making ability of military officers in high-stress, high-visibility situations. These include culture-general knowledge (Mindset), empathic communication skills (Skillset), and a curious, open, and nonjudgmental attitude (Heartset). Participants scored significantly higher, however, when analyzing SJTs situated in more mundane intercultural workplace situations. The study’s findings support the efficacy of utilizing SJTs to develop intercultural competence, especially in online learning environments. Published by Elsevier Ltd.

1. Introduction Critical thinking – the ability to reflect and withhold judgment before making a decision – is essential for success in higher education and beyond (Facione, 2010). Intercultural competence is another set of cognitive skills also widely recognized as integral to academic and professional success. This skillset also promotes effective and appropriate decision-making, but within contexts that are culturally complex (Deardorff, 2009, p. 479). Indeed, the U.S. military views both critical thinking

∗ Corresponding author. E-mail addresses: [email protected], [email protected] (J.W. Miller). http://dx.doi.org/10.1016/j.ijintrel.2015.07.002 0147-1767/Published by Elsevier Ltd.

Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

2

Table 1 The OODA loop and Facione’s six skills.

OODA loop

Six CT skills

Observe The leader detects:

Orient The leader examines:

- Unfolding circumstances - Outside information

-

Interpretation and self-regulation

Analysis and self-regulation

Cultural traditions Genetic heritage New information Previous experience Analyses & synthesis Inference and self-regulation

Decide The leader hypothesizes the best course of action

Act The leader:

Explanation and self-regulation

Evaluation and self-regulation

- Tests & evaluates hypothesis

and intercultural competence as essential to effective leadership (Abbe & Halpin, 2009-2010). Because military leaders are routinely deployed to foreign lands, sometimes at a moment’s notice, the ability to think critically and make effective and appropriate decisions in intercultural environments has attained priority status in education and training programs in the U.S. Air Force (Selmeski, 2009). There is general acknowledgement that the requisite cognitive skills for critical thinking and intercultural competence overlap (Deardorff, 2004, 2006, 2009; J. M. Bennett, 2009; M. J. Bennett, 2013). In fact, M. J. Bennett (2013) has noted that the process for developing intercultural competence “parallels the development of critical thinking” (p. 109). Clearly, both skillsets are related and both are integral to effective communication and leadership. Yet surprisingly little research has focused on the relationship between critical thinking and intercultural competence. The present study seeks to shed light on the linkages between critical thinking and intercultural competence by examining the content of a self-paced, online course for mid-career officers in the U.S. Air Force and the testing data of the more than 3200 students who completed the course. Another goal of this research is to determine which intercultural competencies learners find most challenging. 2. Theoretical background In this section, we review theories related to critical thinking, intercultural competence, online education, constructivist pedagogy, and situational judgment tests. 2.1. Critical thinking models Dewey (1910) described critical thinking as an active process of purposeful reflection during which judgment is suspended. Building on his work, many others have followed in his footsteps (e.g., Ennis, 1979; Glaser, 1941; Paul, 1993), including Facione (1990, 2010) and Boyd (1995). 2.1.1. Facione’s model Facione (2010) asked an interdisciplinary panel of 46 scholars to conceptualize the cognitive and affective aspects of critical thinking, Facione’s expert panel reached consensus on six essential cognitive skills: analysis, interpretation, inference, evaluation, explanation, and self-regulation (p. 6). The panel characterized critical thinking as “the process of purposeful, selfregulatory judgment,” a definition that highlights the importance of self-regulation (p. 22). In his explanation of the results, Facione described self-regulation as a “remarkable” skill that embodies metacognition. Self-regulation is metacognitive because it “double checks” the thinking process by working in concert with the other five cognitive skills as well as itself (p. 7). It is an inner dialogue where personal opinions, beliefs, and assumptions are questioned and initial judgments are reconsidered. The study’s panel of experts did not agree solely on the essential components of critical thinking’s cognitive skills. The panel also reached consensus on what constitutes a “critical spirit,” a set of “dispositions” or attitudes that provide an individual with the proper mental platform for thinking critically (p. 8). These attitudes include qualities such as inquisitiveness, open-mindedness, flexibility, and prudence in suspending judgment. However, before enacting these cognitive and attitudinal skills, critical thinkers must understand how to apply each skill and why each is important. While Facione (2010) does not directly designate this technical knowledge as a specific set of skills, he implies it by providing his readers with a comprehensive description of the cognitive skills. The three skillsets – cognitive, attitudinal, and knowledge skills – form a holistic suite of abilities that, when consciously applied, will result in competent critical thinking. 2.1.2. Boyd’s OODA loop model While Facione’s (2010) model defines critical thinking, Boyd’s (1995) OODA loop describes a recursive process for applying cognitive skills when making decisions or solving problems. Known to virtually all uniformed Air Force personnel, OODA is an acronym that stands for observe, orient, decide, and act (Boyd, 1995, slide 4). The loop is a “time competitive decision cycle” originally created to help fighter pilots respond intelligently to tactical situations. Despite the narrowness of its original Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

3

purpose, OODA can be applied to any situation that requires critical thinking (Osinga, 2007, p. 6). A graphic alignment of both models illustrates how Facione’s six skills can be applied in real world situations within the OODA framework (see Table 1). 2.2. Intercultural competence models Intercultural competence, like critical thinking, is an active process of reflection during which judgment is suspended. Deardorff (2006), J. M. Bennett (2009), M. J. Bennett (2013), and Ting-Toomey (1999) sought to tease out the requisite skills and processes necessary for competent intercultural relations. Other scholars have taken an etic approach (e.g., Hall, 1981; Hofstede, 1991; Triandis, 1995), arguing that cultures can be better understood by comparing values and beliefs that emerge in divergent styles or behaviors (e.g., high/low context dimensions; individualist/collectivist dimensions). 2.2.1. Deardorff’s study of intercultural competence Using research methodology similar to Facione’s (2010), Deardorff (2006) consulted a panel of intercultural scholars to define intercultural competence and identify those cognitive and attitudinal skills essential to the development of competent intercultural communication and decision-making. Deardorff’s panel did not reach consensus on specific cognitive or attitudinal skills. Nonetheless, 80% of the panel did agree on 22 integral components. Among these were two of the six critical thinking skills to emerge from Facione’s research: analysis and interpretation. Other critical thinking skills identified by Deardorff were comparative thinking, cognitive flexibility, and the ability to withhold judgment and relate. These three metacognitive skills require purposeful reflection on one’s own inferences and judgments and are thus examples of self-regulation in practice. In addition, the majority of Deardorff’s (2006) experts also agreed on a suite of attitudes required for competent intercultural communication: respect, openness, and curiosity (p. 254). Two of these align closely with components of the critical spirit articulated by Facione’s (2010) panel: open-mindedness and curiosity. Deardorff’s panel also identified a third group of skills: cultural knowledge. According to Deardorff’s (2006) experts, one must have acquired “[d]eep knowledge and understanding of culture (one’s own and others)” to engage in competent intercultural communication (p. 250). 2.2.2. J.M. Bennett’s model of intercultural competence In an effort to organize the requisite skills for effective intercultural relations, J. M. Bennett (2009) grouped these competencies into three dimensions that roughly align with the three skill groupings to emerge from both Facione (2010) and Deardorff’s (2006) work: knowledge, skills, and attitude. Bennett calls these three dimensions “mindset,” “skillset,” and “heartset” (p. 97). The mindset is the knowledge dimension and encompasses culture-general models and knowledge about cultural awareness along with generalized data about specific cultures. The skillset or behavioral dimension, on the other hand, puts into practice such critical thinking skills as analysis, interpretation, inference, empathy, active listening, and information gathering. Bennett also includes within the skillset the ability to problem solve while building and maintaining relationships, tasks that require the engagement of all the critical thinking skills. The third dimension, the heartset grouping, aligns with Deardorff’s (2006) attitudinal set and Facione’s critical spirit by espousing qualities that lead to effective self-regulation. Bennett includes in this area interpersonal qualities such as “curiosity, initiative, nonjudgmentalness, open-mindedness, [and] tolerance of ambiguity” (p. 97). 2.2.3. M.J. Bennett’s developmental model for intercultural sensitivity J. M. Bennett’s (2013) developmental model for intercultural sensitivity (DMIS) provides a framework for describing how people experience cultural difference. The DMIS describes six types of intercultural sensitivity ranging from “shallow experience. . .to more complex perception,” making it a useful curriculum development tool for determining the level of culture-general content a specific group of learners might need (p. 86). The six stages of development are: denial, defense, minimization, acceptance, adaptation, and integration. The first three (denial, defense, and minimization) are classified as ethnocentric whereas the second group of three (acceptance, adaptation, and integration) are ethnorelative (pp. 84–99). A key DMIS stage for educators tasked with developing curriculum for military personnel is minimization. The DMIS stages progress from the ethnocentric stages of denial, defense, and minimization to the ethnorelative stages of acceptance, adaptation, and integration. Bennett (2013) characterized minimization as a transitional position between the ethnocentric and ethnorelative stages. While behavior at this stage remains essentially ethnocentric, individuals operating within this position are able to recognize differences between their own and other cultures. According to Bennett, they remain prone to judge these differences in behavior or style as a deficiency rather than as indicative of a divergent worldview. Given the typical kinds of work and living arrangements most uniformed military personnel are assigned to when deployed overseas, the majority of midcareer Air Force officers are likely to fit comfortably within the DMIS’s minimization stage. 2.2.4. Ting-Toomey’s observe, describe, interpret, suspend judgment model Ting-Toomey (1999) singled out analysis as a salient intercultural competency which she operationalized into a four-step model: observe, describe, interpret, and suspend judgment or “O-D-I-S” (p. 269). In addition to analysis, three other critical thinking skills are apparent in Ting-Toomey’s model: interpretation, explanation (analogous with “describe”), and selfPlease cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

4 Table 2 The OODA loop and O-D-I-S.

OODA loop

ODIS

Observe The leader detects:

Orient The leader examines:

- Unfolding Circumstances - Outside Information

-

Observe

Describe

Cultural Traditions Genetic Heritage New Information Previous Experience Analyses & Synthesis Interpret

Suspend judgment

Decide The leader hypothesizes the best course of action

Act The leader:

Decide how to engage (unstated step)

Engage appropriately (unstated step)

- Tests & evaluates hypothesis

regulation (analogous with “suspend judgment”). One interesting aspect of O-D-I-S, is its striking similarity to the observe and orient stages of Boyd’s OODA loop (see Table 2). 2.2.5. Etic models of culture Etic models of culture are grounded on the assumption that commonalities can be found in the values and behaviors of divergent cultures. Typically characterized by the use of microcultural dimensions or categories, etic models make it possible to generalize values related to specific behaviors. Examples of microcultural dimensions linked to social practices in a specific cultural setting are Hall’s (1981) low and high context communication styles as well as monochronic and polychronic time orientations (Hall, 1976). Another example is Hofstede’s (1991) individualist and collectivist values. Livermore (2010) categories of rational versus mystical views of the world are another. There are dangers associated with the use of etic models. Unless properly articulated and presented, they can lead to stereotyping and reductionist thinking. Despite these drawbacks, however, they are valuable teaching and assessment tools because the continua representing the contrasting categories of the various models help students to better understand how their own cultural values and the behaviors associated with them differ from those of other cultures. 2.3. Online learning pedagogy and assessment The emergence of the Internet, the World-Wide Web, and mobile communication technologies during the 1980s and 90s have transformed higher education. These innovations, wrote Harasim (2012), challenge the educational community to “rethink and reassess [its] teaching practices and pedagogical approaches” (p. 3). The greatest challenge lies in the development of relevant self-paced online learning environments and assessment instruments. 2.3.1. Constructivist online instruction A constructivist approach to online learning, instruction, and assessment, according to Harasim (2012), must focus on “the role of the learner or group of learners” (p. 68). Constructivist pedagogical assumptions are akin to some of the basic principles underpinning both critical thinking and intercultural competence; learning within the constructivist paradigm emerges through a process of purposeful reflection and dialogue. Indeed, Gold (2001) pointed out another similarity between these two competencies and the constructivist paradigm when he observed that online courses built on constructivist principles typically assign authentic, real-world tasks that help students “integrate other understandings of multiple perspectives through reflection” (p. 36). Building on general assumptions of purposeful reflection, multiple perspective taking, and dialogue, Gold (2001) identified three principles common to all constructivist education programs. These principles are: (1) a curriculum that progresses from “the specific to the general, from the concrete to the abstract”; (2) content that incorporates “prior assimilated structures”; and (3) assessment instruments that test “conceptual understanding,” not rote memorization (p. 38). These three principles present a unique challenge for curriculum developers tasked with designing self-paced online courses. 2.3.2. From the specific to the general, from the concrete to the abstract The introduction of real-world scenarios can lead learners to a better understanding of abstract concepts. In an online instructorless learning environment, an appropriate story or scenario not only draws student interest, but also provides practice in the use of conceptual frameworks to analyze an ambiguous situation. Before course designers can competently create such meaningful content, however, they must first have a clear picture of their students’ educational background, work history, and professional interests. 2.3.3. Use of assimilated models to facilitate learning Using familiar conceptual models to help students acquire new ones is another way to teach unfamiliar abstract concepts. According to Lantolf and Thorne (2006), this learner-centered technique facilitates learning by moving students from the known to the unknown and is best exemplified by the use of “scaffolds” (p. 274). Scaffolds are collaborative learning Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

5

Fig. 1. Vygotsky’s zone of proximal development (ZPD) (Miller, 2011).

strategies that activate the knowledge and skills learners have already mastered to perform tasks or grasp concepts that would otherwise be beyond their abilities. These peer-to-peer, tutor-to-student, and teacher-to-student strategies take place within Vygotsky’s (1978) Zone of Proximal Development (ZPD) (pp. 79–91). The ZPD is a conceptual frontier lying between what the learner is capable of performing unaided and what he or she cannot do even with assistance (see Fig. 1). In a self-paced online environment where peers and an instructor are absent, course designers must build scaffolding into the content through the use of tables, graphics, and online learning tools (e.g., simulations, videos, online quizzes with feedback for incorrect answers). As Harasim (2012) noted, these tools cannot, in and of themselves, “engage the learner in constructivist interactions such as discussion, debates or other knowledge-building interactions” (p. 76). Nonetheless, if course designers and curriculum developers have a clear understanding of the learners’ abilities, it is possible to construct scaffolds that will engage them in an inner dialogue of purposeful reflection. A simple example of scaffolding built into course content appears in Tables 1 and 2. In Table 1, Boyd’s (1995) OODA loop is presented in tandem with Facione’s (2010) critical thinking model while Table 2 scaffolds OODA alongside Ting-Toomey’s (1999) O-D-I-S model. 2.3.4. Assessment of progress through situational judgment testing A student’s motivation is inextricably linked to a conscious awareness that he or she is making progress. As mentioned previously, students in self-paced online learning environments are not able to question instructors or interact with classmates. The absence of dialogue makes it difficult for learners to check their understanding of new content. One way to address this challenge is through situational judgment tests (SJT). SJTs are assessment instruments that provide both a form of dialogue between the student and the content as well as the “authentic tasks” recommended by Gold (2001, p. 36). Hauenstein, Findlay, and McDonald (2010) describe SJTs as “scenario-based assessments designed to simulate judgment processes in the work context” (p. 263). Known originally as “critical incidents” (Ng & Rayner, 2010), SJTs were first introduced by the Air Force during World War II as a way to sort out top performers for aircrews (p. 90). In addition to providing practice in real-world problem solving, SJTs also lend face validity to the assessment process. In her description of this concept, Anastasi (1988) asserted that “the test [must] ‘look valid”’ to the students in order for it to have face validity (p. 144). By composing SJTs that require students to consider how new concepts and skills are applied either in familiar situations or within the context of scenarios to which they can easily relate, course designers ensure that students will not only be engaged in the process, but also learn from the experience and accept that the assessment is a valid one. 2.4. Summary and research questions The primary objective of this study is to gain an understanding of the relationship between critical thinking and intercultural competence. Another objective is to determine which intercultural competencies learners find most challenging. Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

6

Research methodology included a thorough examination of the curriculum, content, testing data, and end-of-course survey results of a non-credit, self-paced, instructorless online course for mid-career Air Force officers. Research questions that guided us in this study are: What are the relationships between critical thinking skills and different types of intercultural competencies? What aspects of intercultural competence (mindset, skillset, and heartset) are more challenging for participants when responding to scenario-based SJT items? 3. Method 3.1. Participants As stated previously, this study drew its data from a self-paced, non-credit orientation course for an Air Force distance learning program for mid-career officers. The program is comprised of 9 self-paced and 3 instructor-led online courses designed to improve the professional knowledge and leadership ability of Air Force majors and major-selects to prepare them for promotion to Lieutenant Colonel. Known by its acronym “ORIN,” this course has been in operation since September 2012. More than 10,400 students have either taken ORIN or are currently enrolled in it. Enrollment for the entire program as of this writing is 8900 students. All students in the program hold at least a bachelor’s degree. Many have earned a master’s degree. Because the vast majority of students are either Air Force Majors or Major-selects, most have been deployed overseas at least once. All have had to reside in diverse regions within the U.S. Students enrolled during the March 2013–June 2013 timeframe completed a voluntary end of course survey (ns = 3205–3336). The data for the analyses investigating the relationship between critical thinking and intercultural competence, item difficulty, pattern of responding, and other psychometric properties of the Lesson 5 SJTs reflected a smaller sample of these students (n = 2241). 3.2. Procedure—overview of the course This study examined the curriculum, assessment instruments, and testing data of ORIN, a self-paced, asynchronous, online course for mid-career officers. Students must take the orientation course before enrolling in the program’s creditbearing courses. ORIN has two goals: (1) to introduce students to the program’s policies and procedures; and (2) to enhance student potential for academic success by introducing lessons on critical thinking, intercultural competence, and academic writing. The course has six lessons. The first introduces the students to the 12-course program and orients them to the Blackboard course management system. The last lesson provides an overview of the requirements for academic writing at the post-baccalaureate level. The middle four lessons, however, are designed to answer four questions students might ask about critical thinking. These are: Lesson 2: “Why is critical thinking important?”; Lesson 3: “What is critical thinking?”; Lesson 4: “How can critical thinking be applied to academic study?” and Lesson 5: “How can I use my critical thinking skills to communicate more effectively across cultures?” While the second lesson gives students a broad introduction to the importance of critical thinking from the perspective of military history and the fourth provides detailed instruction on how to read critically, the third and fifth lessons are the focus of this study. We refer to the third lesson as the critical thinking lesson and the fifth as the intercultural competence lesson. The critical thinking lesson provides a comprehensive introduction to this cognitive skillset. Content is built around Facione’s (2010) six critical thinking skills. Boyd’s OODA loop is also included in the lesson. The intercultural competence lesson’s content introduces general concepts related to intercultural communication, including Bennett’s (2009) taxonomy of intercultural competencies, Ting-Toomey’s (1999) ODIS model, and key micro-cultural dimensions. Desired student learning outcomes for this lesson align with the Air Force’s definition of intercultural competence: The ability to quickly and accurately comprehend, then appropriately and effectively act in a culturally complex environment to achieve the desired effect—without necessarily having had prior exposure to a particular group, region or language. (Selmeski, 2009, p. 1) a Students are able to check their progress by taking scenario-based assessments (SJTs) that simulate dialogue and promote reflection while also delivering robust feedback for each multiple choice answer, whether right or wrong. Lesson 5 begins with an introduction to the basic concepts of intercultural communication followed by content that illustrates how principles of critical thinking can be applied to the task of making decisions in culturally complex situations. The course’s intercultural competence lesson addresses a 5-year Air Force-wide educational initiative to develop “crossculturally competent” (sic) officers and enlisted men and women (Selmeski, 2009, p. 1). Known as the Quality Enhancement Plan or QEP, this initiative was required by the Commission on Colleges of the Southern Association of Colleges and Schools for re-affirmation of accreditation. Because of this initiative, uniformed personnel at all levels have been the beneficiaries of a broad variety of education and training programs focusing on the development of intercultural competence. As a result, the Air Force officers who participated in this research project, have received multiple opportunities to develop their intercultural competence to include the following: (1) pre-deployment training consisting of both online instruction with intercultural competency-based simulations for Iraq and Afghanistan and face-to-face training with regional subject matter experts; and (2) exposure to intercultural competence curricula during their most recent professional military education as Captains whether they took it online or in residence. Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

7

Table 3 Critical thinking SJTs in critical thinking lesson. Critical thinking SJT summary

Critical thinking skills/model

A well-digging project in a developing country

Self-regulation: • Question assumptions

Anti-American sentiment in the Kyrgyz Republic

OODA loop Self-regulation • Move from Observe to Orient stage • Analyze to determine root causes

An ambiguous situation with a shop clerk in an English-speaking country with high context communication style

Observation Self-regulation • Take time to observe • Question assumptions • Ask questions

An ambiguous situation while standing in line in a developing country with a high-context communication style

Observation Self-regulation • Take time to observe • Question assumptions • Ask questions

Building relationships with coalition partners and analyzing a problem

Ethnocentrism Self-regulation • Employ purposeful reflection • Take time to observe • Question assumptions

The content in Lessons 2–5 supports learning objectives across the curricula of the program’s 12 courses while providing a review and reinforcement of concepts related to intercultural competencies previously developed in other Air Force education and training programs. Lesson 5 also articulates directly with three other courses in the program: leadership and command, national security studies, and joint forces planning. Content in these three courses includes case studies and online simulations where students must negotiate and make operational and strategic-level decisions in intercultural contexts. 3.3. Measures 3.3.1. Critical thinking SJTs The assessment for the critical thinking lesson consists of 10 multiple choice items. Five of the 10 questions are SJTs and contain scenarios that are all situated in developing countries. Each SJT presents students with an ambiguous conflict situation triggered by differences in cultural values and behaviors and test students’ critical thinking ability to analyze realworld situations and make appropriate decisions (Table 3). Further, the five SJTs require students to apply their knowledge of critical thinking’s five cognitive skills and the metacognitive skill of self-regulation. The total critical thinking score was calculated by taking an average of the correct responses made on the 10 items (10 points each for each correct response on 5 knowledge-based questions and 5 SJTs; M = .79). Although participants are able to take as many iterations of the test as needed to reach the passing score of 80%, only their scores from the final attempt on the critical thinking assessment were used in the subsequent analyses. When conducting the analyses to obtain an internal consistency reliability estimate, the results produced a negative Cronbach’s Alpha and indicated many negative inter-item correlations. The results of an exploratory factor analyses indicated a six-factor solution with three of the items not loading on any of these six factors. These results, suggesting a multidimensional scale, taken together with the negative reliability estimate (alpha) supports the idea from prior research that the use of internal consistency reliability estimates for SJTs is problematic. Waugh (2004) explained that: Even at the item level, situational judgment tests are multidimensional and heterogeneous in nature. That is, a typical item measures more than one construct and the items measure the various constructs to different degrees. Internal consistency reliability estimates, on the other hand, assume that a single construct or the same set of constructs (to the same degree) underlies the items. Thus, coefficient alpha usually underestimates the reliability of situational judgment tests (p. 6–7). Therefore, in addition to the correlation between the average critical thinking and intercultural competence scores, we provide the correlations between the individual critical thinking items and the average intercultural competence score. The Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

8

items for both the critical thinking and intercultural competence assessments were automatically randomized such that each student received one of 10 different random orders of items each time he or she took the assessments. 3.3.2. Intercultural competence SJTs The intercultural competence assessment consisted of 12 SJTs that were modeled after Cushner and Brislin’s (1996) cultural assimilator methodology and test the students’ ability to practice mindset, skillset, and heartset concepts and skills (see Appendix A). Three scenarios were adapted from Cushner and Brislin’s work, two were based on research conducted with Army Soldiers (Rosenthal, Wadsworth, Paulin, Hooper, & Bhawuk, 2009), four were drawn directly from the course content, one was developed by the first author of the present study from his experiences as a Peace Corps Volunteer, and two were based on tasks performed by Air Force members in overseas locations as specified in Air Force doctrine. The final version of each scenario was reviewed by at least one Air Force officer whose expertise matched the particular tasks being performed in the scenario. Some scenarios were further modified in response to feedback from subject matter experts. Below is an SJT Progress Check from the intercultural competence assessment. To answer correctly, students must employ mindset, skillset, and heartset skills and be willing to disobey a direct order from their commander. Making such a decision with a minimal amount of information requires self-regulation, patience, respect, interest, openness, the ability to adapt, and a certain degree of courage. An officer will not do so lightly. Willfully disobeying an order from a superior without just cause can be considered dereliction of duty and subject to prosecution under the Uniform code of military justice (Article 92; see Fig. 2). As participants are able to take as many iterations of the test as needed to reach the passing score of 80%, their scores were averaged across all of the individual testing iterations to obtain the average intercultural competence score. We decided to average the intercultural competence scores across iterations rather than select only the score from the final iteration because we were interested in examining the variability in responses across multiple iterations of the assessment. Although 12 intercultural competence SJTs were developed for the assessment, the online course randomizes 10 items for each participant and provides 10 points per each correct score. Thus, the total intercultural competence score consists of an average of the correct responses on 10 of the 12 items (M = 81%). To obtain a reliability estimate, all 12 items were coded as 1s (correct) or 0s (incorrect; alpha = .59). Although this reflects a lower reliability estimate, it is a typical estimate for SJTs, which tap multiple constructs across scenarios (Weekley, Ployhart, & Holtz, 2006). This estimate also may be lower because, in order to conduct item analyses with these data, participants may have received a score of 0 for an SJT they did not receive. To address this issue and to better understand the pattern of responding to the intercultural competence SJTs, we provide the correlations between the average critical thinking score and the individual intercultural competence SJT items in addition to the correlation between the average critical thinking and intercultural competence scores. Finally, to analyze the pattern of responding on the distractor items, incorrect responses were subsequently recoded as 1s. 3.3.3. Student end-of-course survey At the completion of the course, students were asked to respond to an end-of-course survey consisting of 10 items on a 6-point Likert scale (1 = strongly disagree, 6 = strongly agree). Students rated the course’s ability to teach critical thinking and intercultural competence skills (see Table 8 for the list of items and results.) The data were collected anonymously. Therefore, it was not possible to investigate relationships between the self-report items and the SJTs. 4. Results In this study, we sought to uncover the relationship between critical thinking and intercultural competence. We were also interested in determining which intercultural competencies students found most challenging to enact as they responded to the intercultural situations presented in the SJT questions. The following results emerged from this investigation. 4.1. The relationship between critical thinking and intercultural competence The results demonstrated that critical thinking was significantly related to the overall measure of intercultural competence (r = .28, p < .001) and significantly related to many of the individual intercultural competence SJT items (Table 4). Further examination of the pattern of relationships indicated that critical thinking was more strongly related to scenarios reflecting mindset and skillset intercultural competencies. This makes sense since these two sets of competencies encompass a working knowledge of culture general concepts such as microcultural dimensions as well as the ability to apply that knowledge to real-world situations. Each of the intercultural competence SJT scenarios either implied or specifically described their cultural contexts using one or more microcultural dimensions. It is not surprising then, that the results indicate use of critical thinking skills to a larger degree when the students applied their understanding of high/low context, individualism/collectivism, time orientations, and rational/mystical religious beliefs to make informed decisions. As the results for the critical thinking measure indicated a multi-dimensional construct, Table 5 provides the correlations between the individual critical thinking items and the overall intercultural competence score. The results demonstrated that the strongest, positive relationships were found among items reflecting self-regulation and inference as well as interpretation and observation skills. The results also revealed an unexpected significant negative correlation between one of the critical Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

Fig. 2. SJT #9 in the intercultural competence lesson ACSC DL 6.0. (2012).

Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

9

G Model IJIR-1163; No. of Pages 17

10

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

Table 4 Nonparametric correlations (Spearman’s Rho) for the critical thinking average percent correct score and the individual intercultural competence SJTs.

*

SJT domains (n = 1732)

Critical thinking Average percent correct

Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures Mindset/Skillset: Cognitive skills; tendencies of high-context cultures Mindset/Skillset: Cognitive skills; Importance of respect Heartset: Self-regulation: Show patience, respect, interest, and openness Heartset: Self-regulation: Suspend judgment; show respect, interest, and openness Heartset: Self-regulation: Show respect, interest, and openness Mindset/Skillset: Cognitive skills; tendencies of high-context cultures Heartset: Self-regulation: Show patience, respect, interest, and openness Mindset/Skillset: Cognitive skills; tendencies of high-context cultures Mindset/Skillset: Cognitive skills; time orientations; tendencies of high-context cultures Mindset/Skillset: Cognitive skills; tendencies of collectivist cultures

.06* .16** .09** .05 .10** .13** .04 .14** .02 .01 .16** .20**

p < .05. p < .01.

**

Table 5 Nonparametric correlations (Spearman’s Rho) for the intercultural competence average percent correct score and the individual critical thinking items.

*

Critical thinking domains (n = 2092)

Intercultural competence Average percent correct

Comprehension question of interpretation Comprehension question of observation Knowledge question of inference Knowledge question of questioning Knowledge question of judgment Self-regulation SJT OODA loop SJT Self-regulation, OODA loop, intercultural context SJT Self-regulation, OODA loop, intercultural context SJT OODA loop, ethnocentrism, intercultural context SJT

−.02 .03 .07** .04 .02 .13** .03 −.05* .06** .04

p < .05. p < .01.

**

Table 6 SJT item difficulty and distractor response results. Intercultural competence skills and microcultural dimensions Correct Response Collapsed Distractor 1 Distractor 2 Distractor 3 % Correct (ns)a % Responded (ns)b % Responded (ns)b % Responded (ns)b (n = 1919) Mindset/Skillset: Collectivist culture Mindset/Skillset: Collectivist culture Mindset/Skillset: Low/High Context Mindset/Skillset: Respect Heartset: Self-regulation Heartset: Self-regulation Heartset: Self-regulation Mindset/Skillset: Low/High Context Heartset: Self-regulation Mindset/Skillset: Low/High Context Mindset/Skillset: Time Orientations; Low/High Context Mindset/Skillset: Collectivist culture

59% (1636) 73% (2024) 64% (1782) 60% (1656) 66% (1822) 66% (1821) 59% (1644) 77% (2138) 25% (704) 47% (1316) 78% (2155) 80% (2213)

16% (438) 18% (504) 38% (1052) 4% (123) 1% (23) 2% (41) 1% (33) 9% (258) 6% (164) 3% (86) 4% (102) 2% (51)

22% (607) 1% (38) 1% (20) 4% (100) 4% (105) 2% (56) 8% (234) 2% (68) 22% (600) 1% (22) 5% (150) 1% (28)

12% (326) 5% (124) 1% (27) 8% (220) 1% (17) 1% (38) 6% (177) 5% (125) 6% (178) 1% (34) 3% (95) 3% (91)

a Students are allowed to keep taking the SJTs until they provide the correct response, thus, correct responses were collapsed across iterations such that a student received a score of #6’7# for any correct response across the iterations attempted. The first attempts at the SJT would reflect much lower percent correct results. b Columns may sum to over 100% because the same students are reflected in more than one column depending on their pattern of erroneous responses across iterations. These columns reflect the percentage of students who provided incorrect responses prior to possibly providing a correct response. On the other hand, columns may not sum to 100% due to missing data (students were only given 10 of the 12 items per attempt).

thinking SJTs and intercultural competence (see item number 8 in Table 5). Some possible explanations for this result are provided in the Section 5. 4.2. Intercultural competence SJT results The results of the item difficulty analysis and patterns of responding for each intercultural competence SJT are presented in this section. Table 6 shows the intercultural competencies and the microcultural dimensions for each SJT as well as the Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

11

Table 7 Point biserial correlations with the overall intercultural competence SJT score. Intercultural competence SJTs

Point biserial correlations* (n = 2241)

SJT1 SJT 2 SJT 3 SJT 4 SJT 5 SJT 6 SJT 7 SJT 8 SJT 9 SJT 10 SJT 11 SJT 12

.44 .24 .42 .21 .21 .23 .26 .18 .25 .22 .15 .09

*

All correlations were significant at the p < .01 level.

percent correct. For these analyses, as students were allowed to retake the SJTs until they could obtain a satisfactory score, correct responses were coded as 1s and were collapsed across testing iterations. Table 6 also shows students’ pattern of incorrect responding which illustrates initial interpretations of the intercultural situations. The range in the percent correct results indicates that some intercultural situations (SJTs) were more challenging for students than others (25–80%; Table 6). Even though the students could retake the assessment, these results indicated that the situations described in some of the SJTs were particularly challenging for them. Only 25% of the students responded correctly to SJT 9 which taps heartset intercultural competencies while requiring students to avoid a typical low context, individualist response declining the invitation to meet with the host nation Minister, and select a more appropriate high context, collectivist response (see Fig. 2 or the Appendix A for a more detailed description of the scenario). Students struggled to espouse the heartset intercultural competencies necessary for effective performance in this intercultural situation such as being open-minded and tolerant of an ambiguous situation. Students needed to incorporate these self-regulatory attitudes in their decision making in order to are necessary to avoid selecting an ethnocentric response. An examination of the pattern of incorrect responding was helpful in determining students’ initial interpretations of the intercultural situations. For example, the results indicated a clear pattern of incorrect responding for SJT 3 (38%, Table 6) in which students provided the reason why the local national indicated a much shorter distance than the real distance (see Appendix A). The results revealed an ethnocentric bias such that the largest percentage of incorrect responses reflected a belief that the local national did not know how to tell time. Further, the results indicated that for this intercultural situation, students did not possess the intercultural mindset and skillset competencies to interpret this situation accurately and select the culturally appropriate answer (that the villager had a high context communication style that prevented him from giving the aid worker bad news). Similarly, the incorrect response selected most often by students in response to SJT 1 (22%) reflects an ethnocentric perspective and a low ability to elicit the necessary mindset and skillset intercultural competencies to respond correctly. The focus of the scenario is on General Zinni and Klotz’s (2007) experience dealing with a refugee crisis along the Turkish–Iraqi border in the aftermath of the first Gulf War (see Appendix A). Students who responded incorrectly to this intercultural situation believed that the most appropriate response would be to enlist the help of Christian missionaries and Peace Corps Volunteers rather than the tribal leaders among the refugees. On the other hand, the high percentage of correct answers to those situations that described real-world examples of interpersonal communication in intercultural contexts suggests that it was easier for students to tease out pertinent microcultural dimensions and use them to analyze an interpersonal situation and recommend an appropriate response. For example, the results for SJT 11 suggested that most of the students responded correctly to this intercultural situation on their first attempt because of the high percent correct (78%, Table 6) and low occurrences of incorrect responding (see Appendix A). The intercultural situation reflected intercultural mindset and skillset competencies and the microcultural dimensions of low and high context communication and time orientation to analyze and interpret an intercultural conflict situation (see Appendix A). Although the scenario was complex due to the amount of information included, it also provided clues that the culture’s preferred communication style was high context. Thus, if the students comprehended the differences between low and high context, they also could provide the correct response. Similarly, most of the students responded correctly to SJT 12 on their first attempt as witnessed by the high percent correct (80%, Table 6) and low occurrence of incorrect responding. Students were likely able to discern the appropriate response by eliminating options that either did not reflect a culturally appropriate response or did not seem plausible given the information contained in the scenario (see Appendix A). Another reason why SJTs 11 and 12 received a high number of correct responses might be related to the scenarios themselves, not only the intercultural competencies and microcultural dimensions represented therein. Both stories are based on fairly common intercultural problems in the workplace that students may have experienced or heard about while deployed, making it easier for them to place themselves in those situations (see Appendix A). To investigate how each individual SJT related to the overall SJT measure, point biserial correlations were obtained between each SJT and a sum of all of the SJTs (Table 7). Higher point-biserial estimates for particular items suggest that students who respond correctly to these items are also responding correctly to the other items in the assessment. As SJTs 1 Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

12

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

Table 8 Student feedback: intercultural competence and critical thinking competencies. This course enhanced my ability to: Strongly agree Agree Slightly agree Slightly disagree Disagree Strongly disagree

Total agree Total disagree

Understand my own cultural values and personal biases 613 1745 665 94 81 28 3023 Frequency 19% 54% 21% 3% 3% 1% 94% Percentage Recognize the cultural information embedded in artifacts, beliefs, values, and assumptions (e.g., physical settings, stories, symbols, rituals, language, and social structure) Frequency 665 1791 617 74 62 27 3073 Percentage 21% 55% 19% 2% 2% 1% 95% Think of more than one reason for a person’s behavior Frequency 795 1739 534 77 59 26 3068 Percentage 25% 54% 17% 2% 2% 1% 95% Understand the points of view of people from different cultural backgrounds Frequency 756 1740 568 76 57 28 3064 Percentage 23% 54% 18% 2% 2% 1% 95% Systematically identify consistencies, patterns and relationships among pieces of cultural information Frequency 614 1763 662 86 61 28 3039 Percentage 19% 55% 21% 3% 2% 1% 95% Incorporate cultural information into my decision-making processes Frequency 727 1744 587 79 56 29 3058 23% 54% 18% 2% 2% 1% 95% Percentage Ask questions to elicit in-depth cultural information and fundamental values Frequency 659 1715 642 93 65 31 3016 Percentage 21% 54% 20% 3% 2% 1% 94% Formulate explanations that reflect an understanding of local cultures 653 1774 604 89 56 29 3031 Frequency 20% 55% 19% 3% 2% 1% 95% Percentage Analyze information and evidence that challenge cultural stereotypes Frequency 680 1775 587 85 61 28 3042 21% 55% 18% 3% 2% 1% 95% Percentage This course improved my critical thinking skills Frequency 657 1514 836 143 109 64 3007 Percentage 20% 46% 25% 4% 3% 2% 90%

Total responses

203 6%

3226

163 5%

3236

162 5%

3230

161 5%

3225

175 5%

3214

164 5%

3222

189 6%

3205

174 5%

3205

174 5%

3216

316 10%

3323

ACSC DL 6.0. (2013).

and 3 related most strongly with the overall score (r = .44, p < .01, r = .42, p < .01, respectively), these items may be particularly good at evaluating students’ mindset and skillset intercultural competencies and their knowledge of collectivist values and low and high context communication styles (see Appendix A). That is, the items more reliably discriminate between good and poor performers on these competencies and dimensions. Interestingly, the two SJTs with the lowest correlations, SJT 11 (r = .15, p < .01) and SJT 12 (r = .09, p < .01), had the highest percent correct scores. Although most of the students were answering the SJTs correctly on their first attempt, these items may not be optimal indicators of improvement in the students’ intercultural competence. This is because it is likely that the students who responded poorly to the other SJTs responded correctly to these items. These SJTs should be further investigated for possible revisions. Nonparametric correlations (Spearman’s Rho) were obtained for all of the SJT items (the full correlation matrix is not provided due to space limitations but can be made available by the first author). Although all of the correlations were significant due to the large sample size of the present study, several relationships are noted because they highlight the need to further examine the content of these intercultural situations. Specifically, SJTs 11 and 12 had one of the strongest positive correlations ( = .89, p < .01) while SJTs 9 and 10 had the only negative correlations ( = −.33,  < .01). These results could be due in part to the idea that the intercultural competencies required to successfully perform in both of these intercultural situations are very different (heartset vs. mindset/skillset). SJT 9 requires students to self-regulate and overcome their ethnocentric biases within a military context whereas SJT 10 requires the students to elicit the mindset and skillset intercultural competencies and determine the most appropriate strategy for working with host country nationals in a complex situation (see Appendix A). Students who possess the skills to perform well in one context may not possess the necessary skills to perform well in the other.

4.3. Student feedback The results for the end-of-course survey indicated that most students agreed that the course developed intercultural competence and critical thinking skills (94–95% agreed across all of the items; Table 8). Further, 90% of the students reported that the course improved their critical thinking skills. Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

13

5. Discussion 5.1. Findings The present study sought to uncover the relationship between critical thinking skills and intercultural competencies while also seeking to tease out those competencies that were most challenging for learners to enact. Assessment data and student feedback metrics from ORIN, a non-credit, self-paced, online course for Air Force Majors were examined to answer these questions. Although one might intuitively surmise that a robust relationship exists between critical thinking and intercultural competence, little research to date has provided the evidence needed for theory building. The findings of the present research provide empirical support for a relationship between critical thinking and intercultural competence and indicate a moderate association between these skillsets. The modest strength of the relationship is somewhat surprising given the overlap of the mindset/skillset dimensions of intercultural competence with critical thinking skills and indicates that an additional set of knowledge, skills and attitudes are needed to perform successfully in intercultural contexts. For example, students seemed to have the most difficulty in responding correctly to the intercultural competency items reflecting Bennett’s (2009) heartset dimension. The results of the SJTs that described low-stress intercultural conflicts in the workplace suggested that students might tend to be more adept at making interculturally competent decisions in such real-world situations. However, the findings also suggest that there are certain cultural differences that are likely to challenge the decision-making capability of military officers in high-stress situations. These areas include mindset skills related to an understanding of differences in time perception (polychronic versus monochronic) as well as skillset (relationship building and information gathering), heartset competencies (risk taking and tolerance of ambiguity), and overcoming an ethnocentric perspective (Bennett, 2009). One such scenario (SJT 9) involved a humanitarian operation in a highly collectivist culture with a high-context communication style (Fig. 2 and Appendix A). Students were asked to decide whether or not to report directly to their commander as ordered or comply with a high-ranking ministry official’s request for a meeting. In a collectivist culture, such a meeting would help to forge a relationship that might well bear fruit later on. Most took the safer route and opted to report to their commander. It seems likely that students perceived the meeting as a risky venture fraught with ambiguity rather than an opportunity for relationship building. These results highlight the dilemma military personnel face in such high challenge situations. As members of an order-driven organization, military officers, when placed in an ambiguous scenario where the stakes are high, must make decisions that, for them, may entail ethical considerations that supersede the desire to display cultural sensitivity. Furthermore, as members of a military organization that is quintessentially hierarchical in nature, mid-career officers would be extremely cautious in agreeing to commit to a meeting with an official at the level of Minister without the prior approval of their commander. Given these caveats, such a question would lend itself better to discussion than a multiple choice SJT with only one correct answer. Yet as an SJT, this scenario can have value when multiple iterations are allowed and robust feedback for answers is provided. Students willing to take the time to re-read the scenario and study the answer feedback learn that making such a culturally sensitive choice could well be beneficial to the mission. Another area of interest to emerge from the data was the negative correlation for self-regulation in critical thinking SJT item number 8 and the overall intercultural competence SJT score. This result was surprising because the scenario tested for the utilization of self-regulation, a critical thinking skill that the students understood and employed successfully in other SJTs. One possible explanation is that the scenario described a communication problem that the students may have encountered while deployed. In this scenario, a shop clerk in a foreign country where English was the official language communicated assent to a question by raising her eyebrows and sharply drawing in her breath. This led many students to assume that the clerk (a) did not speak English, (b) was communicating surprise, or (c) was displaying antiAmerican behavior. (The correct answer was to ask her for clarification.) Situations where one has previously performed successfully can sometimes lead to a misplaced assumption of rightness. Indeed, Ambady, LaPlante, and Johnson (2001) cautioned that an individual’s intuition can be very accurate, but only when operating in highly familiar professional or social situations. Leaders who rely solely on intuition in intercultural contexts run the risk of misinterpreting – or missing completely – a wealth of socio-cultural information, even when the situation, at least on the surface, seems easily understandable. Of all the critical thinking skills, self-regulation was found to be the key to responding correctly to the intercultural scenarios. Prior research has described self-regulation as a malleable skill. When developed, it allows individuals to maintain a high level of performance in stressful situations. However, development of this skill takes time. Baumeister (2000) likened self-regulation to a muscle that becomes depleted over the short term, but strengthens with continuous exercise over time. Military personnel required to work in divergent cultural contexts that are also dangerous will quickly find their selfregulation skills depleted. Yet those who have been educated and trained to self-regulate should be better able to bounce back, recognize their biases, and control their behavior. The results of the SJTs and other data were also helpful in ascertaining the competency level of the students. The course was specifically designed to present content related to critical thinking and intercultural competence that is appropriate for students operating at the minimization level of cultural sensitivity on the DMIS scale (Bennett, 2013, 2009). Taking into consideration that students had more difficulty answering higher challenge, ambiguous questions, but were able to answer Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

14

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

lower challenge questions with some facility, this was most likely an accurate assumption. However, determining the exact level of the DMIS the students had reached by the end of the course was outside the scope of the present study. While Hammer’s (2012) 50-item questionnaire known as the Intercultural Development Inventory has been found to establish an individual’s DMIS level, it was not feasible for inclusion in a study involving thousands of military personnel in a governmentrun educational program. On the other hand, the results of the end-of-course feedback provide a modicum of insight into the students’ selfawareness with regard to cultural differences as well as their overall assessment of the learning experience. Bennett (2009) designated development of cultural self-awareness as the primary task for students at the minimization stage. According to the end of course survey results, 94% of the students who took the course agreed that it had enhanced their understanding of their own cultural values and personal biases and 95% believed that the course had enhanced their ability to “[u]nderstand the points of view of people from different cultural backgrounds.” This suggests that for many, some level of cultural selfawareness was achieved.

5.2. Study limitations The present research has several limitations, due to the procedures involved in conducting both the data collection and the psychometric analyses. First, as students are able to take the assessments as many times as necessary to reach the 80% mark and move forward in the course, the results may have been affected by the responses of some students who tried to achieve 80% during their first attempt by guessing. We tried to minimize these potential errors by averaging the responses from all of the attempts a student made on the Lesson 5 assessments. Averaging across iterations also provided more variability in the scores and reduced the inherent restriction of range in the final responses as by this time most have had multiple opportunities to respond correctly. It is important to note that the nature of the intercultural competence SJTs also pose challenges when analyzing their psychometric properties. For example, the SJTs assessed multiple constructs and are likely heterogeneous at the item level (e.g., responses reflect students’ abilities to tap both cognitive skills and personality/motivation characteristics) in answering the items. In this case, researchers posit that it is inappropriate to use Cronbach’s Alpha to estimate reliability because it underestimates the reliability of the measure (Whetzel & McDaniel, 2009). As test–retest and parallel forms are the most appropriate reliability estimation methods for SJTs (Whetzel & McDaniel, 2009), developers of the course curriculum should consider creating additional SJT items so that these methods could be employed and accurate reliability estimates obtained.

5.2.1. Course structure The most serious limitation to both research and pedagogy, however, is the course’s delivery method. An online, selfpaced course, asynchronous learning environment presents several hurdles for course designers tasked with developing intercultural competence lessons. Heartset learning outcomes such as suspending judgment, ethno-relative perspectivetaking, and tolerance for ambiguity are competencies that would be more effectively taught if collaborative learning tools were compatible with a self-paced online course design. While the SJT instruments examined in the present research have some interactive qualities, particularly when their robust feedback is taken into account, they are delivered in the cold medium of the written word. Other educational and training techniques suitable for asynchronous, self-paced online learning environments such as video-based, interactive SJTs and desktop avatar simulations possess game-like qualities that enhance motivation (Deci & Ryan, 2002; Malone & Lepper, 1987; Rieber & Noah, 2008; Turner & Paris, 1995). Such instructional tools allow individuals to be more fully immersed in an intercultural context and thus have a better opportunity to develop and demonstrate the behaviors associated with the heartset dimension. The development cost for these innovative environments, however, place them outside the reach of many educational programs, as was the case with ORIN. An added hurdle for distance education programs in military institutions is the stringent cybersecurity measures taken to protect their online networks. These constraints make the incorporation of non-secure interactive websites untenable. One way to ameliorate the limitations of self-paced online courseware is to install a “wiki.” Wikis, according to Mackenzie, Forgarty, and Khachadoorian (2013), are “a collection of loosely structured, collaboratively edited, web-linked content on a particular subject.” Mackenzie and Wallace (2013) proposed that wikis are “conducive to a constructivist perspective” because they present students with an interactive staging area to “apply course concepts to their own experiences,” making them ideal for self-paced online courses (p. 247). Installing a wiki would introduce a zone of proximal development allowing students to voluntarily describe intercultural experiences where their self-regulation skills (e.g., the ability to take an alternate perspective or deal with ambiguity) were tested and then analyze their decision-making based on their understanding of the lesson content (Vygotsky, 1978). The wiki platform also allows for other students taking the course at the same time to comment and collaborate. The most compelling of the wiki stories would also be available for curriculum developers to use as SJTs in future iterations of the course. Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model

ARTICLE IN PRESS

IJIR-1163; No. of Pages 17

J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

15

5.3. Future research directions In looking at the positive results of the end-of-course survey, a fruitful area for research would be to further establish the nomological net of critical thinking and intercultural competence. Research along these lines might entail an investigation of motivational constructs such as those proposed by Van Dyne, Ang, and Livermore (2010). In their study of cultural intelligence (CQ), they designated motivation as a separate intercultural competency comprised of three types of motivation: intrinsic, extrinsic, and self-efficacy. Self-efficacy, the belief that “you will be effective in a cross-cultural encounter” (p. 135), may be the most powerful of the three types of motivation for educators and trainers to consider as they develop curricula and lesson plans. If students and trainees emerge from an education or training experience with the basic skills needed to effectively and appropriately plan, communicate, relate, and negotiate in a culturally complex environment, and they are confident about implementing those skills, they are more likely to feel ready to navigate and interact within the culture. In turn, the more effective they are in their intercultural experiences, the more they will be able to rely on intuition to make appropriate decisions. Such confidence, when tempered by the conscious application of self-regulation, also breeds the type of curiosity that Bennett (2009) described, the kind that eventually leads to effective and appropriate decision making through the unconscious use of heartset skills such as patience, respect, interest, openness, multiple perspective-taking, tolerance for ambiguity, and the suspension of judgment. Another area for future research is an investigation of the distal and proximal nature of the intercultural competencies and attributes on intercultural communication behaviors. This approach would provide useful information for both researchers and practitioners. For researchers, the presence of distal predictors often suggests that their effects on performance criteria in intercultural contexts will be indirect and affected by environmental factors and/or other individual characteristics. For practitioners, the presence of proximal predictors suggests that these may be more malleable and thus cost effective to include in training programs. A compelling area for study in the near term would be the unique role of self-regulation. Qualitative research could be conducted with students at the same career level in the Air Force, but who are participating in an instructor-led online course having similar content and objectives. Such a research platform could expand our understanding of the ways this metacognitive skillset is enacted. (At least one such course is currently offered by the Air Force.) Within the context of the online course’s discussion board, students would be invited to become participants and receive one or more of this study’s most challenging SJT questions to ponder and discuss. In this approach, however, the scenarios would be rewritten as openended questions to better stimulate discussion. By sharing stories in writing on the discussion board and commenting on their classmates’ experiences, participants would engage in purposeful reflection and dialogue, essential requirements for learning from the constructivist paradigm. As a research platform, this project would provide opportunities to ask follow up questions on the discussion board and elicit more stories of self-regulatory practices. Qualitative interviews could also be conducted with participants from both instructor-led and self-paced courses later on to tease out more thick descriptions from their experiences. This line of research might well lead to a better understanding of self-regulation. As mentioned previously, another way to generate qualitative data for research within the context of an asynchronous, self-paced course would be to exploit available online technology through the installation of a wiki component. The literature mentions perspective taking, suspension of judgment and questioning of assumptions as requisite heartset competencies. However, these are but two manifestations of self-regulation in intercultural contexts. What others are being successfully employed? Clearly, more research into self-regulation is needed. Intercultural competence requires an abundance of self-regulatory interpersonal skills (e.g., empathy, patience, openness, and tolerance of ambiguity). Even the best critical thinkers in academic or professional contexts may perform at a less than competent level when operating in a culturally complex environment. There is a need for more empirically based research on self-regulation so that education and training programs can maximize the transfer of its benefits to real-world contexts. 6. Conclusion The findings of the present research provide empirical evidence that intercultural competence and the ability to think critically are closely related. Our findings also indicate that the metacognitive ability to self-regulate, a “heartset” skill common to both critical thinking and intercultural competence, is the most challenging to implement. It is our belief that instructional designers for distance education courses can develop online learning with the goal of enhancing intercultural communication behaviors if they take a constructivist approach to curriculum design and create a dialogue between the student and the instructional content. SJTs provide such a vehicle by giving students a rationale for test answers. This is especially helpful with incorrect responses. The SJTs perform the duties of an instructor or classmate by providing scaffolding that lays the groundwork for greater awareness and understanding to take place. Despite the many constraints posed by instructor-less online education programs in military institutions, more can be done to enhance the experience. In the spirit of Vygotsky’s (1978) ZPD, the implementation of wikis that allow for student collaboration in a self-paced online environment can motivate students, increase the potential for learning, and enhance the overall learning experience. Appendix A.

Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

16

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

Intercultural competence lesson Situational judgment tests (lesson 5) A summary 1. Knowledge/Skills: Collectivist Culture Summary: Scenario is based on Gen Zinni’s struggle to work with Kurdish refugees on the Iraq–Turkey border in the aftermath of Desert Storm. You are asked to decide which group to work with to communicate with the refugees: (a) Tribal leaders, (b) Western-educated professionals, (c) Christian missionaries and Peace Corps Volunteers, or (d) European/Middle Eastern/Indian business entrepreneurs/other expatriates who speak the local language 2. Knowledge/Skills: Collectivist Culture Summary: You ask a local mayor to provide workers for a community project, but the workers are inadequate. Does the town tend to have: (a) Collectivist cultural values, (b) Nuclear family system cultural values, (c) Clock time cultural values, or (d) Individualist cultural values? 3. Knowledge/Skills: Low/High Context Summary: You are a USAID worker whose car breaks down in a rural area of a developing nation. You ask an old man in a nearby village how long it would take to walk to your destination. He says you can get there in about 4 hours, but in 4 hours you can only get to the next village where you spend the night. The old man: (a) Had a high-context/indirect communication style, (b) Didn’t know how to tell time, or (c) Was anti-American, or (d) Didn’t realize how slowly Americans walk 4. Knowledge/Skills: Importance of Respect Summary: Col Blaine Holt (now Gen Holt) while commander of Manas Transit Center, Kyrgyz Republic, uncovered several reasons for anti-American sentiment. Which was the most damaging? (a) The U.S. doesn’t respect the Kyrgyz people, (b) Manas’ association with the oppressive government, (c) U.S. money went to Kyrgyz government, not local population, or (d) Little U.S. aid trickled down to the people who needed it 5. 3C Attitude: Self-regulation Summary: You are mentoring the national police force of a developing nation to develop protocols for natural disasters. However, locals don’t seem to be interested in helping out. They tend to believe there’s no point to doing anything since all natural disasters are acts of God, as is survival afterward. You should: (a) Explain to people they already take steps to combat the elements, (a) Ask national government to help, (c) Rank communities by their level of compliance. Those who rank highest get aid first, or (d) Nothing can be done. The project is doomed 6. 3C Attitude: Self-regulation Summary: You are ordered to investigate charges that a U.S. airbase has caused environmental contamination in a developing country. You learn that it’s not the airbase, but a drought causing the problem. However, local populace is skeptical of Western science. You should: (a) Talk to officials and their families; listen to their complaints. Resolution will take time. (b) Tell officials their data is inaccurate. (c) Show empirical scientific evidence that proves your case, or (d) Your evidence is irrefutable. No need to meet with lower level officials. Go see the Minister of Health 7. 3C Attitude: Self-regulation Summary: An atypical snowstorm on an island nation is blocking the runway on a U.S. airbase. You have no heavy equipment. Locals have strong collectivist and spiritual (rather than secular) value system. The snow is a bad omen. They won’t help unless a shaman blesses the airstrip. You should: (a) Have a shaman bless the airbase, (b) Go to highest approval authority and request assistance, (c) Assume it’s a face-saving issue because equipment is damaged. See if Air Force can repair it. (d) Seek alternatives. Superstition shouldn’t get in the way of the mission. Knowledge/Skills: Low/High Context Summary: You are in charge of a construction project in a country with a strong tribal tradition. The land for the project has cultural significance for a nearby village, so they must give you permission. When meeting with the village elders, you notice that some young men are spokespersons. The elders seem to nod in agreement when you ask for permission, so you assume that all is well. But when the contractor arrives to begin the project, entry is blocked. The best explanation is: (a) The locals have a high-context communication style and were loath to say no directly. (b) Because the local culture is hierarchical, you should not have talked with the young spokespersons. (c) People in high-context cultures often change their minds. (d) The decision was superseded by a powerful elder after the meeting ended. 8. 3C Attitude: Self-regulation Summary: As the newly designated TALCE commander of a humanitarian effort in a developing nation, you have been asked to report to your commander upon arrival in-country. However, when you arrive, you learn that the Minister of Health wants to meet with you first. Should you send word that: (a) You’re honored and at his disposal. (b) Pass on your apologies, but you’ve been ordered to report to your commander. (c) You’re honored, but you’ve been ordered to report to your commander. Can you meet with him later? (d) You’re happy to meet with him, but it must brief. You’ve been ordered to report to your commander. Knowledge/Skills: Low/High Context Summary: You must organize the repair of a U.S. airfield in a developing country and hired a local contractor, but local tribesmen started demonstrating—a first for the airbase. The contractor hired workers from a rival tribe. He says the locals have no skilled workers. You meet with a tribal elder who talks about tribal culture and the tribe’s loyalty to the airbase, but not about the demonstrations. When asked, he says “Honor is like money.” The elder is: (a) Telling you indirectly that the airbase’s relationship with the tribe is in jeopardy. (b) Telling you indirectly the problem is a minor one since he talked about loyalty and honor. It would be disloyal and dishonorable to break trust. (c) Telling you indirectly there is no problem since he didn’t respond to your question about the demonstrations. (d) Telling you indirectly that he is willing to accept a bribe. Knowledge/Skills: Time Orientations; Low/High Context Summary: You’re an Acquisitions Officer in a developing country who must negotiate a utilities contract within 3 months. A government agency tells you to obtain permits for the work. Your POC assures you it’s an easy process. However, you are continually required to obtain letter after letter from local agencies. Angry, you do some research and determine the culture tends to be polychronic with a high-context communication style. This helps you understand that: (a) Your POC did not feel a strong urgency to get the permits quickly and also didn’t want to give you bad news directly. (b) In high-context cultures, the more assertive you are, the better. (c) High context cultures are unfazed by an outsider’s anger. (d) Bureaucracies in such cultures tend to add on unnecessary requirements arbitrarily. Knowledge/Skills: Collectivist culture Summary: You’re an ACSC student and are friends with an IO in your class who talks about his extended family. You were placed in the same group to complete a class assignment. You listen to your friend’s briefing, give him some pointers, and ask difficult questions to prepare him. Next day, however, he skipped a planned dinner with you. What’s the problem? (a) His “face” was damaged by your criticism in front of others. (b) He comes from an individualistic culture and felt he didn’t get enough time to practice with the group. (c) He’s overly sensitive. It has nothing to do with culture. (d) He comes from a culture that tends to value low-context communication. He wanted more feedback from your group.

References Abbe, A., & Halpin, S. (2009–2010). The cultural imperative for professional military education and leader development. Parameters, (Winter), 20–31. ACSC DL 6.0., 2012. ACSC DL 6.0 ORIN Lesson 5 progress check: Item 9. [Test item] Maxwell AFB, AL. Air Command and Staff College.

Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002

G Model IJIR-1163; No. of Pages 17

ARTICLE IN PRESS J.W. Miller, J.S. Tucker / International Journal of Intercultural Relations xxx (2015) xxx–xxx

17

ACSC DL 6.0. (March–June 2013). End-of-course survey results: Orientation and introduction course. Maxwell AFB, AL. ESS. Ambady, N., LaPlante, D., & Johnson, E. (2001). Thin slice judgments as a measure of interpersonal sensitivity. In J. A. Hall, & F. J. Bernieri (Eds.), Interpersonal sensitivity: theory & measurement. Mahwah, NJ: Lawrence Erlbaum Publishers. Anastasi, A. (1988). Psychological testing. New York: Macmillan. Baumeister, R. S. (2000). Self-regulation and depletion of limited resources: does self-control resemble a muscle. Psychological Bulletin, 126(2), 247–259. Bennett, J. M. (2009). Transformative training: designing programs for culture learning. In Michael Moodian (Ed.), Contemporary leadership and intercultural competence (pp. 95–110). Thousand Oaks, CA: Sage. Bennett, M. J. (2013). Intercultural practices: international/multicultural education. In Milton J. Bennett (Ed.), Basic concepts of intercultural communication: paradigms, principles, and practices (2nd ed., pp. 105–136). Boston, MA: Intercultural Press. Boyd, J.R., 1995. The essence of winning and losing [PowerPoint slides]. Defense and the National Interest. Retrieved from . Cushner, K., & Brislin, R. (1996). Intercultural interactions: a practical guide (2nd ed.). Thousand Oaks, CA: Sage Publications. Deardorff, D. K. (2004). Internationalization: in search of intercultural competence. International Educator, 13, 13–15. Deardorff, D. K. (2006). Identification and assessment of intercultural competence as a student outcome of internationalization. Journal of Studies in Intercultural Education, 10, 241–266. Deardorff, D. K. (2009). Implementing intercultural assessment. In D. K. Deardorff (Ed.), Sage handbook of intercultural competence (pp. 477–492). Thousand Oaks, CA: Sage. Deci, E. L., & Ryan, R. M. (Eds.). (2002). Handbook of self-determination research. In. Rochester NY: University of Rochester Press. Dewey, J. (1910). How we think. Chicago: D. C. Heath and Co. Ennis, R. H. (1979). A conception of rational thinking. In J. R. Coombs (Ed.), Philosophy of education 1979: proceedings of the thirty-fifth annual meeting of the philosophy of education society (pp. 3–30). Bloomington, IL: Philosophy of Education Society. Facione, P. A. (1990). The Delphi report—critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: California Academic Press. Facione, P. A. (2010). Critical thinking: what it is and why it counts. Millbrae, CA: Peter A. Facione, Measured Reasons and the California Academic Press. Glaser, E. M. (1941). An experiment in the development of critical thinking. New York: AMS Press. Gold, S. G. (2001). A constructivist approach to online training for online teachers. Journal of Asynchronous Learning Networks, 5(1), 35–57. Hall, E. T. (1976). Beyond culture. New York: Anchor Books. Hall, E. T. (1981). The silent language. New York: Anchor Books, Doubleday. Harasim, L. (2012). Learning theory and online technologies. New York: Routledge. Hammer, M. (2012). The intercultural development inventory: a new frontier in assessment and development of intercultural competence. In M. Vande Berg, R. M. Paige, & K. H. Lou (Eds.), Student learning abroad (pp. 115–136). Sterling, VA: Stylus Publishing (Chapter 5) Hauenstein, N. M. A., Findlay, R. A., & McDonald, D. P. (2010). Using situational judgment tests to assess training effectiveness: lessons learned evaluating military equal opportunity advisor trainees. Military Psychology, 22, 262–281. Hofstede, G. (1991). Cultures and Organizations: Software of the Mind. Europe; Maidenhead, Berkshire, UK: McGraw-Hill Book Co. Lantolf, J. P., & Thorne, S. L. (2006). Sociocultural theory and the genesis of second language development. Oxford, UK: Oxford University Press. Livermore, David. (2010). Leading with Cultural Intelligence. New York: AMACOM. Mackenzie, L., Fogarty, P., & Khachadoorian, A., 2013, September 4. A model for online military culture education: Key findings and best practices. Educause, Retrieved from: . Mackenzie, L., & Wallace, M. (2013). Cross-cultural communication contributions to professional military education: a distance learning case study. In R. G. Sands, & A. Greene-Sands (Eds.), Cross-cultural competence for a 21st century military: culture, the flipside of COIN (pp. 239–258). Lanham, MA: Lexington Books. Malone, T. W., & Lepper, M. R. (1987). Making learning fun: a taxonomy of intrinsic motivations for learning. In R. E. Snow, & M. J. Farr (Eds.), Aptitude, learning, and instruction, III: conative and affective process analysis (pp. 223–253). Hillsdale, NJ: Lawrence Erlbaum Associates. Miller, J. W. (2011). The face in the machine: the challenge of online teaching and learning. In Proceedings of distance learning instructor colloquium. Maxwell AFB, AL: Air Command & Staff College (June) Ng, R., & Rayner, S. (2010). Integrating psychometric and cultural theory approaches to formulate alternative measure of risk perception. Innovation: The European Journal of Social Science Research, 23(2), 85–100. Osinga, F. P. B. (2007). Science, strategy and war: the strategic theory of John Boyd. New York: Routledge. Paul, R. (1993). Critical thinking: how to prepare students for a rapidly changing world. Berkeley, CA: Foundation for Critical Thinking. Rosenthal, D. B., Wadsworth, L., Paulin, C., Hooper, A. C., & Bhawuk, D. P. S. (2009). Navigating the human terrain: development of cross-cultural perspective taking skills. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. Selmeski, B. (2009). Air university quality enhancement plan, 2009–2014: cross-culturally competent airmen. Maxwell AFB, AL: Air Force Cultural & Language Center. Ting-Toomey, S. (1999). Communication across cultures. New York: Guilford Press. Triandis, H. C. (1995). Individualism and collectivism. Boulder. CO: Westview Press. Turner, J., & Paris, S. G. (1995). How literacy tasks influence children’s motivation for literacy. The Reading Teacher, 48(8), 662–673. Van Dyne, L., Ang, S., & Livermore, D. (2010). Cultural intelligence: a pathway for leading in a rapidly globalizing world. In K. M. Hannum, B. McFeeters, & L. Booysen (Eds.), Leadership across differences: cases and perspectives. San Francisco, CA: Pfeiffer. Vygotsky, L. (1978). Mind in society: the development of higher psychological processes. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.),. Cambridge, MA: Harvard University Press. Waugh, G. W. (2004). Situational judgment test. In D. J. Knapp, R. A. McCloy, & T. S. Heffner (Eds.), Validation of measures designed to maximize 21st-century army NCO performance (ARI technical report 1145) (pp. 6-1–6-25). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences (DTIC No. ADA 423602). Weekley, J. A., Ployhart, R. E., & Holtz, B. C. (2006). On the development of situational judgment tests: issues in item development, scaling, and scoring. In J. A. Weekley, & R. E. Ployhart (Eds.), Situational judgment tests: theory, measurement, and application (pp. 157–181). Mahwah, NJ: Erlbaum. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: an overview of current research. Human Resource Management Review, 19, 188–202. Zinni, T., & Klotz, T. (2007). The battle for peace: a vision of America’s power and purpose. New York: Palgrave Macmillan.

Please cite this article in press as: Miller, J. W., & Tucker, J.S. Addressing and assessing critical thinking in intercultural contexts: Investigating the distance learning outcomes of military leaders. International Journal of Intercultural Relations (2015), http://dx.doi.org/10.1016/j.ijintrel.2015.07.002