Differentiating between static and complex problems: A theoretical framework and its empirical validation

Intelligence 72 (2019) 1–12

Contents lists available at ScienceDirect

Intelligence journal homepage: www.elsevier.com/locate/intell

Differentiating between static and complex problems: A theoretical framework and its empirical validation☆ Matthias Stadlera,b, Christoph Niepelb, Samuel Greiffb, a b

T

⁎

Ludwig-Maximilians-Universität München, München, Germany University of Luxembourg, Esch-sur-Alzette, Luxembourg

A R T I C LE I N FO

A B S T R A C T

Keywords: Complex problem solving Intelligence Problem solving Framework Cognitive abilities Structural equation modeling

Ever since its first conception, the concept of complex problem solving (CPS) has been debated regarding its role within the conception of human intelligence. The aim of the current paper was to theoretically introduce and empirically test a multifaceted framework of CPS that subsumes different positions and provides testable predictions on the nature of CPS. Following a review of the existing literature on complex problem solving, we conclude that it is necessary to differentiate between the dimensions of connectivity and dynamics. These dimensions are further distinguishable for both phases of CPS, knowledge acquisition and knowledge application, resulting in four facets. Static problems, as used in conventional measures of intelligence, on the other hand, do not include dynamics as much as complex problems. We argue that the differences in CPS and static problem solving proposed by various researchers result from the dimension of dynamics that is highly relevant for CPS but less so for static problem solving. An empirical analysis based on two independent samples supported the assumptions made by the framework. This brings substantial implications for the understanding of CPS as well as the interpretation of previous research, which we discuss.

“Intelligence is adaption” (Piaget, 1950) 1. Introduction The concept of intelligence is generally understood as entailing the “ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, [and] to overcome obstacles by taking thought” (Board of Scientific Affairs of the American Psychological Association; Neisser et al., 1996, p. 77). Thus researchers have long since argued that a major aspect of intelligence is the ability to solve problems, and that careful analysis of problem solving behavior constitutes a means of specifying many of the psychological processes that intelligence comprises (Resnick & Glaser, 1976). Conventional measures of intelligence are, however, mostly state-oriented, employing static problems that do not react to the test-taker or change over time (e.g., Guthke & Stein, 1996; Ren, Wang, Altmeyer, & Schweizer, 2014). Such tests are unable to reflect how people adapt to new problems or provide any opportunity to learn from experience. If “intelligence is what the tests test” (Boring, 1923) though, this implies a mismatch between the conceptual

understanding of intelligence and its empirical operationalization through conventional tests of intelligence. In his seminal work, Dietrich Dörner (e.g., Dörner, 1974) suggested the introduction of complex problem solving (CPS) as a cognitive ability not sufficiently captured by conventional measures of intelligence. CPS describes the ability to solve problems that explicitly require active interaction with the problem to learn from experiences and adapt to new problem situations. Measuring CPS thus requires extending the static problems that make up conventional measures of intelligence towards complex problems that react to the test takers' interactions. Within this paper, we aimed at presenting a framework that combines both static and complex problem solving as essential aspects of intelligence. In that, the current framework of CPS will be refined towards a multifaceted conception of CPS. We will argue how this multifaceted conception can be used to explain the intricate empirical relation between measures of CPS and conventional measures of intelligence (Stadler, Becker, Gödker, Leutner, & Greiff, 2015) on both a theoretical and an empirical level. In the second part of this paper, the utility of this newly developed framework for describing the relation between measures of CPS both to conventional measures of intelligence and to real-world outcomes will be investigated empirically.

☆ ⁎

This research was funded by grants of the Fonds National de la Recherche Luxembourg (ATTRACT “ASKI21”). Corresponding author at: ECCS, University of Luxembourg, Maison des Sciences Humaines, 11 Porte des Sciences, 4366 Esch-sur-Alzette, Luxembourg. E-mail address: samuel.greiff@uni.lu (S. Greiff).

https://doi.org/10.1016/j.intell.2018.11.003 Received 24 January 2018; Received in revised form 12 October 2018; Accepted 7 November 2018 0160-2896/ © 2018 Elsevier Inc. All rights reserved.

Intelligence 72 (2019) 1–12

M. Stadler et al.

2. The theory of problem solving

problems thus allow only for changes in the mental representation of the problem and therefore provide only limited opportunities to learn from experience (within one problem; for learning that occurs over the course of multiple static problems see e.g., Birney, Beckmann, Beckmann, & Double, 2017). Correspondingly, static problems alone cannot fully reflect how people adapt to new problems, which represents a key aspect in most conceptions of intelligence (Neisser et al., 1996). To encompass the concept of intelligence through problem solving, it is therefore necessary to extend the static problems towards more complex problems that do require active manipulation in order to be solved.

In both the everyday world and the psychological laboratory, the ability to solve practical problems is generally regarded as a core cognitive activity and a direct expression of intelligence (Resnick & Glaser, 1976; Sternberg, 1985; Vickers, Lee, Dry, & Hughes, 2003). Solving any problem can, generally speaking, be described as the transition of a system from a current state to a goal state that cannot be achieved by the mere application of routine tasks (Jonassen, 2000). Rather, this transition requires a goal-directed sequence of cognitive operations (Anderson, 1993) with two critical attributes. First, problem solving requires the mental representation of the situation. That is, problem solvers construct a mental model of the problem, known as the problem space (Newell & Simon, 1972). Second, problem solving requires some activity-based manipulation of the problem space, be it an internal mental representation (that can be changed by thinking about it) or an external physical representation (that needs to be interacted with). The terms knowledge acquisition and knowledge application have been established for the formation and the manipulation of the problem space (e.g., Fischer, Greiff, & Funke, 2012). For a practical illustration of these theoretical concepts, take the classical figural matrix item depicted in Fig. 1, which is prototypical for many items in conventional measures of intelligence. The item consists of a 3 × 3 matrix of cells with geometrical symbols. These symbols follow certain rules. In the example, the arrow is rotated by 90° in a counterclockwise direction across each row of the matrix. The circles follow an addition rule in that the circles that appear in the first and second cells of a row are summed in the third cell. The last cell of the item is left empty. To solve this item, the test taker needs to determine which of the response options in the lower part of the item completes the matrix. This requires systematically comparing the separate cells of each row (i.e., knowledge acquisition) in order to find an abstract representation of the rules underlying the cells of each row (i.e., the problem space). Only then can the newly created knowledge be used (i.e., knowledge application) in order to manipulate the problem space according to the determined rules and find the missing cell. Solving a problem like the figural matrix presented in Fig. 1 constitutes an example of static problem solving (Berbeglia, Cordeau, & Laporte, 2010). Static problems, as featured in virtually all conventional measures of intelligence, provide the problem solver with all information necessary to solve the problem (Jonassen, 2000) and do not change over time. It is important to note that this does not make any implications about the problem's difficulty. A static problem can be very simple or extremely difficult. Characterizing them as static problems merely describes the fact that they do not require active interaction (i.e., a physical manipulation by the test taker) or autonomously occurring changes within the problem (Berbeglia et al., 2010). Static

3. CPS as an extension to static problem solving In the early phases of CPS research, five typical qualities of a complex problem were distinguished (Dörner, 1980): (a) complexity, (b) connectivity, (c) dynamics, (d) intransparency (opaqueness), and (e) polytely (i.e., a problem situation having many goals at the same time). Funke (2001) argued however that, first, the (a) complexity and (b) connectivity features are difficult to be distinguished considering the unclear definition of complexity (often understood as “number of variables within a system”). Funke (2001) therefore suggests concentrating on the more comprehensible term “connectivity”. Connectivity is understood as dependency between two or more variables. Connectivity is an important feature of complex systems (Casti, 1979) and requires understanding and handling the connections between the variables; that is, to construct a causal model of the system under consideration. The second important characteristic of a complex problem is (c) dynamics. Complex problems change as a result of the problem-solvers interventions and over time (Frensch & Funke, 1995). They thus require problem-solvers to figure out how the system develops or changes over time and what the short- and long-term effects of specific interventions are. Whereas the aspect of connectivity is related to the structural relationships within a system, the aspect of dynamics is thus related to processes within a system. Finally, the features (d) intransparency and (e) polytely (multiple goals) are not aspects inherent in a problem, but refer to certain decisions of the experimenter, namely, how much information about the system will be given to a subject and what goals the subject is instructed to follow. Because these two features could also be used in static experimental set-ups, they are not specific to systems used in CPS research. One can say that these two features are often part of a complex problem and need to be considered in its design but they are not essential features of the problem, like connectivity or dynamics. Fig. 2 illustrates the underlying structure of a typical complex problem solving task following the MicroDYN approach (Greiff,

Fig. 1. Exemplary matrix task (adapted with permission from Becker et al., 2016).

Fig. 2. Structure of a typical MicroDYN item displaying 3 input (A, B, C) and 3 output (X, Y, Z) variables. 2

Intelligence 72 (2019) 1–12

M. Stadler et al.

in a complex problem, though. Dynamic changes in a system occur from changes within the problem over time rather than due to singular manipulations of the problem. This can either happen through timedelayed interventions by the problem solver or through elements of the system affecting itself (eigendynamics). For an example of time-delayed intervention, consider the temperature setting of a shower. After setting the control to a certain value, the change in temperature does not occur immediately but takes some time. A typical example for eigendynamics on the other hand would be the interest accrual on a bank account. The money in the account will increase over time without any external intervention. Separating dynamic changes in a system from those resulting from external manipulation thus requires observing the problem's change of state over time without external input affecting the system. Such changes within the problem, without any external causes, cannot possibly occur in a static problem, as they require time to be a relevant aspect of the problem (Funke, 2001; Guthke & Stein, 1996). Dynamics should thus distinguish complex problems considerably more distinctly from static problems than connectivity. The resulting multifaceted framework of CPS distinguishing between the dimensions of connectivity and dynamics is illustrated in Fig. 3. As can be seen, CPS and static problem solving share the broad dimensions of connectivity but the dimension of dynamics is proportionally less important for static problem solving than for CPS. On a more specific level, connectivity and dynamics each need to be separated into facets for both the knowledge acquisition and the knowledge application phase resulting in four different cognitive demands (Rigas & Brehmer, 2000). Both the connectivity and dynamics in a complex problem need to be detected in the knowledge acquisition phase and then handled in the knowledge application phase. Static problems on the other hand require mostly the detection and handling of connectivity and include only limited dynamics. Despite several differences regarding the detection and handling of connectivity in static and complex problems, this demand is shared between the two different processes. Detecting and handling dynamics is however far more important to solving complex problems; static problem solving and CPS overlap only little in regards to this demand.

Wüstenberg, & Funke, 2012), one of the most recent developments in CPS assessment (for an overview see Greiff, Fischer, Stadler, & Wüstenberg, 2015). The task consists of three input variables (A to C) and three output variables (X to Z), which are interrelated to some degree (as indicated by the arrows). In this example, changes in A would result in corresponding changes in X and Y but not affect Z. In addition, there is a dynamic change in X, which is independent of any external manipulation. The test taker's task is to first figure out this underlying structure through active manipulation of the input variables and draw it correctly. They then need to utilize this knowledge to reach certain target values. In the actual testing situation, all tasks are given some arbitrary cover story such as coaching a handball team and investigating how three different types of training (input variables) affect three different performance measures (the output variables). The cover stories are kept generic to avoid any influences of previous knowledge or certain expectations. Understanding complex problems (i.e. constructing a mental model of the problem space in a complex problem that is defined by connectivity and dynamics) requires active acquisition of knowledge (Novick & Bassok, 2007) through interaction with the problem. The resulting changes need to be registered and integrated into the mental model. Only after this knowledge is acquired can it be applied to manipulate the state of the problem towards a specific goal state. Just as for static problem solving, it is useful to distinguish between the related phases of knowledge acquisition and knowledge application for a general description of CPS (e.g., Wüstenberg, Greiff, & Funke, 2012). The defining cognitive difference between solving static and complex problems must therefore lie within the concepts of connectivity and dynamics, which can be considered broad dimensions of CPS. 4. A multifaceted framework of CPS To pinpoint how connectivity and dynamics in CPS may cause the difference between static and complex problem solving, it is helpful to first consider connectivity in static problem solving. Klauer (1996) suggests that all static problem-solving tasks consist of detecting regularities and irregularities by finding similarity or difference of attributes or relations with verbal, figural, numerical, or other material (i.e., knowledge acquisition; see also Klauer & Phye, 2008). In this way, a mental model of the problem and its regularities is created. Once a set of common rules is determined, these rules need to be applied to mentally manipulate the problem in order to generate a solution (i.e., knowledge application)1. This strategy of comparing all objects of a static problem systematically to determine the underlying rules can be directly adapted to the dimension of connectivity in CPS. Detecting connectivity in a complex problem requires actively and systematically manipulating the problem. The resulting changes in the problem's current state then need to be incorporated into a mental model of the system's underlying structure and rules. Only then can the acquired knowledge be applied to reach a certain goal state of the complex problem (i.e., handling connectivity). This, again, can only be done in several steps under constant monitoring of changes in the problem's current state and comparing it to the goal state. Whereas this strategy resembles the strategy suggested for static problems (Klauer & Phye, 2008), its active components, as realized in CPS, should add a cognitive demand that is not included in static problem solving (Funke, 2001). The problem space of complex problems must be actively manipulated resulting in changes in the problem state that need to be registered rather than merely mentally anticipating changes in the problem as would suffice with static problems. Neither of these strategies is adequate to detect or handle dynamics

5. CPS and static problem solving Historically, the relation between static problem solving and CPS has been investigated by relating scores in conventional measures of intelligence to scores in measures of CPS. This line of research was mostly fueled by Dörner's (1974) initial theoretical and empirical proposition that CPS as a construct would be (somewhat) independent of intelligence. Theoretically, this view was summarized by the differentdemands hypothesis (Rigas & Brehmer, 2000). To explain the weak correlations that researchers observed between conventional measures of intelligence and CPS performance, the different-demands hypothesis suggests that CPS tasks demand the performance of more complex mental processes than conventional measures of intelligence do. These different demands should then result in low empirical correlations between the two assumed constructs. In contrast, several researchers argued that CPS would be almost redundant to intelligence (Wittmann & Süß, 1999) reporting medium to large correlations between the two constructs (for more recent studies see Kretzschmar, Neubert, Wüstenberg, & Greiff, 2016; Lotz, Sparfeldt, & Greiff, 2016). They argued that all apparent differences between the two constructs were merely due to nonperfect reliabilities of the measures (Rigas, Carling, & Brehmer, 2002). The multifaceted framework introduced above can explain these seemingly contradictory results and advance our understanding of the role of CPS within the construct of intelligence. Previous studies investigating the relation between CPS and conventional measures of intelligence were often based on empirically deducted assumptions and considerations of plausibility rather than a detailed theoretical basis. Correspondingly, they used either single

1 Note that Klauer (1996) used the terms “inductive reasoning” for knowledge acquisition and “inductive inferring” for knowledge application.

3

Intelligence 72 (2019) 1–12

M. Stadler et al.

Fig. 3. Graphical illustration of the multifaceted framework.

problem solving in the prediction of real-world outcomes.

indicators of CPS or merely separated the two phases of knowledge acquisition and knowledge application (Stadler et al., 2015). Based on our framework, this should lead to a biased estimate of the correlation between static problem solving and CPS. The cognitive processes involved in solving complex problems on a broad level are very similar to those involved in solving static problems as used in conventional measures of intelligence such as Raven matrices (Raven, 1938) or number series (Funke, 2001). Solving both complex and static problems requires acquiring and applying knowledge about a new and unknown problem space (Carpenter, Just, & Shell, 1990; Funke, 2001; Wiley, Jarosz, Cushen, & Colflesh, 2011). It is thus plausible to expect high correlations between CPS and conventional measures of intelligence on this broad level. This is supported empirically by meta-analytic findings (Stadler et al., 2015) showing corrected correlations of up to ρ = 0.72 between broad indicators of CPS and conventional measures of intelligence, although the magnitude of this correlation was moderated by the way CPS was assessed. Measures with higher degrees of dynamics showed substantially smaller relations to conventional measures of intelligence (for a discussion on the measurement of CPS, see Greiff, Fischer et al., 2015 as well as Greiff, Stadler, Sonnleitner, Wolff, & Martin, 2017). This is in line with the predictions made by the multifaceted framework of CPS (Fig. 3) as the cognitive processes involved in the CPS facets of detecting and handling connectivity are substantially closer to the cognitive processes involved in static problem solving than the cognitive processes involved in the CPS facets of detecting and handling dynamics. A clear separation of the facets of CPS should accordingly provide a more detailed picture with large correlations between connectivity and static problem solving and substantially smaller correlations between dynamics and static problem solving. In other words, the different-demands hypothesized for CPS almost 20 years ago (Rigas & Brehmer, 2000) that constitutes the added value of complex problems over static problems might hold after all, but could predominantly be driven by the dimension of dynamics in CPS.

Hypothesis 1. Modeling connectivity and dynamics as independent dimensions of complex problem-solving leads to a significantly improved measurement model. Based on the multifaceted framework presented above, detecting and handling connectivity involves different cognitive demands than detecting and handling dynamics. These two defining characteristics of complex problems should thus result in four separable facets of CPS. So far, CPS has been conceptualized and modeled as a two dimensional construct with knowledge acquisition and knowledge application as the related but distinguishable factors (Wüstenberg et al., 2012).2 Separating knowledge acquisition and knowledge application within the two dimensions of connectivity and dynamics (resulting in a four facetted model), should lead to a considerably improved representation of the empirical data. Hypothesis 2. The correlation between static problem solving and connectivity is higher than the correlation between static problem solving and dynamics. Regarding the relation between CPS and static problem solving, we expect to find differences between the four facets. While there are important differences, connectivity requires cognitive processes that are relatively similar to those necessary to solve reasoning tasks (Klauer & Phye, 2008; Wüstenberg et al., 2012). Detecting and handling dynamics on the other hand requires problem solvers to include time as a relevant factor in their mental model of the problem (Funke, 2001). Thus, we expect to find the relation between static problem solving and the detection and handling of connectivity to be substantially larger as opposed to the relation between static problem solving and the detection and handling of dynamics. Hypothesis 3. Connectivity and dynamics show incremental validity over and above each other and static problem solving in the prediction of school grade.

6. Empirical validation

Finally, it is necessary to investigate whether a separation of connectivity and dynamics has any impact on the validity of CPS in the prediction of real-world outcomes. Since a number of studies exist on the relation between CPS tasks and school grades (Greiff, Fischer et al., 2015; Stadler, Becker, Greiff, & Spinath, 2016; Wüstenberg et al., 2012), school grades represents an excellent external criterion to

6.1. Hypotheses In the empirical section of this paper, we aim to test the multifaceted framework of CPS developed above. The framework posits several testable predictions regarding the internal structure of CPS as well as the relation between CPS and static problem solving. We expect a separation of connectivity and dynamics to not only result in an improved representation of the empirical CPS data but also to allow a more detailed view on the relation between CPS and static problem solving as well as the incremental validity of CPS over and above static

2 Note that some publications also modeled a third “strategy” factor that has however been shown to be largely redundant to knowledge acquisition (Wüstenberg et al., 2012).

4

Intelligence 72 (2019) 1–12

M. Stadler et al.

their assumptions about relations between input and output variables in a causal model. In phase 2, the correct model is provided to the students. They are asked to reach target goals for the output variables in no more than four steps by manipulating the input variables accordingly. The target goals are shown as numbers in brackets and as a red line (time frame: 90 s). A comprehensive description of MicroDYN can be found in (Greiff, Fischer et al., 2015) and (Wüstenberg et al., 2012). Six of the tasks used in this study (Task 1–5 and Task 7) included only connectivity while the other three (Tasks 6, 8, and 9) included both connectivity and dynamics. In the knowledge acquisition phase, credit was given if the causal model drawn by students was correct; otherwise, no credit was assigned. In the knowledge application phase, credit was given if all goals were reached using a maximum of four steps; otherwise, no credit was assigned. Overall, the scoring procedures resulted in 18 indicators of CPS performance (capturing knowledge acquisition and knowledge application). 6.2.1.2.3. School grades. Students' grade point average (GPA) was calculated as the mean of their Finnish (mother tongue), math, history, chemistry, and English grades. The grades ranged from 0 to 10 with higher values representing better performance.

investigate whether there is an incremental value of the four facets over and above each other and static problem solving in the prediction of school grades. Based on the framework presented above, we expect to find incremental validity for dynamics and connectivity over and above static problem solving. This incremental validity should however be considerably stronger for dynamics, which share far less overlap with static problem solving, than for connectivity. To investigate these hypotheses, we conducted two studies based on independent samples. 6.2. Study 1 6.2.1. Method and procedure 6.2.1.1. Participants3. In total, our sample consisted of 1476 students attending the ninth grade (668 males, 701 females, 107 missing gender; age: M = 15.24, SD = 0.47) in one Finnish municipality. Students were sampled to be representative for the general Finish population based on several aspects such as socio-economic status as well as parents' education. 6.2.1.2. Measures 6.2.1.2.1. Static problem solving. Static problem solving was measured by a verbal reasoning test and a quantitative reasoning test to ensure a sufficiently comprehensive assessment of cognitive ability (Brunner, 2008). The verbal reasoning task included 8 items and was adapted from the Ross Test of Higher Cognitive Processes (Ross & Ross, 1979). Students were required to identify a valid statement out of five alternatives based on a given premise and conclusion. Items were scored dichotomously as either correct or incorrect. The quantitative reasoning task was an adapted version of Sternberg's Triarchic Test (Hversion) Creative Number Scale (Sternberg, Castejon, Prieto, Hautamäki, & Grigorenko, 2001). In this test, an arithmetical operator is conditionally defined depending on the values of the numbers in the task. More specifically, mathematical expressions (e.g., x lag y = x + y, if x < y; otherwise x lag y = x – y), tasks (e.g., what is 4 lag 7?), and alternatives (e.g., −3, 3, 11, −11) are presented. In this example x (4) is smaller than y (7) and ‘lag’ therefore stands for a plus sign. The solution would thus be 11. The 7 items measuring quantitative reasoning were scored dichotomously as either correct or incorrect. 6.2.1.2.2. Complex problem solving. CPS was measured using nine computer-based problem-solving tasks following the MicroDYN approach (Greiff et al., 2012; Wüstenberg et al., 2012). The MicroDYN approach is particularly well suited for our study as it allows for a clear separation of knowledge acquisition and knowledge application on the one hand and to include items with and without dynamics on the other hand. Tasks based on the MicroDYN approach were part of the assessment of cross-curricular problem solving skills in the PISA 2012 survey and showed sufficient convergent (Greiff et al., 2013) and divergent validity (Schweizer, Wüstenberg, & Greiff, 2013; Wüstenberg et al., 2012). MicroDYN is based on linear structural equations (Funke, 2001) in which (in this study) up to three input variables were related to up to three output variables. Relations were opaque to students, and on some of the tasks, the output variables changed dynamically without specific manipulations of the input variables from the students. The students' task was to apply adequate strategies to acquire knowledge about the problems' structure and to apply that knowledge to achieve certain goals. The assessment procedure can be divided into two phases. In phase 1, students have to use adequate strategies to find out how variables are related (time frame: 180 s). While exploring, students are asked to plot

6.2.1.3. Procedure. Testing took place in the schools' computer rooms, and test administration was divided into two sessions. In the first session, students worked on a test battery comprising of the measures of reasoning and demographic questionnaires as well as several other cognitive and noncognitive measures not used in this study (90 min). Approximately 1 week later, students worked on the CPS tasks (45 min) in a second session. 6.2.1.4. Statistical analyses. We applied confirmatory factor analyses (CFA) and structural equation modeling (Bollen, 1989) to examine measurement models and latent regressions as well as structural models to investigate our research questions. Weighted least squares means and variance adjusted (WLSMV) estimation was applied for all models because CPS scores were based on categorical indicators. SatorraBentler corrections were applied to all X2 values for model comparisons (Satorra & Bentler, 2010). Model fit was evaluated using standard model fit indices such as the root mean square error of approximation (RMSEA), the confirmatory fit index (CFI) and the Tucker Lewis Index (TLI). Analyses were conducted using Mplus 7.11 (Muthén & Muthén, 1998-2012). 6.2.2. Results 6.2.2.1. Measurement models. To investigate whether the separation of knowledge acquisition and knowledge application facets for connectivity and dynamics would lead to a better measurement model for CPS (Hypothesis 1), we chose the 2-dimensional model of CPS suggested by Wüstenberg et al. (2012) as a starting point. In this model, all 9 items measuring knowledge acquisition load onto one factor regardless of whether the items include dynamics or not. Correspondingly all 9 items measuring knowledge application load onto another factor. The two resulting factors are supposed to be highly correlated. Error correlations were allowed between the indicators of knowledge acquisition and knowledge application for each task. As was to be expected, the 2-dimensional measurement model fit well to our data (Table 1). All items loaded significantly and meaningfully on their respective factors and the two factors of knowledge acquisition and knowledge application were strongly correlated (r = 0.82; p < .001). This replicated the results of Wüstenberg et al. (2012) as well as various other studies (e.g. Greiff, Stadler, Sonnleitner, Wolff, & Martin, 2015; Kretzschmar et al., 2016; Stadler et al., 2017)4. 4 The two-dimensional measurement model has already been reported in other publications based on the same data but is repeated here as a starting point.

3

Please note that other studies, investigating different research questions, have been conducted based on this sample. 5

Intelligence 72 (2019) 1–12

M. Stadler et al.

measuring knowledge acquisition and including dynamics as the final model. The model fit the data well (Table 1) and also represented a significant improvement over the 2-dimensional model (χ2 = 27.87; df = 3; p < .001). In this 3-dimensional model all items loaded significantly and meaningfully on their respective factors. The correlation between the two connectivity factors remained strong and significant (r = 0.82; p < .001). This model was more parsimonious than the 4dimensional model and did not result in any significant decrease in model fit. There were no significant cross loadings between the detecting connectivity factor and the detecting dynamics factor. Therefore, the 3-dimensional model was retained as the final model even though it partially contradicted the framework presented above. In summary, the multifaceted framework represented a significant improvement over the 2-dimensional measurement model used in previous studies for knowledge acquisition but not for knowledge application thus partially supporting Hypothesis 1. For the remainder of Study 1, CPS will be modeled using the 3-dimensional measurement model.

Table 1 Fit indices for measurement models and structural models of Study 1. Models Measurement models Static problem solving CPS 2-Dimensional CPS 4-Dimensional CPS 3-Dimensional Structural models CPS and static problem solving CPS and static problem solving predicting GPA

χ2

df

p

RMSEA

CFI

TLI

1.47 303.74 235.64 225.52

2 125 123 121

0.480 < 0.001 < 0.001 < 0.001

0.00 0.03 0.03 0.02

1.00 0.99 0.99 0.99

1.00 0.99 0.99 0.99

418.30

193

< 0.001

0.03

0.99

0.98

574.64

217

< 0.001

0.03

0.98

0.97

Note: df = degrees of freedom, RMSEA = root mean square error of approximation, CFI = comparative fit index, TLI = Tucker–Lewis index, CPS = Complex problem-solving, GPA = Grade Point Average.

To model knowledge acquisition and knowledge application as separate facets of connectivity and dynamics, we defined a 4-dimensional model. All 9 items measuring knowledge acquisition still loaded onto one factor, now only representing detecting connectivity. In addition, the 3 items including dynamics also loaded on a detecting dynamics factor. This was mirrored for knowledge application with all 9 items loading on one handling connectivity factor and the 3 items including dynamics loading on an additional handling dynamics factor. Correlations were defined between the two connectivity factors and the two dynamics factors but no correlations were allowed across the two dimensions of CPS. This four dimensional model fit very well to the data (Table 1) and represented a significant improvement over the 2-dimensional model (χ2 = 55.80; df = 7; p < .001). The measurement model is graphically illustrated in Fig. 4. As can be seen, all items loaded significantly and meaningfully on the two connectivity factors. However, contrary to our expectations, only the items measuring knowledge acquisition, including dynamics, loaded significantly and meaningfully on the respective dynamics factor. None of the three items measuring knowledge application and including dynamics loaded significantly on the respective dynamics factor. The correlation between the two connectivity factors was strong and significant (r = 0.84; p < .001). We therefore defined a 3-dimensional model with two connectivity factors but only one dynamics factor indicated by the three items

6.2.2.2. Structural models. After establishing a measurement model for CPS separating connectivity and dynamics, we investigated the relation between the facets and static problem solving (Hypothesis 2). To derive a measurement for static problem solving and to limit the number of parameters to be estimated in the structural models we aggregated the items for static problem solving to four parcels each containing items from both the numerical and the verbal tests (Little, Cunningham, Shahar, & Widaman, 2002). As can be seen from the fit indicators presented in Table 1, the resulting model of one general factor indicated by all four parcels represented the data well. A model correlating static problem solving with the two connectivity factors and dynamics fit the data well (Table 1). In line with Hypothesis 2, the correlations between detecting connectivity and static problem solving (r = 0.56; p < .001) as well as between handling connectivity and static problem solving (r = 0.62; p < .001) were considerably higher than the correlation between detecting dynamics and static problem solving (r = 0.29; p ≤ .001). This difference was statistically significant (p < .001) for both comparisons. In the final structural models we investigated whether there was an incremental value of the two CPS dimensions connectivity and dynamics over and above each other and static problem solving in the prediction of school grades (Hypothesis 3). To avoid issues of multicollinearity due to the high correlations between detecting connectivity

Fig. 4. 4-Dimensional measurement model for CPS in Study 1, acqX = knowledge acquisition part of task X, appX = knowledge application part of task X, dotted lines indicated non-significant paths. 6

Intelligence 72 (2019) 1–12

M. Stadler et al.

Fig. 5. CPS facets and static problem solving predicting GPA in Study 1, GPA = grade point average, acqX = knowledge acquisition part of task X; appX = knowledge application part of task X, P1-P4 = parcel for static problem solving, dotted lines indicated non-significant paths.

connectivity, which was also confirmed in the analyses. Moreover, detecting dynamics exhibited incremental validity in predicting students´ GPA over and above connectivity and static problem solving as predicted by the theoretical framework. This supports the notion that connectivity and dynamics are the defining aspects of CPS (Funke, 2001) and that dynamics may indeed constitute the different demands (Rigas & Brehmer, 2000) hypothesized to separate CPS from static problem solving most distinctly. Contrary to our predictions, we found no distinct factor of handling dynamics. This may have been due to the relatively high difficulty of our tasks. < 10% of the students solved the knowledge application phases of all three tasks including dynamics. Moreover, the limitation on only three tasks including dynamics may have reduced the reliability of the extracted factor. Regarding our operationalization of static problem solving, we purposefully limited Study 1 to include only measures of reasoning. Kretzschmar et al. (2016) argue however, that this limitation of tasks in conventional measures of intelligence may artificially inflate the incremental validity of CPS (see also Lotz et al., 2016). To investigate the potential influences of the measurement of both CPS and intelligence, we conducted a second study attempting to countervail these three potential limitations.

and handling connectivity, we created a higher order factor of connectivity indicated by the two factors. That way, the regression coefficients remained interpretable (Johnson & Lebreton, 2004). Students' GPA was modeled as a manifest variable and then regressed on the two latent CPS dimensions (detecting dynamics and the higher-order connectivity factor) and the latent static problem solving factor. In addition, the CPS dimensions were regressed on static problem solving with no correlations among the CPS dimensions. Static problem solving explained 46% of the variance in connectivity but was not significantly related to detecting dynamics. The model fit the data well (Table 1). In line with Hypothesis 3, both static problem solving (β = 0.47; p < .001), connectivity (β = 0.10; p < .001), and detecting dynamics (β = 0.31; p < .001) contributed significantly to the prediction of students' GPA with a combined explained variance of R2 = 0.32. The model is graphically illustrated in Fig. 5. 6.2.3. Discussion of study 1 Study 1 confirmed the proposed framework in most parts. Detecting dynamics was clearly distinguished from detecting connectivity, which proved a significant improvement over the generally applied 2-dimensional model that does not separate connectivity and dynamics (Wüstenberg et al., 2012). The framework predicted this new factor of detecting connectivity to show considerably lower correlations with static problem solving than the factors of detecting and handling

6.3. Study 2 Study 2 reanalyzed a dataset originally published by Kretzschmar 7

Intelligence 72 (2019) 1–12

M. Stadler et al.

knowledge application. In this model, all six items measuring knowledge acquisition load onto one factor regardless of whether the items include dynamics or not. Correspondingly, all six items measuring knowledge application load onto another factor. Just as for Study 1, this 2-dimensional measurement model fit the data well (Table 2). All items loaded significantly and meaningfully on their respective factors and the two factors of knowledge acquisition and knowledge application were strongly correlated (r = 0.87; p < .001). To model knowledge acquisition and knowledge application as separate facets of connectivity and dynamics, we defined the same 4-dimensional model as for Study 1. All six items measuring knowledge acquisition still loaded onto one factor, now only representing detecting connectivity. In addition, the four items including dynamics also loaded on a detecting dynamics factor. This was mirrored for knowledge application accordingly. Correlations were defined between the two connectivity factors and the two dynamics factors but no correlations were allowed across the two dimensions of CPS. This 4-dimensional model fit very well to the data (Table 2) and represented a significant improvement over the 2-dimensional model (χ2 = 15.42; df = 3; p < .001). Contrary to Study 1, all items loaded significantly and meaningfully on their respective factors thus fully supporting the 4-dimensional model. In line with our expectations, there were considerable correlations among both the connectivity factors (r = 0.86; p < .001) and the dynamics factors (r = 0.98; p < .001). There were no cross loadings between the connectivity and dynamics factors for both knowledge acquisition and knowledge application. Therefore, this 4-dimensional model was retained as the final model for the remainder of Study 2. In summary, the multifaceted framework represented a significant improvement over the 2-dimensional measurement model used in previous studies thus fully supporting Hypothesis 1.

et al. (2016). The sample consisted of university students who should show higher CPS ability resulting in more appropriate item difficulties (Stadler et al., 2017; Stadler, Becker et al., 2016; Stadler, Niepel, & Greiff, 2016). In addition, the CPS tasks employed consisted of two tasks that included solely connectivity and four tasks that included both connectivity and dynamics thus providing more indicators for the dynamics factor. Finally, static problem solving was assessed using a very broad conventional measure of intelligence covering various facets. Based on this dataset, we tested the very same hypotheses as for Study 1 attempting to validate the proposed multi-facetted model of CPS and its relation to conventional measures of CPS. 6.3.1. Method and procedure 6.3.1.1. Participants. The sample was gathered as part of a larger study conducted at the University of Heidelberg, Germany. The sample consisted of N = 227 university students (73% female; age: M = 22.88, SD = 4.27) who volunteered to take the CPS assessment. Based on the results obtained for Study 1, this sample size should provide sufficient power (> 0.80) to detect model differences (MacCallum, Browne, & Cai, 2006). The full data including a more detailed description is available in an Open Science Framework repository (https://osf.io/qf673). 6.3.1.2. Measures 6.3.1.2.1. Static problem solving. Static problem solving was assessed using the Berlin Intelligence Structure Test (BIS-4 Test; Jäger, Süß, & Beauducel, 1997). Based on the BIS model, the test assesses four operation factors (i.e., reasoning, mental speed, memory, creativity), three content factors (i.e., figural, numerical, verbal), and a g-factor (for an English description of the model and the test, see Süß & Beauducel, 2015). The BIS test contains 45 tasks and takes a total of approximately 2.5 h. In line with Kretzschmar et al. (2016), only the scores for the four operation factors (i.e., reasoning, mental speed, memory, and creativity) were calculated as we made no assumptions regarding content. 6.3.1.2.2. Complex problem solving. CPS was measured using six computer-based problem-solving tasks following the MicroDYN approach (Greiff et al., 2012). Two of the tasks used in this study (Task 1and 2) included only connectivity while the other four (Tasks 3–6) included both connectivity and dynamics.5 6.3.1.2.3. School grades. Academic achievement was measured with self-reported final school grade point average (GPA). In German school systems, school grades range from 1 (excellent) to 6 (insufficient). For our analyses, school grades were reversed so that higher numbers reflected better performance.

6.3.2.2. Structural models. After establishing a measurement model for CPS separating connectivity and dynamics, we investigated the relation between these dimensions and static problem solving (Hypothesis 2). As done for the original publication, we aggregated the items for static problem solving into parcels to derive a parsimonious measurement model for the four static problem solving operations. The resulting model of all four operation factors represented the data very well (Table 2). A model correlating the four static problem solving factors with the two connectivity factors and the two dynamics factors dynamics fit the data well (Table 2). In line with Hypothesis 2, detecting connectivity was significantly correlated with reasoning (r = 0.54; p < .001), speed (r = 0.32; p < .001), and memory (r = 0.29; p < .001). Handling connectivity was significantly correlated with reasoning (r = 0.64; p < .001), speed (r = 0.36; p < .001), creativity (r = 0.18; p < .001), and memory (r = 0.30; p < .001). Both detecting dynamics and handling dynamics were not significantly related to any of the static problem solving factors. In the final structural models, we investigated whether there was an incremental value of the two CPS dimensions connectivity and dynamics over and above each other and static problem solving in the prediction of school grades (Hypothesis 3). As for Study 1, we created higher order factors of connectivity and dynamics to avoid issues of multicollinearity due to the high correlations between the knowledge acquisition phase and the knowledge application phase of the two dimensions. Students' GPA were modeled as a manifest variable and then regressed on the two latent CPS dimensions and the latent static problem solving factors. In addition, the CPS factors were regressed on the static problem solving factors as done in Kretzschmar et al. (2016). Just as for the previous models, there were no correlations across the CPS dimensions. The model fit the data well (Table 2). Reasoning (β = 0.34; p = .002), speed (β = −0.63; p = .005), and memory (β = 0.40; p = .016) predicted students' GPA significantly. In line with Hypothesis

6.3.1.3. Procedure. Testing was conducted in two sessions. The first session (approximately 2 h) included the assessment of CPS, school grades, and demographic data. The second session (approximately 3 h) was usually conducted within one week of the first session. In this second session, participants worked on the intelligence tests and additional questionnaires that were not part of this article. 6.3.2. Results 6.3.2.1. Measurement models. Based on the syntax provided in the Open Science Framework repository (https://osf.io/qf673), we were able to replicate the descriptive statistics and measurement models reported in the original publication (Tables 2 and 3 in Kretzschmar et al., 2016). All correlations were in the expected ranges and direction. Following the suggestions by Wüstenberg et al. (2012), CPS was modeled with the two dimensions of knowledge acquisition and 5 Kretzschmar et al. (2016) also used another measure of CPS (MicroFIN; Neubert, Kretzschmar, Wüstenberg, & Greiff, 2015). This measure does not include dynamics though and could not be used for our study.

8

Intelligence 72 (2019) 1–12

M. Stadler et al.

GPA on the other hand slightly inflated the regression path between memory and GPA (correlation of r = 0.35 as opposed to a regression path of β = 0.40).This final model is illustrated in Fig. 6.

Table 2 Fit indices for measurement models and structural models of Study 2. Models Measurement models Static problem solving CPS 2-Dimensional CPS 4-Dimensional Structural models CPS and static problem solving CPS and static problem solving predicting GPA

χ2

df

p

RMSEA

CFI

TLI

80.35 55.19 46.98

71 47 44

0.210 0.193 0.352

0.00 0.03 0.02

1.00 1.00 1.00

1.00 1.00 1.00

298.77 314.74

277 297

0.176 0.229

0.02 0.02

0.99 0.99

0.99 0.99

6.3.3. Discussion of study 2 The results reported in Study 2 supported the proposed understanding of CPS as a multifaceted construct as well as the proposed parallels and differences to static problem solving. They demonstrate the necessity to distinguish between the dimensions of connectivity and dynamics to comprehensively describe CPS. While Study 1 did not support a distinct factor of handling dynamics, Study 2 fully supported the proposed model. This supports our assumption that the CPS tasks employed in Study 1 may have been too difficult and did not provide sufficient variance among students to indicate a common latent factor. The results of Studies 1 and 2 did not differ substantially regarding the relation between static problem solving and the facets of CPS. As expected, detecting and handling connectivity in CPS tasks was strongly associated with the performance in static problem solving tasks as employed in conventional measures of intelligence, which also require detecting and handling connectivity among elements (Klauer & Phye, 2008). Detecting and handling dynamics in CPS on the other hand requires considering changes over time, which is not necessary in static problem solving tasks. Accordingly, the two dynamics facets were considerably less associated with performance in conventional

Note: df = degrees of freedom, RMSEA = root mean square error of approximation, CFI = comparative fit index, TLI = Tucker–Lewis index, CPS = Complex problem-solving, GPA = Grade Point Average.

3, connectivity (β = 0.30; p = .008) and dynamics (β = 0.45; p = .006) also contributed significantly to the prediction of students' GPA with a combined explained variance of R2 = 0.53. Interestingly, creativity showed a negative path to connectivity whereas speed showed a negative path to GPA. In line with Kretzschmar et al., (2016), these negative paths must be interpreted as suppressor effects. The negative path between creativity and connectivity inflated the regression path between reasoning and connectivity (correlation of r = 0.64 as opposed to a regression path of β = 0.77). The negative path between speed and

Fig. 6. CPS facets and static problem solving predicting GPA in Study 2; GPA = grade point average; acqX = knowledge acquisition part of task X; appX = knowledge application part of task X, dotted lines indicated non-significant paths. 9

Intelligence 72 (2019) 1–12

M. Stadler et al.

Schoppek & Fischer, 2015; see Greiff & Martin, 2014 for a response). Whereas psychometric aspects such as low reliability are certainly an issue that several older measures of CPS suffer from (see Greiff, Fischer et al., 2015), the low correlations between MCS measures and the Tailorshop can be explained by the multifaceted framework of CPS presented in this article. As demonstrated above, a two-dimensional conception of CPS does not differentiate between connectivity and dynamics but only between knowledge acquisition and knowledge application. Performance in CPS tasks with a higher degree of connectivity as compared to the dynamics, such as MCS tasks, will correspondingly be correlated more strongly with other CPS tasks with a higher degree of connectivity than to CPS tasks with a higher degree of dynamics, such as the Tailorshop. Adequately describing the relation between various measures of CPS would therefore require separating the dimensions of connectivity and dynamics with the assumption of high correlations within the dimensions of connectivity and dynamics and low correlations across dimensions. The multifaceted conception of CPS therefore has the potential to resolve the debate on the relation between different measures of CPS. On the one hand, it assumes that there is a common construct of CPS. On the other hand, it proposes that different measures put varying focus on different cognitive demands depending on the degree of connectivity and dynamics they involve. The multifaceted conception of CPS can potentially also resolve the long-standing debate on the relation between measures of CPS and conventional measures of intelligence (Stadler et al., 2015). This debate is summarized by two conflicting hypotheses. The different-demands hypothesis (Rigas & Brehmer, 2000) suggests that CPS tasks demand the performance of more complex mental processes than conventional measures of intelligence do, such as active interaction with the problem to acquire knowledge on the problem environment. This should lead to low correlations between measures of CPS and conventional measures of intelligence (Kluwe, Misiak, & Haider, 1990). The low-reliability hypothesis (Rigas et al., 2002), on the other hand, proposes that the cognitive demands in measures of CPS and conventional measures of intelligence are similar and attributes the low correlations found in some studies to reliability issues within the CPS measures employed. In their meta-analysis on the relation between CPS measures and conventional measures of intelligence, Stadler et al. (2015) found the type of CPS measure to moderate this relation. Measures with a high degree of dynamics showed only a relatively small correlation to conventional measures of intelligence (ρ = 0.34) as compared to the MCS measures with a high degree of connectivity (ρ = 0.59). The authors interpreted these results as support for the different-demand hypothesis but without further specifying the actual demands that are supposed to be different. The multifaceted framework of CPS helps to fill this gap by attempting to specify the cognitive overlaps and differences between complex and static problem solving. This allows for falsifiable predictions on the relation between measures of CPS and conventional measures of intelligence. Higher degrees of dynamics in CPS measures should, based on the framework, result in lower correlations to conventional measures of intelligence that consist of static problems. CPS measures that have a high degree of connectivity should show strong correlations with conventional measures of intelligence. Distinguishing the facets of CPS, as demonstrated in the empirical part of this paper, even allows relating them to conventional measures of intelligence separately. Future research should aim at corroborating the results of this study investigating their generalizability to other approaches to the measurement of CPS. Moreover, strict experimental research is warranted that systematically varies task characteristics of complex problems and observes the resulting effects on correlations with static problem-solving performance.

measures of intelligence. In line with this difference in correlations, the incremental validity of the dynamics aspect of CPS in the prediction of GPA over and above the performance in static problem solving tasks was higher than the incremental validity of the connectivity aspect in both studies. This supports our claim, that connectivity indeed constitutes the different demands (Rigas & Brehmer, 2000) hypothesized to separate CPS from static problem solving most distinctly. Our results did not support the assumption that the incremental validity of CPS tasks over and above conventional measures of intelligence was mostly due to a limitation on measures of reasoning (Kretzschmar et al., 2016; Lotz et al., 2016). Both the reasoning measure employed in Study 1 and the very broad measure employed in Study 2 resulted in similar relations to CPS and to GPA. This further supports the notion that CPS predominantly relates to the facet reasoning in conventional measures of intelligence (e.g. Wittmann & Süß, 1999), which is also the best indicator of general intelligence (Jensen, 1998). 7. General discussion The aim of this paper was to theoretically introduce and empirically test a multifaceted framework of CPS that would explain the seemingly contradictory relations to conventional measures of intelligence. Unlike all previous conceptions of CPS, this framework defines the detection and handling of connectivity and dynamics as four distinct facets of CPS underlying the broad dimensions of connectivity and dynamics. In contrasting these facets with those underlying static problem solving, the framework is able to explain the seemingly contradictory findings on the relation between CPS and static problem solving (Stadler, Becker et al., 2016) and positions CPS and static problem solving as related aspects of intelligence. In that, it integrates CPS within the wider realm of the human intellect adding empirically testable propositions. Understanding CPS as a multifaceted construct has considerable implications on the two major points of contention in this line of research (see Beckmann & Goode, 2017): The relation between different measures of CPS and the relation between measures of CPS and conventional measures of intelligence. Regarding the relation between different measures of CPS, Greiff, Stadler et al. (2015) prominently reported higher correlations within measures of CPS following the multiple complex systems (MCS) approach that uses multiple small tasks to assess CPS (as done in this study; see Greiff, Fischer et al., 2015 for a detailed description of the MCS approach) than between MCS measures and the classical CPS measure “Tailorshop” (Danner et al., 2011). CPS tasks based on the MCS approach, such as the one used in the empirical part of this article, are designed to be highly efficient with only 2–3 input and output variables (see Fig. 2 above) and relatively short knowledge acquisition and knowledge application phases. This way it is possible to combine a number of these problems into one assessment. The Tailorshop on the other hand, as the most prominent representation of a whole group of assessment instruments, is a single complex problem with 24 variables simulating the workings of a clothing manufacturer. 10 of these variables can be directly influenced by the problem solver. The state of the other variables depends on their previous state, the previous state of related variables, and the problem solver's manipulations. Due to its length, an assessment using the Tailorshop usually includes only one attempt to solve the complex problem setup of maximizing the company value in a given number of turns. Greiff, Stadler et al. (2015) interpreted the higher correlations within measures of CPS following the MCS approach than between MCS measures and the Tailorshop as evidence for a higher validity of MCS measures resulting from the use of multiple tasks. This interpretation was heavily disputed, though (Funke, Fischer, & Holt, 2017; Greiff et al., 2017; Kretzschmar, 2017). Some researchers even went as far as suggesting that MCS measures and complex problems such as the Tailorshop were in fact assessing different constructs with different cognitive demands (Funke, 2014; 10

Intelligence 72 (2019) 1–12

M. Stadler et al.

7.1. Conclusion

0146621612439620. Guthke, J., & Stein, H. (1996). Are learning tests the better version of intelligence tests? European Journal of Psychological Assessment, 12, 1–13. https://doi.org/10.1027/ 1015-5759.12.1.1. Jäger, A. O., Süß, H.-M., & Beauducel, A. (1997). Berliner Intelligenzstruktur-test. Göttingen: Hogrefe. Jensen, A. R. (1998). The g factor: The science of mental ability. Human evolution, behavior, and intelligence. Westport, CT: Praeger Publishers. Johnson, J. W., & Lebreton, J. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, 238–257. https://doi. org/10.1177/1094428104266510. Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48, 63–85. https://doi.org/10.1007/ BF02300500. Klauer, K. J. (1996). Teaching inductive reasoning: Some theory and three experimental studies. Learning and Instruction, 6, 37–57. https://doi.org/10.1016/S0959-4752(96) 80003-X. Klauer, K. J., & Phye, G. D. (2008). Inductive reasoning: A training approach. Review of Educational Research, 78, 85–123. https://doi.org/10.3102/0034654307313402. Kluwe, R. H., Misiak, C., & Haider, H. (1990). Learning by doing in the control of a complex system. In H. Mandl, E. de Corte, N. Bennett, & H. F. Friedrich (Eds.). Learning and instruction. Learning and instruction. European research in an international context. Volume 2.1.: Social and cognitive aspects of learning and instruction (pp. 197– 218). Oxford: Pergamon Press. Kretzschmar, A. (2017). Sometimes less is not enough: A commentary on Greiff et al. (2015). Journal of Intelligence, 5, 4. https://doi.org/10.3390/jintelligence5010004. Kretzschmar, A., Neubert, J. C., Wüstenberg, S., & Greiff, S. (2016). Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school grades. Intelligence, 54, 55–69. https://doi.org/10.1016/j.intell.2015.11. 004. Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling: A Multidisciplinary Journal, 9, 151–173. https://doi.org/10.1207/ S15328007SEM0902_1. Lotz, C., Sparfeldt, J. R., & Greiff, S. (2016). Complex problem solving in educational contexts – Still something beyond a “good g ”? Intelligence, 59, 127–138. https://doi. org/10.1016/j.intell.2016.09.001. MacCallum, R. C., Browne, M. W., & Cai, L. (2006). Testing differences between nested covariance structure models: Power analysis and null hypotheses. Psychological Methods, 11, 19–35. https://doi.org/10.1037/1082-989X.11.1.19. McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1–10. https://doi.org/10.1016/j.intell.2008.08.004. Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide (7). Los Angeles, CA: Muthén. Neisser, U., Boodoo, G., Bouchard, T. J., Boykin, A. W., Brody, N., Ceci, S. J., ... Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77–101. https://doi.org/10.1037//0003-066X.51.2.77. Neubert, J. C., Kretzschmar, A., Wüstenberg, S., & Greiff, S. (2015). Extending the Assessment of complex problem solving to finite state automata. European Journal of Psychological Assessment, 31, 181–194. https://doi.org/10.1027/1015-5759/ a000224. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, N. J: Prentice-Hall. Novick, L. R., & Bassok, M. (2007). Problem solving. In K. J. Holyoak (Ed.). The Cambridge handbook of thinking and reasoning (pp. 321–349). Cambridge, MA: Cambridge University Press. Piaget, J. (1950). Psychology of intelligence. London: Routledge and Kegan Paul. Raven, J. C. (1938). Progressive matrices: A perceptual test of intelligence. London: Lewis. Ren, X., Wang, T., Altmeyer, M., & Schweizer, K. (2014). A learning-based account of fluid intelligence from the perspective of the position effect. Learning and Individual Differences, 31, 30–35. https://doi.org/10.1016/j.lindif.2014.01.002. Resnick, R., & Glaser, L. B. (1976). Problem solving and intelligence. In L. B. Resnick (Ed.). The nature of intelligence (pp. 205–230). Hillsdale, N. J., New York, N.Y.: Lawrence Erlbaum Associates Distributed by Halsted Press Division of J. Wiley. Rigas, G., & Brehmer, B. (2000). Mental processes in intelligence tests and dynamics decision making tasks. In P. Juslin, & H. Montgomery (Eds.). Judgment and decision making: Neo-Brunswikian and process-tracing approaches (pp. 45–65). Mahwah, NJ: Psychology Press. Rigas, G., Carling, E., & Brehmer, B. (2002). Reliability and validity of performance measures in microworlds. Intelligence, 30, 463–480. https://doi.org/10.1016/S01602896(02)00121-6. Ross, J. D., & Ross, C. M. (1979). Ross test of higher cognitive processes. Novato, CA: Academic Therapy Publications. Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference Chisquare test statistic. Psychometrika, 75, 243–248. https://doi.org/10.1007/s11336009-9135-y. Schoppek, W., & Fischer, A. (2015). Complex problem solving—Single ability or complex phenomenon? Frontiers in Psychology, 6(27), https://doi.org/10.3389/fpsyg.2015. 01669. Schweizer, F., Wüstenberg, S., & Greiff, S. (2013). Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity. Learning and Individual Differences, 24, 42–52. https://doi.org/10.1016/j.lindif.2012. 12.011. Stadler, M., Becker, N., Gödker, M., Leutner, D., & Greiff, S. (2015). Complex problem solving and intelligence: A meta-analysis. Intelligence, 53, 92–101. https://doi.org/10.

In summary, the multifaceted conception of CPS proposed in this paper is theoretically applicable to explain the majority of seemingly contradictory results on the major points of contention in CPS research. It is at the same time able to subsume various contended positions on the nature of CPS, its assessment, and its relation to other constructs within one comprehensive framework. This could lead to a synthesis of a debate that has lasted decades and could help position CPS into more general theories of human intelligence beyond problem solving such as the Catell-Horn-Caroll theory (McGrew, 2009). This framework can potentially also contribute to other current issues of problem solving and intelligence research such as the stability or trainability of general cognitive abilities. Overall, the framework may lead to a more comprehensive operationalization of intelligence that includes both static and complex problems based on a solid theoretical foundation. References Anderson, J. R. (1993). Problem solving and learning. American Psychologist, 48, 35–44. https://doi.org/10.1037/0003-066X.48.1.35. Becker, N., Schmitz, F., Falk, A., Feldbrügge, J., Recktenwald, D., Wilhelm, O., ... Spinath, F. (2016). Preventing response elimination strategies improves the convergent validity of figural matrices. Journal of Intelligence, 4, 2. https://doi.org/10.3390/ jintelligence4010002. Beckmann, J. F., & Goode, N. (2017). Missing the wood for the wrong trees: On the difficulty of defining the complexity of complex problem solving scenarios. Journal of Intelligence, 5, 15. https://doi.org/10.3390/jintelligence5020015. Berbeglia, G., Cordeau, J.-F., & Laporte, G. (2010). Dynamic pickup and delivery problems. European Journal of Operational Research, 202, 8–15. https://doi.org/10.1016/ j.ejor.2009.04.024. Birney, D. P., Beckmann, J. F., Beckmann, N., & Double, K. S. (2017). Beyond the intellect: Complexity and learning trajectories in Raven's Progressive Matrices depend on selfregulatory processes and conative dispositions. Intelligence, 61, 63–77. https://doi. org/10.1016/j.intell.2017.01.005. Bollen, K. A. (1989). Structural equations with latent variables. 2. [Dr.]. New York [u.a.]: A Wiley-Interscience Publication Wiley. Boring, E. G. (1923). Intelligence as the tests test it. New Republic, 36, 35–37. Brunner, M. (2008). No g in education? Learning and Individual Differences, 18, 152–165. https://doi.org/10.1016/j.lindif.2007.08.005. Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychological Review, 97, 404–431. Danner, D., Hagemann, D., Holt, D. V., Hager, M., Schankin, A., Wüstenberg, S., & Funke, J. (2011). Measuring performance in dynamic decision making. Journal of Individual Differences, 32, 225–233. https://doi.org/10.1027/1614-0001/a000055. Dörner, D. (1974). Die kognitive Organisation beim Problemlösen: Versuche zu einer kybernetischen Theorie der elementaren Informationsverarbeitungsprozesse beim Denken. Bern: Huber. Dörner, D. (1980). On the difficulties people have in dealing with complexity. Simulation and Games, 11, 87–106. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. The Journal of Problem Solving, 4. https://doi.org/10.7771/1932-6246.1118. Frensch, P. A., & Funke, J. (Eds.). (1995). Complex problem solving: The European perspective. Hillsdale (NJ): Lawrence Erlbaum Associates. Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking and Reasoning, 7, 69–89. https://doi.org/10.1080/13546780042000046. Funke, J. (2014). Analysis of minimal complex systems and complex problem solving require different forms of causal cognition. Frontiers in Psychology, 5(185), https:// doi.org/10.3389/fpsyg.2014.00739. Funke, J., Fischer, A., & Holt, D. (2017). When less is less: Solving multiple simple problems is not complex problem solving—A comment on Greiff et al. (2015). Journal of Intelligence, 5, 5. https://doi.org/10.3390/jintelligence5010005. Greiff, S., Fischer, A., Stadler, M., & Wüstenberg, S. (2015). Assessing complex problemsolving skills with multiple complex systems. Thinking and Reasoning, 21, 356–382. https://doi.org/10.1080/13546783.2014.989263. Greiff, S., Fischer, A., Wüstenberg, S., Sonnleitner, P., Brunner, M., & Martin, R. (2013). A multitrait–multimethod study of assessment instruments for complex problem solving. Intelligence, 41, 579–596. https://doi.org/10.1016/j.intell.2013.07.012. Greiff, S., & Martin, R. (2014). What you see is what you (don't) get: A comment on Funke's (2014) opinion paper. Frontiers in Psychology, 5(1061), https://doi.org/10. 3389/fpsyg.2014.01120. Greiff, S., Stadler, M., Sonnleitner, P., Wolff, C., & Martin, R. (2015). Sometimes less is more: Comparing the validity of complex problem solving measures. Intelligence, 50, 100–113. https://doi.org/10.1016/j.intell.2015.02.007. Greiff, S., Stadler, M., Sonnleitner, P., Wolff, C., & Martin, R. (2017). Sometimes more is too much: A rejoinder to the commentaries on Greiff et al. (2015). Journal of Intelligence, 5, 6. https://doi.org/10.3390/jintelligence5010006. Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving. Applied Psychological Measurement, 36, 189–213. https://doi.org/10.1177/

11

Intelligence 72 (2019) 1–12

M. Stadler et al.

Süß, H.-M., & Beauducel, A. (2015). Modeling the construct validity of the Berlin intelligence structure model. Estudos De Psicologia (Campinas), 32, 13–25. https://doi. org/10.1590/0103-166X2015000100002. Vickers, D., Lee, M. D., Dry, M., & Hughes, P. (2003). The roles of the convex hull and the number of potential intersections in performance on visually presented traveling salesperson problems. Memory and Cognition, 31, 1094–1104. https://doi.org/10. 3758/BF03196130. Wiley, J., Jarosz, A. F., Cushen, P. J., & Colflesh, G. J. H. (2011). New rule use drives the relation between working memory capacity and Raven's advanced progressive matrices. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 256–263. https://doi.org/10.1037/a0021613. Wittmann, W. W., & Süß, H.-M. (1999). Investigating the paths between working memory, intelligence, knowledge, and complex problem-solving performances via Brunswik symmetry. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.). Learning and individual differences: Process, trait, and content determinants (pp. 77–108). Washington, DC: American Psychological Association. https://doi.org/10.1037/ 10315-004. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving—More than reasoning? Intelligence, 40, 1–14. https://doi.org/10.1016/j.intell.2011.11.003.

1016/j.intell.2015.09.005. Stadler, M., Becker, N., Greiff, S., & Spinath, F. M. (2016). The complex route to success: Complex problem-solving skills in the prediction of university success. Higher Education Research and Development, 35, 365–379. https://doi.org/10.1080/ 07294360.2015.1087387. Stadler, M., Becker, N., Schult, J., Niepel, C., Spinath, F. M., Sparfeldt, J. R., & Greiff, S. (2017). The logic of success: The relation between complex problem-solving skills and university achievement. Higher Education, 55(469), https://doi.org/10.1007/ s10734-017-0189-y. Stadler, M., Niepel, C., & Greiff, S. (2016). Easily too difficult: Estimating item difficulty in computer simulated microworlds. Computers in Human Behavior, 65, 100–106. https://doi.org/10.1016/j.chb.2016.08.025. Sternberg, R. J. (1985). Reasoning, problem solving, and intelligence. In R. J. Sternberg (Ed.). Handbook of human intelligence (pp. 225–307). Cambridge, MA: Cambridge University Press. Sternberg, R. J., Castejon, J. L., Prieto, M. D., Hautamäki, J., & Grigorenko, E. (2001). Confirmatory factor analysis of the Sternberg Triarchic abilities test (multiple choice items) in three international samples: An empirical test of the Triarchic theory. European Journal of Psychological Assessment, 1–16. https://doi.org/10.1027//10155759.17.1.1.

12