- Email: [email protected]
How handwriting behaviors during problem solving are related to problem-solving success in an engineering course
Contemporary Educational Psychology 58 (2019) 331–337
Contents lists available at ScienceDirect
Contemporary Educational Psychology journal homepage: www.elsevier.com/locate/cedpsych
How handwriting behaviors during problem solving are related to problemsolving success in an engineering course
T
Thomas F. Stahovicha,⁎, Timothy S. Van Arsdaleb, Richard E. Mayerc a
University of California, Riverside, United States Raytheon Company, United States c University of California, Santa Barbara, United States b
ARTICLE INFO
ABSTRACT
Keywords: Problem solving Engineering Expertise Smartpens
If we carefully observe the spatial and temporal organization of students' pen strokes as they solve an engineering problem, can we predict their ability to achieve the correct answer? To address this question, 122 college students were asked to solve exam problems in an engineering course using a smartpen that recorded their writing as digitized timestamped pen strokes. The pen stroke data was used to compute a collection of 10 metrics characterizing various elements of problem-solving fluency including the tendency to progress down the page without revisions, the amount of time with no activity, and the frequency of constructing and using equations. The primary finding is that, on average across 13 different exam problems, these elements of problemsolving process explained 40% of the variance in scores of the correctness of the problem solution. In short, success on generating correct solutions was related to the fluency of the student's problem-solving process (i.e., working sequentially from the top to the bottom of the page, working without detours or long pauses, and working by constructing equations). This work is consistent with the idea that expertise in solving common engineering problems involves being able to treat them like routine rather than non-routine problems.
1. Objective and rationale Problem solving is at the heart of engineering education. College courses in engineering often require students to solve applied physics problems, such as the one in Fig. 1, in homework, quizzes, and exams. By emphasizing instruction, practice, and assessment in solving problems, instructors are making clear that an overarching aim is to help students develop problem-solving expertise. Consider the engineering problem in Fig. 1, taken from an exam in an introductory statics course. Suppose one student solves the problem by working from the top of the page to the bottom without going back to revise previous work and without long pauses. This student also spends a lot of time and effort working on equations. In contrast, suppose another student starts at the top of the page, then jumps to the bottom, and then the middle, and then back up to the top, including making revisions to earlier work and taking long breaks with no activity at all. This student also spends less time and effort on writing and solving equations. Which student is more likely to produce the correct answer? The present study seeks to address this question. The goal of the present study is to determine whether the
handwritten solutions produced by students as they solve exam problems provide clues concerning which problem-solving behaviors are related to success as measured by the correctness of the work. In short, our goal is to determine which elements of the problem-solving process (indicated by the students' pen strokes during problem solving) are predictive of the problem-solving product (indicated by the score assigned by the instructor based on a scoring rubric). Problem-solving transfer — the ability to use what was learned to solve new problems — is a long-standing issue in educational psychology (Mayer & Wittrock, 1996, 2009); a central focus of educational psychology research on learning in STEM disciplines (Star & RittleJohnson, 2016; Tabak, 2016); and a foundational 21st century skill (Bonney & Sternberg, 2017; Pellegrino & Hilton, 2012). In analyzing problem types, Jonassen and colleagues distinguish between classroom problems in engineering, which typically are well-defined problems, and workplace problems in engineering, which typically are ill-defined problems (Jonassen, 2011; Jonassen, Strobel, & Lee, 2006). The classroom problem shown in Fig. 1 is well defined because the givens, goals, and allowable operations are precisely specified (Mayer, 1992). Although such classroom problems may be appropriate as initial work for
⁎ Corresponding author at: Department of Mechanical Engineering, Bourns College of Engineering, A349 Bourns Hall, University of California, Riverside, CA 92521, United States. E-mail address: [email protected] (T.F. Stahovich).
https://doi.org/10.1016/j.cedpsych.2019.04.004
Available online 27 April 2019 0361-476X/ © 2019 Published by Elsevier Inc.
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 1. Exam problem 5 (left panel) and a nearly correct handwritten solution (right panel).
specially instrumented “smartpens” that record every pen stroke with timestamps. We then developed algorithms to extract various measures of fluency from the captured ink data, and examined the relationships between these measures and the correctness of the work. Together, this technology enables us to analyze the problem-solving behaviors of students as they solve typical engineering problems during midterm and final exams, and determine the relation between problem-solving behaviors (as indicated by the spatial and temporal organization of the writing) and their score on the problem solution. Overall, our goal is to determine whether it is possible to predict students' scores on solutions to engineering problems (i.e., the outcome of problem solving) based on their behaviors during the generation of their solutions (i.e., the process of problem solving).
beginning students, the ultimate goal for graduating students is to be able to solve workplace problems, such as through project-based assignments (Jonassen et al., 2006). Classic research (Chi, Glaser, & Farr, 1988; de Groot, 1965; Newell & Simon, 1972; Sternberg & Grigorenko, 2003) and modern research (Mayer, 2012, 2013) on the development of problem-solving expertise in diverse domains suggests that successful problem solving may be characterized by fluency — working from start to end without hesitations and sidetracks, and focusing on the aspects of the problem solution that are relevant. For example, Holvikivi (2007) has shown the need for stronger emphasis in developing problem-solving skills in engineering education and has suggested that fluency is a characteristic of effective problem solving. Taraban (2011) has shown that students entering engineering programs tend to have better problem-solving fluency than non-engineering students, and that experience in solving problems within the engineering curriculum leads to improvements in the development of problem-solving fluency. Grossberg (2013) has shown that cognitive fluency is a result of learning to adapt to new situations. Van Arsdale and Stahovich (2012) provide preliminary evidence that successful engineering problem solvers are more likely to provide spatially neat solutions, working from the top of the sheet to the bottom. Therefore, fluency in producing problem solutions is the focus of this study. In the case of solving engineering problems such as in Fig. 1, fluency involves working through the problem systematically from top to bottom, without breaks and jumps from one part of the page to another. Some indications of fluency in this context include: having few out-oforder pen strokes (i.e., writing added to previously completed areas on the page), having few breaks (i.e., time periods in which no pen strokes are made), and tending to efficiently generate and use all of the relevant equations. If fluency is indicative of successful problem solving, we expect elements of fluency such as these to correlate with problemsolving ability as measured by the grade the student receives on the problem according to a scoring rubric. To test this idea, we conducted a study in which students used
2. Literature review In the field of problem solving, it is customary to make a distinction between routine problems and non-routine problems (Mayer, 2012, 2013). With routine problems, the problem solver may not know the answer but knows how to compute it. For example, for most adults, computing the value of 576 × 29 is a routine problem because they know how to carry out long multiplication. With non-routine problems, the problem solver does not know how to go about computing a solution and so must devise and revise a solution plan. For example, computing the expected life of a rechargeable battery is a non-routine problem because the solution plan is not obvious. Whether or not a problem is routine or non-routine depends on the knowledge of the problem solver. Research shows that successful problem solvers possess schemas for problem types, so that when they see a new problem they can recognize it as fitting with problems of the same type that they know how to solve (Hinsley, Hayes, & Simon, 1977; Mayer, 1981; Riley, Greeno, & Heller, 1983). For example, successful problem solvers often use analogical reasoning (Holyoak, 2005) to solve a new problem (i.e., a target problem) by recognizing a corresponding schema in long-term memory that has the same structural 332
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
characteristics, abstracting the solution method from the schema, and applying it to the target problem. For experienced engineers, most of the problems given to students in introductory engineering courses are routine problems, because experienced engineers possess a large storehouse of schemas reflecting the problem types for commonly used problems in introductory engineering courses. In contrast, beginning engineering students may not possess a large body of problem schemas and therefore must devise and revise solution plans for the problems they encounter in introductory engineering courses. Research on problem-solving expertise shows major differences in the problem-solving processes of experts and novices in a given domain (de Groot, 1965; Mayer, 1992; Newell & Simon, 1972). Experts tend to systematically work forward from the problem statement to the answer because they can simply carry out their solution plan. In solving common engineering problems, expert-like processing is reflected in a high degree of fluency — writing continuously from the top to the bottom of the page with few detours or long pauses. In contrast, novices must develop a solution plan, sometimes through a process of means-ends analysis (Mayer, 2012, 2013; Newell & Simon, 1972) in which problem solvers work backwards from a statement of the goal to establish and execute a series of ever-progressing subgoals. In solving common engineering problems, novice-like processing is reflected in a low degree of fluency — writing in an unstructured sequence of locations on the page, going back to make revisions, and taking long pauses. Creating a solution plan in this way may result in the novice having difficulty in constructing the relevant equations for an engineering problem. It is also customary in the problem-solving literature to distinguish between the process of problem solving and the product of problem solving (Bloom & Broder, 1950; Mayer, 2012, 2013). Problem-solving process refers to the processing the problem solver engages in while generating a solution, including representing the problem, devising a plan, and carrying out the plan. Problem-solving product (or outcome) refers to the solution that was generated. In the present study, we seek to determine whether key expert-novice differences in problem-solving process (i.e., differences in fluency) are related to differences in problem-solving product (i.e., differences in the instructor-assigned score for the problem solution). In short, we explore what can be called the paradox of expertise — the knowledge that experts bring to a common engineering problem that allows them to solve it as a routine problem, which involves recognizing a relevant problem schema and systematically carrying out the corresponding solution plan, whereas novices must go through the creative process of devising a solution plan (reflected in low fluency in handwriting a solution).
can be used to predict student success on the outcome of problem solving. Based on research on expertise in other domains, we expect that expert problem-solving success is related to fluency during the process of generating a solution. In the context of engineering education, this means that the degree to which students possess higher levels of expertise leading to successful problem solving will be related to the degree to which they exhibit problem-solving behaviors during generating a handwritten solution that show signs of fluency — such as working from top to bottom of the page without jumps and long pauses, and the ability to construct and solve the relevant equations. 4. Method 4.1. Participants and design The participants were 122 students from a one-quarter, sophomorelevel course in statics taught at the University of California, Riverside. A total of 113 of the participants completed the course. Demographic data is available for 109 of these students. Most students (85%) were male and the majority were mechanical engineering majors (53%) or other engineering majors (33%). This is an observational study in which all participants solved the same set of 13 exam problems. The predictor variables are based on 10 smartpen metrics derived from behavior in writing solutions to each problem, and the dependent variable is the score assigned on the problem by the instructor based on a scoring rubric. There were between 97 and 122 solutions for each problem, with some data lost on some problems due to technical issues. We followed standards for treatment of human subjects and obtained IRB approval. 4.2. Materials The materials consisted of 13 statics problems that students solved on exams. These are classroom problems typically used in introductory engineering courses rather than workplace problems that would be encountered on the job. The students wrote the solutions on paper using smartpens. Problems 1–3 came from midterm 1, problems 4–6 came from midterm 2, and problems 7–13 came from the final exam. Fig. 1 shows problem 5, which was one of the more difficult problems for students. 4.3. Procedure Students used Livescribe smartpens to solve problems on the midterm and final exams. Students were instructed to show their work as they solved each problem.
3. Theoretical framework Building on long-standing research on expertise (Chi, Glaser, & Farr, 1988; Ericsson, Charness, Feltovich, & Hoffman, 2006; Mayer, 1992; Newell & Simon, 1972; Smith, 1991), we explore the theoretical assertion that experts and novices differ in the way they approach problem solving. Experts possess schemas for problem types, so that when they see a new problem it reminds them of other problems like it that they know how to solve. They can then apply the solution method by working forward from the given information in the problem to compute the answer. In contrast, novices may not possess a schema for a problem they are trying to solve, and therefore must use a means-ends analysis method that involves working backwards from the goal to the givens in a problem. This process will involve some jumping around on the page and even some long pauses, as the student determines what to do next and continually revises the solution plan. In the case of applied physics problems used in engineering education, novices may struggle to construct and use the relevant equations, whereas experts can do so more easily. Fig. 2 summarizes the theoretical framework that guides this study examining how student behaviors during the process of problem solving
4.4. Instrumentation The Livescribe smartpens that students used to write solutions to the exam problems perform the same function as a traditional ballpoint pen, but also digitize each pen stroke as it is written (Rawson, Stahovich, & Mayer, 2017). The smartpens are used with paper preprinted with micro-dots that are used by a camera embedded in the smartpen to determine the location of the pen tip. Each pen stroke is stored on the smartpen as a sequence of timestamped Cartesian coordinates. After the problem-solving session, a USB cable is used to download the digitized data to a PC. 4.5. Scoring and data analysis Table 1 lists 10 problem-solving measures (or metrics) we computed from each student’s pen strokes written during the process of generating a solution to a problem. The measures are intended to capture aspects of problem-solving fluency. In our broader work to develop methods for automatically grading students’ handwritten problem solutions, we 333
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 2. Relation between process and product of problem solving in engineering.
Table 1 Problem-solving measures. Measure
Description
Moderately Out-of-Order Strokes Substantially Out-of-Order Strokes Extensively Out-of-Order Strokes Diagram Activity Equation Activity Digressions Total Breaks Small Breaks Medium Breaks Long Breaks
Number Number Number Number Number Number Number Number Number Number
of of of of of of of of of of
Symbol strokes written out of order by between 40% and 50% strokes written out of order by between 50% and 60% strokes written out of order by at least 60% diagram intervals – amount of activity on diagrams equation intervals – amount of activity on equations digressions break intervals breaks between 2 and 40 sec breaks between 40 and 160 sec breaks between greater than 160 sec
developed a total of 41 features to characterize a solution process. Here we use 10 of these to examine the relationship between problem-solving fluency based on pen stroke data and score on the answer.1 We compute three measures based on the spatial arrangement of the written solution. We characterize the progression of work down a page in terms of the deviation from a reference timeline constructed using a sliding window as shown in Fig. 3. The window is initially placed at the top of the solution. The timestamp of the earliest stroke in the window is used as the reference time at the window's location (i.e., at the top of the window). The window is then slid down the page a small distance. The reference time assigned to the new window location is again that of the earliest stroke in the window, unless that is earlier than the time assigned to the previous window location. In that case, the reference time is the same as that of the previous location. This process is repeated until the bottom of the solution is reached, resulting in a sequence of monotonically increasing reference time values. Once the reference timeline has been constructed, it is used to identify “out-of-order” strokes — i.e., strokes that are inconsistent with a top-down spatial progression. To do this, the reference time at the location of each stroke’s midpoint is computed by linear interpolation of the reference timeline. We consider a pen stroke to be written out-oforder to the extent that the stroke’s timestamp differs from the reference timeline. For example, if a solution took a total of 1000 s and a pen stroke differs from the reference timeline by 250 s, we consider the stroke to be out of order by 250/1000 = 25%. We then count the number of Moderately Out-of-Order strokes that are written out of order by between 40% and 50%. Similarly, Substantially Out-of-Order strokes are those written out of order by between 50% and 60%, and Extensively Out-of-Order strokes are those written out of order by at least 60%. To characterize the sequence of problem-solving activities in a student’s solution, we discretize the work into 400 time intervals and then label each according to the predominant solution activity (Fig. 4). By employing a fixed number of intervals, this representation abstracts away the total solution time. We distinguish between four kinds of activities: drawing diagrams, constructing and solving equations,
MO SO EO DA EA D TB SB MB LB
crossing out work, and digressing (i.e., temporarily working on other problems). Because the digital pens use ink which cannot be erased, students must cross out incorrect work. We identify the activity occurring in an interval using a technique developed by Lin, Stahovich, and Herold (2012) that classifies writing into one of three categories: diagrams, equations, and cross-outs. We label an interval according to the majority category of the writing. For example, if the majority of the pen strokes written during an interval are diagram pen strokes, the interval as a whole is labeled as such. If no writing occurs during an interval, it is labeled as a break. Students typically require roughly 20 min to solve an exam problem. As we divide the work into 400 intervals, this results in intervals that are roughly 3 s in duration, although the exact duration depends on the total solution time for each particular problem. Each digression — a period of time during which the student interrupts his or her work on a problem to work on other problems — is represented as a single interval regardless of how long the digression is. From this representation, we compute four measures of fluency based on the distribution of problem-solving activities. Diagram Activity, Equation Activity, Number of Digressions, and Number of Breaks are the number of intervals of each of those types of activities, respectively. Number of Breaks provides a measure of the total fraction of the time during which the student was not writing. To characterize the size distribution of these periods of inactivity, we consider three additional measures. Small Breaks is the number of breaks lasting between 2 and 40 s, Medium Breaks is the number lasting between 40 and 160 s, and Long Breaks is the number lasting at least 160 s. These features are computed directly from the original timeline of the solution, rather than from the discrete activity sequence. The exam problems were graded by teaching assistants based on rubrics developed by the course instructor. These rubrics assigned credit for the correctness of individual problem-solving elements. The score for each problem was normalized by the number of possible points to produce a continuous value in the range from 0.0 to 1.0 (i.e., partial credit was possible). The rubric for the problem in Fig. 1, for example, considers the four major elements of the solution: (1) The first element is the analysis of pulley B which includes a free body diagram and two force equilibrium equations. This portion of the problem was assigned four points, with 1.5 points for the free body diagram and the remaining 2.5 points divided over the two equations. (2) The second element is the analysis of member BCD which includes a free body diagram, one moment equilibrium equation, two force equilibrium equations, and a force magnitude calculation. This portion of the problem was assigned six points,
1 The 31 features not used here characterize other properties not directly related to fluency or more fine-grained aspects of a student’s handwritten solution including: (1) the organization of the writing into regions containing only equations or only diagrams, (2) the number of pen strokes used to cross out undesired writing, (3) the number and median size of equation, diagram, and cross-out pen strokes, (4) properties of the temporal sequence of activities in the discretized activity sequence (e.g., Fig. 4), and (5) fine-grained characterizations of out-of-order writing, including time delays between individual pen strokes.
334
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 3. Sliding window used to compute a reference timeline and identify out-of-order pen strokes.
would be assigned to the subsequent steps of the problem, even if they were consistent with the erroneous setup. The problems were scored by the graduate student teaching assistants assigned to the class. To determine the reliability of the scoring, we randomly selected 25 students and rescored their problems. We had access to the rubrics for nine of the 13 problems, but not for problems 1, 2, 4, and 6 (because they were accidentally discarded by the teaching assistants). As the rubrics are quite uniform in the amount of credit assigned for various types of elements such as free body diagrams, force equilibrium equations, and moment equilibrium equations, we constructed new rubrics for these four problems for use in the rescoring process. The scores assigned by the original scorers were consistent with those assigned by the new scorer. For the complete set 325 problems (13 problems for each of 25 students), the Pearson correlation between the two sets of scores is r = 0.87. In this comparison, we remedied two outlying data points. First, for problem 13, one student originally received a score of zero, but received full points in the new scoring. This student’s solution was unambiguously correct and the zero may have been a recording error or an administrative issue. Thus, for our analysis, we considered the original grade to be full credit. The second outlier involved one student’s solution to problem four. This student employed an unusual solution approach that was correct for some elements and incorrect for others. A more careful analysis of the student’s solution revealed that the original grader assigned a score that was too low, while the new grader assigned a score that was too high. As this discrepancy is due to the unusual nature of the solution, this problem was eliminated from the analysis as being uncharacteristic of the scoring. If this problem was included, however, the correlation between the two sets of scores would be nearly unchanged at r = 0.86. Table 2 shows the correlations between the original scores and the new scores disaggregated by problem (with the two adjustments for outliers). The correlation coefficients range from 0.60 to 0.96 with an average of 0.85. Overall, these results demonstrate that the scoring of the problems is reliable.
Fig. 4. A portion of a typical discretized activity sequence.
with 1.5 points for the free body diagram and the remaining 4.5 points divided over the other four sub-elements. (3) The third element is the analysis of joint E which includes a free body diagram and two force equilibrium equations. This portion of the problem was assigned five points, with 1.5 points for the free body diagram and the remaining 3.5 points divided over the two equations. (4) The fourth element is the analysis of member FG which includes a free body diagram, one moment equilibrium equation, two force equilibrium equations, and a force magnitude calculation. This portion of the problem was assigned five points, with 1.5 points for the free body diagram and the remaining 3.5 points divided over the other four sub-elements. If a student combined solution steps, then the points for those elements were combined. For example, it is possible for the analysis of the pulley and member BCD to be combined into one step. (The solution in Fig. 1 employs this combined approach, although there is an error in the execution.) In this case, the combined work would be graded out of 10 points. Points were deducted in proportion to the severity of the errors. For example, for the analysis of the pulley, 2.5 points were divided over the two force equilibrium equations. If a student failed to do any work for an equation, 1.25 points would be deducted for that equation. If an equation was attempted, but had small errors, such as an incorrect sign, a small deduction (i.e., 0.5 points) would be made. For a larger error, such as a missing force, a larger deduction (i.e., 1.0 point) would be made. If the equation was completely incorrect, no credit would be assigned. Typically, errors were penalized only once. For example, if a student drew a force in the wrong direction in a free body diagram, and then derived an equation consistent with the incorrect diagram, points would be deducted only for the error in the diagram and not for the corresponding error in the equation. A free body diagram is a tool for deriving the equations. Thus, if the equation is consistent with the diagram, the student has applied the correct reasoning in deriving the equation. However, if the errors in the setup of the problem were particularly severe (for example the student misinterpreted the problem in such a way that all of the main concepts were avoided), no credit
5. Results 5.1. Elements of fluency during problem solving are related to problemsolving score The primary research question concerns whether fluency metrics 335
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Table 2 Reliability of Scoring: Pearson correlation between scores assigned by teaching assistants and scores assigned by a separate scorer. Problem
1
2
3
4
5
6
7
8
9
10
11
12
13
R
0.96
0.90
0.91
0.60
0.91
0.95
0.86
0.90
0.69
0.95
0.84
0.91
0.67
Table 3 Pearson correlation between measures of fluency and solution score. Variable
r
p
Moderately Out-of-Order Strokes Substantially Out-of-Order Strokes Extensively Out-of-Order Strokes Diagram Activity Equation Activity Digressions Total Breaks Small Breaks Medium Breaks Long Breaks
−0.145 −0.167 −0.309 0.037 0.490 −0.222 −0.429 0.273 −0.097 −0.129
<.001 <.001 <.001 .166 <.001 <.001 <.001 <.001 <.001 <.001
Table 4 Coefficients of determination for regression models predicting the score for each problem individually and all problems combined.
(characterizing problem-solving process) are related to score on the exam problems (characterizing problem-solving product). To address this question, we computed the Pearson correlation between each of the 10 metrics and exam score for the combined set of 1411 exam problems, as summarized in Table 3. Our predictions suggest that the ability to work from the top of the page to the bottom without backtracking is a characteristic of expertise. Our results are consistent with this prediction, as all three measures of out-of-order work are negatively correlated with score. Furthermore, the strength of the negative correlation increases monotonically with the degree to which the work is out of order: the number of Moderately Out-of-Order Strokes has a correlation of r = −0.145 (p < .001), the number of Substantially Outof-Order Strokes has a correlation of r = −0.167 (p < .001), and the number of Extensively Out-of-Order Strokes has a correlation of r = −0.309 (p < .001). Consistent with our other predictions, measures that indicate fluency tend to correlate positively with grade. For example, Equation Activity (i.e., the tendency to generate and solve the required equations) which is an indication of fluency, has a strong positive correlation with score (r = 0.490, p < .001). However, the positive correlation between Diagram Activity (i.e., the tendency to generate diagrams) and score does not reach statistical significance (r = 0.037, p = 0.166). Conversely, measures indicating a lack of fluency are negatively correlated with score. For example, Total Breaks (r = −0.429, p < .001) and Digressions (r = −0.222, p = <.001), which are indications of an inability to make continual progress on a problem, are negatively correlated with score. In taking a closer look at breaks, Small Breaks correlates positively with score (r = 0.273, p < .001), whereas both Medium Breaks (r = −0.097, p < .001) and Long Breaks (r = −0.129, p < .001) are negatively correlated with score. This suggests that while small breaks are common even in expert problem solving, longer breaks are an indication of the lack of expertise. We also conducted a multiple regression for each problem with score as the dependent variable and the 10 measures as the predictor variables. We used stepwise linear regression with a probability of 0.05 for entry and 0.10 for removal. Table 4 lists for each problem the coefficient of determination (r2), the significance (p), the measures selected by the regression in the order selected, the number of exam problem solutions (n), and the mean score. In addition to listing the results for each of the 13 problems considered individually, the table also includes the result for all 13 problems considered together (as shown in the last row). For the individual problems, the regression models achieved a coefficient of determination ranging from 0.178 to 0.574 with an average of 0.401. For all problems considered together, the regression model achieved a coefficient of determination of 0.291.
Problem
r2
p
Variables
n
Mean Score
1 2 3 4 5 6 7 8 9 10 11 12 13 All
0.370 0.493 0.564 0.359 0.568 0.178 0.329 0.397 0.232 0.339 0.449 0.574 0.390 0.291
<.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001
EA, LB, SB EA, D EA, SB, D, LB EA, MB EA, SO EA EA, E0 EA, SB EA, E0, TB EA, S0, M0 EA TB, E0 TB, E0 EA, E0, D, SB, LB, DA, TB
122 121 122 110 109 110 104 103 104 104 103 97 102 1411
0.71 0.68 0.53 0.66 0.43 0.72 0.75 0.71 0.81 0.84 0.72 0.62 0.70 0.68
Note: “Variables” are the problem-solving measures selected by the stepwise regression; refer to Table 1 for the list of measures.
In all cases, a set of fluency metrics accounts for a substantial (and significant) proportion of the variance in score. 5.2. The predictive strength of elements of problem-solving fluency is stronger for more difficult problems If the predictive power of the problem-solving elements is based on less successful students treating the problem as a non-routine problem (which is likely to lead to poorer performance), then the strength of the effect should be greatest for more difficult problems. For easy problems, most students may be treating the problem as a routine problem, leaving less variance to be explained. The final column of Table 4 shows the mean score (from 0.0 to 1.0) for each problem. Consistent with this prediction, the Pearson correlation between mean score and percentage of explained variance (r2) is r = 0.56, p < .05, suggesting that the predictive power of problem-solving elements is stronger for more difficult problems. 6. Discussion 6.1. Empirical contributions This study contributes to the emerging field of engineering education (Johri & Olds, 2014) by focusing on a central ingredient in engineering education — i.e., gaining expertise in solving engineering problems. The major empirical finding is that measures of fluency during problem solving are strong predictors of problem solution score, with an average coefficient of determination of 0.401. This finding shows that handwriting behaviors that students exhibit as they work out their solutions to an engineering problem are related to their level of success on the problem. In short, student success in coming up with solutions to engineering problems can be predicted by the spatial and temporal organization of their pen strokes during problem solving without reference to the accuracy of the actual content of the work, as proposed by Van Arsdale and Stahovich (2012). This work also has broader implications for the educational psychology of problem-solving transfer by showing that a characteristic of success in academic problem-solving involves fluency — working from top-to-bottom without hesitations and sidetracks. 336
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
6.2. Theoretical contributions
Acknowledgement
Consistent with long-standing cognitive and educational psychology theories of problem-solving expertise (Chi, Glaser, & Farr, 1988; de Groot, 1965; Mayer & Wittrock, 1996, 2009), successful problem-solving outcomes on engineering problems were associated with elements of fluency — working from the top to the bottom of the page without detours or long pauses, and the ability to generate and construct the relevant equations. One way to conceptualize the theme of this study is that more successful problem solvers behave as if they are solving a routine problem — i.e., they know the solution plan and are simply executing it as efficiently as possible. This suggests that becoming a competent engineering student involves receiving enough training and practice to possess a large repertoire of problem schemas for common engineering problems. Thus, when confronted with a new problem, the student can mentally categorize it and systematically apply the corresponding solution plan for that problem type. The implications for creative problem solving (such as in workplace problems) are less clear.
This material is based upon work supported by the National Science Foundation under Award Numbers 0935239, 1432820, and 1612511. Livescribe, Inc. provided some of the materials used in the project. References Bloom, B. S., & Broder, L. J. (1950). Problem-solving processes of college students. Chicago, IL: University of Chicago Press. Bonney, C. R., & Sternberg, R. J. (2017). Learning to think critically. In R. E. Mayer, & P. A. Alexander (Eds.). Handbook of research on learning and instruction (pp. 175–206). (2nd ed). New York: Routledge. Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The nature of expertise. Hillsdale, NJ: Erlbaum. de Groot, A. D. (1965). Thought and choice in chess. The Hague: Mouton Publishers. Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (Eds.). (2006). The Cambridge handbook of expertise and expert problem solving. New York: Cambridge University Press. Grossberg, S. (2013). Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world. Neural Networks, 37, 1–47. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge. Hinsley, D., Hayes, J. R., & Simon, H. A. (1977). From words to equations. In M. Just, & P. Carpenter (Eds.). Cognitive processes in comprehension (pp. 89–108). Hillsdale, NJ: Erlbaum. Holvikivi, J. (2007). Logical reasoning ability in engineering students: A case study. IEEE Transactions on Education, 50(4), 367–372. Holyoak, K. J. (2005). Analogy. In K. J. Holyoak, & R. G. Morrison (Eds.). The Cambridge handbook of thinking and reasoning (pp. 117–142). New York: Cambridge University Press. Jonassen, D. H. (2011). Learning to solve problems. New York: Routledge. Jonassen, D. H., Strobel, J., & Lee, C. B. (2006). Everyday problem solving in engineering: Lessons for engineering educators. Journal of Engineering Education, 95(2), 1–14. Johri, A., & Olds, B. M. (Eds.). (2014). Cambridge handbook of engineering education research. New York: Cambridge University Press. Lin, H., Stahovich, T., Herold, J. (2012). Automatic Handwritten Statics Solution Classification and its Applications in Predicting Student Performance. In Proceedings of the 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. Mayer, R. E. (1981). Frequency norms and structural analysis of algebra story problems into families, categories, and templates. Instructional Science, 10, 135–175. Mayer, R. E. (1992). Thinking, problem solving, cognition (2nd ed.). New York: Freeman. Mayer, R. E. Problem solving. In V. S. Ramachandran (Ed.). Encyclopedia of Human Behavior (2nd ed.; vol. 3; pp. 181–186), 2012. Oxford, UK: Elsevier. Mayer, R. E. (2013). Problem solving. In D. Reisberg (Ed.). Oxford handbook of cognitive psychology (pp. 769–778). New York: Oxford University Press. Mayer, R. E., & Wittrock, M. C. (1996). Problem solving and transfer. In D. Berliner, & R. Calfee (Eds.). Handbook of educational psychology (pp. 45–61). New York: Macmillan. Mayer, R. E., & Wittrock, M. C. (2009). Problem solving. In E. Anderman (Ed.). Psychology of classroom learning: An encyclopedia (pp. 702–706). Detroit: Macmillan Reference. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: PrenticeHall. Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: National Academies Press. Rawson, K., Stahovich, T. F., & Mayer, R. E. (2017). Homework and achievement: Using smartpen technology to find the connection. Journal of Educational Psychology, 109, 208–219. Riley, M. S., Greeno, J. G., & Heller, J. I. (1983). Development of children's problemsolving ability in arithmetic. In H. P. Ginsburg (Ed.). The development of mathematical thinking (pp. 153–200). New York: Academic Press. Smith, M. U. (Ed.). (1991). Toward a unified theory of problem solving. Hillsdale, NJ: Erlbaum. Star, J. R., & Rittle-Johnson, B. (2016). Toward an educational psychology of mathematics education. In L. Corno, & E. M. Anderman (Eds.). Handbook of educational psychology (pp. 257–268). (3rd ed). New York: Routledge. Sternberg, R. J., & Grigorenko, E. (Eds.). (2003). The psychology of abilities, competencies, and expertise. New York: Cambridge University Press. Tabak, I. (2016). Functional scientific literacy: Seeing the science within the words an across the web. In L. Corno, & E. M. Anderman (Eds.). Handbook of educational psychology (pp. 269–280). (3rd ed). New York: Routledge. Taraban, R. (2011). Information fluency growth through engineering curricula: Analysis of students' text processing skills and beliefs. Journal of Engineering Education, 100(2), 397–416. Van Arsdale, T. S., & Stahovich, T. (2012). Does neatness count? What the organization of student work says about understanding. Proceedings of American Society for Engineering Education Annual Conference. San Antonio, TX: American Society for Engineering Education.
6.3. Practical contributions This is a correlational study rather than an experiment so no causal conclusions can be drawn. However, the results suggest the testable proposal that training and practice with solving common engineering problems can provide students with the knowledge they need to solve problems with fluency. In short, it would be useful to test the effectiveness of interventions aimed at helping students to develop schemas for common types of engineering problems, so they approach them as routine problems. Overall, this work fits in with broader trends in educational psychology on recognizing problem solving as a fundamental 21st century skill (Bonney & Sternberg, 2017; Pellegrino & Hilton, 2012). 6.4. Methodological contributions This work shows the unique contribution of smartpen technology in revealing a level of detail concerning student problem-solving processes that is not otherwise easily accessible. This work also shows the contribution of data mining techniques in educational research, which allow for the construction of metrics of the process of problem solving that are educationally relevant. 6.5. Limitations and future directions This study was carried out in one engineering course at one university, so it would be useful to replicate the study in other contexts. Further empirical and statistical work is needed on refining the 10 measures of problem-solving behavior to consider other aspects of fluency. Overall, our measures of problem-solving behaviors accounted for 40% of the variance in outcome scores, which is educationally significant (Hattie, 2009), but there certainly is room for improvement. A possible next step is to conduct an intervention study to determine if training and practice that increases fluency during problem solving also increases scores on problem solutions. Another possible future study is needed to compare the handwriting behaviors of experts and novices as they solve classroom engineering problems. Declarations of interest None.
337
Contents lists available at ScienceDirect
Contemporary Educational Psychology journal homepage: www.elsevier.com/locate/cedpsych
How handwriting behaviors during problem solving are related to problemsolving success in an engineering course
T
Thomas F. Stahovicha,⁎, Timothy S. Van Arsdaleb, Richard E. Mayerc a
University of California, Riverside, United States Raytheon Company, United States c University of California, Santa Barbara, United States b
ARTICLE INFO
ABSTRACT
Keywords: Problem solving Engineering Expertise Smartpens
If we carefully observe the spatial and temporal organization of students' pen strokes as they solve an engineering problem, can we predict their ability to achieve the correct answer? To address this question, 122 college students were asked to solve exam problems in an engineering course using a smartpen that recorded their writing as digitized timestamped pen strokes. The pen stroke data was used to compute a collection of 10 metrics characterizing various elements of problem-solving fluency including the tendency to progress down the page without revisions, the amount of time with no activity, and the frequency of constructing and using equations. The primary finding is that, on average across 13 different exam problems, these elements of problemsolving process explained 40% of the variance in scores of the correctness of the problem solution. In short, success on generating correct solutions was related to the fluency of the student's problem-solving process (i.e., working sequentially from the top to the bottom of the page, working without detours or long pauses, and working by constructing equations). This work is consistent with the idea that expertise in solving common engineering problems involves being able to treat them like routine rather than non-routine problems.
1. Objective and rationale Problem solving is at the heart of engineering education. College courses in engineering often require students to solve applied physics problems, such as the one in Fig. 1, in homework, quizzes, and exams. By emphasizing instruction, practice, and assessment in solving problems, instructors are making clear that an overarching aim is to help students develop problem-solving expertise. Consider the engineering problem in Fig. 1, taken from an exam in an introductory statics course. Suppose one student solves the problem by working from the top of the page to the bottom without going back to revise previous work and without long pauses. This student also spends a lot of time and effort working on equations. In contrast, suppose another student starts at the top of the page, then jumps to the bottom, and then the middle, and then back up to the top, including making revisions to earlier work and taking long breaks with no activity at all. This student also spends less time and effort on writing and solving equations. Which student is more likely to produce the correct answer? The present study seeks to address this question. The goal of the present study is to determine whether the
handwritten solutions produced by students as they solve exam problems provide clues concerning which problem-solving behaviors are related to success as measured by the correctness of the work. In short, our goal is to determine which elements of the problem-solving process (indicated by the students' pen strokes during problem solving) are predictive of the problem-solving product (indicated by the score assigned by the instructor based on a scoring rubric). Problem-solving transfer — the ability to use what was learned to solve new problems — is a long-standing issue in educational psychology (Mayer & Wittrock, 1996, 2009); a central focus of educational psychology research on learning in STEM disciplines (Star & RittleJohnson, 2016; Tabak, 2016); and a foundational 21st century skill (Bonney & Sternberg, 2017; Pellegrino & Hilton, 2012). In analyzing problem types, Jonassen and colleagues distinguish between classroom problems in engineering, which typically are well-defined problems, and workplace problems in engineering, which typically are ill-defined problems (Jonassen, 2011; Jonassen, Strobel, & Lee, 2006). The classroom problem shown in Fig. 1 is well defined because the givens, goals, and allowable operations are precisely specified (Mayer, 1992). Although such classroom problems may be appropriate as initial work for
⁎ Corresponding author at: Department of Mechanical Engineering, Bourns College of Engineering, A349 Bourns Hall, University of California, Riverside, CA 92521, United States. E-mail address: [email protected] (T.F. Stahovich).
https://doi.org/10.1016/j.cedpsych.2019.04.004
Available online 27 April 2019 0361-476X/ © 2019 Published by Elsevier Inc.
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 1. Exam problem 5 (left panel) and a nearly correct handwritten solution (right panel).
specially instrumented “smartpens” that record every pen stroke with timestamps. We then developed algorithms to extract various measures of fluency from the captured ink data, and examined the relationships between these measures and the correctness of the work. Together, this technology enables us to analyze the problem-solving behaviors of students as they solve typical engineering problems during midterm and final exams, and determine the relation between problem-solving behaviors (as indicated by the spatial and temporal organization of the writing) and their score on the problem solution. Overall, our goal is to determine whether it is possible to predict students' scores on solutions to engineering problems (i.e., the outcome of problem solving) based on their behaviors during the generation of their solutions (i.e., the process of problem solving).
beginning students, the ultimate goal for graduating students is to be able to solve workplace problems, such as through project-based assignments (Jonassen et al., 2006). Classic research (Chi, Glaser, & Farr, 1988; de Groot, 1965; Newell & Simon, 1972; Sternberg & Grigorenko, 2003) and modern research (Mayer, 2012, 2013) on the development of problem-solving expertise in diverse domains suggests that successful problem solving may be characterized by fluency — working from start to end without hesitations and sidetracks, and focusing on the aspects of the problem solution that are relevant. For example, Holvikivi (2007) has shown the need for stronger emphasis in developing problem-solving skills in engineering education and has suggested that fluency is a characteristic of effective problem solving. Taraban (2011) has shown that students entering engineering programs tend to have better problem-solving fluency than non-engineering students, and that experience in solving problems within the engineering curriculum leads to improvements in the development of problem-solving fluency. Grossberg (2013) has shown that cognitive fluency is a result of learning to adapt to new situations. Van Arsdale and Stahovich (2012) provide preliminary evidence that successful engineering problem solvers are more likely to provide spatially neat solutions, working from the top of the sheet to the bottom. Therefore, fluency in producing problem solutions is the focus of this study. In the case of solving engineering problems such as in Fig. 1, fluency involves working through the problem systematically from top to bottom, without breaks and jumps from one part of the page to another. Some indications of fluency in this context include: having few out-oforder pen strokes (i.e., writing added to previously completed areas on the page), having few breaks (i.e., time periods in which no pen strokes are made), and tending to efficiently generate and use all of the relevant equations. If fluency is indicative of successful problem solving, we expect elements of fluency such as these to correlate with problemsolving ability as measured by the grade the student receives on the problem according to a scoring rubric. To test this idea, we conducted a study in which students used
2. Literature review In the field of problem solving, it is customary to make a distinction between routine problems and non-routine problems (Mayer, 2012, 2013). With routine problems, the problem solver may not know the answer but knows how to compute it. For example, for most adults, computing the value of 576 × 29 is a routine problem because they know how to carry out long multiplication. With non-routine problems, the problem solver does not know how to go about computing a solution and so must devise and revise a solution plan. For example, computing the expected life of a rechargeable battery is a non-routine problem because the solution plan is not obvious. Whether or not a problem is routine or non-routine depends on the knowledge of the problem solver. Research shows that successful problem solvers possess schemas for problem types, so that when they see a new problem they can recognize it as fitting with problems of the same type that they know how to solve (Hinsley, Hayes, & Simon, 1977; Mayer, 1981; Riley, Greeno, & Heller, 1983). For example, successful problem solvers often use analogical reasoning (Holyoak, 2005) to solve a new problem (i.e., a target problem) by recognizing a corresponding schema in long-term memory that has the same structural 332
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
characteristics, abstracting the solution method from the schema, and applying it to the target problem. For experienced engineers, most of the problems given to students in introductory engineering courses are routine problems, because experienced engineers possess a large storehouse of schemas reflecting the problem types for commonly used problems in introductory engineering courses. In contrast, beginning engineering students may not possess a large body of problem schemas and therefore must devise and revise solution plans for the problems they encounter in introductory engineering courses. Research on problem-solving expertise shows major differences in the problem-solving processes of experts and novices in a given domain (de Groot, 1965; Mayer, 1992; Newell & Simon, 1972). Experts tend to systematically work forward from the problem statement to the answer because they can simply carry out their solution plan. In solving common engineering problems, expert-like processing is reflected in a high degree of fluency — writing continuously from the top to the bottom of the page with few detours or long pauses. In contrast, novices must develop a solution plan, sometimes through a process of means-ends analysis (Mayer, 2012, 2013; Newell & Simon, 1972) in which problem solvers work backwards from a statement of the goal to establish and execute a series of ever-progressing subgoals. In solving common engineering problems, novice-like processing is reflected in a low degree of fluency — writing in an unstructured sequence of locations on the page, going back to make revisions, and taking long pauses. Creating a solution plan in this way may result in the novice having difficulty in constructing the relevant equations for an engineering problem. It is also customary in the problem-solving literature to distinguish between the process of problem solving and the product of problem solving (Bloom & Broder, 1950; Mayer, 2012, 2013). Problem-solving process refers to the processing the problem solver engages in while generating a solution, including representing the problem, devising a plan, and carrying out the plan. Problem-solving product (or outcome) refers to the solution that was generated. In the present study, we seek to determine whether key expert-novice differences in problem-solving process (i.e., differences in fluency) are related to differences in problem-solving product (i.e., differences in the instructor-assigned score for the problem solution). In short, we explore what can be called the paradox of expertise — the knowledge that experts bring to a common engineering problem that allows them to solve it as a routine problem, which involves recognizing a relevant problem schema and systematically carrying out the corresponding solution plan, whereas novices must go through the creative process of devising a solution plan (reflected in low fluency in handwriting a solution).
can be used to predict student success on the outcome of problem solving. Based on research on expertise in other domains, we expect that expert problem-solving success is related to fluency during the process of generating a solution. In the context of engineering education, this means that the degree to which students possess higher levels of expertise leading to successful problem solving will be related to the degree to which they exhibit problem-solving behaviors during generating a handwritten solution that show signs of fluency — such as working from top to bottom of the page without jumps and long pauses, and the ability to construct and solve the relevant equations. 4. Method 4.1. Participants and design The participants were 122 students from a one-quarter, sophomorelevel course in statics taught at the University of California, Riverside. A total of 113 of the participants completed the course. Demographic data is available for 109 of these students. Most students (85%) were male and the majority were mechanical engineering majors (53%) or other engineering majors (33%). This is an observational study in which all participants solved the same set of 13 exam problems. The predictor variables are based on 10 smartpen metrics derived from behavior in writing solutions to each problem, and the dependent variable is the score assigned on the problem by the instructor based on a scoring rubric. There were between 97 and 122 solutions for each problem, with some data lost on some problems due to technical issues. We followed standards for treatment of human subjects and obtained IRB approval. 4.2. Materials The materials consisted of 13 statics problems that students solved on exams. These are classroom problems typically used in introductory engineering courses rather than workplace problems that would be encountered on the job. The students wrote the solutions on paper using smartpens. Problems 1–3 came from midterm 1, problems 4–6 came from midterm 2, and problems 7–13 came from the final exam. Fig. 1 shows problem 5, which was one of the more difficult problems for students. 4.3. Procedure Students used Livescribe smartpens to solve problems on the midterm and final exams. Students were instructed to show their work as they solved each problem.
3. Theoretical framework Building on long-standing research on expertise (Chi, Glaser, & Farr, 1988; Ericsson, Charness, Feltovich, & Hoffman, 2006; Mayer, 1992; Newell & Simon, 1972; Smith, 1991), we explore the theoretical assertion that experts and novices differ in the way they approach problem solving. Experts possess schemas for problem types, so that when they see a new problem it reminds them of other problems like it that they know how to solve. They can then apply the solution method by working forward from the given information in the problem to compute the answer. In contrast, novices may not possess a schema for a problem they are trying to solve, and therefore must use a means-ends analysis method that involves working backwards from the goal to the givens in a problem. This process will involve some jumping around on the page and even some long pauses, as the student determines what to do next and continually revises the solution plan. In the case of applied physics problems used in engineering education, novices may struggle to construct and use the relevant equations, whereas experts can do so more easily. Fig. 2 summarizes the theoretical framework that guides this study examining how student behaviors during the process of problem solving
4.4. Instrumentation The Livescribe smartpens that students used to write solutions to the exam problems perform the same function as a traditional ballpoint pen, but also digitize each pen stroke as it is written (Rawson, Stahovich, & Mayer, 2017). The smartpens are used with paper preprinted with micro-dots that are used by a camera embedded in the smartpen to determine the location of the pen tip. Each pen stroke is stored on the smartpen as a sequence of timestamped Cartesian coordinates. After the problem-solving session, a USB cable is used to download the digitized data to a PC. 4.5. Scoring and data analysis Table 1 lists 10 problem-solving measures (or metrics) we computed from each student’s pen strokes written during the process of generating a solution to a problem. The measures are intended to capture aspects of problem-solving fluency. In our broader work to develop methods for automatically grading students’ handwritten problem solutions, we 333
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 2. Relation between process and product of problem solving in engineering.
Table 1 Problem-solving measures. Measure
Description
Moderately Out-of-Order Strokes Substantially Out-of-Order Strokes Extensively Out-of-Order Strokes Diagram Activity Equation Activity Digressions Total Breaks Small Breaks Medium Breaks Long Breaks
Number Number Number Number Number Number Number Number Number Number
of of of of of of of of of of
Symbol strokes written out of order by between 40% and 50% strokes written out of order by between 50% and 60% strokes written out of order by at least 60% diagram intervals – amount of activity on diagrams equation intervals – amount of activity on equations digressions break intervals breaks between 2 and 40 sec breaks between 40 and 160 sec breaks between greater than 160 sec
developed a total of 41 features to characterize a solution process. Here we use 10 of these to examine the relationship between problem-solving fluency based on pen stroke data and score on the answer.1 We compute three measures based on the spatial arrangement of the written solution. We characterize the progression of work down a page in terms of the deviation from a reference timeline constructed using a sliding window as shown in Fig. 3. The window is initially placed at the top of the solution. The timestamp of the earliest stroke in the window is used as the reference time at the window's location (i.e., at the top of the window). The window is then slid down the page a small distance. The reference time assigned to the new window location is again that of the earliest stroke in the window, unless that is earlier than the time assigned to the previous window location. In that case, the reference time is the same as that of the previous location. This process is repeated until the bottom of the solution is reached, resulting in a sequence of monotonically increasing reference time values. Once the reference timeline has been constructed, it is used to identify “out-of-order” strokes — i.e., strokes that are inconsistent with a top-down spatial progression. To do this, the reference time at the location of each stroke’s midpoint is computed by linear interpolation of the reference timeline. We consider a pen stroke to be written out-oforder to the extent that the stroke’s timestamp differs from the reference timeline. For example, if a solution took a total of 1000 s and a pen stroke differs from the reference timeline by 250 s, we consider the stroke to be out of order by 250/1000 = 25%. We then count the number of Moderately Out-of-Order strokes that are written out of order by between 40% and 50%. Similarly, Substantially Out-of-Order strokes are those written out of order by between 50% and 60%, and Extensively Out-of-Order strokes are those written out of order by at least 60%. To characterize the sequence of problem-solving activities in a student’s solution, we discretize the work into 400 time intervals and then label each according to the predominant solution activity (Fig. 4). By employing a fixed number of intervals, this representation abstracts away the total solution time. We distinguish between four kinds of activities: drawing diagrams, constructing and solving equations,
MO SO EO DA EA D TB SB MB LB
crossing out work, and digressing (i.e., temporarily working on other problems). Because the digital pens use ink which cannot be erased, students must cross out incorrect work. We identify the activity occurring in an interval using a technique developed by Lin, Stahovich, and Herold (2012) that classifies writing into one of three categories: diagrams, equations, and cross-outs. We label an interval according to the majority category of the writing. For example, if the majority of the pen strokes written during an interval are diagram pen strokes, the interval as a whole is labeled as such. If no writing occurs during an interval, it is labeled as a break. Students typically require roughly 20 min to solve an exam problem. As we divide the work into 400 intervals, this results in intervals that are roughly 3 s in duration, although the exact duration depends on the total solution time for each particular problem. Each digression — a period of time during which the student interrupts his or her work on a problem to work on other problems — is represented as a single interval regardless of how long the digression is. From this representation, we compute four measures of fluency based on the distribution of problem-solving activities. Diagram Activity, Equation Activity, Number of Digressions, and Number of Breaks are the number of intervals of each of those types of activities, respectively. Number of Breaks provides a measure of the total fraction of the time during which the student was not writing. To characterize the size distribution of these periods of inactivity, we consider three additional measures. Small Breaks is the number of breaks lasting between 2 and 40 s, Medium Breaks is the number lasting between 40 and 160 s, and Long Breaks is the number lasting at least 160 s. These features are computed directly from the original timeline of the solution, rather than from the discrete activity sequence. The exam problems were graded by teaching assistants based on rubrics developed by the course instructor. These rubrics assigned credit for the correctness of individual problem-solving elements. The score for each problem was normalized by the number of possible points to produce a continuous value in the range from 0.0 to 1.0 (i.e., partial credit was possible). The rubric for the problem in Fig. 1, for example, considers the four major elements of the solution: (1) The first element is the analysis of pulley B which includes a free body diagram and two force equilibrium equations. This portion of the problem was assigned four points, with 1.5 points for the free body diagram and the remaining 2.5 points divided over the two equations. (2) The second element is the analysis of member BCD which includes a free body diagram, one moment equilibrium equation, two force equilibrium equations, and a force magnitude calculation. This portion of the problem was assigned six points,
1 The 31 features not used here characterize other properties not directly related to fluency or more fine-grained aspects of a student’s handwritten solution including: (1) the organization of the writing into regions containing only equations or only diagrams, (2) the number of pen strokes used to cross out undesired writing, (3) the number and median size of equation, diagram, and cross-out pen strokes, (4) properties of the temporal sequence of activities in the discretized activity sequence (e.g., Fig. 4), and (5) fine-grained characterizations of out-of-order writing, including time delays between individual pen strokes.
334
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Fig. 3. Sliding window used to compute a reference timeline and identify out-of-order pen strokes.
would be assigned to the subsequent steps of the problem, even if they were consistent with the erroneous setup. The problems were scored by the graduate student teaching assistants assigned to the class. To determine the reliability of the scoring, we randomly selected 25 students and rescored their problems. We had access to the rubrics for nine of the 13 problems, but not for problems 1, 2, 4, and 6 (because they were accidentally discarded by the teaching assistants). As the rubrics are quite uniform in the amount of credit assigned for various types of elements such as free body diagrams, force equilibrium equations, and moment equilibrium equations, we constructed new rubrics for these four problems for use in the rescoring process. The scores assigned by the original scorers were consistent with those assigned by the new scorer. For the complete set 325 problems (13 problems for each of 25 students), the Pearson correlation between the two sets of scores is r = 0.87. In this comparison, we remedied two outlying data points. First, for problem 13, one student originally received a score of zero, but received full points in the new scoring. This student’s solution was unambiguously correct and the zero may have been a recording error or an administrative issue. Thus, for our analysis, we considered the original grade to be full credit. The second outlier involved one student’s solution to problem four. This student employed an unusual solution approach that was correct for some elements and incorrect for others. A more careful analysis of the student’s solution revealed that the original grader assigned a score that was too low, while the new grader assigned a score that was too high. As this discrepancy is due to the unusual nature of the solution, this problem was eliminated from the analysis as being uncharacteristic of the scoring. If this problem was included, however, the correlation between the two sets of scores would be nearly unchanged at r = 0.86. Table 2 shows the correlations between the original scores and the new scores disaggregated by problem (with the two adjustments for outliers). The correlation coefficients range from 0.60 to 0.96 with an average of 0.85. Overall, these results demonstrate that the scoring of the problems is reliable.
Fig. 4. A portion of a typical discretized activity sequence.
with 1.5 points for the free body diagram and the remaining 4.5 points divided over the other four sub-elements. (3) The third element is the analysis of joint E which includes a free body diagram and two force equilibrium equations. This portion of the problem was assigned five points, with 1.5 points for the free body diagram and the remaining 3.5 points divided over the two equations. (4) The fourth element is the analysis of member FG which includes a free body diagram, one moment equilibrium equation, two force equilibrium equations, and a force magnitude calculation. This portion of the problem was assigned five points, with 1.5 points for the free body diagram and the remaining 3.5 points divided over the other four sub-elements. If a student combined solution steps, then the points for those elements were combined. For example, it is possible for the analysis of the pulley and member BCD to be combined into one step. (The solution in Fig. 1 employs this combined approach, although there is an error in the execution.) In this case, the combined work would be graded out of 10 points. Points were deducted in proportion to the severity of the errors. For example, for the analysis of the pulley, 2.5 points were divided over the two force equilibrium equations. If a student failed to do any work for an equation, 1.25 points would be deducted for that equation. If an equation was attempted, but had small errors, such as an incorrect sign, a small deduction (i.e., 0.5 points) would be made. For a larger error, such as a missing force, a larger deduction (i.e., 1.0 point) would be made. If the equation was completely incorrect, no credit would be assigned. Typically, errors were penalized only once. For example, if a student drew a force in the wrong direction in a free body diagram, and then derived an equation consistent with the incorrect diagram, points would be deducted only for the error in the diagram and not for the corresponding error in the equation. A free body diagram is a tool for deriving the equations. Thus, if the equation is consistent with the diagram, the student has applied the correct reasoning in deriving the equation. However, if the errors in the setup of the problem were particularly severe (for example the student misinterpreted the problem in such a way that all of the main concepts were avoided), no credit
5. Results 5.1. Elements of fluency during problem solving are related to problemsolving score The primary research question concerns whether fluency metrics 335
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
Table 2 Reliability of Scoring: Pearson correlation between scores assigned by teaching assistants and scores assigned by a separate scorer. Problem
1
2
3
4
5
6
7
8
9
10
11
12
13
R
0.96
0.90
0.91
0.60
0.91
0.95
0.86
0.90
0.69
0.95
0.84
0.91
0.67
Table 3 Pearson correlation between measures of fluency and solution score. Variable
r
p
Moderately Out-of-Order Strokes Substantially Out-of-Order Strokes Extensively Out-of-Order Strokes Diagram Activity Equation Activity Digressions Total Breaks Small Breaks Medium Breaks Long Breaks
−0.145 −0.167 −0.309 0.037 0.490 −0.222 −0.429 0.273 −0.097 −0.129
<.001 <.001 <.001 .166 <.001 <.001 <.001 <.001 <.001 <.001
Table 4 Coefficients of determination for regression models predicting the score for each problem individually and all problems combined.
(characterizing problem-solving process) are related to score on the exam problems (characterizing problem-solving product). To address this question, we computed the Pearson correlation between each of the 10 metrics and exam score for the combined set of 1411 exam problems, as summarized in Table 3. Our predictions suggest that the ability to work from the top of the page to the bottom without backtracking is a characteristic of expertise. Our results are consistent with this prediction, as all three measures of out-of-order work are negatively correlated with score. Furthermore, the strength of the negative correlation increases monotonically with the degree to which the work is out of order: the number of Moderately Out-of-Order Strokes has a correlation of r = −0.145 (p < .001), the number of Substantially Outof-Order Strokes has a correlation of r = −0.167 (p < .001), and the number of Extensively Out-of-Order Strokes has a correlation of r = −0.309 (p < .001). Consistent with our other predictions, measures that indicate fluency tend to correlate positively with grade. For example, Equation Activity (i.e., the tendency to generate and solve the required equations) which is an indication of fluency, has a strong positive correlation with score (r = 0.490, p < .001). However, the positive correlation between Diagram Activity (i.e., the tendency to generate diagrams) and score does not reach statistical significance (r = 0.037, p = 0.166). Conversely, measures indicating a lack of fluency are negatively correlated with score. For example, Total Breaks (r = −0.429, p < .001) and Digressions (r = −0.222, p = <.001), which are indications of an inability to make continual progress on a problem, are negatively correlated with score. In taking a closer look at breaks, Small Breaks correlates positively with score (r = 0.273, p < .001), whereas both Medium Breaks (r = −0.097, p < .001) and Long Breaks (r = −0.129, p < .001) are negatively correlated with score. This suggests that while small breaks are common even in expert problem solving, longer breaks are an indication of the lack of expertise. We also conducted a multiple regression for each problem with score as the dependent variable and the 10 measures as the predictor variables. We used stepwise linear regression with a probability of 0.05 for entry and 0.10 for removal. Table 4 lists for each problem the coefficient of determination (r2), the significance (p), the measures selected by the regression in the order selected, the number of exam problem solutions (n), and the mean score. In addition to listing the results for each of the 13 problems considered individually, the table also includes the result for all 13 problems considered together (as shown in the last row). For the individual problems, the regression models achieved a coefficient of determination ranging from 0.178 to 0.574 with an average of 0.401. For all problems considered together, the regression model achieved a coefficient of determination of 0.291.
Problem
r2
p
Variables
n
Mean Score
1 2 3 4 5 6 7 8 9 10 11 12 13 All
0.370 0.493 0.564 0.359 0.568 0.178 0.329 0.397 0.232 0.339 0.449 0.574 0.390 0.291
<.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001
EA, LB, SB EA, D EA, SB, D, LB EA, MB EA, SO EA EA, E0 EA, SB EA, E0, TB EA, S0, M0 EA TB, E0 TB, E0 EA, E0, D, SB, LB, DA, TB
122 121 122 110 109 110 104 103 104 104 103 97 102 1411
0.71 0.68 0.53 0.66 0.43 0.72 0.75 0.71 0.81 0.84 0.72 0.62 0.70 0.68
Note: “Variables” are the problem-solving measures selected by the stepwise regression; refer to Table 1 for the list of measures.
In all cases, a set of fluency metrics accounts for a substantial (and significant) proportion of the variance in score. 5.2. The predictive strength of elements of problem-solving fluency is stronger for more difficult problems If the predictive power of the problem-solving elements is based on less successful students treating the problem as a non-routine problem (which is likely to lead to poorer performance), then the strength of the effect should be greatest for more difficult problems. For easy problems, most students may be treating the problem as a routine problem, leaving less variance to be explained. The final column of Table 4 shows the mean score (from 0.0 to 1.0) for each problem. Consistent with this prediction, the Pearson correlation between mean score and percentage of explained variance (r2) is r = 0.56, p < .05, suggesting that the predictive power of problem-solving elements is stronger for more difficult problems. 6. Discussion 6.1. Empirical contributions This study contributes to the emerging field of engineering education (Johri & Olds, 2014) by focusing on a central ingredient in engineering education — i.e., gaining expertise in solving engineering problems. The major empirical finding is that measures of fluency during problem solving are strong predictors of problem solution score, with an average coefficient of determination of 0.401. This finding shows that handwriting behaviors that students exhibit as they work out their solutions to an engineering problem are related to their level of success on the problem. In short, student success in coming up with solutions to engineering problems can be predicted by the spatial and temporal organization of their pen strokes during problem solving without reference to the accuracy of the actual content of the work, as proposed by Van Arsdale and Stahovich (2012). This work also has broader implications for the educational psychology of problem-solving transfer by showing that a characteristic of success in academic problem-solving involves fluency — working from top-to-bottom without hesitations and sidetracks. 336
Contemporary Educational Psychology 58 (2019) 331–337
T.F. Stahovich, et al.
6.2. Theoretical contributions
Acknowledgement
Consistent with long-standing cognitive and educational psychology theories of problem-solving expertise (Chi, Glaser, & Farr, 1988; de Groot, 1965; Mayer & Wittrock, 1996, 2009), successful problem-solving outcomes on engineering problems were associated with elements of fluency — working from the top to the bottom of the page without detours or long pauses, and the ability to generate and construct the relevant equations. One way to conceptualize the theme of this study is that more successful problem solvers behave as if they are solving a routine problem — i.e., they know the solution plan and are simply executing it as efficiently as possible. This suggests that becoming a competent engineering student involves receiving enough training and practice to possess a large repertoire of problem schemas for common engineering problems. Thus, when confronted with a new problem, the student can mentally categorize it and systematically apply the corresponding solution plan for that problem type. The implications for creative problem solving (such as in workplace problems) are less clear.
This material is based upon work supported by the National Science Foundation under Award Numbers 0935239, 1432820, and 1612511. Livescribe, Inc. provided some of the materials used in the project. References Bloom, B. S., & Broder, L. J. (1950). Problem-solving processes of college students. Chicago, IL: University of Chicago Press. Bonney, C. R., & Sternberg, R. J. (2017). Learning to think critically. In R. E. Mayer, & P. A. Alexander (Eds.). Handbook of research on learning and instruction (pp. 175–206). (2nd ed). New York: Routledge. Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The nature of expertise. Hillsdale, NJ: Erlbaum. de Groot, A. D. (1965). Thought and choice in chess. The Hague: Mouton Publishers. Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (Eds.). (2006). The Cambridge handbook of expertise and expert problem solving. New York: Cambridge University Press. Grossberg, S. (2013). Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world. Neural Networks, 37, 1–47. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge. Hinsley, D., Hayes, J. R., & Simon, H. A. (1977). From words to equations. In M. Just, & P. Carpenter (Eds.). Cognitive processes in comprehension (pp. 89–108). Hillsdale, NJ: Erlbaum. Holvikivi, J. (2007). Logical reasoning ability in engineering students: A case study. IEEE Transactions on Education, 50(4), 367–372. Holyoak, K. J. (2005). Analogy. In K. J. Holyoak, & R. G. Morrison (Eds.). The Cambridge handbook of thinking and reasoning (pp. 117–142). New York: Cambridge University Press. Jonassen, D. H. (2011). Learning to solve problems. New York: Routledge. Jonassen, D. H., Strobel, J., & Lee, C. B. (2006). Everyday problem solving in engineering: Lessons for engineering educators. Journal of Engineering Education, 95(2), 1–14. Johri, A., & Olds, B. M. (Eds.). (2014). Cambridge handbook of engineering education research. New York: Cambridge University Press. Lin, H., Stahovich, T., Herold, J. (2012). Automatic Handwritten Statics Solution Classification and its Applications in Predicting Student Performance. In Proceedings of the 2012 ASEE Annual Conference & Exposition, San Antonio, Texas. Mayer, R. E. (1981). Frequency norms and structural analysis of algebra story problems into families, categories, and templates. Instructional Science, 10, 135–175. Mayer, R. E. (1992). Thinking, problem solving, cognition (2nd ed.). New York: Freeman. Mayer, R. E. Problem solving. In V. S. Ramachandran (Ed.). Encyclopedia of Human Behavior (2nd ed.; vol. 3; pp. 181–186), 2012. Oxford, UK: Elsevier. Mayer, R. E. (2013). Problem solving. In D. Reisberg (Ed.). Oxford handbook of cognitive psychology (pp. 769–778). New York: Oxford University Press. Mayer, R. E., & Wittrock, M. C. (1996). Problem solving and transfer. In D. Berliner, & R. Calfee (Eds.). Handbook of educational psychology (pp. 45–61). New York: Macmillan. Mayer, R. E., & Wittrock, M. C. (2009). Problem solving. In E. Anderman (Ed.). Psychology of classroom learning: An encyclopedia (pp. 702–706). Detroit: Macmillan Reference. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: PrenticeHall. Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: National Academies Press. Rawson, K., Stahovich, T. F., & Mayer, R. E. (2017). Homework and achievement: Using smartpen technology to find the connection. Journal of Educational Psychology, 109, 208–219. Riley, M. S., Greeno, J. G., & Heller, J. I. (1983). Development of children's problemsolving ability in arithmetic. In H. P. Ginsburg (Ed.). The development of mathematical thinking (pp. 153–200). New York: Academic Press. Smith, M. U. (Ed.). (1991). Toward a unified theory of problem solving. Hillsdale, NJ: Erlbaum. Star, J. R., & Rittle-Johnson, B. (2016). Toward an educational psychology of mathematics education. In L. Corno, & E. M. Anderman (Eds.). Handbook of educational psychology (pp. 257–268). (3rd ed). New York: Routledge. Sternberg, R. J., & Grigorenko, E. (Eds.). (2003). The psychology of abilities, competencies, and expertise. New York: Cambridge University Press. Tabak, I. (2016). Functional scientific literacy: Seeing the science within the words an across the web. In L. Corno, & E. M. Anderman (Eds.). Handbook of educational psychology (pp. 269–280). (3rd ed). New York: Routledge. Taraban, R. (2011). Information fluency growth through engineering curricula: Analysis of students' text processing skills and beliefs. Journal of Engineering Education, 100(2), 397–416. Van Arsdale, T. S., & Stahovich, T. (2012). Does neatness count? What the organization of student work says about understanding. Proceedings of American Society for Engineering Education Annual Conference. San Antonio, TX: American Society for Engineering Education.
6.3. Practical contributions This is a correlational study rather than an experiment so no causal conclusions can be drawn. However, the results suggest the testable proposal that training and practice with solving common engineering problems can provide students with the knowledge they need to solve problems with fluency. In short, it would be useful to test the effectiveness of interventions aimed at helping students to develop schemas for common types of engineering problems, so they approach them as routine problems. Overall, this work fits in with broader trends in educational psychology on recognizing problem solving as a fundamental 21st century skill (Bonney & Sternberg, 2017; Pellegrino & Hilton, 2012). 6.4. Methodological contributions This work shows the unique contribution of smartpen technology in revealing a level of detail concerning student problem-solving processes that is not otherwise easily accessible. This work also shows the contribution of data mining techniques in educational research, which allow for the construction of metrics of the process of problem solving that are educationally relevant. 6.5. Limitations and future directions This study was carried out in one engineering course at one university, so it would be useful to replicate the study in other contexts. Further empirical and statistical work is needed on refining the 10 measures of problem-solving behavior to consider other aspects of fluency. Overall, our measures of problem-solving behaviors accounted for 40% of the variance in outcome scores, which is educationally significant (Hattie, 2009), but there certainly is room for improvement. A possible next step is to conduct an intervention study to determine if training and practice that increases fluency during problem solving also increases scores on problem solutions. Another possible future study is needed to compare the handwriting behaviors of experts and novices as they solve classroom engineering problems. Declarations of interest None.
337