Design timelines: Concrete and sticky representations of design process expertise

Design timelines: Concrete and sticky representations of design process expertise Cynthia J. Atman, Human Centered Design & Engineering, University of Washington, Seattle, WA, 98105, USA This paper presents findings from the analysis of data from 177 individuals who solved 401 separate design problems. The findings from this body of research show that individuals with more expertise have processes that are more complex, consider a broader set of information, spend more time problem solving and are more likely to demonstrate a cascade pattern in their design activities. This paper also demonstrates the utility of a commitment to a single research methodology, creating a design timeline representation that is both concrete (visible) and sticky (memorable), and using that representation to tell a consistent story about design processes. Examples of using the timelines to teach about design processes demonstrate timelines are indeed sticky for students of design. Ó 2019 The Author. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-ncnd/4.0/). Keywords: design expertise, design education, design cognition, protocol analysis, design representations

“Super valuable! Much more compelling to see real data, detail, makes me believe, instead of tuning out “prescribed” info, can’t trust how they derived it b/c don’t know. Spend another day in our class talking about this research, please!”

Corresponding author: Cynthia J. Atman [email protected]

This quote, also seen in Figure 1, is from a student who has just engaged in a learning experience using timeline representations of design processes generated from several decades of research on design expertise. The sentiment behind this quote is the reason I started my research program on engineering design processes in the early 1990s. The representations are design process timelines, or tracings of design activities over time, that make patterns in the process data become visible. The timelines also graphically convey places where it is possible to consider context in design processes. The timelines enable students to both engage in empirical findings from design process research and relate the findings to their own design experiences. www.elsevier.com/locate/destud 0142-694X Design Studies 65 (2019) 125e151 125 https://doi.org/10.1016/j.destud.2019.10.004 Ó 2019 The Author. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Figure 1 Student response to learning activity using timeline representations from design process expertise research (Atman, Arif, Shroyer, Turns, & Borgford-Parnell, 2016)

In this invited paper, I am following the model of Nigel Cross’ invited paper “Developing design as a discipline” where he describes the work he has done over his career (Cross, 2018). At the end of that paper, Cross includes several calls to action to the design research community to further the work of the development of design as a discipline. One specific call he makes is for “. attention to either confirming or refuting some of the early, singlecase studies that are still relied upon as foundational evidence within our discipline” (p. 707, Cross, 2018). I offer this paper in response to that call. As a worked example, I describe a corpus of data that was designed to be a large sample size from the beginning. The analyses of this data lead to concrete (visible) findings about design process expertise that can be presented in sticky (memorable) representations for learners, as demonstrated with the student quote at the beginning of the paper.

1

Prologue - research goal and approach

Motivated by observing the impact of technology on society and the environment e both positive and negative e the overarching goal of my work is to figure out how to teach engineering students to think broadly and consider context as they engage in their work. Understanding context has long been considered to be part of the doing of engineering, as described in a 2004 report from the U.S. National Academy of Engineers, “successful engineers in 2020 will, as they always have, recognize the broader contexts that are intertwined in technology and it application in society” (National Academy of Engineering, p. 56). A major way that engineers bring their technical knowledge into artifacts and systems is through the design process. Therefore, as I embarked on this work in the early 1990s, I concluded that in order to teach engineering students to think broadly, we needed to do this through the teaching of design processes. Using the frames of constructivism and conceptual change from research on learning and expert-novice studies from cognitive science, I decided to focus my research on understanding the design processes of incoming and graduating engineering students, eventually contrasting these findings with expert practicing professionals. This framing was also influenced by my Ph.D.

126

Design Studies Vol 65 No. C November 2019

research (with Baruch Fischhoff in the Department of Engineering and Public Policy at Carnegie Mellon University) in the area of behavioral decision theory e in contrast to decision theory e as a basis for informed risk communication, eventually contributing to the book Risk Communication: A Mental Models Approach (Morgan, Fischhoff, Bostrom, & Atman, 2002). I chose to utilize verbal protocol analysis (VPA) as a research method to provide an account of the process in design at a detailed level. Having taken a cognitive science course from Herbert Simon and Kurt VanLehn my first semester in graduate school, I was intrigued with the detailed level of insights about processes that could be gained with VPA. The common element of the framings I applied to my research in design was a comparison of how an individual “actually does” something in contrast to how they are told they “should do” something, whether it was making decisions, understanding risk or learning science concepts. There were many books and articles that influenced the frames that I used in this work, a sampling of them are listed here (Chi, Slotta, & De Leeuw, 1994; Ericsson & Simon, 1984; Johnson-Laird, 1983; Kahneman, Slovic, Slovic, & Tversky, 1982; Simon, 1979). The aim for my research was to develop descriptive models of how engineers do design that could be used alongside prescriptive models to teach engineering students to think broadly and consider context when they engage in design processes. To accomplish this goal, my colleagues and I conducted multiple studies of individuals solving design problems that resulted in verbal protocol data from 177 individuals with different levels and kinds of expertise. Many of the individual studies represented in this work had large sample sizes. My reasoning for this was due to my intended audience. In my experience, engineers tend to be persuaded more by quantitative than qualitative analyses. My goal was to convince engineering educators that my findings are concrete and convincing. Therefore, I chose to collect verbal data, develop quantitative measures, and run large enough sample sizes that I could do statistical comparisons across the samples. A body of research of this size is not possible to achieve as an individual. Indeed, in my case I have been quite fortunate to have worked with an amazing set of colleagues and graduate students along the way. Therefore, in the rest of the paper, I use the word “we” when I am describing the research that was done, and “I” when I am noting an opinion, research direction, etc. The makeup of the people referred to as “we” through the paper changes over time and can be determined by examining the cited papers.

2

Design process expertise research findings

In this section I describe the corpus of data and the research methods we used and present the findings from the research.

Design process expertise

127

The corpus of data is characterized by the following attributes:  Data from 177 participants who engaged in 401 separate design problems (298 of which were verbal protocols, 103 involved making a list on paper), as detailed in Figure 2.  Participants with different levels of expertise, most of whom were engineers (149 engineering students e both first-year and graduating, 19 practicing engineering experts, 4 experts and one near-expert in the domain of the design problem they were solving, and 4 engineering educators)  Design problems with varying levels of context and time given for completion  Playground problem (PG) - design a playground for a fictitious neighborhood given a set of constraints (contextual elements important; up to 3 h to solve)  Ping-pong problem (PP) - design a ping-pong ball launching device that can hit a specific target (no context, similar to a typical problem at the end of the chapter in an engineering statics book; up to 30 min to solve)  Street crossing (SC) - design a safe way to cross a dangerous street on campus (requiring knowledge of the local context of the situation; up to 30 min to solves)  Midwest Floods (MWF) - list factors that should be considered to design a retaining wall for the Mississippi river after floods in the Midwest (contextual elements important; approximately 10 min to answer)  Different experimental designs and conditions were employed (e.g., within and across subject experimental designs, with “smallish” interventions such as comparing processes of students who had or had not read a chapter on engineering design, or “larger” comparisons such as comparing processes of students and practicing professionals)  Quality of the proposed artifacts/solutions were also assessed Since educating engineering students was our final objective, we developed a coding scheme that utilized design models that were used prescriptively in the teaching of engineering. We did a content analysis of seven textbooks used to teach design to first-year engineering students and synthesized across the models to develop both the activities we used and the order in which they appear (Moore, Atman, Bursic, Shuman, & Gottfried, 1995). Since this was a lab-based study, we excluded the first and last activities (Identifying a need and Implementation). This resulted in eight activities, grouped in three stages, for the coding scheme for our analyses: Problem Scoping: Problem Definition (PD) e define the problem (identify constraints, criteria, etc.) Gather Information (Gath) e search for and collect information

128

Design Studies Vol 65 No. C November 2019

Figure 2 Corpus of data: presented by number of individuals, instances of problem solving and problem type

Generate Alternative Solutions: Generate Ideas (Gen) e think up potential solutions Model (Mod) e detail how to build solution(s) to the problem Feasibility Analysis (Feas) e assess workability of possible solutions Evaluation (Eval) e compare and contrast possible solutions Project Realization: Decision (Dec) e select idea or solution from among alternatives Communication (Com) e communicate the design to others We used verbal protocol analysis as our main research method. We asked individuals to solve design problems in a lab-based setting, concurrently giving a think aloud protocol as they solved the problem. The resulting tape recordings were transcribed, and the transcripts formed the corpus of data. The main coding of the data took two steps. First, each transcript was segmented into idea units. Then, each idea unit was assigned one of the design activity codes that are listed above. Each of these two steps was independently done by two coders, who compared their answers until a satisfactory agreement rate was reached. All disagreements were then discussed until a final code was allocated. The coded data then formed the basis for the quantitative measures

Design process expertise

129

we used to compare the samples on their design processes. In addition to analyzing the participants’ processes, we also looked at several more direct measures of breadth. Specifically, we looked at the ideas the participants considered during their process for each of the verbal protocol studies (PG, PP and SC) by cataloging the number and type of objects that they worked on. For the playground problem we also the cataloged the number and type of information requests they made. Finally, we also assessed the quality of the proposed solutions for the verbal protocol problems with rubrics specifically developed for each problem.

2.1

Designing a playground for a fictitious neighborhood

In this section I describe the findings from 85 individuals who worked on the playground problem. The individuals took up to 3 h to solve the problem. The participants were able to ask for information from the experiment administrator, who then furnished information to the participant when it was available, and also kept track of what information was requested. In addition to analyzing the verbal data as described in the previous section, the solutions that the participants developed were scored with respect to the quality of their design. The quality score rubric for this problem incorporated whether the initial constraints of the problem were met, comparisons to playground standards on specific elements of the proposed design, and Likert-scale scoring on the experience for the users. The participants were all from the U.S. and most of them come from the field of engineering. Here we describe the findings from the four main samples: 26 first-year engineering students, 24 graduating engineering seniors, 19 engineering experts and four playground experts. Details of the problem statement, methods, participants and findings are available in previous publications (Adams, Turns, & Atman, 2003; Atman et al, 1999, 2007). The quantized measures we derived from the qualitative data included: total time spent solving the problem, amount and percent of time spent in each of the design activities in the coding scheme, number and rate of transitions among design activities, number and type of information requests, number and type of objects worked on during the design process and quality score. Here we present the results that are statistically significant for the measures across the three engineering samples: first-year students, graduating seniors and experts. Notably, the measures that show differences between the two student groups differ from the measures that show differences between the experts and the students. These results are found in Atman et al., 2007; Deibel, Atman, & Borgford-Parnell, 2011; Deibel, 2011.

130

Design Studies Vol 65 No. C November 2019

Graduating seniors were significantly more likely than first-year students to .  Design process measures  engage more often in project realization activities (decision and communication)  engage in more transitions among design activities (both total number and rate)  Information gathering  cover a broader set of issues in the information that they gather  Quality  create higher quality products Engineering experts were significantly more likely than students to .  Overall Time  spend more overall time solving the problem  Design process measures  delay a focus on modeling until later in the design process, after engaging in more problem scoping  Information gathering e amount and breadth  gather more information  cover a broader set of issues in the information that they gather  Objects considered e amount and breadth  consider a larger number of objects as they are designing (both directly related to play activities (e.g., swings) and also broader environment elements of a playground (e.g., benches, landscaping))  spend a higher percent time on broader environment objects and lower percent time on play objects than the students  transition more frequently across objects  spend a short amount of time on a large number of the objects they consider, trimming down the set that they work with to a few that they spend a larger amount of time with In a detailed analysis of 16 each of the first-year and graduating students, Robin Adams explored iteration activities in more depth (Adams, 2001). She developed a cognitive model that characterizes the attributes of iteration and what prompted the iteration. Her analysis shows that the graduating seniors demonstrate more effective iteration in multiple ways. The seniors have more iterations, spend more time iterating and cover more design activities in their iterations. Their iterative activities are prompted by selfmonitoring and examining activities, are more likely to be transformative, and more likely to lead to revisions coupled across elements of both the problem and the solution. In addition, graduating seniors are more likely to “. be aware of iterative strategies and their own processes for monitoring, detecting, and resolving design failures” (Adams et al., 2003, p. 280).

Design process expertise

131

Examining the broader set of findings from the list above, many of the design process findings are reinforced by visually examining the design timelines for each of the 73 individual participants, as illustrated in Figure 3. In a timeline representation, time proceeds from left to right, with a tic mark on a line indicating that time was spent in that activity d in essence creating a visual tracing of the design process that was used. Each instantiation of a design process results in a unique design signature for that process. The timelines in Figure 3 are a very reduced version of two example timelines presented in Figure 4. The presentation in Figure 3 is by level of expertise and quality score. To present the timelines by quality for the whole dataset, the quality scores for all the participants were grouped, and then divided into thirds. Timelines from participants who were in the bottom third of the quality scores for their artifact quality are shown in the left-hand column, middle third of the quality scores in the middle column, and top third of the quality scores in the right-hand column. The timelines are also grouped by level and type of expertise, with the first-year students in the bottom row, the graduating seniors in the second row, the engineering experts in the third row and the playground experts in the top row. Visual inspection of the first-year, senior and engineering expert timelines enable us to observe some of the statistical differences of design process measures described earlier as well as some new insights. Note that the data from cataloging the objects considered and type of information requested are not visible in the timelines. Looking across all the participants it is clear that modeling is the dominant activity. Indeed, the percent time spent modeling is the largest category and almost identical amount of time across the three samples: 56%, 57% and 55% for the first-year students, graduating seniors and experts, respectively. Turning to examining the results by quality, it is apparent that the bulk of the individuals with low quality scores are in the first-year group in the bottom row. That group contains most of the timelines that are either very short (with the participants not engaging in much modeling) or have large blocks of uninterrupted time in modeling. As you move up the rows e from first-year to seniors to engineering experts, the ratio of mid-to higher-quality designs in the sample increases, but there were still low-quality scores in both the senior and expert groups. Continuing to examine the first-year students, one can see that few of them engage in the decision and communication activities in the bottom lines of the timeline. Another notable difference looking across the three samples is the amount of time spent and how that time is distributed. The statistically significant difference in total time spent is very apparent e the engineering experts spent significantly more time than the students. Most of the experts used almost the entire 3 h. Many of the participants gather information throughout their process. Close inspection also shows the statistically significant results that the

132

Design Studies Vol 65 No. C November 2019

Figure 3 Design timelines from 73 individuals designing a playground, presented by quality of designed artifact and level of expertise. Within each cell, timelines are sorted with lowest quality score at the top of the left column to highest quality score at the bottom of the right column

Design process expertise

133

Figure 4 Two engineering expert design timelines with a cascade shape Key: Problem Definition (PD), Gather Information (Gath), Generate Ideas (Gen), Model (Mod), Feasibility Analysis (Feas), Evaluation (Eval), Decision (Dec), Communication (Com)

engineering experts spend more time in problem scoping (problem definition and gathering information) before they turn to focus on modeling. Continuing along those lines, after focusing on modeling, the experts are also more likely to then engage in Feasibility, Evaluation, Decision and Communication as they get closer to the end of their process. We have called this sweep of activities e from the upper left to the bottom right of the timelines e a “cascade shape”. We intentionally use the term cascade rather than waterfall to highlight the fact that the participants do not sequentially fall from one activity to another in a linear fashion (as a waterfall model does). Rather, as the participants move through their process, they have many transitions both directions across the activities, both “back” to activities they engaged in previously and “forward” to other activities, exhibiting some of the backward flows or pools you might see across a cascade of a stream down a hillside. A binary coding analysis (cascade shape or not) determined that the expert timelines are more likely to be categorized with a cascade pattern than the student timelines. Figure 4 presents two timelines from the engineering expert sample. Each demonstrates spending the full 3 h to solve the problem, a cascade shape, extended time problem scoping before beginning modeling and gathering information throughout the process. The final observation from Figure 3 for the three samples we are discussing is that the engineering experts’ timelines look more like each other in terms of how time is distributed e like a group e than either of the student samples. The experts have a smaller standard deviation for the total time, number of transitions and transition rate measures than the students do, and this is visually apparent in the timelines. This is particularly noteworthy since the experts were intentionally selected to represent a broad set of engineering disciplines and work experience. The top row of Figure 3 displays timelines from the four participants who were experts in the domain of the problem e playground design. One of the playground experts had some background in mechanical engineering. All four of

134

Design Studies Vol 65 No. C November 2019

the playground experts’ designs received high quality scores. As a researcher, this was excellent to see, as it was one way to validate the quality scoring rubric. Other observations that can be made from the figure, all the playground experts used almost the entire 3 h, spent a significant time problem scoping and modeling, and most of them exhibit a cascade shape. In the other measures, like the engineering experts, all the playground experts considered a large number of objects. However, not surprising for a small sample, there is a large variation in the amount of information gathered and number of categories of information covered. Finally, we conducted two additional analyses using new measures to compare the playground experts to a small subset of the engineering experts. In the first analysis we compared the four playground experts to five of the engineering experts and found that the playground experts considered more social context factors and thought more holistically about the people engaging in their designs than this subset of engineering experts. Also, the playground experts predominantly used their professional knowledge during their design, while the engineering experts also used their professional knowledge, but brought in their personal knowledge of playgrounds when they lacked professional knowledge in some arenas (Krause, Atman, Borgford-Parnell, & Yasuhara, 2013). In a second analysis, we compared three of the playground experts with three engineering experts with high quality artifacts from the perspective of the kinds of questions they asked. In this analysis, building on Eris’ work (Eris, 2004), we found that both playground and engineering experts ask “decrease ambiguity” questions to understand the basics about the problem they are solving. They both also effectively ask “increase ambiguity” questions to help understand and reframe the problem and help generate options. Both sets of experts asked questions in alignment with their expertise (Scalone et al., in press). The playground problem yielded a rich set of findings that describes how samples with various levels of expertise approach design problems. A second study of 6 first-year and 8 graduating seniors from a second university yielded multiple observations that were consistent with the original study (Deibel, Atman, Saleem, Kang, & Ng, 2007). We now turn to findings from a complimentary set of design problems referred to in Figure 2.

2.2

Three shorter design problems and with-in subject growth over time

In this section we discuss findings from the other set of design problems in the corpus of data. Did we get similar results with other design problems and other experimental designs? The short answer is yes.

Design process expertise

135

This set of design problems include the ping-pong launching device (PP), street crossing (SC) and Midwest floods (MWF) problems described earlier. Both the ping-pong launcher and street crossing problems were conducted with verbal protocols. In the Midwest Floods task, participants wrote down the factors they thought were important on paper. The methods and results I describe in this section can be seen in more detail in (Atman, Cardella, Turns, & Adams, 2005; Atman & Turns, 2001). We ran three studies with these problems with the following interventions: 1) a first semester of an engineering curriculum, 2) reading a chapter on design from a textbook, and 3) a four-year engineering curriculum. Thirty-two students participated in the first study, half before and half after their first semester in university (Mulllins, Atman & Shuman, 1999). In this study we determined that this research method was able to capture differences in design behavior and documented that the post-semester students for the PP and SC problems spent longer solving the problems, considered more criteria and had more transitions across design activities. In the MWF problem, which asked for a list of factors on paper, we find that post-semester students identified significantly more factors than pre-semester students. Sequentially, this is the first verbal protocol study I ran. Knowing that the methods captured these process differences enabled us to confidently invest the time to develop the longer, more involved playground problem described in the previous section. It also led to the design of our textbook study. Essentially, we wanted to figure out the smallest meaningful intervention we could think of that might induce a change in an individual’s design process, which we speculated could be passively reading about design from a textbook. We ran a small study in which five students read text about the design process before solving the PP, SC and MWF problems, and five did not. Similar to the semester study, even with the small numbers of participants and small intervention, we saw that the students who read the text spent longer solving the problem, and had more transitions among design activities in the PP and SC problems and identified more factors in the MWF problem (Atman & Bursic, 1996). Evidently, even reading about design can influence behavior. In our final experiment with these three problems, 18 of the original 32 participants in the first-semester study repeated the experiment the year that they graduated (four years after participating in the initial experiment). We also recruited 43 additional graduating seniors, so we were able to conduct both within-subject and across-subject analyses (Atman et al., 2005). We found that, for the PP and SC problems, compared to the first-year students, the graduating seniors 1) spent longer solving the problems, 2) had a higher number of transitions, 3) engaged more often in project realization activities (decision and communication), 4) had higher quality designs, and 5) for the SC problem had a larger number of alternative solutions. Many of these differences are consistent with the differences we see between first-year students

136

Design Studies Vol 65 No. C November 2019

Figure 5 One student’s design timelines for ping-pong and street crossing problems both as a first-year student and graduating senior Key: Problem Definition (PD), Gather Information (Gath), Generate Ideas (Gen), Model (Mod), Feasibility Analysis (Feas), Evaluation (Eval), Decision (Dec), Communication (Com)

and seniors in the playground study. However, looking across the three problems (PP, SC, PG), there are some differences across the of the length of the problem being solved, the number of transitions and transition rate that appear to be an interaction that would be interesting to delve into more deeply. Perhaps the most interesting way to display the process findings from this experiment is again with timelines. Among the 18 repeat student participants, we found that some did not show much progression in their processes and a few who had less sophisticated processes as a senior. Most of the students, however, demonstrated increased sophistication in their design behavior upon graduation as compared to when they entered college. An example of one student’s first-year and graduating senior processes for the PP and SC problems can be seen in Figure 5 (Cardella, Atman, Turns, & Adams, 2008). As an educator, it is gratifying to see the greater complexity in most of the senior design timelines. It is also important to think about why some students did not show growth.

2.3

Summing up - what do we see across the research findings

Looking across all the 177 participants of various levels and types of expertise (first-year and graduating students, engineering and playground design experts) who engaged in 298 design problems (playground, ping-pong, street crossing and Midwest floods) in studies that used both across-subject and within-subject comparisons, what do we see?

Design process expertise

137

In very general terms, the findings from this body of research can be summarized by stating: individuals with more expertise use processes that demonstrate a higher level of complexity/sophistication and consider a broader set of information and objects during their process. They also take longer solving the problem and are more likely to demonstrate a cascade pattern in their tracing across design activities. We also see that undergraduate students move along the path towards these behaviors across the four years of their degrees, also growing in their capacity to identify factors that are relevant to consider during design processes. The patterns in the design process that we see offer concrete opportunities to point out places for students to learn to incorporate context, the original motivation for this body of research. These particularly include the problem scoping process, the transition behavior and gathering relevant information throughout the process. We have found that the timeline representations provide a compelling way to convey the results to people. So many people remarked about the similarity to musical notations that we sonified the data. You can reinforce the visual representations by listening to them at http://bit.ly/celtsoundtracks (Atman et al., 2010) I suggest that you first listen to the bells version of contrasting timelines to hear the difference. As we know, most design problems are too large for just one individual and design is most often done in teams. Fortunately, the Design Thinking Research Symposium 7 meeting provided an opportunity to use our coding scheme to analyze a team design process (McDonnell & Lloyd, 2009). We analyzed the engineering team data from that meeting with our coding scheme (as well as a few others) and found that 1) we were able to apply these codes in a team design situation, 2) we observed the beginning of a cascade shape in this team data (the data were from the early stages of the design process), and 3) we were also able to capture a focus on context factors from broad to specific that mapped along with the move from problem scoping to modeling (Atman et al., 2009b; Borgford-Parnell, Deibel, & Atman, 2014). One key observation here, regarding the attributes and limitations within which these data were collected and analyses conducted, is that there is a need for triangulation for these findings with respect to findings from research with other methods, in other settings, with teams and those who do and do not have domain expertise. In the next section we investigate the affordances of design timelines when compared to more traditionally used models of the design process and also describe several ways in which the timelines have been found to be useful in teaching about design processes.

138

Design Studies Vol 65 No. C November 2019

3

Teaching with timelines

My design challenge, as I moved further into this work, was to figure out how to use the findings to teach engineering students about design processes and opportunities to consider context while engaging in design. Specifically, I wanted to encourage students to have the kinds of insights demonstrated in Figure 6 by a student who had engaged in a classroom activity based on the timelines that is described below. Responding to what the most important thing they learned in the activity, one student wrote: “Realizing that taking your time is important, realizing that higher quality designs gather data and define the problem more thoroughly BEFORE modeling. Which is SO COOL to see as statistically relevant because now I can PROVE to people that understanding the problem FIRST is crucial for success”. In this section, I will describe the affordances of design timeline representations when compared to node-and-arc diagrams of design and describe several instances of the use of timelines to teach about design processes. It is important to understand the starting point from a teaching perspective in engineering classrooms. As we were collecting the student data for the playground problem described in the previous section, we also collected protocols from four educators of engineering students: two mechanical engineering faculty, one industrial engineering faculty, and one practicing nuclear physicist who taught nuclear engineering (Atman, Turns, Cardella, & Adams, 2003). This sample of four individuals showed the largest variability across our experimental measures than any of our other small samples. Using the quality score grouping of thirds presented in Figure 3, two of the educators received scores in the middle third and two of the educators received scores in the bottom third. Only one of the four educator’s process followed a cascade shape typical of an expert practitioner. Our results should not be surprising - engineering educators are rarely hired for their ability as expert designers. It does, however, highlight the fact that as we are describing effective ways to teach undergraduate engineers, we need to take into account the various starting points for the educators themselves.

Figure 6 Student response to learning activity using timeline representations from design process expertise research (Scalone, Joya, Shroyer, & Atman, 2019)

Design process expertise

139

In engineering education, design is taught in many ways (Atman, BorgfordParnell, McDonnell, Eris, & Cardella, 2014; Dym, Agogino, Eris, Frey, & Leifer, 2005; Sheppard, Macatangay, Colby, & Sullivan, 2008). Design is often taught through experiences, including most consistently, capstone design. In recent years, engineering students are more often encountering design experiences in other places in their undergraduate degree, including cornerstone classes, service-learning experiences, projects in maker spaces, and problem-based learning activities attached to many types of courses. It is in these experiential learning activities that engineering students are sometimes pointed to abstract models of design processes, and it is in this area that our research can be useful, as described in the next section.

3.1

Design timelines: concrete and sticky

A model is, by definition, a simplified representation of some concept, entity, process, etc. A model of something as amorphous as a design process is necessarily an abstract representation that leaves out many details, and much information is lost (McDonnell & Atman, 2015). In engineering, typical models used to describe design are node-and-arc models with boxes to denote activities and arrows connecting the boxes to denote movement among activities (e.g., Cross, 2008; Dym & Little, 1999; Pahl & Beitz, 1988). When asked to sketch their design models, 11 of the 19 engineering experts in our study sketched a traditional linear node-and-arc model (Mosborg et al., 2005). The Dubberly Compendium of design models pulls together scores of examples of design models (Dubberly, 2004). In teaching, models such as these are often used prescriptively to teach about the activities of design and a rough order in which the activities can (or should) be performed. One big take-away from the Dubberly Compendium is that there is no “one model” of design. As Box states in his well-known quote, “all models are wrong, some are useful” (Box, Hunter, & Hunter, 2005, p. 440). One of the things that is lost in typical models of design is the potential complexity of the flow of design processes. The timeline, in contrast has different affordances (Atman, Deibel & Borgford-Parnell, 2009a). Each timeline is unique, offering a representation of a tracing through the design activities in a model. Each timeline is one specific instance, making that design tracing (or design signature) visible, with time as an explicit element of the representation. These timelines, grounded in data, make the abstract concept of a design process more concrete, or visible. The idea of a transition between activities, or an iteration to go back to previous activities is evident. In essence, timelines can convey the potential complexity of design processes. We have found that the timeline representations are also memorable, or sticky. They are visually appealing. In countless conversations we find that the people find the timelines relatable and they stick around in people memories. People

140

Design Studies Vol 65 No. C November 2019

make personal connections to them, pointing to one of the timeline patterns and saying “yes, we hire people with processes like these .” or “yes, this is my process .“. When we then play sound versions of the timelines (http:// bit.ly/celtsoundtracks) this reinforces the connections and relatedness for people. I am not suggesting that our team is the first set of researchers to use time or timelines to convey important aspects of design. Indeed, one of the original Design as a Discipline series of papers in Design Studies is titled “A timeline theory of planning and design” (Nadler, 1980). Many abstract models of design include the concept of time (see, for example, the model of coevolution of problem and solution) (see Dorst’ article in this issue, 2019) or models of convergent/divergent processes (Cross, 2008). Many other researchers have used timelines to convey their empirical findings (see, for example, three researchers who were part of the Delft protocol analysis research: Gunther, Frankenberger, & Auer, 1996; Mazijoglou, Scrivener, & Clark, 1996; Radcliffe, 1996). This is an example of an idea where triangulation across other findings is important. What I am suggesting is that these timelines (and others) can be useful companions to teach with the more abstract models that are typically used. The next two sections present some of the ways that we have used specific timelines that come from our research that have resonated particularly well with students, as demonstrated by the quote at the beginning of this section.

3.2

Teaching with timelines e two specific instances

A robust finding from the learning sciences is that “[h]ow students organize knowledge influences how they learn and apply what they know” (Donovan, Bransford & Peligrino, 1999, p. 4). In addition, an individual is more likely to be able to transfer what they have learned to a new context if the way their knowledge is organized is in a meaningful conceptual framework (Ambrose et al., 2010; Bransford, Brown, & Cocking, 2000). The timelines might be helping students learn about design by providing an innovative way for them to organize the design process information they are learning. This section presents two teaching experiences centered around the timelines. We worked with over 75 undergraduate students to code our verbal protocol data. Over the years, after engaging so deeply with the coding scheme and rigorous coding process, we consistently heard from the students the following “Oh, NOW I get what design is!“. This was such a compelling, consistent experience for us that we ran a verbal protocol study of the learning affordances of the coding processes in verbal protocol analysis. We affectionately called it our VPA(VPA) study. We found that, indeed, the students who engaged in coding developed deep, new insights about design processes (Scott, Turns & Atman,

Design process expertise

141

2000, 2001). It was clear, however, that engaging students in time consuming coding experiences would not scale to large numbers of students, so we turned to other ideas. Fortunately, we were also having another consistent experience, and that was the resonance that people had with the timeline representations. We built a 45min classroom exercise around six timelines e three example timelines for firstyear students and three for graduating students, one each to demonstrate attributes of low, middle and high-quality scores for the designed artifact as shown in Figure 7. The timelines were chosen from each quality range to characterize empirical findings from the analyses. Students were given a presentation to explain the experimental methods, design activity definitions and timeline representation. They then responded to two prompts with a class discussion in between. Prompt 1: What similarities and differences do you see between the first-year and graduating senior engineering students? Do these similarities also involve the quality scores? How so? Prompt 2: What are the most important things you learned today? Why?

Figure 7 Design timelines presented by level of expertise and quality of designed artifact used in classroom activity. Key: Problem Definition (PD), Gather Information (Gath), Generate Ideas (Gen), Model (Mod), Feasibility Analysis (Feas), Evaluation (Eval), Decision (Dec), Communication (Com) (Reprinted with permission, Borgford-Parnell, Deibel, & Atman, 2010)

142

Design Studies Vol 65 No. C November 2019

Figure 8 Sample student observations from design timelines classroom activity in Figure 7

We have used this activity in the classroom many times, and each time students make insightful connections about effective design processes (Atman et al., 2016; Borgford-Parnell, Deibel, & Atman, 2010; Scalone et al., 2019). Figure 8 presents some of these connections. The quotes in Figure 8 are responses to the second prompt. The annotation one student added to the timeline of the high artifact quality senior and labeled “the ideal project envelope” coincides with the cascade shape identified in our previous analysis. Students have made similar observations each time we have run this classroom activity. In our more recent teaching activities, being cognizant that design knowledge alone is not sufficient to change design behavior, we have also been including questions about design intentions after engaging in the activity (Scalone, 2019). When asked if the exercise will affect future designs, students respond with statements such as “Yes, I’ll be more active checking my design and trying to bring in new ideas, rather than sit in a puddle with just a few ideas.” While the timelines can quickly communicate important aspects of design processes, they can also be used to invoke a deeper engagement with design processes. Here we turn to a second example of teaching with timelines in more depth. Janet McDonnell and Anegrette Molhave developed two design briefs for a workshop with master’s students in Communication Design in a design school in the United Kingdom (McDonnell & Atman, 2015). In the first brief, similar to the above activity, students were given a version of the classroom activity described above as well as three timelines from engineering experts, one each of low, middle and high quality artifact. The students were then given spreadsheets with the coded data for the nine timeline representations and given a week to create new representations from the original data. Following a debrief session with each other and the instructors, students engaged with the second brief where they were given a design problem to solve that could take a couple of hours. They had to choose a way to record their design process, and

Design process expertise

143

then represent their own process in a new representation. The resulting representations students generated demonstrated some deep insights. Figure 9 displays representations from undergraduate engineering students from a school in the Pacific Northwest of the U.S. as they engaged with the two design briefs (9a) and (9b) as part of a 10-week seminar on design processes (Atman, McDonnell, Campbell, Borgford-Parnell, & Turns, 2015). The figure also includes a third representation of design as a memory aid (9c), where students represented the concepts they wanted to remember when they engaged in design in the future. A content analysis of the student representations shows that the students are incorporating many design process concepts in their representations. The students also report meaningful learning experiences, as demonstrated by one student description: “. it was the work we did designing the representation [for Design Brief One] . that really helped me to understand the benefits of diverse tasking during a design process . I hadn’t realized that planning out how you are going to design can actually assist you in creating better and quicker solutions. Often, I have just chugged away at a design task, jumping to whatever I think needs to happen next, and don’t have much a plan going forward. Understanding this, I am better able to comprehend prototyping and coming up with alternative solutions, instead of generally just one which is what I am prone to do.”

Figure 9 Sample student design process representations for three assignments: a) Design Brief 1, b) Design Brief 2, c) Memory Aid

144

Design Studies Vol 65 No. C November 2019

3.3 Summing up e how can we teach from these research findings? The teaching examples presented in this paper are a few of the ways we have taught with timelines. Other examples include 1) students using the coding scheme in a classroom “fishbowl activity” to code design team activities in real time as they engage in a short design challenge (this activity builds on work by Chong, Foster, & Irish, 2011), 2) student design teams tracking their design activities over the course of a project, and 3) a physical sorting task where students sort cards with timelines printed on them to discover patterns of design expertise. I have found it to be most effective to engage students directly with the timeline representations and let them develop insights themselves. Reinforcing the visual timelines with the timeline sonifications (http://bit.ly/celtsoundtracks) is particularly informative e sometimes leading to transformative “aha” moments. We continue to investigate ways in which the timelines and research results can inform teaching. We are currently running a seminar where undergraduate students are investigating the idea of “design awareness”. They are developing methods to monitor their own design processes with a goal of modifying their process when they think they are going astray (Joya et al., 2019). Students have said that this seminar has helped them “see the rest of the iceberg” of design processes, and that they are “learning the rules to break them”. These are the future designers we need.

4

Conclusions

Findings from research with sample sizes as large as this still need to be triangulated. While I can state the findings from my research with confidence, I also strongly advocate the need to interpret these findings within the context of research done with other research methods (e.g., ethnographies, interviews, phenomenographic studies, research through design), with other models of design (e.g., see the Dubberly Compendium, 2004), and with results from other design process research (e.g., co-evolution of solution and problem, fixation). To offer an example, we mapped our findings to Sch€ on’s Reflective Practitioner lens, connecting our work to his concepts of problem setting and backtalk (Adams et al., 2003). Researchers investigating the amorphous concept of the doing of design are familiar with the concept of trying to grasp something that is not graspable. McDonnell (2015) provides a compelling description of how the design research community has approached this difficult task. The research presented here contributes to that body of scholarship by providing a worked example of findings that can be developed from a commitment to use a single research methodology consistently across a large number of participants. It also demonstrates the utility of creating a representation that is both concrete and

Design process expertise

145

sticky, and using that representation to tell a consistent story about design processes.

5

Epilogue

My hope is that others in the design research and education community can see connections between the work presented in this paper and the way they engage in both their research and teaching; the corpus of findings contributes to the conversation of design as a discipline as Cross calls for; and the learning affordances from these findings and representations can be a useful addition to the many ways that design is already taught and enacted. I have presented these research findings with timeline representations for quite some time, and while the findings are generated from a sample of mostly engineers from some time ago, the representations themselves continue to tell a story of design in a way that resonates with people from across disciplines. This includes not only engineers and designers but also people from the learning sciences, math education, science education, computer science education, cognitive science, people from industry and the medical field, artists and musicians. When Clarence Acox, a band director and jazz musician, saw the three example graduating senior timelines, he saw a musical score for a full jazz band and observed, “Generating Ideas . from the jazz process, that automatically comes from the creative aspect of improvisation”. The representations also stick with students who I hear from long after they take my classes. As I stated in the prologue e the overall goal for my research was to develop descriptive models of how engineers “actually do” design that could be used alongside prescriptive models to teach engineering students to think broadly and consider context when they engage in the design process. Did I achieve that goal? I think that the answer is both yes and no. While I think that I have contributed to the body of knowledge of the discipline of design and I have created compelling teaching activities to enable designers to learn to think broadly as they engage in design processes, there have been a limited number of design students who have encountered these activities. My hope is that this paper may convince other teachers of design to give these representations a try, and I invite anyone who is interested in using any of my materials to see (http://bit.ly/DesignTimelines) or contact me at [email protected] so we can expand the number of students who might say “. much more compelling to see real data, detail, makes me believe, instead of tuning out ‘prescribed’ info .“. Returning to the idea of studying how an individual “actually does” design to add to the body of work that says how one “should do” design. As I have been doing this work over the decades, I have had an aspiration for the design research field that we could have the same kind of impact on society that

146

Design Studies Vol 65 No. C November 2019

the field of behavioral decision theory has had. Consider the popular success of Kahneman’s recent book Thinking: Fast and Slow (Kahneman, 2011), or the fact that you see referrals to cognitive biases in the popular press. Understanding the discipline of design and helping the contributions of design research scholars reach the larger community of designers is essential for the world at large. On a daily basis, people across the globe live with both the amazing benefits and the unintended consequences of designed artifacts and systems. It is possible that some of those unintended consequences could have been avoided if different decisions had been made during the design process. Imagine if the insights that are generated by the design research community could have an impact on even a fraction of the large number of design processes that are engaged in every day. Imagine if people doing design were intentionally thinking about how their solution was co-evolving with their understanding of the problem they are solving, or thought about where they might be in a cascade shape of a design process? Might the resulting products and services be better? Might there be fewer unintended consequences? How can we get the complex eco-system of design teachers and doers to know more about design research? Continuing to do our research and getting the results out to the design doers in the world is exceedingly important as our society and globe move forward into an uncertain future.

Acknowledgements This work was supported by the U.S. National Science Foundation (RED9358516, ROLE-0125547), the Center for Engineering Learning & Teaching at the University of Washington, the Mitchell T. and Lella Blanche Bowie Endowment and the Mark and Carolyn Guidry Foundation. I would like to thank all the participants in the studies and students in the classes I have taught that contributed to this work. I would also like to thank Nigel Cross, both for his support over the years and his invitation to write this article, which was a great honor. The opportunity to write this article gave me the chance to reflect on the large number of people it takes to conduct research of this type. While I have worked with over 50 colleagues and graduate students and more than 75 undergraduate students over the years, I would specifically like to mention a few colleagues whose close collaboration made this work over the years extremely rewarding, including Jennifer Turns, Robin Adams, Monica Cardella, Janet McDonnell, Jim Borgford-Parnell, Ken Yasuhara, Lauren Thomas and Terri Lovins. I would also like to thank the many colleagues who gave me feedback on this paper. Finally, I would like to thank Indira Nair for her many insights over the years; and my husband and kids, Mike, Abby and Toby Meyer, who have been putting up with design timelines for several decades.

Design process expertise

147

Declaration of Competing Interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References Adams, R. (2001). Cognitive processes in iterative design behavior. (Unpublished doctoral dissertation). Seattle, WA: University of Washington. Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of reflective practice. Design Studies, 24(3), 275e294. Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., Norman, M. K., & Mayer, R. E. (2010). How learning works: Seven research-based principles for smart teaching (1st ed.). San Francisco, CA: Jossey-Bass. Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., & Saleem, J. J. (2007). Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education, 96(4), 359e379. Atman, C. J., Arif, A., Shroyer, K. E., Turns, J. A., & Borgford-Parnell, J. (2016). “Spend another day in our class talking about this research please”: Student insights from a research-based design thinking exercise. Design Research Society, 2016 Design Research Society 50th Anniversary Conference (DRS). Brighton, UK. June 27-30, 2016. Atman, C., Blair, J., Bodle, C., Borgford-Parnell, J., Deibel, K., Goist, Z., et al. (2010). Seeing and hearing design: Exploring how visual representations and soundtracks could be used to teach design. In Design thinking and research symposia. In proceedings of design thinking and research symposia (pp. 25e37), Sydney, Australia. Atman, C., Borgford-Parnell, J., & Deibel, K. (2009). The process of engineering design: A comparison of three representations. International conference on engineering design (ICED). In Proceedings of the international conference on engineering design. Stanford: CA. Atman, C., Borgford-Parnell, J., Deibel, K., Kang, A., Kilgore, D., Ng, W., et al. (2009). Matters of context in design. In J. McDonnell, & P. Lloyd (Eds.), About designing: Analysing design meetings (pp. 399e416). London, UK: Taylor and Francis Group. Atman, C., Borgford-Parnell, J., McDonnell, J., Eris, O., & Cardella, M. (2014). Engineering design education: Research, practice, and examples that link the two. Atman, C. J., & Bursic, K. M. (1996). Teaching engineering design: Can reading a textbook make a difference? Research in Engineering Design, 8, 240e250. Atman, C. J., Cardella, M. E., Turns, J., & Adams, R. (2005). Comparing freshman and senior engineering design processes: An in-depth follow-up study. Design Studies, 26(4), 325e357. Atman, C. J., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A comparison of freshman and senior engineering design processes. Design Studies, 20(2), 131e152. Atman, C. J., McDonnell, J., Campbell, R., Borgford-Parnell, J., & Turns, J. A. (2015). Using design process timelines to teach design: Implementing research results. American Society of Engineering Education Annual Conference (ASEE). Seattle, WA. June 14-17, 2015. Atman, C., & Turns, J. (2001). Studying engineering design learning: Four verbal protocol studies. In C. Eastman, W. McCracken, & W. Newstetter (Eds.),

148

Design Studies Vol 65 No. C November 2019

Design knowing and learning: Cognition in design education (pp. 37e60). New York: Elsevier. Atman, C. J., Turns, J., Cardella, M. E., & Adams, R. S. (2003). The design processes of engineering educators: Thick descriptions and potential implications. In Proceedings of the expertise in design: Proceedings of the design thinking research Symposium 6. Sydney: Australia, November 2003. Borgford-Parnell, J., Deibel, K., & Atman, C. J. (2010). From engineering design research to engineering pedagogy: Bringing research results directly to the students. International Journal of Engineering Education, 26(4), 748e759. Borgford-Parnell, J., Deibel, K., & Atman, C. (2014). Engineering design teams: Considering the forest and the trees. In J. Figueiredo, J. Trevelyan, & B. Williams (Eds.), Engineering practice in global context (pp. 79e99). EH, Leiden, The Netherlands: CRC Press/Balkema. Box, G., Hunter, J., & Hunter, W. (2005). Statistics for experimenters (2nd ed.). Hoboken: NJ: John Wiley & Sons, Inc. Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn, 11. Washington, DC: National academy press. Cardella, M. E., Atman, C. J., Turns, J., & Adams, R. S. (2008). Students with differing design processes as freshmen: Case studies on change. International Journal of Engineering Education, 24(2), 246e259. Chi, M. T., Slotta, J. D., & De Leeuw, N. (1994). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction, 4(1), 27e43. Chong, A., Foster, J., & Irish, R. (2011). (First-year) design fishbowl. Workshop, Vancouver, BC, Canada. Cross, N. (2008). Engineering design methods: Strategies for product design (4th ed.). Chichester: John Wiley and Sons Ltd. 2008. Cross, N. (2018). Developing design as a discipline. Journal of Engineering Design, 29(12), 691e708. Deibel, K. (2011). Playground object analyses (CELT technical report CELT11e04). Seattle, WA: Center for Engineering Learning & Teaching. Deibel, K., Atman, C., & Borgford-Parnell, J. (2011). The role or expertise in shaping the dominance of modeling activities in engineering design (CELT Technical Report CELT-11e01). Seattle, WA: Center for Engineering Learning & Teaching. Deibel, K., Atman, C., Saleem, J., Kang, A., & Ng, W. H. (2007). Comparing two populations of undergraduate student design processes on the playground design task, (CELT technical report CELT-07-11). Seattle: Center for Engineering Learning and Teaching. 2007. Donovan, M. S., Bransford, J. D., & Pellegrino, J. W. (1999). How people learn: Bridging research and practice. Washington DC: National Academy Press. Dorst, K. (2019). Co-evolution and emergence in design. Design Studies, 65, 60e77. Dubberly, H. (2004). How do you design. A compendium of Models. Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 94(1), 103e120. Dym, C. L., & Little, P. (1999). Engineering Design: A Project-Based Introduction. John Wiley and sons. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. The MIT Press. Eris, O. (2004). Effective inquiry for innovative engineering design. Springer Science & Business Media, 10.

Design process expertise

149

Gunther, J., Frankenberger, E., & Auer, P. (1996). Investigation of individual and team design processes. Analysing design activity, 117e132. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness (No. 6). Harvard University Press. Joya, A., Barar, G., Gray, A., Jordan, K., Sweem, R., Washington, N., et al. (2019). What the hell is water? Exploring connections between design process, design doing, and design awareness (CELT technical report CELT 19-01). Seattle, WA: Center for engineering learning & teaching. Kahneman, D. (2011). Thinking, fast and slow. Macmillan. Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge university press. Krause, K., Atman, C., Borgford-Parnell, J., & Yasuhara, K. (2013 June). Designing for communities: The impact of domain expertise. ASEE Annual Conference & Exposition. In Proceedings of the 2013 ASEE annual conference & exposition. Atlanta, GA. Mazijoglou, S., Scrivener, S., & Clark, S. (1996). Representing Design.Displays within a design team. Analysing Design Activity, 343e364. McDonnell, J. (2015). Gifts to the future: Design reasoning, design research, and critical design practitioners. She Ji: The Journal of Design, Economics, and Innovation, 1(2), 107e117. McDonnell, J., & Atman, C. J. (2015). Paying attention to the design process: Critically examining personal design practice. In LearnxDesign conference. Chicago, IL. June 28-30, 2015. McDonnell, J., & Lloyd, P. (2009). Introduction to about: Designing-analysing design meetings. Taylor and Francis Group. Moore, P., Atman, C., Bursic, K., Shuman, L., & Gottfried, B. (1995). Do freshman design texts adequately define the engineering design process? American society of engineering education annual conference. In Proceedings of the American society of engineering education annual conference. Anaheim, CA. Morgan, M. G., Fischhoff, B., Bostrom, A., & Atman, C. J. (2002). Risk communication: A mental models approach. Cambridge University Press. Mosborg, S., Adams, R., Kim, R., Atman, C., Turns, J., & Cardella, M. (2005). Conceptions of the engineering design process: An expert study of advanced practicing professionals. American society for engineering education annual conference. In Proceedings of the American society for engineering education annual conference. Portland: OR. Mullins, C. A., Atman, C. J., & Shuman, L. J. (1999). Freshman engineers’ performance when solving design problems. IEEE Transactions on Education, 42(4), 281e287. Nadler, G. (1980). A timeline theory of planning and design. Design Studies, 1(5), 299e307. Pahl, G., & Beitz, W. (1988). Engineering design: A systematic approach. Springer science & business media. Radcliffe, D. (1996). Concurrency of actions, ideas and knowledge displays within a design team. Analysing design activity, 343e364. Scalone, G., Atman, C.J., Meija, K., Twigg-Smith, H., Shroyer, K., & Joya, A. (2020). “Dealing with ambiguity: Leveraging different types of expertise to guide design questioning”, International Journal of Engineering Education, in press. Scalone, G., Joya, A., Shroyer, K., & Atman, C. (2019). Design intentions: Engineering students looking ahead to their future design behavior. American Society for engineering education annual conference. In Proceedings of the

150

Design Studies Vol 65 No. C November 2019

American society for engineering education annual conference. St. Petersburg, FL. Scott, C., Turns, J., & Atman, C. (2000). Learning about engineering design through an examination of student design strategies. American Society for Engineering Education Conference. In Talk presented at the 2000 American society for engineering education conference. St. Louis, MO. Scott, C., Turns, J., & Atman, C. (2001). Mastering concepts through the coding of design protocols in a collaborative setting (CELT technical report CELT01e11). Seattle, WA: Center for Engineering Learning & Teaching. Sheppard, S. D., Macatangay, K., Colby, A., & Sullivan, W. M. (2008). Educating engineers: Designing for the future of the field. Book highlights. Carnegie Foundation for the Advancement of Teaching. Simon, H. A. (1979). Models of thought, 352. Yale University Press.

Design process expertise

151