Building undergraduate STEM majors’ capacity for delivering inquiry-based mathematics and science lessons: An exploratory evaluation study

Studies in Educational Evaluation 64 (2020) 100833

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/locate/stueduc

Building undergraduate STEM majors’ capacity for delivering inquiry-based mathematics and science lessons: An exploratory evaluation study

T

Xiaoxia A. Newton*,1, Edward P. Tonelli Jr.2 Cato College of Education, University of North Carolina, Charlotte, USA

A R T I C LE I N FO

A B S T R A C T

Keywords: Inquiry pedagogy Secondary mathematics and science content STEM majors Teacher preparation

Recruiting and preparing STEM majors for teaching has become one of the major efforts at improving mathematics and science teacher quality at secondary level. One question is whether STEM majors who have not had the chance to experience active learning in mathematics and science classes as secondary students themselves know what inquiry pedagogy is. Secondly, it is unclear whether those who experienced inquiry in their college introductory discipline courses will be able to utilize the pedagogy in teaching secondary content. We address these questions through studying an undergraduate research methods course designed to improve STEM majors’ capacity for delivering inquiry-based mathematics and science lesson. Analysis of data from pre-and-post course surveys and students’ written research reports including students’ reflection on their inquiry projects suggests that offering future STEM teachers opportunities to conduct inquiry and reflect explicitly on how inquiry can be used to teach secondary content is important and beneficial.

1. Introduction Improving American students’ opportunities to learn and performance in mathematics and science has been of major concern for several decades. The success of student learning in part depends on teachers. Consequently, there have been calls for recruiting from those with the strongest quantitative backgrounds (e.g., STEM majors) (Schmidt, Houang, & Cogan, 2011). Sustained efforts at recruiting undergraduate STEM majors into teaching have been launched through programs such as 100k10 in New York, UTeach in Texas, and UTeach replication sites across the country (Newton & Poon, 2015; Dailey, Bunn, & Cotabish, 2015; Hutchison, 2012; Mervis, 2007). Besides recruitment, these programs also pay a parallel attention to the approach through which these STEM majors are prepared as future secondary mathematics and science teachers. One approach gaining increasing traction is the interdisciplinary collaboration, especially between STEM and education faculty. Examples include Colorado Learning Assistant Model (Otero, Pollock, & Finkelstein, 2010), UTeach and its replication programs in higher education institutions across the

United States (UTeach Institute, 2019). These interdisciplinary STEM teacher preparation programs have a heavy emphasis on teaching mathematics and sciences through inquiry (Sanders, 2009). The interdisciplinary nature and the heavy emphasis on inquiry as the signature pedagogy common across these programs as a way to address the STEM teacher quality seem to rest on two assumptions. First, recruiting STEM majors will ensure future STEM teachers are highly qualified in terms of their content understanding; secondly, utilizing pedagogical approaches that emphasize active student learning will address the pedagogical issues that plague secondary mathematics and science classrooms. What has not been brought to the forefront is the need for an explicit integration of how inquiry pedagogy can be utilized to teach secondary mathematics and science content (York & Dallas, 2018). This omission is problematic for a variety reasons. Chief among them is the question of whether STEM majors who have not had the chance to experience active learning in mathematics and science classes as secondary students themselves know what inquiry pedagogy is and how to use the pedagogy that is widely called for by reformers.

⁎

Corresponding author. E-mail address: [email protected] (X.A. Newton). 1 Newton is an applied methodologist whose scholarship is centrally concerned with using methodological tools in creative and productive ways to tackle educational problems pertinent in urban communities, with an emphasis on mathematics education and STEM (Science, Technology, Engineering and Mathematics) participation. 2 Tonelli is a high school science faculty who is interested in K-12 mathematics education and science education. This research was conducted while he was completing his doctoral thesis at College of Education, University of Massachusetts Lowell. Tonelli now works at Saint John’s High School in Shrewsbury, Massachusetts, USA. https://doi.org/10.1016/j.stueduc.2019.100833 Received 12 November 2018; Received in revised form 26 November 2019; Accepted 18 December 2019 0191-491X/ © 2019 Elsevier Ltd. All rights reserved.

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

2.1. Inquiry frameworks

Secondly, it is unclear whether those who experienced inquiry pedagogy in their college level introductory discipline courses will be able to utilize the pedagogy in teaching mathematics and science concepts to secondary students (i.e., grades 6–12). An intentional and explicit emphasis of integrating inquiry and secondary content is important because research has shown that experts with advanced domain knowledge in a field tend to be unaware of struggles confronted by students or novices. This phenomenon is called “expert blind spot” (Nathan, Koedinger, & Alibali, 2001). We address these questions through our own study of an undergraduate research methods course designed to improve STEM majors’ capacity for delivering inquiry-based mathematics and science lessons. This research methods class is offered as part of the core courses in our university’s UTeach replication program. UTeach provides an excellent context for investigating these issues, because the program recruits from STEM majors, involves interdisciplinary approach to teacher training, emphasizes inquiry and has a wide range of higher education institutions across the country replicating the program. Through sharing our study, other institutions could replicate our research efforts. The following research question guided our study: How did course enrollees’ research skills evolve and in what ways did they connect inquiry with secondary mathematics and science content? We found that offering future STEM teachers opportunities to learn to conduct scientific inquiry and reflect explicitly on how inquiry can be used to teach secondary (grades 6 through 12) mathematics and science concepts is important and beneficial. In the sections that follow, we first review relevant literature so as to provide a rationale and conceptual basis for the design of the research methods course. We then describe various aspects of our study methods and present the findings. We conclude with a discussion of the implications of the study findings and suggest areas for further research.

The word “inquiry” is a popular term used to describe the pedagogical approach that science education reformers aspire for (see the National Science Education Standards in NRC 1996, 2001). In the mathematics education reform community, terms such as constructivist, discovery, hands-on, problem-based, experiential, reformoriented (as opposed to traditional), and authentic, have been used to describe the ideal approach to the teaching and learning of mathematics (Jaworski, 2002; Pauli, Reusser, & Grob, 2007; Savery & Duffy, 1995; Smith, Desimone, & Ueno, 2005). Regardless of the terminology, the pedagogical emphasis by reformers in both mathematics and science education communities shares a common belief that students learn best when they are actively engaged in the learning process as opposed to being lectured to by a knowledgeable authority (i.e., a teacher). These pedagogical beliefs, championed by Piaget (1964) and Vygotsky (1978), are rooted in the constructivist learning theory that has since characterized the work of other well-known scholars (e.g., Bandura & Walters, 1977; Gallagher & Reid, 2002; Wertsh & Tulviste, 1990; Zimmerman, 2013). Educational researchers have described inquiry teaching and learning with various features (e.g., Furtak et al., 2012). One oftenreferenced conceptual framework of inquiry was based on Duschl’s work (2003, 2008). Duschl’s inquiry framework consists of three components: (a) conceptual (facts, theories, and principles that constitute the scientific disciplinary knowledge), (b) epistemic (knowledge generation through scientific investigation), and (c) social (knowledge dissemination through scientific collaborations and communication). This inquiry framework was further extended by other researchers with the addition of a fourth dimension, that is, procedural knowledge. The procedural dimension refers to activities involved in a scientific investigation such as generating scientific hypothesis and questions, designing experiments and data collection procedures, collecting and analyzing data, and disseminating findings (Furtak et al., 2012). The procedural dimension of Duschl’s inquiry framework partly informed the design of the three inquiry projects that formed the core curricular component of our Research Methods course. Our course design was further enhanced by Pedaste et al.’s work. Pedaste et al. proposed five phases of inquiry-based learning framework based on synthesis of research on inquiry-based learning (Pedaste et al., 2015). According to Pedaste et al., the five phases consisted of orientation (i.e., the process of stimulate curiosity), conceptualization (i.e., the process of stating research hypothesis or questions), investigation (i.e., the process of planning exploration or experimentation, collecting and analyzing data), conclusion (i.e., the process of drawing conclusion based on data), and discussion (i.e., the process of presenting findings, communicating with each other, and engaging in reflective activities). Pedaste et al.’s five phases of inquiry learning guided how we scaffold course enrollees through each of the three inquiry projects. In other words, each inquiry project requires students to generate a hypothesis, design an experiment, collect and analyze data, and communicate findings through a written report and an oral presentation. Table 1 shows the alignment of the inquiry learning framework phases (Pedaste et al., 2015), the key elements of the inquiry domain in an investigation cycle (Furtak et al., 2012; Van Uum et al., 2016), class topics across the semester, and the three course inquiry projects. As shown in Table 1, the common skills that were embedded (or assessed) in all three projects included students’ demonstrated ability to: (a) explain how data were generated (inquiry design); (b) present data from multiple trials accurately through tables and/or graphs; (c) analyze data using appropriate techniques, (d) interpret data properly and draw conclusions, and (e) share research process and findings through well organized written reports with minimal grammatical errors. In addition to these five skills, Inquiry 1 paid special attention to several foundational skills, including the ability to clearly articulate a researchable question, to justify why the question is researchable, and

2. Literature review The existing literature informed our research in two ways. First, we were able to build our work on scholarly contributions from Duschl (2003, 2008) who differentiated three domains of scientific knowledge, namely, conceptual, epistemic, and social. Duschl’s framework was further extended by educational researchers to include a fourth dimension: procedural knowledge. Procedural knowledge focuses on formulating research questions and drawing conclusions to the research questions (Furtak, Seidel, Iverson, & Briggs, 2012; Van Uum, Verhoeff, & Peeters, 2016). Our course design was further enhanced by Pedaste et al.’ five phases of inquiry-based learning framework (Pedaste et al., 2015). The inquiry frameworks and their key concepts helped us to design the course projects and provided a lens through which we defined and taught inquiry to course enrollees. Secondly, our work is enhanced by the seminal contribution of Shulman, who proposed the concept of pedagogical content knowledge (PCK), which he defines as “knowledge which goes beyond knowledge of subject matter per se to the dimension of subject knowledge for teaching” (Shulman, 1986, p. 9). Shulman’s PCK concept prompted us to offer course enrollees the opportunities to reflect explicitly how inquiry could be used to teach secondary mathematics and science concepts. In addition to Shulman, our work is inspired by cognitive scientists’ work on “expert blind spot” (e.g., Nathan et al., 2001). “Expert blind spot” refers to the tendency of experts in a domain subject not being aware of the struggles that novices face. We argue that while STEM majors in our program likely possess high level STEM disciplinary knowledge, their advanced knowledge might impede their ability to teach secondary mathematics and science content effectively to students in those grade levels because of “expert blind spot”. Therefore, these STEM majors need opportunities to make explicit connection between inquiry pedagogy and secondary mathematics and science content.

2

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

Table 1 Alignment of Inquiry Elements and Class Topics. Pedaste et al. Framework (General Phases)

Duschel et al. Framework (Procedural Dimension)

Orientation

Question

Class Topics

Inquiry Assignments

Inquiry 1: Fostering curiosity

- Researchable questions and hypotheses - Justify your research questions and literature review

Conceptualization

Design

-

Investigation

Data

-

Conclusion

Conclusion

Discussion

Communication

-

a

Inquiry 2: Conducting disciplinary based experiment - Lab Safety - Research design - Calibration and errorsa Data collection Calibration and errors Human subject research and IRB Design a survey assessment Evaluate a survey instrument (face/ content validity, reliability) Data/statistical analysis Model data Evaluate a model Identify statistical errors Code open survey responses Represent data Graph data Peer review Oral presentation Report writing

Inquiry 3: Teaching slope concept through its STEM applications: Assessing students’ prior understanding of slope

This topic is relevant to both research design and data collection domains.

evaluated on the criteria of long term retention of knowledge and skills and transferability of such knowledge and skills to their broader application settings (e.g., applying knowledge and skills learned from a calculus course to disciplines such as physics and economics). Examining the debates between education researchers and cognitive scientists, we observed that cognitive scientists challenge the advocacy for constructivist-oriented pedagogy to teaching mathematics and science on the grounds of theoretical and empirical basis. Additionally these cognitive scientists believe whatever labels have been used to describe these pedagogical approaches, they are essentially the same (Anderson et al., 1999; Kirschner et al., 2006). In contrast, educational scholars differentiate different forms of inquiry. Several widely cited meta-analyses revealed more nuanced patterns and findings with regards to the efficacy of inquiry-approach to science education (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011; Furtak et al., 2012; Lott, 1983; Schroeder, Scott, Tolson, Huang, & Lee, 2007; Wise & Okey, 1983). One conclusion from these synthesis studies is that engaging students in a guided inquiry process led to greater students’ learning gains than lecturing or unguided instruction (Furtak et al., 2012). Despite the evidence on the positive impact of guided inquiry learning, classroom mathematics and science teaching at the secondary level remains counter to what reformers aspire for (Hiebert et al., 2005; National Council of Teachers of Mathematics, 2000; National Research Council, 2001). Teacher education plays an important role in moving classroom mathematics and science teaching towards guided inquiry, yet most research on pre-service teacher preparation has focused primarily on elementary school teachers (e.g., Yoon, Joung, & Kim, 2012). Little research has been done on secondary mathematics and science teachers. Building on prior scholars’ work, our research focused on STEM majors enrolled in our research methods course who were being trained to teach secondary students. Our research methods course intends to teach STEM majors what inquiry is and how to do it by engaging them in the process of research. Furthermore, the course intends to build STEM majors’ capacity for delivering inquiry-based lessons by requiring them to make the explicit connection between the inquiry pedagogy with the mathematical and/ or scientific concepts underlying their inquiry projects. We argue this

to accurately classify key variables into different categories (e.g., dependent variables, independent variables, variables held constant in an experiment, etc.).

2.2. Inquiry pedagogy and secondary mathematics and science content While widely popular in the education community, inquiry approach to teaching has faced criticisms. Several prominent cognitive scientists (e.g., Anderson, Reder, & Simon, 1999; Kirschner, Sweller, & Clark, 2006) have argued that constructivist pedagogical beliefs are based on the hypothesis that students learn best by discovering for themselves disciplinary concepts and principles. Kirschner et al. regard constructivist pedagogical beliefs as faulty because they confuse “the teaching of a discipline as inquiry (i.e., a curricular emphasis on the research processes within a science) with the teaching of the discipline by inquiry (i.e., using the research process of the discipline as a pedagogy for learning)” (Kirschner et al., 2006; p. 78). This confusion, Kirschner et al. further argue, at the fundamental level has failed to integrate content and pedagogy. Furthermore, Kirschner et al. believe such a hypothesis is incompatible with what cognitive scientists know about human cognitive architecture (Kirschner et al., 2006). More specifically, Kirschner et al. (2006) argue that equating learning a discipline with experiencing the processes and procedures of the discipline is a faulty assumption (Clark & Estes, 1998, 1999; Estes & Clark, 1999; Kirschner, Martens, & Strijbos, 2004). And this assumption is incompatible with the learning theory (human cognitive architecture, short-term and long-term memory). Additionally, they argue this assumption lacks empirical evidence and in fact, the evidence suggests directed instruction where learners are provided with partial or complete information is more effective than constructivist instructional approach where learners are more or less in charge of their own learning (Kirschner et al., 2006). Similar criticism has been made about the mathematics education community’s advocacy for more constructivist-oriented mathematics teaching and learning. Anderson et al. (1999) argue that instead of relying on short-term outcomes such as test scores, evidence of constructive pedagogies on students’ learning outcomes should be 3

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

deliberate effort at connecting content and pedagogy is one way through which PCK manifests itself in the teaching and learning of secondary mathematical and scientific concepts. In addition, we argue connecting inquiry pedagogy and secondary mathematics and science content explicitly is one way to address the “expert blind spot” problem and cognitive scientists’ criticism that constructive pedagogy fails to integrate content and pedagogy.

skills to the design and implementation of the course. One of us was trained in social science research methodology with an extensive background in applied statistics, measurement, and evaluation, while the other in mathematics education with substantive knowledge of secondary school mathematics and sciences curriculum and state standards through secondary school teaching experiences (mathematics, biology, physics, and chemistry).

2.3. Significance of our study

3.2. Course objectives, inquiry framework, and course projects

Our study intends to contribute to the field in two related aspects. First, we focus on undergraduate STEM majors. This focus is important because recent initiatives aimed at improving mathematics and science teacher quality has focused on recruiting STEM majors, and yet we know little about their inquiry skills. Based on the literature synthesis, we focus on inquiry skills defined by existing inquiry framework. Secondly, we believe it is important to provide these STEM majors with opportunities to think explicitly about the connection between the mathematical and scientific concepts embedded in their inquiry projects, with the goal that they would be teaching future secondary students the mathematical and scientific concepts through the inquiry projects. In other words, we emphasize the explicit connection between the inquiry and the secondary mathematics and science content embedded in the inquiry project. This explicit connection is one way to address the pedagogy-content separation criticism. In addition, this explicit connection helps to develop STEM majors’ awareness of “expert blind spot” problem when they become STEM teachers down the road. Our study site provides a unique opportunity to investigate these issues because we have an undergraduate STEM teacher preparation program that is part of the Uteach replication programs across the U.S. UTeach is an interdisciplinary undergraduate STEM teacher preparation program that engages faculty from Arts and Sciences, Education, and Engineering. The program recruits undergraduate STEM majors and offers them education courses so that these STEM majors learn how to teach while studying for their majors. Given the target population of the program, our study sample consisted of both mathematics and physical sciences/engineering majors. While inquiry might come naturally for science and engineering majors, it is not necessarily so for mathematics majors. We therefore argue it is of great importance to engage mathematics majors in a course such as ours where mathematics majors have opportunities to approach the task of designing inquiry projects for teaching secondary mathematics concepts.

The Research Methods course has two intertwined objectives: (1) to help course participants develop skills in conducting scientific inquiries; and (2) to help course participants reflect on how scientific inquiries can be used as an effective pedagogy teaching relevant mathematical or scientific concepts to middle or high school students. To accomplish the two objectives, we anchored the course curriculum around three inquiry projects that students needed to complete. The design of these inquiry projects was modeled after the inquiry framework proposed by Pedaste et al. (2015), Duschl (2003, 2008) and colleagues. Specifically, the first inquiry was a simple experiment that could be done in any kitchen. Students were asked to look around their kitchens, come up with a research question and/or hypothesis, and design and carry out an experiment to investigate the question or test the hypothesis they came up with. The second inquiry was discipline specific. Students were asked to conceptualize and carry out an experiment in their own disciplinary field which required the use of more complex laboratory equipment and more precise measurements than the kitchen experiment (i.e., the first assignment). The third inquiry asked enrollees to pretend to be a teacher of middle or high school mathematics or science course who were about to teach his/her students topics that involve the concept of slope and its applications in mathematics or sciences. This project required enrollees to develop and administer an assessment tool. In addition, enrollees analyzed the assessment data with the purpose of finding out their pretend-students’ prior knowledge and understanding of the concept of slope and its applications in different STEM disciplines (i.e., using assessment information for lesson planning purposes). In teaching this research methods course, we focused in-class time on equipping students with knowledge and skills needed to carry out their independent projects. Our pedagogical approach aimed for a balance among different teaching methods, including interactive instructor-led discussions, group activities, students-led class discussions, and in-class group experiments in order to promote active student learning.

3. Study context 3.1. Research methods course for STEM majors

3.3. Premise and conception underlying the course design and our research program

Our study took place in a public research university in the Northeast of the United State. The university is one of the replication sites of the UTeach program, an undergraduate interdisciplinary secondary mathematics and science teacher preparation program started at UT Austin in the late 1990s. Uteach allows STEM majors to take specially designed courses that will prepare them for teaching while pursuing their majors. Since its inception, UTeach has expanded to 44 universities in 23 states and the District of Columbia. As of Spring 2019, the total number of UTeach graduates reached 5, 278 (UTeach Institute, 2019). Our study site became one of the UTeach expansion programs in Spring 2012 and has graduated 16 STEM teachers as of Spring 2017 (UTeach UMass Lowell Newsletter, 2018). One of the hallmark features of the UTeach teacher preparation program is the emphasis on research inquiry and its role in teaching secondary mathematics and science content. Research Methods is one of the core program courses through which STEM majors are exposed to the process of research inquiry for the purpose of teaching mathematics and sciences to future secondary students. The co-authors of the study were the co-instructors who brought complementary knowledge and

Our hypothesis is that by providing course enrollees with inquiry projects that resonate with what scientists do and by giving enrollees opportunities to make explicit the connection between the inquiry approach (i.e., pedagogy) and the underlying secondary mathematical or scientific concepts (i.e., content) and to reflect how the inquiry could be utilized to teach these concepts, enrollees will deepen their own understanding of the mathematical and scientific concepts and build their capacity for teaching the concepts to secondary students using an inquiry or research approach. Fig. 1 depicts the theory of action underlying our course design and functions as a guide for this line of research. 4. Evaluation question and approach Our current research focused on the first two stages (i.e., the first two columns of boxes in Fig. 1). Specifically, our research investigated the extent to which the course inquiry projects were able to build course enrollees’ knowledge of inquiry skills and their capacity in 4

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

Fig. 1. Translating Course Learning into Teaching to Impact K-12 Students.

questions students might have, the program coordinator (who was not involved in our research) obtained the consent forms and stored the forms in a sealed envelope locked in her office cabinet. We were able to access the consent forms only after the semester was over and all students’ grades had been submitted to the university registrar’s office. To alleviate the burden on students, we closely aligned our research data with course assignments and activities. Table 2 provides an overviewing of the overarching question, intended outcomes, types of data, and analysis. Details of the inquiry projects were provided in the previous section. Here we describe the surveys and students’ reflections.

implementing inquiry based instruction by achieving short-term course outcomes. 4.1. Sample The sample consisted of 13 students enrolled in our course. All enrollees were undergraduate STEM majors interested in pursuing a teaching career as middle and high school mathematics and science teachers. More than half of the enrollees (seven) were math majors. Two were engineering majors (one with a double major in physics) and the rest consisted of biological sciences (three) and chemistry (one). The enrollees were roughly split half-and-half between sophomores and juniors. Approximately 60 % of the enrollees were female and 40 % were male students.

4.2.1. Pre-and-post course surveys Our Research Methods course emphasized the centrality and logical priority of an empirical question over research design and procedures. This emphasis is consistent with the concept of “orientation” in existing inquiry framework. Hence the course starts with helping students understand what a researchable question is, why a question is researchable, and how to develop one. On the first day of the class, students filled out a survey. The survey consisted of questions about a hypothetical laboratory experiment and also a foundatiaonal mathematical concept (slope) that has important

4.2. Data and analysis Since the co-authors were also co-instructors of the course, we made every attempt to be transparent with the students about our research following the IRB guidelines. For instance, we introduced our research on the first day of the class. After explaining its purpose and clarifying 5

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

Table 2 Research Skills, Data, and Analysis. Question: How did course enrollees’ research skills evolve and in what ways did they connect inquiry with K-12 content? Research Skills - Question - Design - Data - Conclusion - Communication

Types of Data Written reports: - Inquiry 1: A cooking experiment - Inquiry 2: A scientific experiment - Inquiry 3: A survey research: Assessing students’ prior knowledge of slope Pre-and-Post Surveys (Open-ended questions): - A hypothetical experiment Students’ reflections: - Reflection section of the written reports

Analysis - Rubric (see Table 3 for an example)

- Content analysis for themes emerged from the responses - Content analysis for themes emerged from the written text

rubric consists of several criteria (e.g., research design, data analysis, interpretation, etc.), each of which represents a skill and is given certain points (e.g., 10, 20, etc.). The same skill might not be given the same points for different assignments (e.g., research design will be given 10 points for assignment 1 but 20 points for assignment 2). To compare scores across the assignments on the same skill (e.g., research design), we first calculated the weighted scores for each criterion so as to make the scores on a common metric across different assignment (e.g., a student earning 9 points out 10 on research design for assignment one would be 90 on a scale of 0–100; this would be equivalent to someone who scored 18 out of 20 on the same skill, research design, for assignment 2). We then calculated the class average for each sub skill within an assignment. The total score for an assignment was scaled at 0–100, so the average class total score for an assignment is simply the average total scores for that assignment. Secondly, we conducted content analysis of the reflection section of the report in order to examine students’ thoughts on the relevance of inquiry to K-12 teaching and also ways through which inquiry (i.e., pedagogy) could be integrated with content (i.e., mathematical or scientific concepts underlying the hypothesis embedded in their inquiry projects). Our analysis followed five general steps in qualitative data analysis with an emphasis on content analysis (Renner & Taylor-Powell, 2003): (a) get to know the data: the researchers read the data so as to familiarize with the content of the responses; (b) focus the analysis: with our focus as the guide (i.e., students’ research skills at the baseline), we focused on students’ responses that had to do with the research process; (c) categorize information: Once we zoomed in on the responses, we categorized students’ responses based on the concepts in the inquiry framework (e.g., research hypothesis, identify variables, etc.; (d) identify patterns: with the categories in mind, we paid close attention to the quality of students’ responses (e.g., was the hypothesis clearly specified? Were variables defined correctly? Etc.); and (e) interpret the data: Looking across students’ responses (i.e., patterns), we developed a list of key findings with regards to the quality of students’ skills defined by the inquiry framework. We conducted similar content analysis of the pre-and-post survey responses (i.e., our survey questions elicit open-ended responses) and looked for patterns and themes emerged from students’ responses. To ensure credibility and consistency of the content analysis, we utilized several strategies. First, the codes used in light of the research question we asked are of low inference. Reliability and validity issues with low inference codes are lesser of a concern than codes of high inference. Secondly, the co-authors of the study were also co-instructors for the course. We did the initial coding and theme summarizing independently and then compared key takeaways we found in the data. We also triangulated and checked for consistency in findings from the quantitative scores of the written reports with patterns emerged from the qualitative data. Finally, we shared preliminary findings at various conferences both externally and internally (where program staff and instructors attended). The first author organized and chaired a panel discussion session at the State summit on STEM education. The panel

applications in advanced mathematics and sciences. The hypothetical laboratory experiment was designed to gauge students’ baseline knowledge of research skills, including identifying research the research question and hypothesis, justifying why the question and hypothesis are researchable, identifying and classifying key variables embedded in a research hypothesis, anticipating potential sources of measurement errors (extraneous variables), and designing strategies that will minimize sources of measurement errors. Specifically, the survey consists of a scenario question that describes an experiment done by a high school student testing whether the boiling point of water (i.e., dependent variable) might be affected by the volume of water and types of container (i.e., independent variables). The hypothesis was not given to students; rather the scenario described what the high school student did and the data he collected. At the end of the scenario, eight (8) questions were asked. These eight questions asked students to: (a) identify the research question/hypothesis embedded in the experiment, (b) why the question/hypothesis is researchable or can be tested, (c) what variables were considered and/or measured, which ones were dependent variable (DV), independent variable (IV), or held constant, and (d) what variables were potentially missing and what are the potential sources of measurement errors. In other words, we wanted to establish a baseline profile of these STEM majors’ research skills in terms of their ability to link a research question with operationalized variables and critical aspects of the research design that will generate data in order to test the hypothesis (e.g., what to vary/manipulate, what to hold constant, how to account for potential measurement errors, etc. through experimentation). On the last day of the class (roughly 15 weeks apart), students filled out the same survey. The end-of-course survey helps us to gauge students’ knowledge of research skills after having gone through the course. 4.2.2. Students’ written reflections Students were asked to reflect on their written reports why inquiry approach might be relevant to the teaching and learning of middle and high school mathematics and sciences. In particular, depending on their specific inquiry projects, they were asked to state explicitly the connection between the questions or research hypothesis underlying their project and the state mathematics and/or science standards for grades 6 through 12 students. This type of reflections was one way through which we wanted these STEM majors to be explicit about the content and pedagogy connection. 4.3. Data analysis Students’ written reports on their projects were analyzed in two ways. First, to examine students’ performance on the inquiry projects, we used a rubric (see Table 3 for an example). Each instructor independently scored students’ report using the rubric. We then reconciled our scores to reach a consensus where we differed. Once we agreed on the score, it was recorded as the final score for a student. The 6

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

Table 3 Example of Inquiry Report Scoring Rubric. Points earned

Points Pos-sible

Criterion

Description of Criterion and Performance

20

Inquiry Design

20

Presentation of data

10

Choice of analysis tools

20

Interpre-tation of data

20

Lessons learned (Final draft only)

10

Quality of writing

Is the report clear about how you gathered the data and how you accounted for error? High score: Writes a clear progression of steps by which data were collected plus tools and methods used to limit errors. Low score: Fails to articulate the logical progression of steps used to gather data, does not describe tools or how errors were addressed. Is data appropriately represented, using tables and graphs? High score: Data tables and graphs show all relevant data. Graphical representations are appropriate. Low score: Fails to present adequate data. Graphical representations not appropriate or not shown at all. Is data analysis justified and appropriate? High score: Statistical analysis matches the data and the inquiry question. Low score: Statistical analysis is omitted or fails to match the data and inquiry question. Are results accurate and correctly interpreted? High score: Statistics are calculated correctly, and the results are clearly and accurately explained. Low score: Incorrect calculations and explanations of results are unclear or inaccurate. Does reflection show thoughtful ways to retain and amend the question and design? High score: Identifies shortcomings of the inquiry question or methods and suggests reasonable alterations. Low score: Fails to identify shortcomings of the inquiry question or methods or to identify reasonable alterations. Grammar, spelling, organization, and syntax? High score: Writing well-organized; grammar, spelling, and syntax are sound; appropriate scientific terms used. Low score: Writing poorly organized—no section headers; misspelled words, poor grammar and syntax; colloquial or incorrect terms used.

100

Total

overwhelmingly focused only on the dependent variable (i.e., the boiling water temperature) when asked what the research question the experiment was attempting to answer. All but one student responded that the research question was, “Is the boiling point of water 100 degree Celsius (C)?” Consequently, it was not surprising that most students were not able to list all variables entailed in the experiment, correctly identify which variables were held constant and which variables were varied (i.e., IVs). Most students were able to identify that the temperature of the boiling water was the DV. A couple of students mistakenly considered the DV as the variable that was being varied. This signals the confusion between the IV as the variable being manipulated and varied and DV as the outcome, which leads to self-contradictory responses (i.e., the temperature being both a DV and as the variable that is being manipulated through the experiment which is the IV). In summary, this simple high school experiment revealed the conceptual hole among these STEM majors in their understanding of the critical research concepts defined by the inquiry framework (i.e., components that tie different procedural aspects of an experiment to the underlying domain knowledge such as research hypothesis, central constructs embedded in the hypothesis, relationship between the constructs, and how to investigate such a relationship by accounting for

discussion involved a student representative presenting one of the inquiry projects. We view this collaborative effort and various dissemination activities as one of the member checking mechanism in order to ensure key findings emerged from the study were consistent with students’ perspectives and transparent to key program stakeholders. 4.4. Findings Our overarching question is how course enrollees’ research skills evolve and in what ways they are able to connect inquiry with secondary mathematics and science content. In this section, we present our findings based on various data sources. Table 4 highlights key findings based on various data sources. 4.4.1. Finding 1: At the baseline, students showed weaknesses in identifying research questions or hypothesis, justifying why the question is researchable, and connecting questions with the research design Several observations can be made based on students’ responses to the survey questions at the baseline. Across board, students struggled the most with being able to identify the research question or central hypothesis embedded in the experiment. Specifically, students Table 4 Highlights of Key Findings. Research Focus

Themes

Inquiry Elements: Orientation/question Conceptualization/design Investigation/data Conclusion Discussion/communication Importance of inquiry for teaching math/science to secondary students:

Qualitative improvement in: - identifying research questions/hypothesis - justifying why the question is researchable - connecting questions with research design

- responses to open-ended survey questions - written inquiry reports

Inquiry: - motivates students to learn math/science as it helps them see the relevance - enables students to see the iterative nature of scientific investigation - allows students to learn from mistakes Sophistication of inquiry and experimentation needs to take into account students’ grade levels

- written reports reflections

Content & Pedagogy Connection:

Data Sources

7

- written reports reflections

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

skills as Inquiries 1 and 2 (see Table 5). Inquiries 1 and 2 represent laboratory research that models after the hard sciences (i.e., experimental designs that consist of hypothesis testing with precise measurements and repeating the same measurement procedures in order to test the accuracy of the results, etc.). Nonetheless, similar research procedural skills were involved in the survey research, including data representation, analysis and interpretation. Slightly different research skills needed for Inquiry 3 included considerations for the sample selection, the representativeness of the sample, and the validity/reliability of the survey instrument. Overall, students did quite well on all these domains as the scores indicated (average weighted scores ranged from 83 to 87, see Table 5). Last but not the least, students’ writing improved significantly over the course of the semester (see the scores for writing in Table 5). The improvement included both the substantive aspect of the writing and mechanical aspects such as grammar, punctuations, and so on. Students’ course evaluation and the focus group discussions revealed that the improvement in writing was most likely due to the consistent feedback from the instructors and peers that was built into the writing process of each inquiry report.

measurement errors through experimentation, etc.). 4.4.2. Finding 2: At the end of the course, students demonstrated qualitative improvement in the areas where they showed weaknesses at the baseline Analysis of students’ responses to end-of-course survey suggested that all but three students qualitatively improved in terms of their understanding of concepts defined by the inquiry framework. Most notably, those who framed the research question purely in terms of the dependent variable were able to correctly specify the research question. For instance, whereas a typical pre-survey response reads, “Is the temperature of boiling water 100 °C”, the end-of-course response was more accurately stated as “Does the volume and type of water container impact the boiling point of water?” Associated with the improvement in stating the research question, most students were also able to justify why the question is researchable in terms of the key hypothesis tested by the research question, the underlying constructs and measured variables, and data generated through the experiment that would allow researchers to test the hypothesis. In comparison, at the baseline, most students only broadly stated that the question is researchable because we could do an experiment without getting into any of the specific aspects of the research process. A typical example of baseline responses to why the question is researchable is, “Yes, because he [high school student] was able to design and complete an experiment to approach it”. Finally, students’ responses indicated that they gained skills at correctly classifying experimental variables and identifying sources of measurement errors. Overall, students’ end-of-course responses showed a qualitative improvement in their knowledge of the connection between a researchable question and the conceptualization of the research design to investigate the question.

4.4.4. Finding 4: when reflecting on why inquiry is important for teaching secondary mathematics and science content, course enrollees believe teaching secondary mathematics and science content through inquiry is important because inquiry helps students see the relevance and thereby motivates them to learn Course enrollees in general were very thoughtful about how inquiry could advance secondary students’ thinking and learning of mathematics and science. The reason they believe this is true is because they think inquiry helps students see the relevance of mathematical and/or scientific concepts to real life problems. The connection between science and real life problems seems to be a dominant reason for why inquiry is needed and important. For instance, one student commented that inquiry approach to teaching could help students connect concepts with real life problems and lead to a deep understanding of the concept: In this experiment students are using what they know about height, base, angle and equilibrium to learn about stable equilibrium and center of mass by applying these concepts to a real life scenario; Connect learning with real life experiences. This strategy helps in engaging students. It gives them more reasons on why they should learn the material being taught. And the reality is that learning is about developing understanding. When a student is actually learning, the student is able to use the learned concepts in any given problem and in real life scenarios. If students can connect concepts with real life problems, then the student has developed a deep understanding of the concept. In addition, course enrollees believe inquiry enables students to see the iterative nature of scientific investigations. Students argued that inquiry could enable students to come up with new questions leading to further inquiries, or replication of the same experiment in different contexts. This iterative investigative process is essential for developing deeper understanding of concepts, as one student pointed out: Also, when students find their own answers is more likely that they also come up with questions and some of these questions may require further research or more experiments. Again, this shows that students have developed a deep understanding and therefore they can connect and create new bridges to learn new concepts and apply what they learned to different scenarios. At the end of this experiment a student who gained a deep understanding of the concepts and investigation may want to create another experiment where instead of testing cylinders they will test triangles, cones, squares, etc. Finally, course enrollees believe inquiry allows students to learn from mistakes or to correct misconception. Students emphasized that teaching through inquiry could offer opportunities to learn from self-made mistakes, as the following example showed: In a classroom research or experiment I believe this is actually good

4.4.3. Finding 3: the observations based on the survey data were supported by their performance on the inquiry projects over the course of the semester Table 5 displays the weighted average scores for research subskills and total scores for each of the three inquiry reports. Based on weighted average of sub scores on Inquiry 1 (see Table 5), students struggled the most on stating explicitly a question (71.1), justifying why the question is researchable (63.7), identifying variables underlying in the research question (65.6), and also conceptualizing the design elements to minimize measurement errors (71.9). The average total score for the research report was 74.6. These results were consistent with findings from the baseline survey. By inquiry 2, students improved their performance on all areas of the research process (average weighted scores ranged from 89.4 to 89.4). The average total score was 89.9 for the research report, a 15points improvement (scaled at 0–100), compared to assignment 1 (average total score of 74.6). Inquiry 3 differs qualitatively from Inquiries1 and 2. Whereas Inquiries 1 and 2 focus on skills entailed in conducting traditional STEM laboratory experimentation, Inquiry 3 is essentially a social science survey research project, even though Inquiry 3 assesses some common Table 5 Mean Scores of Different Aspects of the Three Inquiry Project Reports. Inquiry 1 Identify research question Justify research question Classify variables Inquiry design Present data Analyze/interpret data n/a n/a Writing Total score

71.1 63.7 65.6 71.9 77.8 86.7

67.8 74.6

Inquiry 2

Inquiry 3

n/a n/a n/a Inquiry design Present data Analyze data Interpret data Reflections Writing Total score

n/a n/a n/a Inquiry design Present data Analyze data Interpret data Reflections Writing Total score

89.4 89.4 90 89.4 86.1 83.3 89.9

83 87 86 87 84 84 89

8

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

because the purpose of an experiment is for students to find their own answers, to learn from their mistakes and then to share with the whole class what went wrong and how could this have been improve. Also, the main purpose of this research is for students to learn about stable equilibrium and center of mass on their own with the materials and procedures provided. Students also commented that inquiry teaching gave students opportunities to correct misconception through direct observations. For instance, one student commented: The data provided by this research are highly valuable to high school students in a physics classroom. Misconceptions in such an environment are common, and students are particularly prone to confusion when dealing with phenomena such as an object’s moment of inertia, or the idea that objects accelerate at the same rate. From the perspective of a high school student, it would be contradictory to observe a rolling object outpacing another rolling object of the same mass and radius. This intentionally causes a disequilibrium, forcing the learner to reconsider this aspect of this data. The experiments outlined here would generate both the theory and experiential knowledge for more meaningful learning in the realm of physics.

given an opportunity to learn that mathematical proof is more rigorous than just showing through examples: In high school courses, the focus of the experiment should be to show the need for proofs in mathematics. Physical examples are subject to human error no matter how well accuracy and precision are tended to and do not show that the concept works for every single example. In constructing each of the physical examples, the trigonometric ratios can be emphasized as a means to find the interior angles of each polygon, and/or the formula for the size of the interior angles in a regular polygon can be derived. And this student ties exploratory proof (i.e., more inductive and theory generation so to speak) and mathematical proof together, arguing that experimentation helps students understand the importance of knowing the mathematical history and of abstract thinking ability: There are two main reasons this experiment would fulfill a need within a math class. The first is that it gives insight into the history of mathematics. Math is a subject where students are often expected to learn concepts without any context as to how they were discovered or why it is important. The most background information found in a math textbook may be a biography of someone whose name is attached to a theorem. While it is quicker to learn the concepts without the history, it devalues the fact that it may have taken thousands of years to come to that conclusion. Prior to Archimedes, measuring was the only way this sort of estimation was possible. With theoretical computation, Archimedes was able to get a value much more precise than ever before. Along with this, physically drawing out and measuring a shape with more than twenty sides is tedious and time consuming. Obtaining the raw data took approximately ten hours and is not something that would be feasible in an average class setting. Computation reduces this time into something far more manageable. Another science student explained how the experiment she designed could help students see applications of calculus in two different contexts: one in cooking and one in forensic science. As calculus concepts are hard to grasp, this student argued that showing the applications of calculus of different sophistication in real life could help to sustain secondary students’ interests and ultimately foster deep content learning. We regard this student’s idea another way of how content and pedagogy can be integrated to help students learn mathematics. Students who conduct this experiment will learn about an application of calculus that is outside of the classroom and one that could potentially be useful in everyday life. Though cool-down times for food do not seem very important nor do they serve a very meaningful purpose it is still more interesting for students to learn this concept in an applicable way. There is also an application of this law to forensic sciences. …forensic scientists’ use and application of this law to determine approximate time of death. Students interested in this application could research about this application as well. Many calculus concepts are hard for teachers to illustrate so teaching derivatives in context of something real, like cool-down times, may engage students more than doing practice problems out of the back of a book. These findings showed that well-thought out inquiry projects were a useful way to integrate content and pedagogy because inquiry taught through our research methods course allowed them to see the importance of teaching students what inquiry means as opposed to what they learned in high school lab class which was to follow instructions, as this student summed up nicely: Students who are interested in the field of science gain valuable experience if taught to view experimentation as a process of design, rather than just following instructions. In conclusion, results based on students’ three inquiry assignments, and their responses to the pre- and post-course survey suggest that providing opportunities for future STEM teachers to conduct scientific inquiry and reflecting on how inquiry can be a pedagogy for teaching mathematics and/or science concepts is beneficial.

4.4.5. Finding 5: A few STEM majors pointed out that when teaching secondary students, the sophistication of inquiry and experimentation needs to take into account students’ grade level. We argue this is an important aspect of connecting content and pedagogy (i.e., a hint at PCK) and the awareness of “expert blind spot” problem A few students demonstrated their ability to think about how aspects of the inquiry and the sophistication of the concepts involved could be adjusted based on levels of secondary students’ capacity. We regard these students’ content understanding as good examples of what PCK embodies (i.e., understanding content deeply so as to flexibly adjust the pedagogical approach in order to make content accessible to learners). For instance, the following description made by an engineering student demonstrates a deep understanding of concepts underlying his inquiry project, while at the same time, making the concepts accessible to high school students: Resistance is a fundamental concept of electrical systems and therefore is vital to understanding how technology is designed and the theory behind instrumentation in mechanical engineering. Students will be able to directly observe this relationship because the data generated through this experiment will show a clear trend relating the resistance to temperature, with only minor influence by sources of error. Fundamentally, this was a sound inquiry that produced acceptable results. The inquiry was driven by a clearly stated research question. Additionally, it was defensibly relevant to mechanical engineering because it relates to setting design limitations on models the theory behind instrumentation and how an engineer might design a digital temperature probe. The results were also relevant to educating students about engineering because the experiment produced the very strong linear trend suggested by the theory, and relatively accurately calculated the temperature coefficient resistance for copper. Students could simply and safely conduct this inquiry both to directly observe the effects of temperature on resistance and to investigate the engineering theories behind instrumentation and design limitations. Similar to this engineering student, one of the math majors commented on the differentiated purposes based on the grade levels of students. For instance, at lower grades experimentation helps students to develop an intuitive understanding of mathematical proof: In lower level math courses, this experiment could provide insight into problems involving perimeters, circumference, and measurement precision. Rather than having the students create the various polygons, they could be given premade ones from which to make their estimations. Accuracy in measurements will lead to more accurate estimations of pi. Once the physical experimentation is completed, students must be led towards the concept that proving something works for a few examples in mathematics does not necessarily mean that it works in all circumstances. This student explained that at higher grades, students should be

4.5. Discussion and further research Recruiting and preparing STEM majors for a teaching career has 9

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

practices. These advantages of teaching mathematics and science through inquiry are consistent with why researchers have argued for the emphasis of inquiry pedagogy (Deslauriers et al., 2011; Silverstein et al., 2009). Interestingly, while almost all STEM majors were able to identify with the advantages of inquiry approach to teaching secondary mathematics and science, only a few of them recognized the need to adjust the sophistication of inquiry and experimentation with the grade levels of students. For instance, one math major pointed out that experimentation helps students to develop an intuitive understanding of mathematical proof at the lower grades but emphasized that at higher grades, students should be given opportunities to learn that mathematical proof is more rigorous than just showing through a few examples. Similarly, an engineering major demonstrated a deep understanding of concepts underlying his inquiry project, while at the same time making the concepts accessible to high school students. We regard these students as good examples of what PCK embodies. In other words, they understood content deeply so as to be able to flexibly adjust the pedagogical approach in order to make content accessible to learners. The opportunity to make explicit the pedagogy and content connection has brought out an awareness of “expert blind spot” problem among these students. This problem could be investigated further among programs that recruit and prepare STEM majors for teaching secondary mathematics and sciences. In closing, we would like to reflect on our research effort and connect our work with the broader research community. Improving K-12 students’ opportunities to learn and performance in mathematics and science has been of major concern for several decades. Policymakers around the world have a strong interest in reforming mathematics and science education because these subjects are seen as critical for economic competitiveness through their impact on innovation in scientific research and technology (Augustine, 2005; National Research Council, 1991; Sahlberg & Oldroyd, 2010; Sahlberg, 2006; Steen, 1987). Historically, mathematics education reformers have advocated teaching for broad skills, emphasizing the important applications of mathematics and encouraging teaching for understanding and proficiency (Fennema & Romberg, 1999; Geary, 1994; Herrera & Owens, 2001; Rotherham & Willingham, 2009). Similar efforts are evident in science education reforms, as various organizations have advocated for an inquiry approach to the teaching and learning of science where learners gain scientific knowledge by immersing themselves in an inquiry process that is similar to what scientists do to advance their fields (American Association for the Advancement of Science & Project 2061, 1998American Association for the Advancement of Science, 1998American Association for the Advancement of Science & Project 2061, 1998; Mullis, Martin, Ruddock, O’Sullivan, & Preuschoff, 2009; National Research Council & Mathematics Learning Study Committee, 2001; Organisation for Economic Co-operation & Development Staff, 2009). Despite waves of education reforms on K-12 mathematics and science teaching and learning, American students’ performance on these subjects remain lackluster when compared to their international peers (Loveless, 2013; National Center for Education Statistics, 2004; Programme for International Student Assessment, 2003). To improve students’ opportunities to learn mathematics and science, we need a parallel attention to their teachers’ opportunities to learn (Stigler & Hiebert, 1999). Higher education institutions play a critical role in this effort because they are where future mathematics and science teachers gain disciplinary knowledge and pedagogical skills necessary for them to function effectively as classroom teachers (Schmidt et al., 2011). By attending to a critical component of the mathematics and science education system (i.e., pre-service mathematics and science teacher education), we intend to make a contribution to the field by producing teachers who not only are strong in their disciplinary knowledge, but also can utilize research methods effectively to educate the next generation of STEM talents. The inquiry frameworks we reviewed from

increasingly become one of the major efforts at improving mathematics and science teacher quality at secondary level (Mervis, 2007). Programs such as 100k10 in New York, UTeach in UT Austin, and UTeach replication sites across the country represent some of the most wellknown and established ones. These interdisciplinary STEM teacher preparation programs have a heavy emphasis on using inquiry to teach middle and high school mathematics and sciences (Newton & Poon, 2015; Dailey et al., 2015; Hutchison, 2012; Mervis, 2007). This emphasis is not surprising as inquiry has featured prominently in various calls for mathematics and science reforms. Despite widespread belief in its positive impact on student learning, secondary mathematics and science classroom teaching and learning remain a stark contrast to what reforms push for. Teachers play an important role in not only what is learned but also how content is taught. One question is whether STEM majors who had little chance to experience active learning as secondary students know what inquiry is. An associated second question is whether STEM majors who experienced inquiry pedagogy in their disciplines at college level will be able to conceptualize how to use inquiry to teach mathematics and science content at secondary level (i.e., grades 6–12). We investigated these questions through an undergraduate research methods course designed to improve undergraduate STEM majors’ capacity for delivering inquirybased mathematics and science lessons. Our study yielded several interesting findings. One of the key findings is that we cannot take for granted that STEM majors know the basic concepts defined by the inquiry frameworks. For instance, at the beginning of the course, we found that students struggled with basic concepts such as dependent, independent, and extraneous variables. In addition, though they were good at designing an experiment and collecting data, they had a hard time justifying why they did what they did (i.e., clearly articulating the research hypothesis and/or questions). To some extent, this finding should not be surprising, because these students are used to the step-by-step laboratory activities that tend to dominate high school science classes in the U.S. On the other hand, our data also shows that a research course like the one we taught gives students opportunities to unlearn what they have learned about scientific inquiry in their high schools and relearn how scientists typically conduct their scientific investigations. This type of research course is needed for future STEM teachers if we expect them to teach secondary mathematics and science through inquiry (Deslauriers, Schelew, & Wieman, 2011; Silverstein, Dubner, Miller, Glied, & Loike, 2009). We also found that STEM majors we studied identified several reasons why teaching mathematics and science to secondary students through inquiry is important. They believed inquiry would help students see the relevance of mathematics and science and thereby motivate them to learn. As an example, one student pointed out the coolingdown time for food in kitchen experiment could help students learn law of cooling and calculus. While this daily mundane chore might not be significant or meaningful, the same concepts (i.e., law of cooling and calculus) are used by forensic scientists in their daily work to determine time of death, which is an important application of mathematics and science and “may engage students more than doing practice problems out of the back of a book”. Additionally, inquiry allows students to experience and see the iterative nature of scientific investigation. Specifically, students argued that inquiry could enable K-12 students to come up with new questions, leading to further investigations or replications of the same experiment in different contexts. This iterative investigative process is essential for developing deeper understanding of concepts. Finally, inquiry allows students to learn from mistakes, because inquiry approach to teaching could offer opportunities to learn from self-made mistakes or correct misconceptions through direct observations. The survey assessment inquiry also helps students recognize the importance of using assessment tools to gauge students’ prior knowledge, which is an important aspect of good teaching and learning 10

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

other scholars provided a way to both design our research methods course and offered a conceptual framework to guide our research on whether the course met its intended objectives or not. Our work is based on a small sample of STEM majors and is limited to our undergraduate STEM teacher preparation program. In addition, our research does not have information on how these students will do in terms of classroom teaching and learning once they become K-12 teachers. Despite these limitations, we believe this line of investigation has important implications. Our study provided insights into STEM majors’ thinking on inquiry and content. By sharing our research findings with higher education scholars and educators of teacher education, we hope they may replicate our research attempt at other undergraduate STEM teacher preparation programs such as the Uteach and Uteach replication programs in different higher education institutions across the U.S. Collectively, these small scale exploratory studies could generate a knowledge base for what works in STEM teacher preparation. Over time, we as a research community will advance the thinking on how to best implement research-based teaching practices among pre-service mathematics and science teachers so that future students taught by these program graduates will benefit greatly.

Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332, 862–864. https://doi.org/10.1126/science. 1201783. Duschl, R. A. (2003). Assessment of inquiry. In J. M. Atkin, & J. Coffey (Eds.). Everyday assessment in the science classroom (pp. 41–59). Arlington, VA: NSTA Press. Duschl, R. (2008). Science education in three-part harmony: Balancing conceptual, epistemic, and social learning goals. Review of Research in Education, 32(1), 268–291. https://doi.org/10.3102/0091732X07309371. Estes, F., & Clark, R. E. (1999). Authentic educational technology: The lynchpin between theory and practice. Educational Technology, 39(6), 5–13. Fennema, E., & Romberg, T. A. (Eds.). (1999). Mathematics classrooms that promote understanding. Mahwah, NJ: Lawrence Erlbaum Associates. Furtak, E. M., Seidel, T., Iverson, H., & Briggs, D. C. (2012). Experimental and quasiexperimental studies of inquiry-based science teaching: A meta-analysis. Review of Educational Research, 82(3), 300–329. Gallagher, J. M., & Reid, D. K. (2002). The learning theory of Piaget and Inhelder. iUniverse. Geary, D. C. (1994). Children’s mathematical development: Research and practical applications. American Psychological Association. Herrera, T. A., & Owens, D. T. (2001). The" new math?” two reform movements in mathematics education. Theory into Practice, 40(2), 84–92. Hiebert, J., Stigler, J., Jacobs, J., Givvin, K., Garnier, H., Smith, M., ... Gallimore, R. (2005). Mathematics teaching in the United States today (and tomorrow): Results from the TIMSS 1999 video study. Educational Evaluation and Policy Analysis, 27(2), 111–112. Hutchison, L. F. (2012). Addressing the STEM teacher shortage in American schools: Ways to recruit and retain effective STEM teachers. Action in Teacher Education, 34(5–6), 541–550. Jaworski, B. (2002). Investigating mathematics teaching: A constructivist enquiry. Routledge. Kirschner, P. A., Martens, R. L., & Strijbos, J. W. (2004). CSCL in higher education? What we know about CSCL. Dordrecht: Springer3–30. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. Lott, G. W. (1983). The effect of inquiry teaching and advance organizers upon student outcomes in science education. Journal of Research in Science Teaching, 20(5), 437–451. Loveless, T. (2013). The 2013 Brown Center Report on American Education: How well are American students learning? With sections on the latest international tests, tracking and ability grouping, and advanced math in 8th gradeAccessible at: http://www.brookings.edu/∼/media/research/files/reports/2013/03/18-brown-center-loveless/ 2013-brown-center-report-web.pdf. Mervis, J. (2007). A new twist on training teachers. Science, 316(5829), 1270–1277. Mullis, I. V., Martin, M. O., Ruddock, G. J., O’Sullivan, C. Y., & Preuschoff, C. (2009). TIMSS 2011 assessment frameworks. International Association for the Evaluation of Educational Achievement. Nathan, M. J., Koedinger, K. R., & Alibali, M. W. (2001). Expert blind spot: When content knowledge eclipses pedagogical content knowledge. Proceedings of the third international conference on cognitive science April (Vol. 644648). National Center for Education Statistics (2004). Highlights from the trends in International mathematics and science study (TIMSS) 2003. DC: Author. National Council of Teachers of Mathematics (2000). Principles and standards for school mathematics. Reston, VA: Author. National Research Council (1991). Mathematical sciences, technology, and economic competitiveness. National Academies Press. National Research Council (1996). National science education standards. National Academies Press. National Research Council (2001). Classroom assessment and the national science education standards. National Academies Press. National Research Council, & Mathematics Learning Study Committee (2001). Adding it up: Helping children learn mathematics. National Academies Press. Newton, X., & Poon, R. (2015). Mathematical content understanding for teaching: A study of undergraduate STEM majors. Creative Education, 6(10), 998–1031. https://doi.org/ 10.4236/ce.2015.610101. OECD Publishing, & Organisation for Economic Co-operation and Development Staff (2009). Creating effective teaching and learning environments: First results from TALIS. Organisation for Economic Co-operation and Development. Otero, V., Pollock, S., & Finkelstein, N. (2010). A physics department’s role in preparing physics teachers: The Colorado learning assistant model. American Journal of Physics, 78(11), 1218–1224. Pauli, C., Reusser, K., & Grob, U. (2007). Teaching for understanding and/or self-regulated learning? A video-based analysis of reform-oriented mathematics instruction in Switzerland. International Journal of Educational Research, 46(5), 294–305. Pedaste, M., Mäeots, M., Siiman, L. A., De Jong, T., Van Riesen, S. A., Kamp, E. T., ... Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review, 14, 47–61. Piaget, J. (1964). Part I: Cognitive development in children: Piaget development and learning. Journal of Research in Science Teaching, 2(3), 176–186. Renner, M., & Taylor-Powell, E. (2003). Analyzing qualitative data. Programme development & evaluation. University of Wisconsin-Extension Cooperative Extension1–10. Programme for International Student Assessment (2003). First results from PISA 2003. Paris, France: Author. Rotherham, A. J., & Willingham, D. (2009). 21st century. Educational Leadership, 67(1), 16–21. Sahlberg, P. (2006). Education reform for raising economic competitiveness. Journal of Educational Change, 7(4), 259–287.

5. Conclusion Recruiting and preparing STEM majors for teaching has become one of the major efforts at improving mathematics and science teacher quality at secondary level. The common assumptions are that STEM majors have strong disciplinary knowledge and are familiar with inquiry which is the basis for various scientific and technological discoveries. What has not been brought to the forefront is the need for an explicit integration of how inquiry pedagogy can be utilized to teach secondary mathematics and science content. This omission is problematic because research has shown that experts with advanced domain knowledge in a field tend to be unaware of struggles confronted by students or novices, a phenomenon called “expert blind spot” (Nathan et al., 2001). Our research methods course provides a means for an intentional and explicit emphasis on integrating inquiry and secondary content. Results showed that offering future STEM teachers opportunities to conduct inquiry and reflect explicitly on how inquiry can be used to teach secondary content is important and beneficial. This line of research helps to address the “expert blind spot” problem and building future STEM teachers’ capacity for delivering inquiry-based mathematics and science lessons. Acknowledgement The authors would like to thank all students who enrolled in the research methods course and gave their permission to use the data for research purposes. References Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1. American Association for the Advancement of Science, & Project 2061 (American Association for the Advancement of Science) (1998). Blueprints for reform: Science, mathematics, and technology education. Oxford University Press. Anderson, J. R., Reder, L. M., & Simon, H. A. (1999). Applications and misapplications of cognitive psychology to mathematics education. Augustine, N. R. (2005). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Bandura, A., & Walters, R. H. (1977). Social learning theory, Vol. 1. Englewood Cliffs, NJ: Prentice-hall. Clark, R. E., & Estes, F. (1998). Technology or craft: What are we doing? Educational Technology, 38(5), 5–11. Clark, R. E., & Estes, F. (1999). The development of authentic educational technologies. Educational Technology, 39(2), 5–16. Dailey, D., Bunn, G., & Cotabish, A. (2015). Answering the call to improve STEM education: A stem teacher preparation program. Journal of the National Association for Alternative Certification, 10(2), 3–16.

11

Studies in Educational Evaluation 64 (2020) 100833

X.A. Newton and E.P. Tonelli

Stigler, J., & Hiebert, J. (1999). The teaching gap: Best ideas from the world’s teachers for improving education in the classroom. New York: Free Press. UTeach Institute (2019). Snapshot of UTeach program expansion. Retrieved from:https:// institute.uteach.utexas.edu/sites/institute.uteach.utexas.edu/files/uteach-nationalsnapshot.pdf. UTeach UMass Lowell Newsletter (2018). Retrieved from: https://XXXX/UTeach %20Newsletter_tcm18-279959.pdf. Van Uum, M. S., Verhoeff, R. P., & Peeters, M. (2016). Inquiry-based science education: Towards a pedagogical framework for primary school teachers. International Journal of Science Education, 38(3), 450–469. Vygotsky, L. (1978). Interaction between learning and development. Readings on the Development of Children, 23(3), 34–41. Wertsh, J. V., & Tulviste, P. (1990). Apprenticeship in thinking: Cognitive development in social context. Science, 249(4969), 684–686. Wise, K. C., & Okey, J. R. (1983). A meta‐analysis of the effects of various science teaching strategies on achievement. Journal of Research in Science Teaching, 20(5), 419–435. Yoon, H. G., Joung, Y. J., & Kim, M. (2012). The challenges of science inquiry teaching for pre-service teachers in elementary classrooms: Difficulties on and under the scene. Research in Science Education, 42(3), 589–608. York, M. K., & Dallas, U. (2018). STEM content and pedagogy are not integrated. Zimmerman, B. J. (2013). Theories of self-regulated learning and academic achievement: An overview and analysis. Self-regulated learning and academic achievement. Routledge10–45.

Sahlberg, P., & Oldroyd, D. (2010). Pedagogy for economic competitiveness and sustainable development. European Journal of Education, 45(2), 280–299. Sanders (2009). STEM, STEM Education, STEMmania. The Technology Teacher. December/ January 2009, 20–26. Retrieved from: https://vtechworks.lib.vt.edu/bitstream/ handle/10919/51616/STEMmania.pdf?sequence. Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational technology, 35(5), 31–38. Schmidt, W. H., Houang, R., & Cogan, L. S. (2011). Preparing future math teachers. Education Forum, Science, 332, 1266–1267. Schroeder, C. M., Scott, T. P., Tolson, H., Huang, T. Y., & Lee, Y. H. (2007). A meta‐analysis of national research: Effects of teaching strategies on student achievement in science in the United States. Journal of Research in Science Teaching, 44(10), 1436–1460. Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. Silverstein, S. C., Dubner, J., Miller, J., Glied, S., & Loike, J. D. (2009). Teachers’ participation in research programs improves their students’ achievement in science. Science, 326, 440–442. https://doi.org/10.1126/science.1177344. Smith, T. M., Desimone, L. M., & Ueno, K. (2005). “Highly qualified” to do what? The relationship between NCLB teacher quality mandates and the use of reform-oriented instruction in middle school mathematics. Educational Evaluation and Policy Analysis, 27(1), 75–109. Steen, L. A. (1987). Mathematics education: A predictor of scientific competitiveness. Science, 237(4812), 251–302.

12