Students’ guided inquiry with simulation and its relation to school science achievement and scientific literacy

Journal Pre-proof Students’ guided inquiry with simulation and its relation to school science achievement and scientific literacy Cai-Ting Wen, Chen-Chung Liu, Hsin-Yi Chang, Chia-Jung Chang, Ming-Hua Chang, Shih-Hsun Fan Chiang, Chih-Wei Yang, Fu-Kwun Hwang PII:

S0360-1315(20)30032-4

DOI:

https://doi.org/10.1016/j.compedu.2020.103830

Reference:

CAE 103830

To appear in:

Computers & Education

Received Date: 5 September 2019 Revised Date:

7 January 2020

Accepted Date: 2 February 2020

Please cite this article as: Wen C.-T., Liu C.-C., Chang H.-Y., Chang C.-J., Chang M.-H., Fan Chiang S.-H., Yang C.-W. & Hwang F.-K., Students’ guided inquiry with simulation and its relation to school science achievement and scientific literacy, Computers & Education (2020), doi: https://doi.org/10.1016/ j.compedu.2020.103830. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier Ltd.

GUIDED INQUIRY WITH SIMULATION

Students’ Guided Inquiry with Simulation and Its Relation to School Science Achievement and Scientific Literacy

Cai-Ting Wen: Conceptualization, Methodology, Software, Formal analysis, Writing Original Draft, Writing - Review & Editing Chen-Chung Liu: Conceptualization, Methodology, Formal analysis, Writing - Original Draft, Writing - Review & Editing Hsin-Yi Chang: Conceptualization, Methodology, Formal analysis, Investigation, Writing Original Draft, Writing - Review & Editing Chia-Jung Chang: Methodology, Software, Validation Ming-Hua Chang: Software, Validation Shih-Hsun Fan Chiang: Software, Validation Chih-Wei Yang: Validation, Supervision Fu-Kwun Hwang: Validation, Supervision

1

GUIDED INQUIRY WITH SIMULATION

Students’ Guided Inquiry with Simulation and Its Relation to School Science Achievement and Scientific Literacy Cai-Ting Wen1, Chen-Chung Liu1, Hsin-Yi Chang2*, Chia-Jung Chang3, Ming-Hua Chang1, Shih-Hsun Fan Chiang1, Chih-Wei Yang4, and Fu-Kwun Hwang5 1

2

National Central University, Taiwan. Program of Learning Sciences, School of Learning Informatics, National Taiwan Normal

University, Taiwan. 3

Takming University of Science and Technology

4

National Taichung University of Education

5

Department of Physics, National Taiwan Normal University, Taiwan.

*corresponding author Hsin-Yi Chang: Postal address: Program of Learning Sciences, National Taiwan Normal University, #162, Sec.1, Heping E. Rd., Taipei City, 106, Taiwan Phone: 886-2-7734-5633; [email protected] Conflict of Interest All authors of this manuscript declare that we have no conflict of interest. Source of Funding This work was supported by Ministry of Science and Technology, Taiwan, under Grants MOST106-2511-S-008-012-MY3, MOST107-2511-H-008-003-MY3, MOST107-2811-H008-006, and MOST108-2628-H-003-001-MY3. It was also financially supported by the Institute for Research Excellence in Learning Sciences of National Taiwan Normal University (NTNU) from the Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan.

1

Students’ Guided Inquiry with Simulation and Its Relation to School Science Achievement and Scientific Literacy Abstract The study investigates the effect of an interactive simulation with embedded inquiry support, which was seamlessly embedded within the simulation, on students’ scientific literacy and school science achievement, and explores the relationships among prior school science achievement, inquiry processes and scientific literacy. A total of 49 eighth-grade students at a public junior high school in northern Taiwan participated. Data collected include the students’ pre- and post-scores for school science achievement, logging data that indicate their inquiry processes, and pretest, posttest, and delayed-test data that measure their scientific literacy. The results provide evidence that the designed simulation and inquiry support had a long-term effect on the students’ scientific literacy. Replacing conventional teaching with inquiry activities did not harm the students’ school science achievement performances. Moreover, compared to school science achievement, students’ scientific literacy seems a better predictor of their inquiry behavior, especially in the aspect of making conclusions. Analyses of the students’ inquiry processes indicate that the so-called low science achieving students conducted more data analyses than the other students, and demonstrated adequate inquiry engagement. The students with middle level school science achievement demonstrated the most active engagement in inquiry and showed good gains of scientific literacy after the learning. These results indicate that a guided inquiry learning environment can support students with different levels of school science achievement to highly engage in science inquiry. Implications and future studies are discussed. Keywords: Applications in subject areas; Simulations; Pedagogical issues; Improving classroom teaching; Secondary education; Inquiry; Scientific Literacy.

1

1. Introduction Science education standards worldwide stress the importance of engaging students in learning science through inquiry (e.g., NGSS Lead States, 2013; Taiwan Ministry of Education, 2014). This learning approach is called inquiry learning, which situates learning in investigations of complex problems or phenomena with an emphasis on learning activities that involve, but are not limited to, the process of posing questions and investigating them with empirical data (Bell, Urhahne, Schanze, & Ploetzner, 2010; Hmelo-Silver, Duncan, & Chinn, 2006). Specifically, computer simulations can be incorporated into inquiry learning environments to enable students’ investigations. Research has found positive impacts of student inquiry with simulations on students’ understanding of science concepts (Chiu, DeJaegher, & Chao, 2015), development of inquiry skills (Efstathiou et al., 2018) or science epistemic beliefs (Huang, Ge, & Eseryel, 2017). However, few studies have investigated whether and how the use of simulations in science inquiry learning environments may facilitate students’ scientific literacy, given that developing students’ scientific literacy has been recently emphasized in science education standards globally and locally (NGSS Lead States, 2013; OECD, 2016; Taiwan Ministry of Education, 2014). Moreover, a practical concern involves a common reluctance of science teachers to engage students in inquiry learning because many teachers may concern that implementing inquiry learning activities may result in a decrease in students’ school science achievement scores. However, we found little research that was able to address this practical concern since

2

studies often developed their own instruments to measure students’ conceptual understanding of science (e.g., Author, 2017; Chiu et al., 2015; Thacker & Sinatra, 2019) rather than using school science achievement scores. To address the issues that have not been investigated much in past studies, in this study, we investigated the effect of the guided inquiry with simulations environment on learners’ scientific literacy and school science achievements. Another central issue remains regarding how to design and guide student inquiry with simulation to best benefit learning (Donnelly, Linn, & Ludvigsen, 2014; Scalise et al., 2011; Rutten, van Joolingen, & van der Veen, 2012). For example, students may have difficulties conducting mindful and purposeful inquiry with simulations, given the openness of an interactive simulation environment (McElhaney & Linn, 2011). In this study, a new interactive simulation focusing on the concept of buoyancy has been developed via the CoSci platform (http://cosci.tw/) (Authors, 2018) and was used in this study. Learners’ inquiry with the simulation is guided through students’ self-generated inquiry maps. This guidance employs an innovative approach to support students’ inquiry by providing elements of inquiry and allowing students to create their own inquiry map (detailed in the methods section). We examine the effectiveness of the design in this study. Despite the existence of the studies documenting successful interventions of science inquiry with simulations and their effects on learning outcomes, there is also still a need for studies investigating students’ inquiry processes and linking these processes to their pre- and post-performances. For example, Gal, Uzan, Belford, Karabinos, and Yaron (2015) analyzed log files collected from an open-ended virtual laboratory environment to make sense of students’

3

actions during inquiry. Their purpose was to develop an automatic approach to analyzing students’ problem-solving processes that can be used in the future to develop technology for adaptive feedback. With this focus the study only investigated the learning process. Issues still remain such as what kinds of inquiry processes are more or less productive in terms of leading to a more or less successful inquiry learning outcome. As initial steps to address such issues, in this study we linked the students’ inquiry learning processes to their inquiry learning outcomes and characterized the students based on their learning processes and performances. Such work can provide insight into the development of science inquiry learning environments that address the needs of different students. The research questions addressed include: (1) What is the quality of the students’ inquiry with the simulation? (2) What is the effect of the guided inquiry with the interactive simulation on students’ scientific literacy and school science achievement? (3) What are the relationships, if any, among the students’ prior school science achievement, their inquiry process with the interactive simulation, and their developed scientific literacy? (4) How can the students’ performance be characterized according to their school science achievement, quality of inquiry process and scientific literacy? The present study aimed to provide insight into the question of how to augment effects of inquiry with simulation learning environments, such as effective designs of inquiry with interactive simulations adapted for students with different science achievement levels to promote scientific literacy. It is hoped that the results of the study contribute to theory and practice in that they address the theoretical perspectives and practical

4

concerns regarding whether and how inquiry learning with simulations has impact on students’ scientific literacy and school science achievement.

1.1 Background 1.1.1 Guided science inquiry with interactive simulation Simulations refer to computer programs that simulate scientific phenomena through systems of representations or models. Interactive features are usually programmed so that students can interact with the simulation to conduct virtual experiments. Virtual experiments involve the use of technologies to engage students in “What-If” explorations where the outcomes of the experiments can be immediately accessed through the use of a simulation (Hennessy et al., 2007). Virtual experiments are particularly beneficial for supporting students’ explorations on phenomena that cannot be easily observed or investigated in real-life situations. Reviews on the use of computer simulations to support learning have indicated overall positive effects of computer simulations on learning outcomes (e.g., Authors, 2015; Donnelly, Linn, & Ludvigsen, 2014; Scalise et al., 2011; Rutten, van Joolingen, & van der Veen, 2012). For example, Scalise et al. (2011) reviewed 79 articles reporting on the learning effects of using computer simulations to support science learning, and found that 53.2% of the studies reported gains and 17.7% reported gains under the right conditions, such as effective interfaces that consider students’ cognitive load, or inquiry supports that guide students’ learning with the simulation, whereas 25.3% had mixed results, and only 3.8% showed no gains. It can be

5

evidenced that computer simulations often support science learning, but a central issue remains, that is, how to design and guide student inquiry with simulation to best benefit learning. Blanchard et al. (2010) categorized inquiry instruction into four levels: Level 0 refers to verification laboratory instruction in which the teacher provides students with the question and methods of investigation, and guides them toward an expected conclusion; Level 1 refers to structured inquiry in which students are provided with a question and a method but are responsible for interpreting the result; Level 2 refers to guided inquiry in which students design their own method of investigation and generate interpretation of the results; Level 3 refers to open inquiry in which students generate a question and take responsibility for all major aspects of the investigation. Studies have shown that the verification or cookbook-style laboratory instruction has an unfavorable influence and the least effect on student learning (Blanchard et al., 2010; Scalise et al., 2011). In contrast, the results from the study by Adams, Paulson, and Wieman (2009) suggest that students are more engaged and learn better with minimal but nonzero guidance as they conduct inquiry with simulations. Similarly, one study found that the participants in the guided inquiry with simulation condition conducted more systematic and comprehensive investigations and reported a lower level of cognitive load than the participants in the unguided inquiry with simulation condition (Moon & Brockway, 2019). In light of the relative effect of structured and guided inquiry with simulation, Author (2017) found that structured inquiry supported students in conducting more virtual experiments with simulations, but such student behavior did not lead to better conceptual understanding of the science concepts targeted in the simulation, given that both the students in the guided and structured

6

inquiry conditions developed equally good conceptual understanding. As a result, the guided inquiry approach led to better learning efficiency than the structured inquiry approach did, because the guided inquiry group conducted fewer experiments but gained comparable conceptual understanding of the science concepts. Donnelly et al. (2014) reviewed 30 effective science inquiry learning environments with visualization or simulation, and found that the majority of the environments employed guided inquiry. Overall, consistent evidence indicates that guided inquiry with simulations can benefit student learning. However, there are variations of guided inquiry and their effects need further differentiation and investigation. The guided inquiry approach is also employed in the current study in which elements of inquiry are provided for students to create their inquiry map which guides students’ inquiry process (detailed in the methods section). The design of such guidance is consistent with design principles suggested for inquiry learning environments (Donnelly et al., 2014; Quintana et al., 2004; Scalise et al., 2011). Salient components of inquiry are revealed via the inquiry map, and hints or prompts are provided in each component to support students’ thinking and inquiry process. Multiple paths are allowed so that students can set and test their own hypotheses with virtual experiments. We examine the effectiveness of such a design in the current study. 1.1.2 Learning outcomes of inquiry with simulation Well-designed science inquiry with simulation learning environments can be effective in promoting at least three types of learning outcomes, as research has provided evidence. First, guided interaction with simulations can promote students’ conceptual understanding of the content targeted in the simulation (e.g., Author, 2017; Chiu et al., 2015; Thacker & Sinatra,

7

2019). For example, Chiu et al. (2015) engaged 45 eighth-grade students in guided science inquiry with a molecular simulation. They found that the students made significant gains in terms of their understanding of gas properties from the pretests to the posttests. Second, guided inquiry with simulations can enhance students’ general inquiry skills (Efstathiou et al., 2018). In the study by Efstathiou et al. (2018), the fifth-grade students used the Go-Lab simulation and computer-supported inquiry learning environment in which they were guided to conduct virtual experiments to learn science concepts related to sinking, floating and relative density. The Experiment Design Tool of the Go-Lab environment provides the experimental group students with feedback regarding their design of virtual experiments. In contrast, the control group only used paper-based worksheets that supported the students’ design of virtual experiments without feedback. The experimental group outperformed the control group on assessment measuring inquiry skills including four aspects of designing an experiment: identifying variables, stating hypotheses, operationally defining, and designing investigations. Since the items of the assessment involve the four aspects of inquiry skills across various contexts and events, the assessed inquiry skills indicate more domain-general than domain-specific inquiry skills. The results of the study provide evidence for the use of guided inquiry with simulation environments to facilitate students’ general inquiry skills. Third, one study found that engaging students in guided inquiry with simulations also enhanced the students’ science epistemic beliefs (Huang et al., 2017). The students significantly improved in their beliefs about the source or certainty of science knowledge after the intervention lasting 10 days with one 45-minute session each day in which the students were

8

guided to conduct an inquiry with two PhET simulations about motion and force. The guidance was prompting questions to engage students in inquiry phases of predict-observe-explain (White & Gunstone, 1989). The study provided evidence that students’ fundamental beliefs about science can also be changed and developed during guided inquiry with computer simulations. In the current study we focus on scientific literacy, which has received relatively little attention as a learning outcome in research on guided inquiry with simulations. Developing future citizens’ scientific literacy involves the goal of educating young people to become critical users of scientific knowledge, including developing their ability to explain phenomena scientifically, evaluate and design scientific inquiry, and interpret data and evidence scientifically (OECD, 2016). This goal has been emphasized in science education standards in Taiwan (Taiwan Ministry of Education, 2014) and globally (e.g., NGSS Lead States, 2013). Compared to general inquiry skills often measured in studies, assessment of scientific literacy requires students to apply science knowledge or concepts and may need to take advantage of multiple formats of assessment such as multiple-choice and constructed-response items. Therefore, in this study we developed assessment of scientific literacy in the case of science concepts relating to buoyancy. By examining the guided inquiry activities the simulation environment provides, we hypothesize that the intervention can enhance the students’ scientific literacy in the case of buoyancy in five aspects: (1) identifying the question explored in a given scientific study, (2) offering explanatory hypotheses, (3) interpreting data and drawing appropriate conclusions, (4) evaluating ways of exploring a given question scientifically, and (5)

9

proposing a way of exploring a given question scientifically (OECD, 2016). Moreover, we further examined the students’ inquiry processes and related them to their scientific literacy performance. Such links between inquiry processes and outcomes reveal patterns of students’ inquiry that lead to productive or less productive learning outcomes, and provide insight into designing adaptive learning activities for students with different characteristics. However, researchers have noticed that it is challenging to use specific innovations such as inquiry-based instruction to raise scores on external achievement tests such as some high-stakes achievement tests or school achievement science tests, probably because of the way the achievement tests are constructed (Hickey & Zuiker, 2012). Many of the science achievement tests at schools in Taiwan consist of only multiple-choice items, cover a wide range of science concepts, and do not focus on the critical thinking or reasoning that scientific literacy emphasizes. Therefore, researchers suggest using such achievement tests with caution since they may be faulty indicators of how much individuals have learned from an intervention (Hickey & Zuiker, 2012). In this regard, we did not expect that the inquiry activity in the treatment of this study would better increase students’ school science achievement scores after the intervention than did the traditional instruction. In fact, many science teachers in Taiwan think that implementing inquiry learning activities may result in a decrease in students’ achievement scores because traditional science instruction is usually specifically tailored for helping students perform well on achievement tests whereas inquiry instruction is not. However, we think that this perspective may be a misconception because research has shown that inquiry can foster students’ conceptual understanding of science concepts (e.g., Author, 2017; Chiu et

10

al., 2015; Thacker & Sinatra, 2019). Therefore, in this study we compared the impact of the treatment which used an inquiry approach versus the control which used the traditional approach on students’ school science achievement scores after the unit. Our hypothesis was that the inquiry treatment would not harm the students’ performance on the achievement tests. We also collected the students’ school science achievement scores before the intervention to help characterize the students’ school science achievements and to relate them to their inquiry processes. Doing so would provide insight into understanding how students with different prior school science achievements may perform and learn during guided inquiry with simulation to spur discussion and reflection on designing adaptive activities for different students.

2. Methods We employed a quasi-experimental research design to investigate the impact of the guided inquiry with simulation on students’ scientific literacy and school science achievement. We also employed content analysis of students’ written responses and logging data to examine the students’ inquiry processes. Both the impact of inquiry with simulation on scientific literacy and school science achievement and the learning processes of inquiry with simulation in relation to the learning outcomes have been little addressed in previous studies. 2.1 Participants The participants in this study were 49 eighth-grade students (26 female, 23 male) in two classes taught by the same science teacher at a public junior high school in northern Taiwan. One of the classes was randomly assigned as the treatment group (N=24) who used the guided

11

inquiry environment to interact with a simulation relating to the phenomenon of sinking and floating to learn science concepts relating to buoyancy. The treatment group conducted inquiry with the simulation in a computer laboratory, and each of the students worked on one desktop computer while they were encouraged to discuss with their peers and the teacher. In contrast, the other class was assigned as the control group (N=25) which received conventional instruction on buoyancy in a regular classroom. The teacher followed the textbook and delivered lectures on concepts relating to buoyancy, and had the students practice test problems. Both the treatment and control groups spent three class periods (45 minutes per period) learning the concepts relating to buoyancy. The design of the research comparison was driven by a practical issue about the impact of inquiry versus traditional instruction. The inclusion of the control group is only to provide the baseline information regarding how the learning outcomes of the treatment group compared to the learning outcomes of students receiving conventional instruction. No students in the treatment or control group had learned scientific concepts relating to buoyancy before this study. None of them had prior experience with the guided inquiry learning environment used in this study. Moreover, their school science achievements as measured by the school science mid-term examination prior to this study indicate no significant difference between the two groups in terms of their school science achievements prior to this study (t=0.007, p=.995). Neither did the two groups differ in terms of their prior scientific literacy as measured by the pretests (t=1.44, p=.156). The science teacher has 7 years’ science teaching experience. She received her bachelor’s degree in science and her master’s degree in educational technology. She has rich

12

experience in using technology to support her teaching, and has experience in guiding students' learning with web-based inquiry science environments. She is also very confident and comfortable employing conventional instructional approaches such as textbook-based teaching since such approaches are still dominant in science classrooms and are generally requested by parents in Taiwan. We designed the treatment group inquiry activities as follows. 2.2 Treatment Using the Guided Inquiry with Simulation Learning Environment In the first class period, the science teacher explained to the treatment group that the inquiry task was to investigate why objects sink or float on a liquid by interacting with and running the simulation (Figure 1). Students could change values of three variables: the density of the liquid, the object’s mass and the object’s volume, and could run the simulation to observe the status of the object in terms of sinking into or floating on a liquid. The teacher also demonstrated how to use the functions provided in the learning environment step by step. -----------------------------------------------------------------------Insert Figure 1 about here -----------------------------------------------------------------------In the next two class periods the students in the treatment group conducted their inquiry with the simulation by following the guidance provided in the environment. An interface has been developed on the CoSci platform (http://cosci.tw/) for students to create their own inquiry map that supports learners’ inquiry process (Authors, 2018, Figure 2). The interface provides six inquiry phases for students to create their own inquiry map that guides the inquiry process, including understanding the task, generating hypotheses, designing

13

experiments, collecting data, analyzing data and making conclusions. The details of each phase are described as follows. − Understand the task: The learning task was described during this phase. The students clicked the icon to enter the phase which provided them with an explanation of the context of the simulation, and the goal of the learning task. − Generate hypotheses: The students clicked the “Generate hypotheses” icon to enter this phase in which they were asked to generate hypotheses by selecting one of the provided hypotheses (such as: the heavier the object, the greater its buoyancy), or generating their own hypotheses. − Design experiments: The students were prompted to design experiments by setting the values of the three variables in this phase which would be used for conducting the experiment in the next phase. They were allowed to set the mass of the object, the volume of the object and the density of the liquid to explore the relationship between those variables and the magnitude of the buoyancy force, and if those relate to whether the object sinks or floats. They were also asked to explain their design ideas. − Collect data: The interactive simulation about buoyancy was provided in this phase. The students could conduct the experiment based on the previous phase. The data were automatically recorded by the system and were shown in a dynamic data table. The students could select the data that they needed from the data table in this phase. − Analyze data: In this phase, graphing and other tools were provided for the students to organize and analyze the data collected by running the interactive simulation. For

14

example, the data browsing tool allows students to see all the details of the data collected from previous phases. The graphing tool supports them to generate the diagrams to represent relationships between variables. The table generating tool enables them to select values of variables from the experiments to form a table. − Make conclusions: The students were asked to make conclusions based on the results of the data analysis phase to verify if the hypothesis is correct or needs revision. The students were prompted to use the data as scientific evidence to support their conclusions. -----------------------------------------------------------------------Insert Figure 2 about here -----------------------------------------------------------------------To facilitate the students’ awareness of the inquiry process, they were asked to go through each phase strictly. In other words, only when one phase was complete were they allowed to create and access the next phase. The students could proceed by creating a node from the previous phase. They could also generate multiple nodes to form more than one inquiry cycle. For example, they could reflect on the result and create a new hypothesis phase node to form another inquiry cycle based on it. All the behavior during the inquiry process was logged for us to investigate the quality of the students’ inquiry with the simulations. 2.3 Data Collection and Analysis 2.3.1 Pre-, post-, and delayed-tests to assess scientific literacy in the context of sinking and floating

15

All of the participants in the two groups took the scientific literacy test (45 minutes) before (the pretests) and after (the posttests) the learning activity using the inquiry map. Five months later the two groups took the scientific literacy delayed test. The science teacher confirmed that during the five months no other inquiry activities or review sessions on the concepts of buoyancy were implemented. The pretests consisted of eight constructed-response items that asked students to make a claim and provide a reason for the claim. The eight items were developed to measure scientific literacy in the context of applying scientific concepts relating to buoyancy, in the four aspects that the inquiry learning environment particularly aims to facilitate, including (1) identifying the question explored in a given scientific study, (2) offering explanatory hypotheses, (3) interpreting data and drawing appropriate conclusions, and (4) evaluating ways of exploring a given question scientifically. The posttests consisted of three additional constructed-response items (a total of 11 items, among which eight were identical to the pretests) that measured another important aspect of scientific literacy, which is (5) proposing a way of exploring a given question scientifically. This aspect was only tested in the posttests and delayed posttests since it involves an advanced level of scientific literacy that should be developed after students have experiences of conducting scientific inquiry. The delayed posttests were very similar to the posttests. The only difference was that in the delayed posttests the three additional items were revised from a pure constructed-response format to a two-tier multiple-choice format. In the two-tier multiple-choice format, four choices of a claim were provided for students to select from, followed by four choices of the reason for the claim also

16

provided for students’ selection. We generated the choices by examining students’ responses in the posttests. The tests went through several rounds of revision by two science educators and one assessment expert to establish the construct and content validity. Detailed scoring rubrics were developed to code the students’ responses to the pre-, post- and delayed-tests. In general, for the claim part, one point was given for an appropriate claim and zero for an inappropriate one. For the reason part, two points were given for a completely adequate response, one point for a partially adequate response, and zero for an incomplete or irrelevant one. Two independent raters coded 10 tests and the inter-rater reliability (Lombard, Snyder-Duch, & Bracken, 2002) reached 95%. Cohen’s Kappa is 0.91. Inconsistent codes were discussed and resolved. Tests of normality indicated that the pre-, post-, and delayed-tests were all normally distributed (pretests: W = 0.98, p = .66; posttests: W = 0.98, p = .69; delayed tests: W = 0.96, p = .07). The paired sample t test was employed for the identical eight items of both the pre- and post-tests to indicate whether there were significant gains in the scientific literacy from the pretests to the posttests. Independent sample t tests were employed for the comparison between the experimental and control groups. 2.3.2 Students’ inquiry process logged via the learning environment The students’ behavior to create inquiry maps and responses to the embedded prompting questions in the inquiry phases were logged by the system and collected and analyzed in this study to indicate students’ inquiry processes. Such data were only collected for the experimental group since the control group did not conduct inquiry. Detailed scoring rubrics

17

were developed to code the students’ responses to the inquiry prompting questions embedded during the students’ inquiry with the simulation. Table 1 presents the overall scoring rubric for coding the quality of the students’ inquiry process. The phase “Understand the task” was not coded since it only provided contextual information for students and did not require them to write down their responses. The reliability of coding the students’ quality of inquiry process is Cohen’s Kappa 0.88. In addition, the numbers of nodes generated by students in each phase were summed and calculated to indicate their inquiry behavior. -----------------------------------------------------------------------Insert Table 1 about here -----------------------------------------------------------------------2.3.3 Students’ pre- and post-scores for school science achievement The students’ school science achievement scores as measured by the school science mid-term examinations prior to and after this study were collected for both the experimental and control groups. The school science mid-term examinations consisted of only multiple-choice items that mainly measured factual knowledge covering a wide range of science topics based on the science textbooks. The school science mid-term examination after the study consisted of items measuring factual knowledge relating to concepts of buoyancy. To investigate the relationships among students’ prior school science achievement, their inquiry process during the learning activity, and their scientific literacy developed after the learning, only the data from the experimental group were analyzed since the control group did not conduct inquiry. Pearson correlation was employed. In addition, the k-means cluster analysis was performed for examining how different types of students (in terms of their prior

18

school science achievement and performance of inquiry process) would gain different degrees of scientific literacy after their learning with the simulation. 3. Results 3.1 Students’ Inquiry Process with the Simulation To examine how the students engaged in the inquiry with the support of the environment, the numbers of nodes generated by students in each phase were calculated. The results are listed in Table 2, which shows that most of them completed more than one inquiry cycle. In other words, they were able to experience more than one inquiry by going through every phase in the learning environment. -----------------------------------------------------------------------Insert Table 2 about here -----------------------------------------------------------------------In order to investigate the students’ inquiry process in each node, we further analyzed their behavior and content logged inside the nodes. The results are summarized in Tables 3 and 4. As shown in Table 3, on average each student generated more than one hypothesis (M=1.71, SD=0.7), which suggests that the students tried to explore the simulation from multiple viewpoints. Note that the average number of generated hypotheses is less than the average number of the hypothesis nodes they created. This is because some of the students abandoned some nodes by keeping them blank, which caused the numbers of hypotheses generated to be less than the number of hypothesis nodes generated. In the “Design Experiment” node, students can plan multiple experiments. On average, each student planned eight experiments (M=8.00, SD=2.72) to test their hypotheses. In the “Collect Data” node, the

19

students can run the simulation as trials of their experiments. On average, each student conducted 19 trials (M=19.08, SD=9.44). These results indicate the students’ behavior during the inquiry process, showing that overall the students were highly engaged behaviorally in the learning environment. This also provides evidence for the effectiveness of the inquiry map approach supporting students in highly engaged inquiry. -----------------------------------------------------------------------Insert Table 3 about here -----------------------------------------------------------------------The quality of the students’ inquiry process is indicated by our analysis of the quality of the content in the nodes using the scoring rubric in Table 1. Note that in Table 1, for example, the maximum score for the first aspect, experiment design quality A, is one. Since a student may create multiple nodes, the actual maximum score each student obtained can be more than one. Table 4 lists the actual maximum scores of the students’ data. It shows that, for example, the maximum score for the first aspect, experiment design quality A, is two, which indicates that the best performance of the students was a student creating two Design Experiment nodes, and the content of the nodes was rated 1 for each, and summed as 2 as a result. The score of each student in the five aspects was summed respectively for each aspect, and the average of the summed scores gained by the students of the experiment group was calculated. The results are summarized in Table 4. Most of the students designed experiments based on their selected inquiry question (M=1.00, SD=0.72), were able to identify variables that were needed for analysis (M=1.46, SD=1.18) and made proper conclusions using the evidence (M=1.00, SD=1.18). However, the results show that the students encountered

20

difficulty in conducting consistent trials and selecting data from the trials to address the inquiry question (M=0.08, SD=0.28). The results may suggest that the complexity of the real-time data generated from the execution of the trials might hinder the students conducting inquiry.

-----------------------------------------------------------------------Insert Table 4 about here -----------------------------------------------------------------------3.2 Effects of the Guided Inquiry with the Simulation Learning Environment on Scientific Literacy and School Science Achievement The results of the paired sample t tests indicate that both the experimental and control groups made significant gains from the pretests (eight items) to the posttests (considering only the eight items identical to the pretests) [Experimental group: t=5.64, p<.001; Control group: t=4.01, p=.001]. The results indicate that both the conventional textbook-based instruction and the guided inquiry with simulation approach can facilitate students’ scientific literacy in the case related to the phenomenon of sinking and floating. The results of independent sample t tests further indicate that there is no significant difference between the treatment group and the control group in the posttest scores of the scientific literacy assessment (t=0.689, p=.494). However, the treatment group outperformed the control group on the delayed-posttests (t=2.522, p=.015) (Table 5). This suggests that the guided inquiry with simulation learning environment is helpful for students to develop more permanent scientific literacy.

21

In addition to scientific literacy, we compared the two groups’ performance on their school science achievement scores after the study. The result of the independent sample t test indicates that there is no significant difference between the treatment group and the control group in their school science achievements after the unit (t=-0.177, p=.860). -----------------------------------------------------------------------Insert Table 5 about here -----------------------------------------------------------------------3.3 Relationships Among Prior School Science Achievement, Process of Inquiry, and Scientific Literacy Table 6 shows the relationships among the students’ prior school science achievement, process of inquiry, and scientific literacy. Overall, judged by the total scores of the inquiry quality, the students’ inquiry process was significantly related to their (both pre- and post-instructional) scientific literacy and prior school science achievement. We further differentiated the results by examining in detail the students’ performance on the phases or aspects of their inquiry process. The results show positive relations between the number of nodes generated for making conclusions and both the pre- and the post- instructional scientific literacy. This suggests that students who have higher prior scientific literacy would try to make more different conclusions, and would have better scientific literacy performance after the activity. However, students’ prior school science achievement was not related to any of their node-generation process or inquiry behavioral engagement. Compared to school science achievement, students’ scientific literacy seems a better predictor of their inquiry behavior, especially in the aspect of making conclusions.

22

On the other hand, both prior school science achievement and scientific literacy were positively related to the quality of the students’ inquiry process. Specifically, how well the students designed experiments based on the inquiry question and experiment principle, and made adequate conclusions based on the data collected as evidence, were highly or moderately correlated with their scientific literacy and school science achievement. In comparison, how well the students were able to conduct aligned trials and to identify variables for analysis were not related to their scientific literacy or their school science achievement. -----------------------------------------------------------------------Insert Table 6 about here -----------------------------------------------------------------------3.4 Characterizing Students According to Prior School Science Achievement, Inquiry Process and Scientific Literacy We conducted cluster analysis to characterize the students. Since the number of participants is small, we used the cluster analysis results for only exploratory purposes. We considered the results with descriptive statistics and their limitations. The results are presented in Table 7 and Figures 3, 4, and 5. Cluster 3 (M) consisted of five students at the middle level of prior school science achievement. They highly engaged in the inquiry, especially for the aspects of planning experiments (10.99) and conducting trials (27.96 times) to collect data. Their high engagement seemed to reflect on their improvement (the posttest scientific literacy scores minus the pretest scientific literacy scores) of scientific literacy (a mean of 3.4 points), which is the greatest gain among the 4 clusters.

23

Compared to Cluster 3, Cluster 4 (H) consisted of eight students with higher level of prior school science achievement. These students made more nodes, especially nodes of making conclusions (2.12). Their other behavioral engagement of inquiry was also higher than the average, such as the number of hypotheses they generated (2.12) and the trials they conducted (21.16 times). Meanwhile, they maintained a high quality inquiry process, especially for the aspects of experiment design (1.75 points) and data analysis (2.75 points). Their scientific literacy also improved (2.5 points), and was only inferior to Cluster 3. There might be a ceiling effect since the Cluster 4 students had already scored higher than the other students in the pretests. On the other hand, Clusters 1 (L1) and 2 (L2) consisted of students with low level prior school science achievement but demonstrated different types of engagement in the inquiry process. More specifically, the Cluster 1 students generated more nodes and conducted more trials than the Cluster 2 students who on average focused on only one hypothesis. However, both groups of students needed to improve the quality of their inquiry performance. This indicates that the Cluster 1 students might have struggled during the inquiry process and used the strategy of trial and error repeatedly as they generated a high number of hypothesis nodes (2.20) but demonstrated a low quality inquiry process. Also, neither cluster of students performed well on the scientific literacy assessment. It is suggested that more adaptive guidance should be provided to these students who engaged differently in the guided inquiry environment. -----------------------------------------------------------------------Insert Table 7 about here

24

-----------------------------------------------------------------------Figure 3 shows the patterns of the four clusters in their node-generation process. It is encouraging to observe that the guided inquiry with simulation learning environment enabled not only the high but also the low school science achievement students to engage in inquiry by making the nodes. Cluster 2 (L2) could be characterized as motivated data analyzers since they created more data analysis nodes than the others. We also observed that Cluster 1 (L1) demonstrated similar patterns to Cluster 4 (H) except that the high achieving (H) students created more “Collect Data,” “Analyze Data” and “Make Conclusions” nodes. -----------------------------------------------------------------------Insert Figure 3 about here -----------------------------------------------------------------------Moving beyond the preliminary engagement in inquiry by making the nodes, the patterns of the four clusters in terms of their main behavioral engagement of inquiry are presented in Figure 4. It is apparent that Cluster 3 (M) students were active inquiry engagers since they planned more experiments and conducted more trials. This result is also encouraging as it shows that the inquiry learning environment can successfully engage students who might not score the highest in school science tests. We believe that such engagement is important since it provides different opportunities for these students to get to know what science is about, which may in turn help foster their motivation to learn and achieve in science. The Cluster 3 students gained the most in the scientific literacy assessments in this study. In comparison, Cluster 4 (H) can be characterized as moderate inquiry engagers, and Cluster 1 (L1) as average inquiry engagers. -----------------------------------------------------------------------Insert Figure 4 about here

25

-----------------------------------------------------------------------The patterns of the students’ quality of inquiry process are presented in Figure 5. Overall Cluster 4 (H) students maintained high quality of their inquiry process. In comparison, the motivated data analyzers (L2), the average inquiry engagers (L1), and the active inquiry engagers (M) need support to improve their quality of the four aspects of inquiry, namely, designing experiments aligned with the selected inquiry question, designing experiments based on the experiment principle, identifying variables needed to be analyzed in order to address the inquiry question, and making proper conclusions based on the evidence. The average inquiry engagers (L1) demonstrated better quality of conducting trials of experiments aligned with the selected inquiry question, but judged by the scores they received, all students need further support in this aspect. -----------------------------------------------------------------------Insert Figure 5 about here -----------------------------------------------------------------------4. Discussion and Conclusions The study involved a relatively small number of participants engaging in guided inquiry with an interactive simulation. The generalizability of the results may be limited due to this small number and the context of the study. Nevertheless, we were able to collect and analyze the data from the students’ inquiry process as well, aiming to provide a full picture of students’ inquiry. We discuss the lessons learned, implications and future studies as follows. 4.1 Effects of the Guided Inquiry with Simulation Learning Environment on Scientific Literacy and School Science Achievement

26

Both the experimental and control groups made significant gains from the scientific literacy pretests to the posttests. This is reasonable given that both instructional approaches helped the students learn science concepts relating to sinking and floating, and that proficiency in scientific literacy requires students to make critical use of science knowledge (OECD, 2016), and in many science inquiry performances science knowledge plays a significant and central role (Author, 2018). In terms of the between group comparison, the two groups did not differ in the scientific literacy posttests. However, the experimental group outperformed the control group in the scientific literacy delayed posttests. A learning curve predicted by Horton (2001) (Figure 6) indicates that students’ knowledge may decrease but proficiency of performance may increase some time after an intervention takes place. Our findings further suggest that different learning curves exist for different instructional approaches. The guided inquiry with simulation learning environment did a better job than the conventional science teaching at retaining the students’ scientific literacy, which requires both application of science knowledge and performance of science inquiry. Research has indicated that students’ inquiry learning with computer simulations can facilitate students’ understanding of science concepts (Chiu et al., 2015), development of inquiry skills (Efstathiou et al., 2018) or science epistemic beliefs (Huang et al., 2017). This current study is among the very few studies that have started to examine the effect of inquiry learning with computer simulations on students’ development of scientific literacy, an aspect that has been emphasized in recent science education reforms (NGSS Lead States, 2013; OECD,

27

2016; Taiwan Ministry of Education, 2014). The results of the study indicate that the guided inquiry with a simulation learning environment has an effect on the students’ scientific literacy. Moreover, this effect has a long-term influence on students’ scientific literacy. In addition to scientific literacy, we compared the experimental and control groups’ performance on their school science achievement scores after the study. This comparison addresses a practical concern that many science teachers may fear that implementing inquiry learning activities may result in a decrease in students’ achievement scores. However, we think that this concern may stem from a misconception, because research has shown that inquiry can foster students’ conceptual understanding of science concepts (e.g., Author, 2017; Chiu et al., 2015; Thacker & Sinatra, 2019). Nevertheless we found little research that was able to address this practical concern since these studies did not use school science achievement scores but rather developed their own instruments to measure students’ conceptual understanding of science. In the current study, we found that replacing lectures on concepts relating to buoyancy with the buoyancy simulation did not harm the students’ school science achievement performances. Despite the fact that high-stakes achievement tests or school achievement science tests may not be sensitive enough to indicate the benefits of some innovative learning approaches such as the inquiry-based learning approach (Hickey & Zuiker, 2012), in the current study we used and compared the different groups’ school science achievement scores to provide evidence that the inquiry-based learning approach has a similar effect on aspects of science learning outcomes that are valued in school science achievement tests. -----------------------------------------------------------------------Insert Figure 6 about here

28

-----------------------------------------------------------------------4.2 Productive Inquiry Process Connecting to Scientific Literacy Little research has linked students’ inquiry processes to their performances after inquiry. Only a few qualitative studies have reported on cases of students’ learning trajectories during inquiry with simulations (e.g., Thacker & Sinatra, 2019). In our study, we found that how well the students purposefully designed their experiments to address their inquiry question and adequately made conclusions based on the evidence obtained from the inquiry were significantly correlated to their performance on the scientific literacy assessments. This result suggests that more attention or scaffolding is needed to support students’ mindful engagement in these aspects of science inquiry, in order to develop scientific literacy that is important for the future citizens of the 21st century (NGSS Lead States, 2013; Taiwan Ministry of Education, 2014). A recent study has focused on scaffolding students’ learning using an experiment design tool to apply the ‘‘vary one thing at a time’’ [VOTAT] heuristic when designing their experiments (Efstathiou et al., 2018). The experiment design tool has found it to be effective for fostering students’ domain-general inquiry skills. Based on our results, we further suggest other areas of scaffolding, including students’ reflection on how well their designs address the inquiry question, and how well their conclusions are supported by the evidence from the inquiry. One study found that their designed meta-conceptual scaffolding (e.g., prompting students to reflect on and elaborate their thinking when they predicted and explained a simulation) enhanced the students’ conceptual understanding of the concepts represented in the simulation (Huang et al., 2017). For future studies, it seems

29

equally important to design and incorporate meta-inquiry scaffolding and to investigate its impact on students’ inquiry processes and outcomes. In comparison, the students’ behavioral engagement of inquiry, namely, number of nodes created, hypotheses tested, experiments planned, or trials conducted, was not significantly related to their scientific literacy or school science achievement performances. Author (2017) found that structured inquiry may increase students’ behavioral engagement of inquiry, but this behavioral engagement may not necessarily lead to better learning outcomes. In our guided inquiry environment, the students’ variations of behavioral engagement were allowed. In general, the students were highly behaviorally engaged, indicating that the learning environment was able to engage students in conducting inquiry. Students’ behavioral engagement in inquiry can be a basic indicator of the successfulness of an inquiry environment. The students in this study, regardless of their prior school science achievement levels, were highly engaged in the inquiry activities. This provides evidence for the effectiveness of the inquiry map support. Future studies are needed to investigate how to take advantage of students’ behavioral engagement to further scaffold them to conduct quality inquiry with interactive simulation. 4.3 Designing Guided Inquiry with Simulation Learning Environments for High and Low Prior School Science Achievement Students Conventional, high-stakes achievement tests continue to dominate educational systems, while researchers have started to call for alternative assessments that are sensitive to innovative science curricula (Hickey, Taasoobshirazi, & Cross, 2012; Hickey & Zuiker, 2012).

30

In this study we demonstrated how using logging data of students’ inquiry combined with their prior and post performance in science can provide evidence of the effectiveness of innovative science learning activities. Specifically in Taiwan, many school science achievement tests such as mid-term examinations follow the standardized tests, consisting of only multiple-choice items covering a wide range of science facts and drill-and-practice problems. Students’ motivation to learn science and their ideas about science may suffer when too much emphasis is put on the achievement tests and too little attention is paid to innovative and authentic ways of doing and learning science and to formatively assessing and revealing their impacts on student learning of science. In this study, we explored the impact of the guided inquiry with simulation learning environment by characterizing the students considering their prior school science achievements, inquiry processes, and gained scientific literacy. The results suggest that the so-called low science achieving students can be motivated data analyzers or average inquiry engagers in an inquiry learning environment. Moreover, the students with middle level school science achievement demonstrated the most active engagement in inquiry and showed good gains in scientific literacy after the learning. These results provide evidence of how a guided inquiry with a simulation learning environment can promote students with different school science achievement levels to learn science with different trajectories. Specifically, high-achieving students were more able to engage in scientific inquiry focusing on designing experiments based on the inquiry question and experiment principle and making conclusions based on the collected data as evidence. In comparison, low-achieving students tended to practice their

31

inquiry through generating more hypothesis nodes, conducting more trials, and creating more data analysis nodes. These are good inquiry practices to start with, and future designs of interventions need to address how to scaffold students’ inquiry learning from personal inquiry (e.g., trial-and-error or free exploration) to scientific inquiry (e.g., conducting valid experiments to make adequate conclusions to address the inquiry question). Future studies are needed to address the common difficulties in inquiry with simulations, including designing experiments and conducting trials aligned with the selected inquiry question, designing experiments based on the experiment principle, identifying variables which need to be analyzed, making judgments about what data are needed in order to address the inquiry question, and making proper conclusions based on the evidence.

Conflict of Interest All authors of this manuscript declare that we have no conflict of interest. References Adams, W. K., Paulson, A., & Wieman, C. E. (2009). What levels of guidance promote engaged exploration with interactive simulations? In H. Charles, S. Mel, & H. Leon (Eds.), Proceedings of AIP Conference: Vol. 1064. 2008 Physics Education Research Conference (pp. 59-62). Edmonton, Alberta, Canada: American Institute of Physics. doi:10.1063/1.3021273 Authors (2015). Author (2017).

32

Author (2018). Authors (2018). Bell, T., Urhahne, D., Schanze, S., & Ploetzner, R. (2010). Collaborative inquiry learning: Models, tools, and challenges. International Journal of Science Education, 32, 349-377. doi:10.1080/09500690802582241 Blanchard, M. R., Southerland, S. A., Osborne, J. W., Sampson, V. D., Annetta, L. A., & Granger, E. M. (2010). Is inquiry possible in light of accountability? A quantitative comparison of the relative effectiveness of guided inquiry and verification laboratory instruction. Science Education, 94, 577-616. doi:10.1002/sce.20390 Chiu, J. L., DeJaegher, C. J., Chao, J. (2015). The effects of augmented virtual science laboratories on middle school students' understanding of gas properties. Computers & Education, 85, 59-73. doi: 10.1016/j.compedu.2015.02.007 Donnelly, D. F., Linn, M. C., & Ludvigsen, S. (2014). Impacts and characteristics of computer-based science inquiry learning environments for precollege students. Review of Educational Research, 84, 572-608. doi:10.3102/0034654314546954 Efstathiou, C., Hovardas, T., Xenofontos, N. A., Zacharia, Z. C., deJong, T., Anjewierden, A., & van Riesen, S. A. N. (2018). Providing guidance in virtual lab experimentation: The case of an experiment design tool. Education Technology Research and Development, 66, 767-791. doi: 10.1007/s11423-018-9576-z

33

Gal, Y. a., Uzan, O., Belford, R., Karabinos, M., & Yaron, D. (2015). Making sense of students’ actions in an open-ended virtual laboratory environment. Journal of Chemical Education, 92, 610-616. doi:10.1021/ed500531a Hennessy, S., Wishart, J., Whitelock, D., Deaney, R., La Velle, L., McFarlane, A., . . . Winterbottom, M. (2007). Pedagogical approaches for technology-integrated science teaching. Computers & Education, 48, 137-152. doi:10.1016/j.compedu.2006.02.004 Hickey, D. T., Taasoobshirazi, G., & Cross, D. (2012). Assessment as learning: Enhancing discourse, understanding, and achievement in innovative science curricula. Journal of Research in Science Teaching, 49, 1240-1270. doi:10.1002/tea.21056 Hickey, D. T., & Zuiker, S. J. (2012). Multilevel assessment for discourse, understanding, and achievement. Journal of the Learning Sciences, 21, 522-582. doi:10.1080/10508406.2011.652320 Hmelo-Silver, C., Duncan, R., & Chinn, C. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99-107. doi:10.1080/00461520701263368 Horton,W. (2001). Evaluating e-learning. Alexandria, VA : American Society for Training & Development. Huang, K., Ge, X., & Eseryel, D. (2017). Metaconceptually-enhanced simulation-based inquiry: Effects on eighth grade students’ conceptual change and science epistemic beliefs. Educational Technology Research and Development, 65, 75-100. doi:10.1007/s11423-016-9462-5

34

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587-604. doi:10.1111/j.1468-2958.2002.tb00826.x McElhaney, K. W., & Linn, M. C. (2011). Investigations of a complex, realistic task: Intentional, unsystematic, and exhaustive experimenters. Journal of Research in Science Teaching, 48(7), 745-770. Ministry of Education (MOE). (2014). Curriculum outlines of the 12-year basic education curriculum-the master outline. National Research Council Press, Taipei. Moon, J. A., & Brockway, D. (2019). Facilitating learning in an interactive science simulation: The effects of task segmentation guidance on adults’ inquiry-based learning and cognitive load. Journal of Research on Technology in Education, 51, 77-100. doi:10.1080/15391523.2019.1566038 NGSS Lead States. (2013). Next generation science standards: For states, by States. Washington, DC:The National Academies Press. OECD (2016), PISA 2015 Assessment and analytical framework: Science, reading, mathematic and financial literacy. Paris: OECD Publishing, Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., . . . Soloway, E. (2004). A scaffolding design framework for software to support science inquiry. The Journal of the Learning Sciences, 13, 337-386. doi:10.1207/s15327809jls1303_4

35

Rutten, N., van Joolingen, W. R., & van der Veen, J. T. (2012). The learning effects of computer simulations in science education. Computers & Education, 58, 136-153. doi:https://doi.org/10.1016/j.compedu.2011.07.017 Scalise, K., Timms, M., Moorjani, A., Clark, L., Holtermann, K., & Irvin, P. S. (2011). Student learning in science simulations: Design features that promote learning gains. Journal of Research in Science Teaching, 48, 1050-1078. doi:10.1002/tea.20437 Thacker, I., & Sinatra, G. M. (2019). Visualizing the greenhouse effect: Restructuring mental models of climate change through a guidedonline simulation. Education Sciences, 9, 14. doi:10.3390/educsci9010014 White, R. J., & Gunstone, R. F. (1989). Metalearning and conceptual change. International Journal of Science Education, 11, 577-586. doi:10.1080/0950069890110509

36

Table 1 The scoring rubric for the quality of the students’ inquiry process Inquiry Criteria Phase Score 0 Design experiments A Irrelevant -Student designed description, blank, experiments aligned with or not aligned with the selected inquiry the inquiry question question. Design experiments Designs not based Design experiments B on the experiment -Student designed principle, namely, experiments based on the varying one variable experiment principle at one time Irrelevant Trials conducting description, blank, Student conducted trials or trials not aligned of experiments aligned with the inquiry with the selected inquiry question question. Collect and analyze data Data analysis - Student Irrelevant identified variables description or blank needed to be analyzed in order to address the inquiry question. Irrelevant description or blank. Conclusions - Student Make made proper conclusions conclusions based on the evidence.

Scoring Rubric Score 1 Description of the designs aligning with the inquiry question

Score 2

Designs based on the experiment principle

Correct description of what data were selected from the trials that were used to address the inquiry question Identification of only some of the variables related to the inquiry question Conclusion made consistent with the scientific principle of buoyancy

Identification of all variables related to the inquiry question

Conclusion made consistent with the scientific principle of buoyancy and data used as evidence

37

Table 2 The average numbers of nodes generated by the students in each phase. M

S.D.

Max

Min

Generate Hypotheses Node

1.83

0.70

3

1

Design Experiment Node

2.33

1.09

6

1

Collect Data Node

2.04

0.75

4

1

Analyze Data Node

2.25

1.03

5

1

Make Conclusions Node

1.46

0.66

3

0

Total

9.92

2.54

3

1

Table 3 Indicators of the students’ inquiry process inside the nodes. M

S.D.

Max

Min

Number of hypotheses generated

1.71

0.69

3

1

Number of experiments planned

8.00

2.72

14

3

Number of trials conducted

19.08

9.44

44

5

Table 4 The quality of the students’ inquiry process. Aspects of the inquiry process

Average

S.D.

Max

Min

Experiment design quality A

1.00

0.72

2

0

Experiment design quality B

0.96

0.75

2

0

Trials conducting quality

0.08

0.28

1

0

Data analysis quality

1.46

1.18

4

0

Conclusions quality

1.00

1.18

4

0

Total

4.50

3.20

12

0

Table 5 Means and standard deviations of the posttest, delayed-test, and post achievement scores Scientific Literacy Posttest

Scientific Literacy

Post School Science

Delayed-test

Achievement Score

M

S.D.

M

S.D.

M

S.D.

The treatment group

14.92

6.23

19.08

6.53

64.20

15.01

The control group

13.73

5.94

14.44

6.58

63.44

15.75

Independent sample t tests

t=0.689, p=.494

t=2.522, p=.015

t=0.177, p=.860

38

Table 6 Pearson bivariate correlation coefficients *<.05; **<.01 Inquiry Process

Phases or Aspects

Pretest

Posttest

Prior school science

Scientific

Scientific

achievement

literacy

literacy

Number of

Generate Hypotheses Node

.023

.044

.098

nodes made

Design Experiment Node

-.139

-.065

.150

Collect Data Node

-.175

-.052

.179

Analyze Data Node

-.113

-.124

-.090

Make Conclusions Node

.405*

.435*

.389

Behavioral

Number of hypotheses generated

.092

.102

.108

engagement

Number of experiments planned

.036

.235

.382

of inquiry

Number of trials conducted

.151

.259

.378

Quality of

Experiment design quality A

.585**

.563**

.661**

inquiry

Experiment design quality B

.536**

.491*

.543**

Trials conducting quality

.029

.031

-.207

Data analysis quality

.364

.342

.321

Conclusions quality

.762**

.731**

.624**

Total

.675**

.640**

.606**

39

Table 7. The cluster analysis results Cluster 1

Cluster 2

Cluster 3

Cluster 4

(L1)

(L2)

(M)

(H)

Mean

(n=5)

(n=6)

(n=5)

(n=8)

60.04

44.66

45.96

67.62

75.42

1.83

2.20

1.00

1.80

2.25

Design Experiment Node

2.33

2.59

1.50

2.20

2.88

Collect Data Node

2.04

1.80

1.50

2.80

2.12

Analyze Data Node

2.25

1.60

3.00

2.00

2.25

Make Conclusions Node

1.46

1.20

1.00

1.20

2.12

1.71

1.80

1.00

1.80

2.12

8.00

6.99

5.50

10.99

8.63

19.08

17.01

10.68

27.96

21.16

1.00

0.60

0.67

0.60

1.75

0.96

0.60

0.67

0.40

1.75

Trials conducting quality

0.08

0.40

0.00

0.00

0.00

Data analysis quality

1.46

0.80

1.16

0.40

2.75

Conclusions quality

1.00

0.80

0.50

0.20

2.00

1.92

0.60

1.00

3.40

2.50

Characteristic

Variable

Prior school

Prior mid-term

science

examination score

achievement Number of

Generate Hypotheses

nodes made

Node

Behavioral

Number of hypotheses

engagement of

generated

inquiry

Number of experiments planned Number of trials conducted

Quality of

Experiment design quality

inquiry process

A Experiment design quality B

Scientific literacy

Total Gain

40

Figure 1. The buoyancy simulation

41

Figure 2. A screenshot of an inquiry map consisting of nodes created by a student

Figure 3. The four clusters in relation to the numbers of nodes made

42

Figure 4. The four clusters in relation to their behavioral engagement of inquiry

Figure 5. The four clusters in relation to their inquiry process quality

43

Figure 6. Predicted learning curve by Horton (2001)

44

Highlights • The designed simulation and embedded inquiry support had a long-term effect on the students’ scientific literacy. • Compared to school science achievement, students’ scientific literacy seems a better predictor of their inquiry behavior. • The low science achieving students conducted more data analyses and demonstrated adequate inquiry engagement. • The middle-level achieving students demonstrated the most active engagement in inquiry and gains of scientific literacy.