- Email: [email protected]
Application adjunct post-questions and conceptual problem solving
CONTEMPORARY
EDUCATIONAL
PSYCHOLOGY
17, 89-97 (1992)
BRIEF RESEARCH Application
Adjunct
REPORT
Post-Questions Problem Solving
and Conceptual
RICHARD HAMILTON Universiry
of Housion
This study evaluated the influence of matched and unmatched application adjunct questions (preceded by definition adjunct questions) on the acquisition of problems solving skills from a prose passage. The study included 78 undergraduates from a large southwestern university. Subjects studied a passage containing matched or unmatched application questions preceded by definition adjunct questions; or definition adjunct questions only. In the unmatched application questions, critical and noncritical attributes of the target concepts were varied across the four multiple-choice alternatives, while in matched application questions only the critical attributes were varied across the alternatives. Unmatched application adjunct post-questions produced significantly superior problem solving, classification of novel examples, and recall of concept definitions in comparison to definition adjunct questions only. In contrast, matched application adjunct post-questions significantly influenced only the classification of novel examples in comparison to definition adjunct questions only. B 1992 Academic Press, Inc.
The ability to successfully use concepts to solve novel problems involves both the categorization of a problem as relevant to target concepts and the creation of a solution which involves these target concepts. In order to successfully categorize problems and create solutions, learners need to possess appropriate conceptual, procedural, and contextual knowledge. Conceptual knowledge consists of descriptions of the critical defining dimensions of the concepts, typical examples of concepts and connections with related knowledge and experience. In contrast, procedural knowledge consists of the rules that govern the application of one’s conceptual knowledge to novel situations. Finally, contextual knowledge consists of cues which indicate when it is appropriate to apply specific knowledge. Conceptual knowledge is developed through the storage and integration of conceptual information; procedural knowledge is developed through retrieval of conceptual knowledge and application of this knowledge to novel situations; and contextual knowledge is developed through the application of conceptual and procedural knowledge to a variety of relevant novel situations (cf. Tennyson & Cocchiarella, 1986). Within a prose learning setting, the acquisition of appropriate conceptual, proceData for this study were collected under the Faculty Opportunity Grant from the College of Education, University of Houston. Reprint requests should be addressed to Dr. Richard Hamilton, Department of Educational Psychology, University of Houston, 4800 Calhoun Road, Houston, TX 77204-5874. 89 0361-476X/92 $3.00 CopyrightQ 1992 by Academic All rights of reproduction
Press, inc. in any form reserved.
90
RICHARD
HAMILTON
dural, and contextual knowledge would require that learners actively process target concept definitions and teaching examples and apply these concepts to a variety of novel situations. One area of research that seems applicable to these concerns involves inserted questions, specifically application adjunct post-questions. Although the effects of application adjunct questions have been somewhat inconsistent on higher level outcomes, some experiments have found positive effects of application questions on problem solving and classification outcomes. For example, Felker and Dapra (1975), in an experiment which focused specifically on the influence of application adjunct post-questions on readers’ ability to apply psychological concepts to problem situations, found that these questions significantly improved subjects’ problem solving ability. Also, Hamilton (1989) evaluated the effects of presenting definition adjunct questions prior to application adjunct questions on classification outcomes within two separate studies. The definition adjunct questions were intended to reactivate appropriate conceptual knowledge to be used during application activities. In addition, two different types of application adjunct questions (i.e., unmatched and matched) were used to evaluate the effects of presenting a wide variety of alternatives versus a narrow set of alternatives. In the unmatched application questions, critical and noncritical attributes of the target concepts were varied across the four multiple-choice alternatives, while in matched application questions only the critical attributes were varied across the alternatives. Within both studies, only unmatched adjunct questions preceded by definition adjunct questions produced significantly better classification performance than the definition adjunct question only control group. Finally, when application adjunct questions were presented without definition adjunct questions and compared to a no question control group, no significant differences were found. Both matched and unmatched application adjunct questions when preceded by definition adjunct questions should induce the development of appropriate conceptual knowledge, i.e., storage and integration of critical defining dimensions of the target concepts. However, the positive effects of unmatched questions on classification outcomes can be attributed to the development of appropriate procedural and contextual knowledge. That is, the unmatched questions induced readers to practice classifying novel examples in varied contexts which developed appropriate rules for applying target concepts and induced readers to associate a variety of cues with the target knowledge. Given the importance of these two types of knowledge for the acquisition of appropriate problem solving skills, one would predict that unmatched application questions preceded by definition questions should also significantly improve readers’ conceptual problem solving skills.
ADJUNCT
POST-QUESTIONS
91
The present study is aimed at evaluating the influence of matched and unmatched application adjunct post-questions on the acquisition of problem solving skills from a prose passage. As indicated earlier, definition adjunct questions are seen to be helpful in the establishment of appropriate conceptual knowledge; consequently, both types of application adjunct questions were preceded by a definition adjunct question. In order to evaluate the specific effects of manipulating type of application adjunct question on criterion outcomes, the control group consisted of the presentation of definition adjunct questions only. In addition to subjects’ ability to solve novel problems with target concepts, other learning outcomes were also measured: free recall of concept definitions, free recall of teaching examples, and classification of novel examples. It was expected that only the unmatched application questions (with definition adjunct questions) would produce better classification and problem solving performance than the definition questions only control. METHODS
Subjects and Design The subjects were 78 volunteers from four sections of an undergraduate introductory psychology course. There were 26 subjects in each of three treatment conditions: 1) matched application adjunct post-questions, 2) unmatched application adjunct post-questions, and 3) definition adjunct post-questions only (control). All three groups received definition adjunct questions. Subjects participated in the experiment in groups of 10 to 20 and, within each group session, subjects were randomly assigned to one of the three conditions.
Experimental Text and Treatments The experimental text focused on positive and negative reinforcement and positive and negative punishment. The first three paragraphs of the experimental text included an introductory paragraph followed by two paragraphs on the critical attributes of target concepts, i.e., aversive/attractive events, added/subtracted events. The following four paragraphs each presented one of the four target concepts. Each paragraph presented the definition of the concept, four examples of the concept, and discussion of how each example was illustrative of the concept. After each paragraph, all subjects were given a multiple-choice definition adjunct post-question. The two treatment groups were given either two matched or two unmatched multiple-choice application adjunct post-questions immediately after the definition adjunct question (see Hamilton, 1989, for sample questions). The definition adjunct post-question consisted of four alternatives, each of which consisted of a description and one of the four critical attributes or a combination of two of the critical attributes. The application adjunct post-questions consisted of four alternatives, each of which was an example of one of the four coordinate target concepts. The alternatives were either matched or unmatched in terms of the noncritical attributes of the target concepts. After each application adjunct question, subjects were given feedback which consisted of re-presenting the same question with the correct alternative identified. The definition questions were not followed with feedback for two reasons: a) the function of the questions was to prompt the appropriate definition as an aid for answering the application questions, and b) previous experiments found perfect levels of performance on these questions (see Hamilton, 1989).
92
RICHARD
HAMILTON
Procedures Subjects participated in the following four tasks in order during the 2-h experimental sessions: (1) a reading efftciency test, (2) a prior knowledge test, (3) an experimental text and criterion test, and (4) a mental skills test. All tests and text were presented in paper and pencil form. All subjects were told not to turn back once they had turned the page in the experimental booklet. All subjects also were asked to record the time (hour, minutes, seconds) from looking at the clock at the front of the room when they reached particular portions of the experimental text and criterion test (i.e., at the beginning of the experimental text, after finishing the text, and after they finished each part of the criterion test). These recorded times were used to estimate the time subjects spent reading the text and completing each of the criterion tasks.
Dependent Measures The criterion test consisted of three parts: (1) a free recall test, (2) a classification test, and (3) a problem-solving test. The free recall test asked subjects to recall the definition of each of the four coordinate concepts as well as the examples used in the experimental text. The definitions were scored as correct if the subjects presented the gist of the definition accurately. The examples were scored similarly. Each definition and example was reduced to its essential proposition or propositions, and each definition or example was judged on the basis of whether it captured the gist of one of these propositions. The objectivity of this scoring procedure was evaluated by comparing two independent scorings of the definitions and examples (r = .95). The classification test asked subjects to classify statements describing a behavior and a contingent event (same format as the alternatives used in application questions), each of which was an example of one of the four coordinate target concepts. There were six examples of each target concept. A Cronbach Alpha test found the internal consistency of the classification test to be .93. The problem-solving test consisted of two scenarios describing a classroom situation. Subjects were to create sets of contingencies representing each of the four target concepts to alleviate disruptive behaviors described in the classroom scenarios. When using each concept subjects were asked to identify the disruptive behavior and the event which was contingent upon the behavior. Subjects received one point for correctly setting up a set of contingencies which was an example of each of the four target concepts described in the text. In order to be marked correct, each answer had to clearly describe the behavior to be modified and a consequence that was an example of the target concept. The objectivity of this scoring procedure was evaluated by comparing two independent scorings of the adequacy of the description of target behaviors and consequences (r = 90).
Ability Measures Subjects were given three ability tests: prior knowledge, reading efftciency, and mental skills. The prior knowledge test consisted of six multiple-choice questions to test knowledge of target concepts. If subjects received a score of four or better on the prior knowledge test, they were eliminated from the experimental analysis. The reading efftciency subtest from the Iowa Silent Reading Test was used to evaluate the speed and accuracy of subjects’ reading ability. The mental skills tests consisted of 20 multiple-choice questions which include analogies, reasoning, and mathematical word problems.
RESULTS
Given the very low overall performance on recall of teaching examples (no higher than 12% recall in any condition), the following analyses did
93
ADJUNCT POST-QUESTIONS
not include this criterion variable. Alpha was set at .05 for all analyses. Performance on the problem solving test, free recall of definitions, and classification test were significantly positively correlated (.62 < r” < .69). An ANOVA indicated that reading efficiency scores significantly differed between treatment groups [F(2,75) = 3.58, MS, = 151.011.Consequently, reading efficiency scores were integrated into the following analyses as a covariate. Given the significant correlation between performance on the three criterion measures, overall differences were initially analyzed within a multivariate analysis of covariance. The MANCOVA found a significant main effect [F(6,144) = 4.01, Wilks’ Lambda = 0.731 (see Table 1). Post hoc multivariate contrasts (for each contrast, alpha = .025) found significant effects for a) matched questions versus definition questions only [F(3,72) = 2.91, Wilks’ Lambda = 0.891, and b) unmatched questions versus definition questions only [F(3,72) = 6.58, Wilks’ Lambda = 0.791. Finally, in order to evaluate the original hypotheses, a series of Tukey post hoc contrast (for each contrast, alpha = .008) between each type of application adjunct question treatment group and the control group for each criterion measure found the following differences: a) DEFINITIONS: Unmatched application questions produced significantly better performance than no application questions [F( 1,74) = 13.65, MS, = 22.021, while there was no significant difference between matched application questions and no application questions [F(1,74) = 1.59, MS, = 2.581; b) CLASSIFICATION: Both matched and unmatched application questions produced better performance than no application questions [MATCHED: F(1,74) = 8.01, MS, = 143.06; UNMATCHED: F(1,74) = 12.72, MS, = 227.261; c) PROBLEM SOLVING: Unmatched application questions produced better performance than no application TABLE 1 SCORES FOR PERFORMANCE ON CRITERION TESTS
Question group Matched Criterion tests” Recall of definitions Classification of novel examples Problem solving scenarios
Unmatched
Control
Meanb
SD
Mean
SD
Mean
SD
2.34 (2.19) 17.23 (16.91) 4.54 (4.16)
1.44
2.96 (3.02) 17.58 (17.64) 5.27 (5.41)
1.21
1.65 (1.75) 13.39 (13.65) 2.96 (3.19)
1.26
4.31 2.30
4.34 2.09
4.08 2.09
a Best possible score for free recall of concept definitions = 4.0; classification of novel examples = 24.0; problem solving scenarios = 8.0. b Adjusted means are listed in parentheses.
94
RICHARD HAMILTON
questions [F(1,74) = 7.15, MS, = 68.351, while there was no significant difference between the matched application questions and no application questions [F(1,74) = 3.23, MS, = 13.071. An ANOVA found significant main effects for time involved reading the prose passage [F(1,74) = 66.353, MS, = 893.21. A series of Tukey post hoc contrast (for each contrast, alpha = .025) found significant differences between each of the application adjunct post-question groups and the control group [MATCHED vs CONTROL; F(1,74) = 85.265, MS, = 114.9; UNMATCHED vs CONTROL; F(1,74) = 110.148, MS, = 148.21 (see Table 2). A series of Pearson product-moment coefficient correlations were calculated between time involved reading the prose passage and the criterion outcomes within each treatment group. No significant relationships were found, with the coefficients ranging from -0.29 to +0.036. Finally, there was no significant difference between treatment groups for performance on the definition and application adjunct post-questions (see Table 2). DISCUSSION When the effects of treatments on criterion measures were evaluated within a multivariate framework, both unmatched and matched application adjunct post-questions were found to produce superior performance in comparison to a definition question only control. In contrast, when the effects of treatments on each criterion measure were evaluated separately, unmatched application adjunct post-questions significantly improved subjects’ problem solving, classification of novel examples, and recall of concept definitions, while matched application adjunct postquestions significantly improved only the classification of novel examples. The latter analysis allows for the identification of the differential TABLE 2 MINUTES SPENT READING EXPERIMENTAL TEXT AND SCORES FOR PERFORMANCE ON DEFINITION ADJUNCT QUESTIONS AND APPLICATION ADJUNCT QUESTIONS
Question group Matched
Time reading text Definition adjunct questions” Application adjunct question?
Unmatched
Control
Mean
SD
Mean
SD
Mean
SD
19.03
4.41
20.84
4.51
10.19
2.35
3.69
0.55
3.89
0.33
3.62
0.49
10.89
1.03
10.73
1.15
a The best possible score for definition adjunct questions = 4.0; and application adjunct questions = 12.0.
ADJUNCT
POST-QUESTIONS
95
effects of the two treatments on the criterion measures and suggests the two types of application adjunct post-questions differentially influenced subjects’ conceptual, procedural, and contextual knowledge. Tennyson and Cocchiarella (1986) indicates that the type of practice used during the acquisition of target knowledge will dramatically influence the accessibility of that knowledge. That is, activities that force learners to use knowledge in a variety of contexts should produce greater access to target knowledge in future use situations. Unmatched adjunct questions vary the contexts of target concepts by presenting examples and nonexamples which vary in their noncritical attributes, i.e., contexts of application. This contrast with matched questions which use examples and nonexamples sets which possess similar noncritical attributes. The increased diversity of contexts presented within unmatched questions may partially explain its effects on problem solving outcomes. As indicated under Results, subjects in the definition question only control condition took significantly less time to read the prose passage than subjects in the matched and unmatched application question conditions. Although some would argue that the superior performance of application/definition question conditions was due to extra study time, one set of results mitigate against this interpretation. In none of the conditions was the correlation between time spent reading the text and performance on any of the criterion measures statistically or practically significant. There are three important ways in which the present evaluation of application questions differs from previous research on application questions: the presentation of two application questions per concept, the presentation of feedback, and the pairing of definition questions with the application questions. The increased practice due to two application questions may increase the probability of effects of application questions on criterion outcomes, however, previous research has found that the presentation of two application questions without definition questions did not significantly influence performance on criterion outcomes in comparison to the presentation of no questions at all (Hamilton, 1989). These latter results suggests that the results of the present experiment are not due to an increase in the number of application questions. In contrast to previous research on application questions, the present experiment presented feedback to the application questions. Recent research on the effects of feedback suggests that both the presence of feedback and the nature of feedback will influence the effectiveness of the targeted questions (Kulhavy & Stock, 1989). Within the present context, the presence of feedback may have influenced the results in at least two ways. First, the presence of feedback may have been responsible for the significant effects of matched questions on classification performance. As indicated earlier, matched and unmatched adjunct questions produced
96
RICHARD
HAMILTON
superior classification performance in comparison to the control group. These results are not consistent with those found in Hamilton (1989), i.e., no effects for matched application adjunct questions. However, there is one important difference between the procedures used by Hamilton (1989) and the present experiment, i.e., subjects were not given feedback to the application questions in the study by Hamilton (1989). Matched practice derives its benefits from allowing a comparison of the critical attributes of examples and nonexamples. Since the target examples were not clearly identified during practice in Hamilton (1989) but were in the present experiment, benefits for matched questions would only be realized in the present experiment. The results of the current experiment are consistent with the majority of previous concept learning experiments (cf. Tennyson & Cocchiarella, 1986). In these previous experiments, the examples of the target concepts were always clearly identified through feedback and/or expository directions. Second, it is possible that without feedback or with less elaborate feedback, the unmatched application questions would not have produced significant far transfer effects. Although there is some evidence that these types of application questions without feedback can influence transfer outcomes (i.e., classification) (Hamilton, 1989), there is clearly a need to evaluate the interactive effects of the nature of feedback, the presence of feedback, and the nature of application questions on learning concepts from prose. Finally, given the results of Hamilton (1989), definition questions were paired with the application questions within the present experiment. Performance on the definition questions was very high (~90%) in all three conditions and there were no differences between conditions. These results suggest comparable levels of knowledge of definitions derived from the text in all three conditions. However, given the importance of feedback and its potential effects, future research should evaluate the effects of feedback to definition questions within the present context, i.e., paired with application questions. In terms of instructional implications, it is important to reemphasize that the unmatched application questions were presented with feedback and were preceded with definition questions. Accordingly, both the definition questions and the feedback would need to accompany the presentation of unmatched questions in order for similar effects to occur within instructional situations. Two further points need to be mentioned concerning the generalizability of these results. There is a need to look at the effects of these inserted aids both within more typical instructional materials, e.g., chapters from textbooks, etc., and within a framework which allows for evaluation of long-term effects. In summary, unmatched application questions were again found to produce positive overall effects. The present results underscore the impor-
ADJUNCT
POST-QUESTIONS
97
tance of looking specifically at the nature of application adjunct postquestions when evaluating their effectiveness within a prose learning setting. Future research should also monitor the interactive effects of the following variables when accessing the influence of different types of application questions on problem solving outcomes: the number of questions per concept, the presence of and nature of feedback, retention intervals, and the nature of experimental text. REFERENCES FELKER, D., & DAPRA, R. (1975). Effects of question type and questions placement on problem solving ability from prose material. Journal of Educational Psychology, 67, 380-384.
HAMILTON, R. (1989). The role of concept definition, teaching examples and practice on concept learning from prose. Contemporary Educational Psychology, 14, 357-365. KULHAVY, R., & STOCK,W. (1989). Feedback in written instruction: The place of response certitude. Review of Educational Research, l(4), 279-308. TENNYSON,R. D., & COCCHIARELLA,M. (1986). An empirically-based instructional design theory for teaching concepts. Review of Educational Research, 56(l), 40-71.
EDUCATIONAL
PSYCHOLOGY
17, 89-97 (1992)
BRIEF RESEARCH Application
Adjunct
REPORT
Post-Questions Problem Solving
and Conceptual
RICHARD HAMILTON Universiry
of Housion
This study evaluated the influence of matched and unmatched application adjunct questions (preceded by definition adjunct questions) on the acquisition of problems solving skills from a prose passage. The study included 78 undergraduates from a large southwestern university. Subjects studied a passage containing matched or unmatched application questions preceded by definition adjunct questions; or definition adjunct questions only. In the unmatched application questions, critical and noncritical attributes of the target concepts were varied across the four multiple-choice alternatives, while in matched application questions only the critical attributes were varied across the alternatives. Unmatched application adjunct post-questions produced significantly superior problem solving, classification of novel examples, and recall of concept definitions in comparison to definition adjunct questions only. In contrast, matched application adjunct post-questions significantly influenced only the classification of novel examples in comparison to definition adjunct questions only. B 1992 Academic Press, Inc.
The ability to successfully use concepts to solve novel problems involves both the categorization of a problem as relevant to target concepts and the creation of a solution which involves these target concepts. In order to successfully categorize problems and create solutions, learners need to possess appropriate conceptual, procedural, and contextual knowledge. Conceptual knowledge consists of descriptions of the critical defining dimensions of the concepts, typical examples of concepts and connections with related knowledge and experience. In contrast, procedural knowledge consists of the rules that govern the application of one’s conceptual knowledge to novel situations. Finally, contextual knowledge consists of cues which indicate when it is appropriate to apply specific knowledge. Conceptual knowledge is developed through the storage and integration of conceptual information; procedural knowledge is developed through retrieval of conceptual knowledge and application of this knowledge to novel situations; and contextual knowledge is developed through the application of conceptual and procedural knowledge to a variety of relevant novel situations (cf. Tennyson & Cocchiarella, 1986). Within a prose learning setting, the acquisition of appropriate conceptual, proceData for this study were collected under the Faculty Opportunity Grant from the College of Education, University of Houston. Reprint requests should be addressed to Dr. Richard Hamilton, Department of Educational Psychology, University of Houston, 4800 Calhoun Road, Houston, TX 77204-5874. 89 0361-476X/92 $3.00 CopyrightQ 1992 by Academic All rights of reproduction
Press, inc. in any form reserved.
90
RICHARD
HAMILTON
dural, and contextual knowledge would require that learners actively process target concept definitions and teaching examples and apply these concepts to a variety of novel situations. One area of research that seems applicable to these concerns involves inserted questions, specifically application adjunct post-questions. Although the effects of application adjunct questions have been somewhat inconsistent on higher level outcomes, some experiments have found positive effects of application questions on problem solving and classification outcomes. For example, Felker and Dapra (1975), in an experiment which focused specifically on the influence of application adjunct post-questions on readers’ ability to apply psychological concepts to problem situations, found that these questions significantly improved subjects’ problem solving ability. Also, Hamilton (1989) evaluated the effects of presenting definition adjunct questions prior to application adjunct questions on classification outcomes within two separate studies. The definition adjunct questions were intended to reactivate appropriate conceptual knowledge to be used during application activities. In addition, two different types of application adjunct questions (i.e., unmatched and matched) were used to evaluate the effects of presenting a wide variety of alternatives versus a narrow set of alternatives. In the unmatched application questions, critical and noncritical attributes of the target concepts were varied across the four multiple-choice alternatives, while in matched application questions only the critical attributes were varied across the alternatives. Within both studies, only unmatched adjunct questions preceded by definition adjunct questions produced significantly better classification performance than the definition adjunct question only control group. Finally, when application adjunct questions were presented without definition adjunct questions and compared to a no question control group, no significant differences were found. Both matched and unmatched application adjunct questions when preceded by definition adjunct questions should induce the development of appropriate conceptual knowledge, i.e., storage and integration of critical defining dimensions of the target concepts. However, the positive effects of unmatched questions on classification outcomes can be attributed to the development of appropriate procedural and contextual knowledge. That is, the unmatched questions induced readers to practice classifying novel examples in varied contexts which developed appropriate rules for applying target concepts and induced readers to associate a variety of cues with the target knowledge. Given the importance of these two types of knowledge for the acquisition of appropriate problem solving skills, one would predict that unmatched application questions preceded by definition questions should also significantly improve readers’ conceptual problem solving skills.
ADJUNCT
POST-QUESTIONS
91
The present study is aimed at evaluating the influence of matched and unmatched application adjunct post-questions on the acquisition of problem solving skills from a prose passage. As indicated earlier, definition adjunct questions are seen to be helpful in the establishment of appropriate conceptual knowledge; consequently, both types of application adjunct questions were preceded by a definition adjunct question. In order to evaluate the specific effects of manipulating type of application adjunct question on criterion outcomes, the control group consisted of the presentation of definition adjunct questions only. In addition to subjects’ ability to solve novel problems with target concepts, other learning outcomes were also measured: free recall of concept definitions, free recall of teaching examples, and classification of novel examples. It was expected that only the unmatched application questions (with definition adjunct questions) would produce better classification and problem solving performance than the definition questions only control. METHODS
Subjects and Design The subjects were 78 volunteers from four sections of an undergraduate introductory psychology course. There were 26 subjects in each of three treatment conditions: 1) matched application adjunct post-questions, 2) unmatched application adjunct post-questions, and 3) definition adjunct post-questions only (control). All three groups received definition adjunct questions. Subjects participated in the experiment in groups of 10 to 20 and, within each group session, subjects were randomly assigned to one of the three conditions.
Experimental Text and Treatments The experimental text focused on positive and negative reinforcement and positive and negative punishment. The first three paragraphs of the experimental text included an introductory paragraph followed by two paragraphs on the critical attributes of target concepts, i.e., aversive/attractive events, added/subtracted events. The following four paragraphs each presented one of the four target concepts. Each paragraph presented the definition of the concept, four examples of the concept, and discussion of how each example was illustrative of the concept. After each paragraph, all subjects were given a multiple-choice definition adjunct post-question. The two treatment groups were given either two matched or two unmatched multiple-choice application adjunct post-questions immediately after the definition adjunct question (see Hamilton, 1989, for sample questions). The definition adjunct post-question consisted of four alternatives, each of which consisted of a description and one of the four critical attributes or a combination of two of the critical attributes. The application adjunct post-questions consisted of four alternatives, each of which was an example of one of the four coordinate target concepts. The alternatives were either matched or unmatched in terms of the noncritical attributes of the target concepts. After each application adjunct question, subjects were given feedback which consisted of re-presenting the same question with the correct alternative identified. The definition questions were not followed with feedback for two reasons: a) the function of the questions was to prompt the appropriate definition as an aid for answering the application questions, and b) previous experiments found perfect levels of performance on these questions (see Hamilton, 1989).
92
RICHARD
HAMILTON
Procedures Subjects participated in the following four tasks in order during the 2-h experimental sessions: (1) a reading efftciency test, (2) a prior knowledge test, (3) an experimental text and criterion test, and (4) a mental skills test. All tests and text were presented in paper and pencil form. All subjects were told not to turn back once they had turned the page in the experimental booklet. All subjects also were asked to record the time (hour, minutes, seconds) from looking at the clock at the front of the room when they reached particular portions of the experimental text and criterion test (i.e., at the beginning of the experimental text, after finishing the text, and after they finished each part of the criterion test). These recorded times were used to estimate the time subjects spent reading the text and completing each of the criterion tasks.
Dependent Measures The criterion test consisted of three parts: (1) a free recall test, (2) a classification test, and (3) a problem-solving test. The free recall test asked subjects to recall the definition of each of the four coordinate concepts as well as the examples used in the experimental text. The definitions were scored as correct if the subjects presented the gist of the definition accurately. The examples were scored similarly. Each definition and example was reduced to its essential proposition or propositions, and each definition or example was judged on the basis of whether it captured the gist of one of these propositions. The objectivity of this scoring procedure was evaluated by comparing two independent scorings of the definitions and examples (r = .95). The classification test asked subjects to classify statements describing a behavior and a contingent event (same format as the alternatives used in application questions), each of which was an example of one of the four coordinate target concepts. There were six examples of each target concept. A Cronbach Alpha test found the internal consistency of the classification test to be .93. The problem-solving test consisted of two scenarios describing a classroom situation. Subjects were to create sets of contingencies representing each of the four target concepts to alleviate disruptive behaviors described in the classroom scenarios. When using each concept subjects were asked to identify the disruptive behavior and the event which was contingent upon the behavior. Subjects received one point for correctly setting up a set of contingencies which was an example of each of the four target concepts described in the text. In order to be marked correct, each answer had to clearly describe the behavior to be modified and a consequence that was an example of the target concept. The objectivity of this scoring procedure was evaluated by comparing two independent scorings of the adequacy of the description of target behaviors and consequences (r = 90).
Ability Measures Subjects were given three ability tests: prior knowledge, reading efftciency, and mental skills. The prior knowledge test consisted of six multiple-choice questions to test knowledge of target concepts. If subjects received a score of four or better on the prior knowledge test, they were eliminated from the experimental analysis. The reading efftciency subtest from the Iowa Silent Reading Test was used to evaluate the speed and accuracy of subjects’ reading ability. The mental skills tests consisted of 20 multiple-choice questions which include analogies, reasoning, and mathematical word problems.
RESULTS
Given the very low overall performance on recall of teaching examples (no higher than 12% recall in any condition), the following analyses did
93
ADJUNCT POST-QUESTIONS
not include this criterion variable. Alpha was set at .05 for all analyses. Performance on the problem solving test, free recall of definitions, and classification test were significantly positively correlated (.62 < r” < .69). An ANOVA indicated that reading efficiency scores significantly differed between treatment groups [F(2,75) = 3.58, MS, = 151.011.Consequently, reading efficiency scores were integrated into the following analyses as a covariate. Given the significant correlation between performance on the three criterion measures, overall differences were initially analyzed within a multivariate analysis of covariance. The MANCOVA found a significant main effect [F(6,144) = 4.01, Wilks’ Lambda = 0.731 (see Table 1). Post hoc multivariate contrasts (for each contrast, alpha = .025) found significant effects for a) matched questions versus definition questions only [F(3,72) = 2.91, Wilks’ Lambda = 0.891, and b) unmatched questions versus definition questions only [F(3,72) = 6.58, Wilks’ Lambda = 0.791. Finally, in order to evaluate the original hypotheses, a series of Tukey post hoc contrast (for each contrast, alpha = .008) between each type of application adjunct question treatment group and the control group for each criterion measure found the following differences: a) DEFINITIONS: Unmatched application questions produced significantly better performance than no application questions [F( 1,74) = 13.65, MS, = 22.021, while there was no significant difference between matched application questions and no application questions [F(1,74) = 1.59, MS, = 2.581; b) CLASSIFICATION: Both matched and unmatched application questions produced better performance than no application questions [MATCHED: F(1,74) = 8.01, MS, = 143.06; UNMATCHED: F(1,74) = 12.72, MS, = 227.261; c) PROBLEM SOLVING: Unmatched application questions produced better performance than no application TABLE 1 SCORES FOR PERFORMANCE ON CRITERION TESTS
Question group Matched Criterion tests” Recall of definitions Classification of novel examples Problem solving scenarios
Unmatched
Control
Meanb
SD
Mean
SD
Mean
SD
2.34 (2.19) 17.23 (16.91) 4.54 (4.16)
1.44
2.96 (3.02) 17.58 (17.64) 5.27 (5.41)
1.21
1.65 (1.75) 13.39 (13.65) 2.96 (3.19)
1.26
4.31 2.30
4.34 2.09
4.08 2.09
a Best possible score for free recall of concept definitions = 4.0; classification of novel examples = 24.0; problem solving scenarios = 8.0. b Adjusted means are listed in parentheses.
94
RICHARD HAMILTON
questions [F(1,74) = 7.15, MS, = 68.351, while there was no significant difference between the matched application questions and no application questions [F(1,74) = 3.23, MS, = 13.071. An ANOVA found significant main effects for time involved reading the prose passage [F(1,74) = 66.353, MS, = 893.21. A series of Tukey post hoc contrast (for each contrast, alpha = .025) found significant differences between each of the application adjunct post-question groups and the control group [MATCHED vs CONTROL; F(1,74) = 85.265, MS, = 114.9; UNMATCHED vs CONTROL; F(1,74) = 110.148, MS, = 148.21 (see Table 2). A series of Pearson product-moment coefficient correlations were calculated between time involved reading the prose passage and the criterion outcomes within each treatment group. No significant relationships were found, with the coefficients ranging from -0.29 to +0.036. Finally, there was no significant difference between treatment groups for performance on the definition and application adjunct post-questions (see Table 2). DISCUSSION When the effects of treatments on criterion measures were evaluated within a multivariate framework, both unmatched and matched application adjunct post-questions were found to produce superior performance in comparison to a definition question only control. In contrast, when the effects of treatments on each criterion measure were evaluated separately, unmatched application adjunct post-questions significantly improved subjects’ problem solving, classification of novel examples, and recall of concept definitions, while matched application adjunct postquestions significantly improved only the classification of novel examples. The latter analysis allows for the identification of the differential TABLE 2 MINUTES SPENT READING EXPERIMENTAL TEXT AND SCORES FOR PERFORMANCE ON DEFINITION ADJUNCT QUESTIONS AND APPLICATION ADJUNCT QUESTIONS
Question group Matched
Time reading text Definition adjunct questions” Application adjunct question?
Unmatched
Control
Mean
SD
Mean
SD
Mean
SD
19.03
4.41
20.84
4.51
10.19
2.35
3.69
0.55
3.89
0.33
3.62
0.49
10.89
1.03
10.73
1.15
a The best possible score for definition adjunct questions = 4.0; and application adjunct questions = 12.0.
ADJUNCT
POST-QUESTIONS
95
effects of the two treatments on the criterion measures and suggests the two types of application adjunct post-questions differentially influenced subjects’ conceptual, procedural, and contextual knowledge. Tennyson and Cocchiarella (1986) indicates that the type of practice used during the acquisition of target knowledge will dramatically influence the accessibility of that knowledge. That is, activities that force learners to use knowledge in a variety of contexts should produce greater access to target knowledge in future use situations. Unmatched adjunct questions vary the contexts of target concepts by presenting examples and nonexamples which vary in their noncritical attributes, i.e., contexts of application. This contrast with matched questions which use examples and nonexamples sets which possess similar noncritical attributes. The increased diversity of contexts presented within unmatched questions may partially explain its effects on problem solving outcomes. As indicated under Results, subjects in the definition question only control condition took significantly less time to read the prose passage than subjects in the matched and unmatched application question conditions. Although some would argue that the superior performance of application/definition question conditions was due to extra study time, one set of results mitigate against this interpretation. In none of the conditions was the correlation between time spent reading the text and performance on any of the criterion measures statistically or practically significant. There are three important ways in which the present evaluation of application questions differs from previous research on application questions: the presentation of two application questions per concept, the presentation of feedback, and the pairing of definition questions with the application questions. The increased practice due to two application questions may increase the probability of effects of application questions on criterion outcomes, however, previous research has found that the presentation of two application questions without definition questions did not significantly influence performance on criterion outcomes in comparison to the presentation of no questions at all (Hamilton, 1989). These latter results suggests that the results of the present experiment are not due to an increase in the number of application questions. In contrast to previous research on application questions, the present experiment presented feedback to the application questions. Recent research on the effects of feedback suggests that both the presence of feedback and the nature of feedback will influence the effectiveness of the targeted questions (Kulhavy & Stock, 1989). Within the present context, the presence of feedback may have influenced the results in at least two ways. First, the presence of feedback may have been responsible for the significant effects of matched questions on classification performance. As indicated earlier, matched and unmatched adjunct questions produced
96
RICHARD
HAMILTON
superior classification performance in comparison to the control group. These results are not consistent with those found in Hamilton (1989), i.e., no effects for matched application adjunct questions. However, there is one important difference between the procedures used by Hamilton (1989) and the present experiment, i.e., subjects were not given feedback to the application questions in the study by Hamilton (1989). Matched practice derives its benefits from allowing a comparison of the critical attributes of examples and nonexamples. Since the target examples were not clearly identified during practice in Hamilton (1989) but were in the present experiment, benefits for matched questions would only be realized in the present experiment. The results of the current experiment are consistent with the majority of previous concept learning experiments (cf. Tennyson & Cocchiarella, 1986). In these previous experiments, the examples of the target concepts were always clearly identified through feedback and/or expository directions. Second, it is possible that without feedback or with less elaborate feedback, the unmatched application questions would not have produced significant far transfer effects. Although there is some evidence that these types of application questions without feedback can influence transfer outcomes (i.e., classification) (Hamilton, 1989), there is clearly a need to evaluate the interactive effects of the nature of feedback, the presence of feedback, and the nature of application questions on learning concepts from prose. Finally, given the results of Hamilton (1989), definition questions were paired with the application questions within the present experiment. Performance on the definition questions was very high (~90%) in all three conditions and there were no differences between conditions. These results suggest comparable levels of knowledge of definitions derived from the text in all three conditions. However, given the importance of feedback and its potential effects, future research should evaluate the effects of feedback to definition questions within the present context, i.e., paired with application questions. In terms of instructional implications, it is important to reemphasize that the unmatched application questions were presented with feedback and were preceded with definition questions. Accordingly, both the definition questions and the feedback would need to accompany the presentation of unmatched questions in order for similar effects to occur within instructional situations. Two further points need to be mentioned concerning the generalizability of these results. There is a need to look at the effects of these inserted aids both within more typical instructional materials, e.g., chapters from textbooks, etc., and within a framework which allows for evaluation of long-term effects. In summary, unmatched application questions were again found to produce positive overall effects. The present results underscore the impor-
ADJUNCT
POST-QUESTIONS
97
tance of looking specifically at the nature of application adjunct postquestions when evaluating their effectiveness within a prose learning setting. Future research should also monitor the interactive effects of the following variables when accessing the influence of different types of application questions on problem solving outcomes: the number of questions per concept, the presence of and nature of feedback, retention intervals, and the nature of experimental text. REFERENCES FELKER, D., & DAPRA, R. (1975). Effects of question type and questions placement on problem solving ability from prose material. Journal of Educational Psychology, 67, 380-384.
HAMILTON, R. (1989). The role of concept definition, teaching examples and practice on concept learning from prose. Contemporary Educational Psychology, 14, 357-365. KULHAVY, R., & STOCK,W. (1989). Feedback in written instruction: The place of response certitude. Review of Educational Research, l(4), 279-308. TENNYSON,R. D., & COCCHIARELLA,M. (1986). An empirically-based instructional design theory for teaching concepts. Review of Educational Research, 56(l), 40-71.