- Email: [email protected]
Another Node On the interNet
Computers & Geosciences 27 (2001) 375–378
Another Node On the interNet Kim Kastens Lamont-Doherty Earth Observatory of Columbia University, PO Box 1000, 61 Route 9W, Palisades, N.Y. 10964, USA
How to identify the ‘‘best’’ resources for the reviewed collection of the Digital Library for Earth System Education The Digital Library for Earth System Education (‘‘DLESE’’; http://www.dlese.org/) is envisioned as a facility that provides: (a) easy access to high-quality educational materials about the Earth for use by educators and learners at all levels, (b) information, tools and services to maximize the usefulness of the materials provided, and (c) a community center that fosters interaction, collaboration and sharing among Earth science educators and learners. Development of the library is underway as a nation-wide distributed effort, coordinated by the DLESE Program Center at UCAR, and funded by the National Science Foundation. Rollout of a test version of DLESE is scheduled for summer 2001. The library’s collection will have a reviewed section and a non-reviewed section. The rationale for establishing a Reviewed Collection is to help library users find quality teaching and learning materials, and to help resource creators achieve academic career recognition. The rationale for maintaining an unreviewed or Open Collection is to provide the broadest possible range of resources, and to provide a forum in which resource users can provide feedback to creators to iteratively improve the quality of individual resources. The DLESE Collections Subcommittee has defined a set of seven selection criteria by which resources will be judged for inclusion in DLESE’s Reviewed Collection: * * *
* *
Scientifically accurate. Importance or significance. Pedagogical effectiveness: has student occurred? Well-documented. Ease of use for students and faculty.
learning
E-mail address: [email protected] (K. Kastens).
PII: S 0 0 9 8 - 3 0 0 4 ( 0 0 ) 0 0 1 1 5 - 1
* *
Power to inspire or motivate students. Robustness/sustainability as a digital resource.
This is a demanding set of criteria } both for the resource creators and for the gatekeepers/editors of the Reviewed Collection. This set of criteria requires that we judge attributes that are normally judged by quite different communities through quite different procedures. Scientific accuracy is most commonly assessed by peer review by scientist-experts recruited by a journal editor. Pedagogical effectiveness is commonly evaluated by pedagogy-experts through classroom observation, interviews, questionnaires, and other instruments. Robustness of a digital resource is commonly evaluated by QA (quality assurance) specialists as part of a software development effort. The DLESE Reviewed Collection aspires to do this all } in a coordinated, coherent way. The following premises have guided the design of the DLESE review procedure: (1) In order to be useful, DLESE has to contain a critical mass of resources. (a) In the growth of DLESE, teachers, classrooms and students will be abundant resources. (b) The rate-limiting resource in DLESE’s growth will be professional staff-time. (2) The materials in the DLESE Reviewed Collection must be classroom-tested. However: (a) Testimony from the creator of a resource that learning occurred in his or her classroom is insufficient. The presence of an expert in the classroom can bridge over gaps or blurry spots in the material that would greatly diminish the resource’s effectiveness in the hands of another teacher. (b) It is not realistic to pay for professional evaluators to go into classrooms to evaluate whether learning has occurred for every potential DLESE resource. (c) Experienced educators can tell whether or not their own students are learning effectively from
376
K. Kastens / Computers & Geosciences 27 (2001) 375–378
Table 1 Overview of review process Step
Selection criterion
How to implement
1 2
Well documented Importance/significance
3
Pedagogical effectiveness Ease of use for students and faculty Inspirational or motivational for students Scientifically accurate Robustness/sustainability
Review by DLESE staff More than N (threshold number to be chosen) educators from DLESE community tried this resource in their classroom On-line questionnaire filled out by educators who used resource in their classroom
4 5
an educational resource. Their experience and intuition, in the aggregate, are valuable and valid indicators of pedagogical effectiveness. (d) It is easier to answer: ‘‘Did your students learn?’’ than ‘‘Do you think students would learn?’’ With these premises in mind, we have designed what we call a ‘‘community review’’ filtering system. This system taps into the power of the World Wide Web and the strength of numbers of the DLESE community. A summary of the process is provided in Table 1 and Fig. 1. Step 1: The first step in the review procedure will occur in parallel with metadata tagging. As the library staff person reviews the resource to attach the metadata, he/she will check to make sure that the resource is ‘‘welldocumented’’. The meaning of ‘‘well-documented’’ varies for each resource type. For example, assessment tools should include an answer key or scoring rubric; data should include documentation of how, when, where the data were acquired; field trips guides should include maps. If the resource is incompletely documented, the creator will be invited to upgrade it. Step 2: As each user accesses a resource, DLESE will determine if he or she is an educator, through an automated user-authorization infrastructure. Non-educators will be invited to send suggestions for improving the resource to the creator (Fig. 1, pathway 2a), but only evaluations from educators will contribute to the formal review process. Educators will be asked to fill out an evaluation questionnaire as a web-based form. The questionnaire will first ask if you have used the resource in your classroom. The ‘‘importance/significance’’ criterion will be judged to have been passed when a threshold number of members of the community attest that they have used the resource in their classroom (Fig. 1, pathway 2b). Even if they did not like the resource, the very fact that they tried it will be considered as de facto evidence that the content or understanding or skill addressed by the resource is important to the community.
Invited review by a scientist, recruited by an editor QA testing (functionality and configuration testing)
For educators who respond, No, they did not use the resource in their classroom, the questionnaire seeks to determine why the resource was not used. The evaluator/educator can click choices such as ‘‘too hard’’, ‘‘too easy’’, ‘‘too time consuming’’, ‘‘decided not to teach this topic’’, ‘‘found a mistake, which was _______’’, ‘‘found a better resource, which was ______’’, etc. The information from educators who looked at the resource but decided not to use it will be forwarded automatically but anonymously to the resource creator (Fig. 1, pathway 2c), and archived. In addition, this information will be aggregated and forwarded to the keepers of the DLESE Discovery system for use in refining the search procedures to cut down on the number of not-useful resources explored by DLESE educators (Fig. 1, pathway 2d). Step 3: For educators who respond that, Yes, they did use the resource in their classroom, the questionnaire will dig more deeply. Educators will be asked to provide numerical ratings, evaluating the resource on the criteria of pedagogical effectiveness, ease of use for faculty and students, and power to motivate or inspire students. Separate text fields will be provided for an open-ended review of the resource, and for teaching tips that other educators might find useful in using the resource. Finally, the questionnaire would gather contextual information about the setting in which the resource was used. The response to the write-in questions and the numerical scores will be sent, anonymously and automatically, to the creator of the resource (Fig. 1, pathway 3a), and archived. The creator may chose to modify the resource to take into account the reviewers’ comments. After revision, the creator may choose to wait for better reviews to accrue, or ‘‘wipe the slate clean’’ and omit all the pre-revision scores from the accounting. Educators’ responses to the teaching tips question will be posted with the resource (Fig. 1, pathway 3b). Examples include: suggestions on how to adapt the resource for certain categories of students (e.g. disabled, older or younger than original target); references to
K. Kastens / Computers & Geosciences 27 (2001) 375–378
Fig. 1.
377
378
K. Kastens / Computers & Geosciences 27 (2001) 375–378
print or web-based material pertinent to some aspect of the resource; warnings of difficulties that students have encountered. When a given resource has accumulated more than a threshold number of reviews from educators who have used the resource in their classroom, and the numerical scores have risen above a threshold value, the entire packet of reviews will be sent automatically to the ‘‘editor’’ or ‘‘gatekeeper’’ of the Reviewed Collection (Fig. 1, pathway 3c). The threshold numerical scores could be a sum of the three scores, or weighted more heavily for pedagogical effectiveness than for the others, or each criteria could have a separate threshold. We will experiment with different threshold values. The editor will review the packet and trigger the more staff-intensive steps in the evaluation process: the evaluations for ‘‘scientifically accurate’’, and ‘‘robustness/sustainability’’. Step 4: For ‘‘scientifically accurate’’, the editor will recruit a scientist-reviewer, a specialist in the content domain of the resource, who will search for errors or weaknesses in the science presented (Fig. 1, pathway 4). This process would be similar to the familiar peer-review process for scientific manuscripts. Step 5: For ‘‘robustness/sustainability’’, the resource will be run through a quality-assurance process (Fig. 1, pathway 5). For a simple resource such as an image collection, this will be an automated check for broken hyperlinks, and a verification that the images display and print successfully on a range of browsers and hardware. For a resource with embedded software, such as a simulation or animation, the robustness check will include a check for compatibility with different hardware and operating system configurations, and an attempt to break the software by exercising it with a wide range of inputs, conditions and parameters. The reviews and bug list from these staff-mediated review steps will be sent to the creator of the resource, who will then modify the resource as needed. A staff member will check to make sure that the revisions accomplished the changes suggested by the reviews and tests, and certify when the resource is ready to move into the Reviewed Collection. If either the QA or the science review necessitate revisions that substantially change the
pedagogical approach or educational level, the editor/ gatekeeper could send the resource back to the community review process. The community review mechanism will remain in place even after an item has moved into the reviewed collection. Teaching tips sent in by educator-evaluators will continue to accrue and be posted with the resource. The numerical scores on the pedagogical effectiveness, ease of use, and inspirational/motivational criteria for items in the reviewed collection will be used for two additional purposes. First, those resources that continue to receive the highest scores on the community review evaluation, and also are accessed by a large number of users will be rewarded with ‘‘best of DLESE’’ awards. These rewards will be given annually, in a several categories, to encourage appropriate academic career recognition for the creators of excellent educational resources. At the other end of the spectrum, those resources which begin to receive subpar numerical scores in the community review evaluation will be re-examined. If appropriate, the creator will be offered an opportunity to update the resource; if that does not happen, the resource may be pruned from the Reviewed Collection. The major strengths of the planned review system, as we see it, are as follows: *
*
* *
*
Resources in the resulting Reviewed Collection should be of consistently high quality. The system is economical and scalable to large numbers of resources. Scarce talents are applied at the end of the process, to the smallest number of items. All resources are classroom-tested. The system balances community input and expert input. The system contributes to the DLESE vision of a library built ‘‘by the community, for the community’’. John C. Butler Associate Editor University of Houston, Houston, TX 77204, USA E-mail address: [email protected]
Another Node On the interNet Kim Kastens Lamont-Doherty Earth Observatory of Columbia University, PO Box 1000, 61 Route 9W, Palisades, N.Y. 10964, USA
How to identify the ‘‘best’’ resources for the reviewed collection of the Digital Library for Earth System Education The Digital Library for Earth System Education (‘‘DLESE’’; http://www.dlese.org/) is envisioned as a facility that provides: (a) easy access to high-quality educational materials about the Earth for use by educators and learners at all levels, (b) information, tools and services to maximize the usefulness of the materials provided, and (c) a community center that fosters interaction, collaboration and sharing among Earth science educators and learners. Development of the library is underway as a nation-wide distributed effort, coordinated by the DLESE Program Center at UCAR, and funded by the National Science Foundation. Rollout of a test version of DLESE is scheduled for summer 2001. The library’s collection will have a reviewed section and a non-reviewed section. The rationale for establishing a Reviewed Collection is to help library users find quality teaching and learning materials, and to help resource creators achieve academic career recognition. The rationale for maintaining an unreviewed or Open Collection is to provide the broadest possible range of resources, and to provide a forum in which resource users can provide feedback to creators to iteratively improve the quality of individual resources. The DLESE Collections Subcommittee has defined a set of seven selection criteria by which resources will be judged for inclusion in DLESE’s Reviewed Collection: * * *
* *
Scientifically accurate. Importance or significance. Pedagogical effectiveness: has student occurred? Well-documented. Ease of use for students and faculty.
learning
E-mail address: [email protected] (K. Kastens).
PII: S 0 0 9 8 - 3 0 0 4 ( 0 0 ) 0 0 1 1 5 - 1
* *
Power to inspire or motivate students. Robustness/sustainability as a digital resource.
This is a demanding set of criteria } both for the resource creators and for the gatekeepers/editors of the Reviewed Collection. This set of criteria requires that we judge attributes that are normally judged by quite different communities through quite different procedures. Scientific accuracy is most commonly assessed by peer review by scientist-experts recruited by a journal editor. Pedagogical effectiveness is commonly evaluated by pedagogy-experts through classroom observation, interviews, questionnaires, and other instruments. Robustness of a digital resource is commonly evaluated by QA (quality assurance) specialists as part of a software development effort. The DLESE Reviewed Collection aspires to do this all } in a coordinated, coherent way. The following premises have guided the design of the DLESE review procedure: (1) In order to be useful, DLESE has to contain a critical mass of resources. (a) In the growth of DLESE, teachers, classrooms and students will be abundant resources. (b) The rate-limiting resource in DLESE’s growth will be professional staff-time. (2) The materials in the DLESE Reviewed Collection must be classroom-tested. However: (a) Testimony from the creator of a resource that learning occurred in his or her classroom is insufficient. The presence of an expert in the classroom can bridge over gaps or blurry spots in the material that would greatly diminish the resource’s effectiveness in the hands of another teacher. (b) It is not realistic to pay for professional evaluators to go into classrooms to evaluate whether learning has occurred for every potential DLESE resource. (c) Experienced educators can tell whether or not their own students are learning effectively from
376
K. Kastens / Computers & Geosciences 27 (2001) 375–378
Table 1 Overview of review process Step
Selection criterion
How to implement
1 2
Well documented Importance/significance
3
Pedagogical effectiveness Ease of use for students and faculty Inspirational or motivational for students Scientifically accurate Robustness/sustainability
Review by DLESE staff More than N (threshold number to be chosen) educators from DLESE community tried this resource in their classroom On-line questionnaire filled out by educators who used resource in their classroom
4 5
an educational resource. Their experience and intuition, in the aggregate, are valuable and valid indicators of pedagogical effectiveness. (d) It is easier to answer: ‘‘Did your students learn?’’ than ‘‘Do you think students would learn?’’ With these premises in mind, we have designed what we call a ‘‘community review’’ filtering system. This system taps into the power of the World Wide Web and the strength of numbers of the DLESE community. A summary of the process is provided in Table 1 and Fig. 1. Step 1: The first step in the review procedure will occur in parallel with metadata tagging. As the library staff person reviews the resource to attach the metadata, he/she will check to make sure that the resource is ‘‘welldocumented’’. The meaning of ‘‘well-documented’’ varies for each resource type. For example, assessment tools should include an answer key or scoring rubric; data should include documentation of how, when, where the data were acquired; field trips guides should include maps. If the resource is incompletely documented, the creator will be invited to upgrade it. Step 2: As each user accesses a resource, DLESE will determine if he or she is an educator, through an automated user-authorization infrastructure. Non-educators will be invited to send suggestions for improving the resource to the creator (Fig. 1, pathway 2a), but only evaluations from educators will contribute to the formal review process. Educators will be asked to fill out an evaluation questionnaire as a web-based form. The questionnaire will first ask if you have used the resource in your classroom. The ‘‘importance/significance’’ criterion will be judged to have been passed when a threshold number of members of the community attest that they have used the resource in their classroom (Fig. 1, pathway 2b). Even if they did not like the resource, the very fact that they tried it will be considered as de facto evidence that the content or understanding or skill addressed by the resource is important to the community.
Invited review by a scientist, recruited by an editor QA testing (functionality and configuration testing)
For educators who respond, No, they did not use the resource in their classroom, the questionnaire seeks to determine why the resource was not used. The evaluator/educator can click choices such as ‘‘too hard’’, ‘‘too easy’’, ‘‘too time consuming’’, ‘‘decided not to teach this topic’’, ‘‘found a mistake, which was _______’’, ‘‘found a better resource, which was ______’’, etc. The information from educators who looked at the resource but decided not to use it will be forwarded automatically but anonymously to the resource creator (Fig. 1, pathway 2c), and archived. In addition, this information will be aggregated and forwarded to the keepers of the DLESE Discovery system for use in refining the search procedures to cut down on the number of not-useful resources explored by DLESE educators (Fig. 1, pathway 2d). Step 3: For educators who respond that, Yes, they did use the resource in their classroom, the questionnaire will dig more deeply. Educators will be asked to provide numerical ratings, evaluating the resource on the criteria of pedagogical effectiveness, ease of use for faculty and students, and power to motivate or inspire students. Separate text fields will be provided for an open-ended review of the resource, and for teaching tips that other educators might find useful in using the resource. Finally, the questionnaire would gather contextual information about the setting in which the resource was used. The response to the write-in questions and the numerical scores will be sent, anonymously and automatically, to the creator of the resource (Fig. 1, pathway 3a), and archived. The creator may chose to modify the resource to take into account the reviewers’ comments. After revision, the creator may choose to wait for better reviews to accrue, or ‘‘wipe the slate clean’’ and omit all the pre-revision scores from the accounting. Educators’ responses to the teaching tips question will be posted with the resource (Fig. 1, pathway 3b). Examples include: suggestions on how to adapt the resource for certain categories of students (e.g. disabled, older or younger than original target); references to
K. Kastens / Computers & Geosciences 27 (2001) 375–378
Fig. 1.
377
378
K. Kastens / Computers & Geosciences 27 (2001) 375–378
print or web-based material pertinent to some aspect of the resource; warnings of difficulties that students have encountered. When a given resource has accumulated more than a threshold number of reviews from educators who have used the resource in their classroom, and the numerical scores have risen above a threshold value, the entire packet of reviews will be sent automatically to the ‘‘editor’’ or ‘‘gatekeeper’’ of the Reviewed Collection (Fig. 1, pathway 3c). The threshold numerical scores could be a sum of the three scores, or weighted more heavily for pedagogical effectiveness than for the others, or each criteria could have a separate threshold. We will experiment with different threshold values. The editor will review the packet and trigger the more staff-intensive steps in the evaluation process: the evaluations for ‘‘scientifically accurate’’, and ‘‘robustness/sustainability’’. Step 4: For ‘‘scientifically accurate’’, the editor will recruit a scientist-reviewer, a specialist in the content domain of the resource, who will search for errors or weaknesses in the science presented (Fig. 1, pathway 4). This process would be similar to the familiar peer-review process for scientific manuscripts. Step 5: For ‘‘robustness/sustainability’’, the resource will be run through a quality-assurance process (Fig. 1, pathway 5). For a simple resource such as an image collection, this will be an automated check for broken hyperlinks, and a verification that the images display and print successfully on a range of browsers and hardware. For a resource with embedded software, such as a simulation or animation, the robustness check will include a check for compatibility with different hardware and operating system configurations, and an attempt to break the software by exercising it with a wide range of inputs, conditions and parameters. The reviews and bug list from these staff-mediated review steps will be sent to the creator of the resource, who will then modify the resource as needed. A staff member will check to make sure that the revisions accomplished the changes suggested by the reviews and tests, and certify when the resource is ready to move into the Reviewed Collection. If either the QA or the science review necessitate revisions that substantially change the
pedagogical approach or educational level, the editor/ gatekeeper could send the resource back to the community review process. The community review mechanism will remain in place even after an item has moved into the reviewed collection. Teaching tips sent in by educator-evaluators will continue to accrue and be posted with the resource. The numerical scores on the pedagogical effectiveness, ease of use, and inspirational/motivational criteria for items in the reviewed collection will be used for two additional purposes. First, those resources that continue to receive the highest scores on the community review evaluation, and also are accessed by a large number of users will be rewarded with ‘‘best of DLESE’’ awards. These rewards will be given annually, in a several categories, to encourage appropriate academic career recognition for the creators of excellent educational resources. At the other end of the spectrum, those resources which begin to receive subpar numerical scores in the community review evaluation will be re-examined. If appropriate, the creator will be offered an opportunity to update the resource; if that does not happen, the resource may be pruned from the Reviewed Collection. The major strengths of the planned review system, as we see it, are as follows: *
*
* *
*
Resources in the resulting Reviewed Collection should be of consistently high quality. The system is economical and scalable to large numbers of resources. Scarce talents are applied at the end of the process, to the smallest number of items. All resources are classroom-tested. The system balances community input and expert input. The system contributes to the DLESE vision of a library built ‘‘by the community, for the community’’. John C. Butler Associate Editor University of Houston, Houston, TX 77204, USA E-mail address: [email protected]