Educational metadata and brokerage for learning resources

Computers & Education 38 (2002) 351–374 www.elsevier.com/locate/compedu

Educational metadata and brokerage for learning resources Luis E. Anido*,1, Manuel J. Ferna´ndez, Manuel Caeiro, Juan M. Santos, Judith S. Rodrı´guez, Martı´n Llamas ETSE Telecomunication, Universidade de Vigo, Spain Received 30 June 2001; accepted 10 December 2001

Abstract The learning technology standardization process is one of the key research activities in computer-based education. Institutions like the IEEE, the US Department of Defense and the European Commission have set up committees to deliver recommendations and proposals in this area. The objective is to allow the reuse of learning resources and to offer interoperability among heterogeneous e-learning systems. The first part of this paper is devoted to the presentation of an up-to-date survey on one of the most prolific fields of the learning technology standardization: educational metadata. The second part shows how these data models are applied by actual software systems to facilitate the location of learning resources. Educational brokerage is a promising field that lets learners find those computer-based training resources that best fit their needs. We identify the main actors involved, their roles, and open issues and trends. # 2002 Elsevier Science Ltd. All rights reserved. Keywords: Standardization; Interoperability; Data models; Learning recources reuse

1. Introduction Advances in information and communication technologies, and specifically in Multimedia, Networking and Software Engineering promoted a new generation of computer-based training systems. Internet is today the ubiquitous supporting environment for virtual and distributed learning environments. As a consequence, many institutions, both public and private, take advantage of new technologies to offer training products and services at all levels. * Corresponding author. Fax: +34-986-812116. E-mail address: [email protected] (L.E. Anido). 1 Project Team expert hired by CEN/ISSS/LT, in charge of LOM Internationalisation. 0360-1315/02/$ - see front matter # 2002 Elsevier Science Ltd. All rights reserved. PII: S0360-1315(02)00018-0

352

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Nomenclature ACTS ADL AICC ANSI ARIADNE ATM CBT CEN DC DCAC DC-ED DCMI DCMES EdNA ERIC GAIA GEM GEMSTONES GESTALT IEC IEEE IETF IMS-SEL ISSS ISO KPS LDAP LOM LTSC PAPI PROMETEUS QoS RDS

Advanced Communication Technology and Services Advanced Distributed Learning. Initiative from the US DoD Aviation Industry CBT Committee American National Standardization Institute Alliance of Remote Instructional Authoring and Distribution Networks for Europe Asynchronous Transfer Mode Computer-Based Training Comite` Europe´en de Normalization, European Committee for Standardization Dublin Core Dublin Core Advisory Committee Dublin Core metadata for Education Dublin Core Metadata Initiative Dublin Core Metadata Element Set Education Network Australia Educational Resources Information Center Generic Architecture for Information Availability Gateway to Educational Materials GESTALT Extensions to Metadata Standards for ON-Line Education Systems Getting Educational Systems Talking Across Leading edge Technologies International Electrotechnical Commission The Institute of Electrical and Electronics Engineers The Internet Engineering Task Force IMS Standard Extension Library Information Society Standardization System Subcommittee, hosted by CEN International Standardization Organization ARIADNE’s Knowledge Pool System Lightweight Directory Access Protocol Learning Object Metadata Learning Technologies Standardization Committee, hosted by the IEEE Public And Private Information PROmoting Multimedia access to Education and Training in EUropean Society Quality of Service Resource Discovery Service

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

SCORM TFADLT XML

353

Sharable Content Object Reference Model Total Force Advanced Distributed Learning Action Team Extended Markup Language

In this situation, educational systems and resources proliferate, and a need for standardization becomes apparent. Like in other standard-driven initiatives, standardization applied to learning technologies will enable reuse and interoperation among heterogeneous software systems. To achieve this, a consensus is needed on architectures, services, protocols, data models and open interfaces. Thus, institutional users of educational software are joining their efforts to achieve standards and recommendations to support the interoperation of heterogeneous learning systems. This is an active, continuously evolving process that will last for years to come, until a clear, precise and generally accepted set of standards for educational-related systems is developed. This is a complex process, which occurs at several levels and is supported by many different related initiatives. The areas of interest in the standardization of computer-based education cover student models; equipment, formats and low level aspects; software components and runtime environments; course organization and behaviour; course packaging and transfer; educational resource searching, location and access, including brokerage services and harvesting; and educational metadata. A key aspect in networked educational systems is to define, as precisely as possible, the resources and services offered to potential users. Information on offered courses, related contents, target audience or technical requirements should be made available in a way that permits searching, location and, eventually, access. The trend is to describe this information using metadata. Metadata (Day, 2001) are traditionally defined as data about data, information about information, and are used to describe document contents and structure, and to provide information about accessibility, organization of data, relations among data items, and the properties of the corresponding data domains. Metadata are also useful to provide textual descriptions for non-textual objects, for example, to enable the representation of multimedia document properties in a structured way, simplifying document management and retrieval (Yoon & Kim, 2000). Today, hundreds of collections worldwide already adopted metadata as the basic tool for resource description. Following this trend, metadata recommendations for educational systems are one of the most productive activities in the standardization of computer-based learning systems (Sutton, Lankes, Small, & Eisenberg, 1998). Besides the uses discussed above, educational metadata is used to offer learning resource brokerage (Adam & Atluri, 2001). A broker is a mediator for the searching and location of resources. Typically, brokers are independent actors that reach agreements with resource providers (e.g. digital libraries, information repositories) to implement new services to facilitate resource discovery and other activities related to information search and retrieval. These services may be customized for registered users, including notifications, profiling, statistics generation, premium services, etc.

354

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

In other words, brokerage enriches the traditional searching services (Schwartz, 1998) to include resource acquisition, distribution and billing, and perfectly adapts to a scenery with multiple, independent content providers. It enables clients to locate, select and access content providers in a rapid and efficient way. Furthermore, brokerage offers benefits also to information providers, offering tailored policies for catalog updating, and support for marketing, customer service, content delivery, and even information management. In an educational environment, brokers collect information about education providers and their products to make them available to potential users. Typically, users provide the broker with their preference profile, which is used to return customized results and to offer other value-added services (e.g. notifications, a common front-end to providers, consolidated results from several providers, etc.). Thus, educational brokerage systems facilitate an efficient use of available educational resources, improving existing searching and location methods. As brokerage often takes place among different institutions, it also provides an adequate model for federated searching (Powell & Fox, 1998; Sharon & Frank, 2000). In Task Force on Metadata (1999) the reader can find a comprehensive collection of definitions related to metadata, metadata schemas, and interoperability. In this paper we offer a survey on the current state of the art of the standardization of educational metadata. We try to identify the key aspects of this process, the actors, and future trends. Along the next sections we will discuss in some detail the issues outlined in the previous paragraphs, including brief descriptions of those systems that currently offer educational brokerage capabilities. First, we will present the key institutions and organizations involved. Section 3 is devoted to present the current trends in the standardization of educational metadata. Then, in Section 4, we show some of the most important educational brokerage systems based on standardized metadata. Finally, we end with some conclusions and identify future trends.

2. The actors Involved institutions and organizations in the learning technology standardization process are typically North American or European entities, both public and private, that massively use educational software products. As a consequence, they are conscious of the need of recommendations and standards to simplify and promote software reuse and system interoperability. In the next paragraphs we identify the most active actors in this field, which are summarized in Table 1. 2.1. The Institute of Electrical and Electronics Engineers The Learning Technologies Standardization Committee (LTSC, 2001) from the IEEE covers practically all aspects related to computer-based education. Its main objective is to develop technical standards, recommended practices and guidelines for software components, tools, technologies and design methods to facilitate the development, implementation, maintenance and interoperation of educational systems and their contents. The 15 subcommittees composing the LTSC are organized into five working areas: general, content-related, learner-related, data and metadata, and management systems and applications.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

355

The first proposals from LTSC are related to Computer Based Training (CBT) system architecture and reference model, student model and educational metadata. As we will see in Section 3, the LTSC leads the standardization of educational metadata models. 2.2. International Standardization Organization The 36th subcommittee of the first joint International Standardization Organization and International Electrotechnical Commission Committee (ISO/IEC JTC1 SC36, 2001) was launched in 1999 to cover all aspects related to the standardization in the field of learning technologies. Its focus is on interoperability, not only at the technical level, but also taking into account social and cultural issues. This subcommittee has tight relations with other related groups, like those devoted to user interfaces (SC35), data management and exchange (SC32), programming languages (SC22) or character coding (SC2). Additionally, explicit links were established with the LTSC in the areas of architecture and reference model, learner-related activities, content-related activities, data and metadata, and management systems and applications. It is expected that the first ISO educational metadata standard, based on IEEE LTSC work, will appear in the near future. 2.2.1. Dublin Core Metadata Initiative The Dublin Core Metadata Initiative (DCMI, DCI; DC, 2001) is an organization dedicated to the promotion of interoperable metadata standards, and to the development of metadata vocabularies to describe resources to significantly improve information retrieval and discovery systems. DCMI activities are organized around working groups and workshops. The first workshop where the initial proposal for Dublin Core was issued took place in Dublin, Ohio in 1995. Since Table 1 Main educational standardization initiatives Acronym

Organization

Initiative

IEEE—LTSC JTC12 SC36 IMS AICC ADL DC-ED GEM NSDL Metadata ARIADNE GESTALT PROMETEUS CEN/ISSS/LT EdNA

IEEE ISO and IEC EDUCAUSE US AI US DoD DCMI US DoE NSDL SWG EC EC EC CEN

Learning Technologies Standardization Committee Joint Committee for the Standardization of Learning Technologies IMS Project & Consortium Aviation Industry CBT Committee Advanced Distributed Learning Dublin Core Educational Metadata Gateway to Educational Materials National Science. Mathematics. Engineering, and Technology Education Digital Library Alliance of Remote Instructional Authoring and Distributed Networks for Europe Getting Educational Systems Talking Across Leading edge Techonolgies PROmoting Multimedia access to Education and Training in EUropean Society Learning Technologies Workshop Education Network Australia

JTC: Joint Technical Comniittee; SC: Subcommittee; DoD: Department of Defence; DoE: Department of Education; AI: Aviation Industry: NSDL SWG: NSDL Standards Working Group; DCMI: DublinCore Metadata Initiative; EC: European Community; CEN: European Standardization Committee.

356

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

then, DCMI activities have been targeted to refine a core foundation of metadata elements to provide semantic information about Web resources. 2.3. EDUCAUSE—IMS The IMS2 (IMS, 2001) project was launched by EDUCAUSE, formerly EDUCOM, a consortium of North American educational institutions and their industrial partners to define technical standards for the interoperation of distributed learning applications and services. Recently, EDUCAUSE was awarded the management of the .edu Internet domain by the US Department of Commerce. Their first efforts were targeted to the definition of an architecture and reference model for distributed learning systems. However, their activities were redefined when they detected that they needed a common data model to describe resources, structures and other elements handled by the intended architecture. Today, the main results of the IMS are in the fields of metadata, content packaging, test definition, and student and group profiling and management. Each one of these data models are instantiated in three different documents: model definition, XML (Bray, 2000) model specification, and model implementation guide. 2.4. North American Aviation Industry The Aviation Industry CBT Committee (AICC, 2001) is the natural response to the educational standardization challenge from one of the largest users of educational software. Recommendations from AICC are published in three different formats: AICC Guidelines and Recommendations (AGR), technical reports, and working documents. The AGR reflect the official position of the aviation industry and are the basic reference for educational software providers. The activities of the AICC are targeted, among others, to the definition of software and hardware requirements for student computers, needed peripherals, multimedia formats for course contents, and user interface properties. Its proposal for runtime environments includes standalone systems where communication is supported by file transfers and Web-based learning systems. The AICC has a close relationship with US Department of Defense’s Advanced Distributed Learning (ADL) initiative. No metadata recommendation or brokerage proposal has been delivered by the AICC so far. 2.5. US Department of Defense In 1997, the US DoD and the White House Science and Technology Bureau launched the Advanced Distributed Learning (ADL, 2001) initiative. ADL contributes to satisfy the needs of one of the largest software consumers in the world, and to promote better educational services. ADL was targeted from the very beginning to Web-based education. Its work is coordinated with other organizations like IEEE, IMS and AICC. Coordination is of paramount importance to identify aspects of Web-based learning systems where common interface specifications are needed.

2 IMS stands for Instructional Management System. However, the IMS Consortium prefers not to use the acronym to avoid misinterpretations. The aim of IMS goes beyond instructional management systems.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

357

As a result of this joint work, the Sharable Content Object Reference Model (SCORM; Dodds, 2001) was produced. This proposal includes a reference model for educational sharable software objects, a runtime environment, a metadata model and a content structure model. In 1999 ADL established Co-Lab as the working group responsible for testing and validating new ADL recommendations, for verifying the degree of compliance with SCORM of external products, and for developing SCORM-compliant prototypes. Finally, the Total Force Advanced Distributed Learning Action Team (TFADLAT) serves as the interface between the ADL initiative and the US Secretary of Defense. It makes recommendations to the DoD concerning educational software and systems, and promotes the implementation of developed standards. 2.6. Other US initiatives Project GEM (GEM, 2001), Gateway to Educational Materials, provides a unified framework for the publication and location of educational resources available through the Internet. This project was promoted by the US Departament of Education, and was born in 1997 as an special project within the ERIC Clearinghouse on Information & Technology. Educational resource providers register their resources according to the GEM cataloging standards, and users query the system to obtain a list of matching resources. The definition of metadata to represent educational resources is also part of the GEM work. Starting in 1996, the National Science Foundation (NSF) addressed the development of a US digital library for science, mathematics, engineering and technology education (SMETE; Wattenberg, 1998). Building on work supported under the multi-agency Digital Libraries Initiative, the National Science Foundation developed the National SMETE Digital Library (NSDL) program to found a national digital library that will constitute an online network of learning environments and resources for science, mathematics, engineering, and technology (SMET) education at all levels. The NSDL has a Standards Working Group devoted mainly to crosswalking metadata standards, taking into consideration proposals from Dublin Core, ADL/SCORM, LTSC/IMS, and others. 2.7. Education Network Australia Education Network Australia, EdNA (EdNA, 2001b) is targeted to promote the Internet as a supporting tool for computer-based learning among the Australian educational community, from students to content providers. Like GEM and NSDL, EdNA’s main objective is to offer access to educational resources and services. They have developed their own version of educational metadata. 2.8. European Community Within the European Community, we can identify four main initiatives related to the standardization of computer-based education: ARIADNE (2001), GESTALT (2001), PROMETEUS (2001) and CEN/ISSS/LT (2001). The Alliance of Remote Instructional Authoring and Distribution Networks for Europe (ARIADNE) was part of the European Commission’s fourth framework program. The main

358

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

working fields of this alliance are: computer networks for education and learning; methodologies for the development, management and reuse of educational contents; syllabus definition for computer based training; and educational metadata. One of the main contributions of this initiative is a proposal on educational metadata, which was made in collaboration with IMS, that set the basis for the IEEE’s proposal. Another European initiative is project GESTALT, Getting Educational Systems Talking Across Leading edge Technologies. GESTALT establishes a reference framework for the development of distributed, heterogeneous, scalable and compatible educational systems. The general objective of this framework is to enable users to discover educational resources, and to provide access to the discovered resources through a conveniently managed network infrastructure. In addition to the above mentioned objectives, GESTALT has made important contributions to the definition of data models for networked educational systems, specifically for the definition of educational metadata and student profiles and preferences. PROMETEUS, PROmoting Multimedia access to Education and Training in EUropean Society is another European initiative that gets together more than 400 institutions involved in computer-based education. This initiative is organized around a collection of mailing lists devoted to several CBT-related aspects, and meetings taking place in different European cities. Although participation in discussions and meetings is intense, at the time of writing this paper PROMETEUS had not produced significative outcomes. The European Committee for Standardization (Comite` Europe`een de Normalization, CEN) hosts the Information Society Standardization System (ISSS) subcommittee. Educational standardization activities at ISSS take place within the Learning Technologies Workshop (CEN/ ISSS/LT). The main efforts are devoted to reuse and interoperation of educational resources, educational collaboration, metadata for educational contents, and learning process quality taking into account European cultural diversity. At the time of this writing several CEN project teams were working on internationalization of educational metadata recommendations. 2.9. Relations among actors and their contributions As the reader will infer from the different proposals discussed in this brief survey, in many cases they are the result of the joint efforts of several institutions involved in the standardization process. Usually, an activity is initiated at the same time by different institutions, and links are established along the process to define the final recommendations. In most cases, LTSC collects proposals from all actors and converts them into common recommendations after a general agreement is reached. Finally, those proposals approved by the IEEE initiate a more rigorous process to become ANSI or ISO standards. This process is outlined in Fig. 1.

3. Educational metadata Educational metadata is one of the most prolific fields in the standardization of education technologies. Most of the initiatives presented in Section 2 have made proposals in this area, and

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

359

a consensus that will serve as the basis for new generally accepted recommendations and standards is expected in the near future. Educational metadata provides information about educational resources. An educational resource is an entity that can be used or referred to along a learning process. Multimedia contents, books, manuals, programs, tests, software applications, tools, people, organizations are examples of educational resources. As available educational resources grow and grow, the need for metadata becomes apparent. The lack of information about the properties, location or availability of a resource could make it unusable. This situation is even more patent in an open, unstructured environment like the Internet. Metadata contributes to solve this problem by providing a standard and efficient way to conveniently characterize resource properties. This way, resource location, sharing, construction or customization is made simpler. In the next paragraphs we discuss the most relevant proposals on metadata for educational resources. The first proposal presented, Learning Object Metadata (LOM), will probably become the standard for educational metadata. The rest of the proposals, based on LOM, are extensions or specific instantiations of LOM, or try to keep compatible with it. 3.1. Learning Object Metadata One of the main contributors to the definition of educational metadata is the LTSC from the IEEE. In 1998, IMS and ARIADNE submitted a joint proposal to the IEEE, which formed the foundation of the current IEEE LOM Base Document. Presently, LOM version 6.0 is under development (Hodgins, 2001). LOM specifies the syntax and semantics of learning object metadata, defined as the attributes required to fully and adequately describe a learning object. This includes element names, definitions, data types, taxonomies, vocabularies, and field lengths. LOM is focused on the minimal set of attributes needed to allow these learning objects to be managed, located and evaluated. Relevant attributes of learning objects to be described include type of object, author, owner, terms of

Fig. 1. Relations among the actors.

360

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Table 2 LOM topmost categories Group

Information collected

General Lifecycle Meta-metadata Technical Educational Rights Relation Annotation Classification

Context-independent features plus the semantic descriptors for resources. Features linked to the lifecycle of resources. Features of the description itself, rather than those of the resource being described. Technical features of resources. Educational and pedagogic features. Features that deal with conditions of use. Features of resources that link theni to other resources. Comments on the educational use of resources. Characteristics of resources described by entries in classifications.

distribution, and format. Where applicable, learning object metadata may also include pedagogical attributes such as teaching or interaction style, grade level, mastery level, and prerequisites. Educational resources are heterogeneous in nature, and this diversity should be reflected in metadata descriptions. LOM is organized into nine categories that group 60 different metadata elements. Among other tasks, LOM metadata descriptions support version management and maintenance, resource storage and recovery (searching, location, instantiation, packaging, editing, etc.), and resource sharing. Although not explicitly stated by the LOM proposal, LOM metadata may also be useful for other related tasks like intellectual property rights management or electronic commerce. As outlined above, LOM conceptual model defines a hierarchy with nine topmost categories (c.f. Table 2). LOM is described using the ISO/IEC 11404 (ISO, 1996) data structure specification. Besides, some LOM instances are under implementation using XML. Specifically, IMS and ADL versions of LOM are available in XML. 3.2. Dublin Core Metadata Element set for education Dublin Core Metadata Element Set (DCMES, DC; Hillmann, 2001) is a general-purpose and widely adopted metadata scheme targeted to resource location developed within the Dublin Core Metadata Initiative. It is compact and its elements are the result of a wide consensus. As a consequence DC has become the foundation for other initiatives (c.f. Section 3.7). In August 1999, the Dublin Core Advisory Committee (DCAC) formed the DC-Education Working Group (DC-Ed WG; Mason & Sutton, 2000) to develop and make a proposal for the use of Dublin Core metadata for the description of educational resources. Essentially, its task is to propose extensions to the DC metadata set to describe these kind of resources, taking LOM and the IMS proposal as a basis. There are five information areas of particular interest to existing DC-based educational metadata projects (c.f. Table 3). Specifically, they will include the following elements for educational purposes: Audience, Standards, Quality, InteractivityType, InteractivityLevel, and TypicalLearningTime. The last three are taken directly from the LOM specification.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

361

Table 3 DC Education interest areas Area

Scope

Users

 Grade, age, academic/vocational/training level.  Administrators of the resource.  Student audience (e.g. target group,learnimmg context, beneficiary).

Duration

Focus on use time (e.g. how long it will take to learn something) as opposed to technical duration (e.g. play time for a video tape)

Learning processes characteristics

Student groupings, teaching methods, mechanisms of assessment. learning prerequisites, interactivity type and level, material type from a didactic viewpoint, type of use in a scholastic mnihieu, difficulty, semantic density. etc.

Standards

 National and/or internatiomial curricula.  National and/or international content/process standards.  Domain specific standards and benchmarks (e.g., U.S. Departments of Labor amid Defense training benchmarks, etc.).

Quality

 Unstructured assessments (e.g., third-party reviews/annotations).  Structured assessments (i.e. assessment based on established evaluative criteria).

3.3. IMS metadata The IMS project detected that an agreement on metadata for educational resources was one of the first tasks to be considered in the learning standardization process. Since 1998, when they made a joint proposal with ARIADNE to create LOM, IMS contributes regularly to its evolution. Indeed, they are using LOM metadata in their specifications. For example, the last IMS metadata specification (Anderson & Wason, 2000; Smythe & Shepherd, 2000a, 2000b) is based on LOM version 3.5 (LTSCWG12, 1998b). However, IMS considers that the number or items defined for LOM is too large. Many organizations within the IMS community recommended that a reduced core of basic elements must be identified to simplify initial implementation efforts. IMS metadata try to make LOM metadata more flexible by providing two different specifications: IMS Core (19 LOM elements), which describes a reduced set of fundamental metadata, as considered by IMS, and IMS Standard Extension Library (IMS-SEL), which collects the remaining LOM metadata. To implement metadata, IMS recommends XML (Bray, 2000). Examples provided by IMS are written in this language, and an XML DTD is provided to validate metadata descriptions. 3.4. Gestalt GEMSTONES The metadata structure used in the first versions of the GESTALT demonstrator was based in LOM version 2.5 (LTSC WG12, 1998a). From this, they developed a new proposal that included the basic LOM data model and additional metadata attributes, named extensions, identified along the design process. This work is known as GEMSTONES (Gestalt Extensions to Metadata

362

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Standards for ON-Line Education Systems; Foster, Kraner, & Grazino, 2000). These additions to LOM included, among others, new types to define technical requirements, a richer support for personal information, external rights management, use of numeric enumerated lists to operate the vocabularies across different natural languages, definition of metadata for assessment, and the creation of a mapping category to match required element tag names to their equivalents in the learner profile specifications. However, their main contribution is the creation of a new metadata category to extensively handle quality of service aspects. Quality of Service (QoS) relates to two distinct categories, Client Configuration and Network. The first is targeted primarily to transparent content negotiation. The second one addresses network-related aspects. The QoS Metadata in GEMSTONES is based on the definitions of ATM for QoS (ATM, 2001) and IETF standards IntServ and DiffServ (IETF, 2001). Presently, the GESTALT metadata model, GEMSTONES (Foster et al. 2000), is based on LOM version 4.0 (Hodgins, 2000). Many of the LOM Version 2.5 extensions identified in GESTALT helped to shape parts of the LOM Version 4.0 data model. They continue to propose the inclusion of a QoS metadata category, together with a mapping category to corresponding PAPI fields (Farance 2000). PAPI (Public and Private Information) is a standard proposed by the LTSCthat defines the syntax and semantics of a student information model, organized around a set of completely defined records. 3.5. ADL SCORM metadata Although in most cases XML implementations are available, the above mentioned LOM-based metadata specifications do not describe how to apply metadata to particular, specific systems. SCORM has adopted the set of metadata elements described in the IMS Learning Resource Meta-data Information Model (Anderson & Wason, 2000), which in turn is based on LOM. It also references the IMS XML binding specification to validate their implementations. SCORM’s original contribution is a mapping of metadata elements into three learning content elements: raw media, content and course. In this way, they provide the missing link between general metadata specifications and specific content models. SCORM identifies three types of learning content metadata: 1. Raw media metadata are metadata that can be applied to assets such as illustrations, documents, or media streams that provide descriptive information about the assets independently of learning content. These metadata are used to facilitate reuse and discoverability, mainly during learning content creation of assets within an asset repository. 2. Content metadata can be applied to block contents or assignable units to provide descriptive information about the learning contents, independently of the particular content aggregation. These metadata are used to facilitate reuse and discoverability of such learning content within a learning content repository. 3. Course metadata describe content aggregations defined as courses, similar to the SCORM content structure format. These metadata are used to facilitate reuse and discoverability within a courseware repository, and to provide descriptive information about the content aggregation.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

363

3.6. ARIADNE metadata ARIADNE’s primary goal is to promote the sharing and reuse of electronic pedagogical material. In order to support this goal, they have built their Knowledge Pool System: a Europe-wide distributed repository for pedagogical documents (c.f. Section 4.2), with associated indexing and query tools. One of the main features of these tools is the metadata schema used. Their interest is centered in the development of a metadata system that works in a multilingual and multicultural environment, neutral with regard to both the language of the original document being indexed and the language used to create the metadata. They use a metadata scheme based in LOM 3.8 (LTSC WG12, 1999). Their metadata are grouped into five categories: general information of the resource, semantics, technical characteristics, conditions for use, and meta-metadata information. 3.7. GEM, NSDL and EdNA metadata As discussed above, GEM main objective was to provide a solution for the location and publication of educational resources available through the Internet. For this, they provide a tool, GEMcat (GEMCat, 2001), which assists the content provider in the cataloging of the resources to be made available to the public. These resources can then be accessed through the GEM gateway (GEM Gateway, 2001). This cataloging process is supported by a metadata model (Morgan, 1997) based on Dublin Core and Waldo (2001), and has been defined in collaboration with IMS (Sutton, 1998). The integration into the LOM framework is also being considered. The GEM metadata element set has 23 elements, 15 of which comprise the Dublin Core, and an array of controlled vocabularies. The eight domain specific elements added to the Dublin Core represent information important to teachers and educators searching for curricula, lesson plans and other educational materials. Examples include Grade, Essential Resources and Pedagogy. The set of metadata proposed by the NSDL Standards Workgroup (Baca, 1998) has 20 elements, and is based on Dublin Core, which contributes with its 15 elements to the set. Dublin Core Education (Audience and Standards) and IEEE’s LOM (InteractivityType, InteractivityLevel and TypicalLearningTime) provide the remaining five elements. In the same line, EdNA defined a metadata scheme (EdNA, 2001a) based on Dublin Core, extending it with eight new elements that describe information about the classification of available resources. These proposals are targeted to a specific problem: resource location and retrieval. It could be argued that their role in the standardization process is not as clear as in the previous cases. 3.8. Learning metadata compendium Figs. 2 and 3 summarize the learning metadata schemes discussed so far. LOM has not experienced any significant change since version 4.0, so its metadata are almost the same used presently by GESTALT and IMS. Minor changes in LOM since version 4.0 affect mainly to element types and descriptions.

364

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Fig. 2. Learning metadata summary (I).

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Fig. 3. Learning metadata summary (II).

365

366

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

The purpose of this compendium is to provide a reference for the learning metadata elements proposed by the organizations involved in their standardization. Except for minor exceptions, LOM subsumes all relevant metadata proposals, and has been taken as the basic reference to present all other discussed metadata schemes. Thus, we have included a mapping from the LOM elements to Dublin Core, Dublin Core Education, ARIADNE, GEM, NSDL and EdNA. Entries in Figs. 2 and 3 have the following fields: 1. Element name and type, as discussed below. 2. SCORM mapping established by ADL to identify what elements to use with the different types of resources (raw media, contents, courses). 3. Mappings to Dublin Core, Dublin Core Education, ARIADNE, GEM, NSDL and EdNA, when defined. The subdivision of IMS metadata into IMS Core and IMS SEL is denoted by a shadowed box for IMS Core metadata elements. Dublin Core metadata are identified by a thick border. Metadata not included in LOM but included in other proposals are denoted by a dotted box, whereas boxes for metadata not available in SCORM or IMS are dashed. Elements can be of one of the following basic types: 1. CharString represents a string of characters. 2. LangString is composed by two CharString elements, one to indicate the language, and the other to describe the content in that language. 3. Date has a CharString element to describe a date, and a LangString element to provide a language-dependent description. 4. Vocabulary defines the terms in a vocabulary as a LangString. It includes an additional LangString to select a certain term of such vocabulary.

4. Brokerage for educational systems Efficient searching and location services for educational contents is a key aspect of distributed, open computer-based educational systems. For this to be possible, we need an efficient description of those resources to be located. The previous section showed the main trends in the description of e-learning resources through standardized metadata. In this section, we show how different institutions use these metadata to offer services for searching and locating educational resources. In the present-day Internet environment, classical search engines likeYahoo! (2001), Altavista (2001) or Excite (2001) do not provide an adequate support for computer-based, distributed education. Digital libraries, and particularly those devoted to educational resources like Ilumina (Ilumina Project, 2001) or SMETE (SMETE.ORG, 2001), are an important step forward, whereas new techniques like metadata harvesting (Lynch, 2001) will help to speed up the generalization of metadata as the common framework for the representation of resource properties. However, as discussed in the introduction, a new approach to resource searching and discovery, based on the brokerage concept, will facilitate the introduction of new services that will definitely enrich the learning experience (Anido et al., 2001). Along the next paragraphs we discuss the most

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

367

outstanding contributions to educational resource brokerage from the institutions and organizations interested in the standardization of computer based education. First, we present the brokerage service offered by GESTALT. We will also offer some insight into the ARIADNE, GEM and EdNA approaches to brokerage. 4.1. GESTALT Within the GESTALT initiative, there exists a brokerage service for educational resources to facilitate users the selection of courses, and the preferred institution to access these courses. This service is named Resource Discovery Service (RDS), and is based on the GAIA architecture that is briefly described inSection 4.1.1. Section 4.1.2 offers a more detailed discussion about the application of the Resource Discovery Service in GESTALT. 4.1.1. Generic Architecture for Information Availability The Generic Architecture for Information Availability (GAIA Partners, 2000) was developed under the Advanced Communication Technology and Services (ACTS, 2001) initiative from the European Commission’s fourth framework program. The main objective of GAIA was to define an architecture for information access and electronic brokerage. The main goal of GAIA was to provide searching and location of services and information, content provider location, quality of service, delivery and price negotiation support, digital domain delivery, and authentication, billing and payment management. The main results of this project are discussed in the GAIA standard, including a reference model and a functional architecture for brokerage that supports scalability and diversity in contents and formats. The GAIA reference model defines four different roles: customer, broker, supplier and helper. Customers access the GAIA server to search and locate information and products. The GAIA server offers several tools to assist the customer in the searching process. Brokers serve as interfaces between customers and suppliers. They provide services and products to customers according to their explicit needs. They also fetch and provide information about price, delivery conditions, etc. Suppliers provide information to customers through brokers. They describe their products and services to their brokers to make them available to customers. Helpers provide additional services and support, like authentication, payment management or transaction security. The GAIA Reference model also defines four actions that can be performed in the system: search, locate, order and deliver:  Search occurs when a customer asks a broker to find some products or services. The result of this action is a list of unique identifiers for the products or services satisfying customer needs.  Location takes place when the customer asks the broker to provide the coordinates of a product or service. For this, the customer produces the unique identifier for the desired resource, and obtains, as a result, a list of unique identifiers for resource location, together with information about delivery conditions, price, etc.

368

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

 The Order action is triggered when a customer asks a resource, after it has been located. This action evolves through two different phases: negotiation and purchase.  The process ends with a Deliver action. Through this action, the broker delivers the acquired product. 4.1.2. Resource discovery service GESTALT’s RDS services allow users to explore which courses and modules are available from every registered institution. Following the concepts and architecture developed within the GAIA project, RDS is composed by a CORBA-compliant (Siegel, 1999) broker and some additional components to support course location. However, GESTALT does not support actions related to course ordering or delivery. RDS architecture is outlined in Fig. 4. Fig. 5 shows the relationship among RDS and other system components. RDS can be contacted through a Web gateway. Learners use available broker services to discover the existence of educational resources. It is also possible for the learner to restrict the search by providing a personal profile. User profile information is exchanged using PAPI (Farance, 2000) through the LDAP (Yeong, Howes, & Kille, 1995) directory service. RDS exchanges metadata information with the Asset Management Service (AMS), which provides access to valued educational resources and publicizes educational resources to the RDS. This service is accessed through the Asset Metadata Server, a database that contains metadata descriptions for educational resources.

Fig. 4. RDS architecture.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

369

Fig. 5. RDS integration in GESTALT.

4.2. ARIADNE’s Knowledge Pool System ARIADNE’s Knowledge Pool System (KPS) is this project’s module for resource location. KPS contains Pedagogical Elements (PEs). Each PE is composed by a Pedagogical Document (PD), which actually holds the educational contents, and a Pedagogical Header (PH), which describes the attached PD using LOM 3.8 metadata (see Section 3.1). Information about a PD provided by a PH is organized into four categories: general, technical, semantic and pedagogical aspects. To each category corresponds a set of attributes, which are used for indexing purposes. Technically, the KPS is implemented as a relational database management system (RDBMS) to manage PE. This RDBMS supports element searching, transaction handling, multiuser access and back-up. The relational approach is a suitable solution to manage the textual data in PHs. However, when KPS was designed, relational systems lacked adequate support for multimedia information and, as a consequence, PDs were stored in a classical file system. Nevertheless, migration to a full relational system is scheduled, and will occur as soon as relational databases provide adequate support for multimedia objects. To make a better use of available bandwidth, and therefore to improve quality of service, the KPS is implemented as a distributed system. Fig. 6 reflects the status of the pool in 2000. The central server is located in Leuven, Belgium, and other local servers (Local Knowledge Pools, LKP) are scattered around Europe, hosted by participating institutions. To guarantee consistence in this distributed environment, and to make the best possible use of available storage resources, three PE categories have been defined, according to different replication policies:  Standard PEs, which are accessible from any location. Corresponding PHs are available in the local server where they were first introduced, and replicated in the central server and all local servers. PDs are replicated in the central servers, and those local servers interested in the corresponding contents.

370

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Fig. 6. KPS distribution.

 Restricted PEs, which are not freely available. The replication scheme is similar to that for standard PEs, but local servers interested in a given content may replicate a PD only if they have previously acquired the corresponding licenses.  Local PEs, which are only available in a given LKP. They are not replicated. Content searching is based on predefined queries, parameterized with user-provided attribute values (general, technological, semantic, pedagogical). 4.3. GEM and EdNA approaches to resource brokerage The GEM and EdNA projects were intended to provide a tool to search and fetch educational resources available on the Internet. In both cases, resource providers should characterize their educational resources using a metadata scheme defined as an extension of Dublin Core (see Section 3.2). However, GEM plans to make its proprietary metadata scheme LOM-compliant. For GEM, metadata can be embedded into documents, as meta-tags, or made available as standalone HTML files. GEM cataloguer is responsible for deciding which of the new proposed resources should be made available through the GEM system. Cataloguing of new documents is made using the GEM cataloguing tool, named GEMcat (2001), which provides an easy way to create GEM metadata for the new documents. EdNA provides a metadata editor (EdNA Metadata Editor) to insert metadata into HTML documents. Instead of the GEM’s approval procedure for open proposals, EdNA restricts the entities allowed to introduce new resources to some designated institutions or individuals. As for ARIADNE’s KPS, information is stored in a relational database, but in this case information is kept in a single central server. Queries return resource records reflecting available metadata information: title, description, keywords, format, data, language, location, etc.

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

371

5. Conclusions Educational technologies standardization is a challenging process. There are many issues that have to be addressed. Many different institutions and organizations are contributing to this. From a strictly methodological point of view, this field is a melting pot where traditional approaches to standardization, driven by rigid, slow-reacting, classical standardization bodies producing recommendations from scratch, merge with new approaches driven by users or industry, which produce standards from already available expertise and products. Eventually, some of the contributions presented in this paper will become generally accepted standards or recommendations. For this, collaboration among actors will be of paramount importance. This trend can be observed even at this moment. With respect to educational metadata, Learning Object Metadata (LOM) will likely play this role. It is the reference metadata scheme for many other proposals, and LOM interoperability seems to be a must for other metadata approaches. Nevertheless, there is a debate among organizations interested in an extensive set of metadata, trying to support the resource description needs for the present and future, and institutions, mainly from industry, interested in a relatively reduced set of metadata elements needed by typical present-day applications, to be able to provide metadata-enhanced versions of their products as soon as possible. This controversy is reflected by the IMS initiative, which decided to divide LOM metadata into two groups, a reduced set of metadata elements considered fundamental, and the rest. Other approaches adapt LOM to their particular needs, and also contribute to the enrichment of the LOM proposal. Ongoing work on educational metadata in Dublin Core is also promising. They have a large experience in the field of metadata standardization, and it is very that likely their proposals will be taken into account. Brokerage, as defined here, is implemented only by GESTALT. Other systems, like GEM, ARIADNE or EdNA, offer a front end to access a relational database where resources are maintained. For GESTALT, searches imply a three-step procedure where the broker role is clearly identified. The broker receives student queries, and tries to find the corresponding resources in other modules. When resources are located, the broker returns them to the user who made the query. This intermediation step (i.e. collecting available resources in other modules on behalf of the querying user, to offer consolidated results to that user) is clearly missing in other approaches. The basic difference between GEM and ARIADNE is the distributed nature of ARIADNE. The relationship between brokerage and metadata definitions is obvious. All these systems have their own educational metadata schemes or contributed to the definition of external specifications. In a near future, it is expected that brokerage systems will add a new layer of interoperability by offering open software interfaces to be used by external systems to automatically update or query the broker system.

Acknowledgements Authors would like to thank Diego Conde-Pe´rez, Fernando Go´mez-Ferna´ndez, Jorge M. Domı´nguez-Rodrı´guez and Juan Aguiar-Ferna´ndez for their contribution to this work. We

372

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

would also like to thank an anonymous reviewer for his/her suggestions to improve this paper. This work has been partially supported by the European Union under grants FEDER 1FD97– 0100 and 1FD97–0282.

References ACTS. (2001). Advanced Communication Technology and Services. WWW site. http://www.infowin.org/ACTS. [Last access 16 November 2001]. Adam, N. R., & Atluri, I. A. V. (2000). SI in digital libraries. Communications of the ACM, 43(6), 64–72. ADL. (2001). US Department of Defense, Advanced Distributed Learning (ADL)initiative. WWW site. http:// www.adlnet.org. (Last access November 16th, 2001). AICC. (2001). Aviation Industry Computer Based Training Committee. WWW site. http://www.aicc.org. [Last access 16 November 2001]. Altavista. (2001). Altavista Search Engine. WWW site. http://www.altavista.com. [Last access 16 November 2001]. Anderson, T., & Wason, T. (2000). IMS Learning Resource Meta-data Information Model. Version 1.1. Technical report (June), IMS Global Learning Consortium, Inc. Available at http://www.imsproject.org/metadata/mdinfov1p1.html. [Last access 16 November 2001]. Anido, L., Rodrı´guez, J., Ferna´ndez, M., Caeiro, M., Santos, J., Llamas, M., & Pavo´n, P. (2001). Educational brokerage using standardized metadata and software interfaces. In Procs. of the UICEE 2001 Conference (2001), pp. 301–305. UNESCO: UNESCO-ICEE. ARIADNE. (2001). The Alliance of Remote Instructional Authoring and Distribution Networks for Europe. WWW site. http://ariadne.unil.ch. [Last access 16 November 2001]. ATM. (2001). The Asynchronous Transfer Mode (ATM) forum. WWW site. http://www.atmforum.com. [Last access 16 November 2001]. Baca, M. Ed. (1998). Introduction to metadata. Pathway to digital information. J. Paul Getty Trust. Bray, T. (2000). Extensible Markup Language (XML). Technical report, W3 Consortium. Available at http:// www.w3c.org/xml/REC-xml-20001006. [Last access 16 November 2001]. CEN/ISSS/LT. (2001). European Committee for Standardization (CEN), Information Society Standardization Systems (ISSS), Learning Technologies Workshop (LT). WWW site. http://www.cenorm.be/isss/Workshop/lt/. [Last access 16 November 2001]. Day, M. (2001). Metadata in a nutshell. Information Europe, 6, 2. DC. (2001). Dublin Core Metadata Initiative. WWW site. http://purl.org/dc. [Last access 16 November 2001]. Dodds, P. (2001). ADL Sharable Content Object Reference Model (SCORM). Version 1.1. Technical report (January), Advanced Distributed Learning Initiative. Available at http://www.adlnet.org/Scorm/docs/SCORM1.1.zip. [Last access 16 November 2001]. EdNA. (2001a). EdNA Metadata Homepage. WWW site. http://standards.edna.edu.au/metadata/index.html. [Last access 16 November 2001]. EdNA. (2001b). Educational Network Australia (EdNA). WWW site. http://www.edna.edu.au. [Last access 16 November 2001]. Excite. (2001). Excite Search Engine. WWW site. http://www.excite.com. [Last access 16 November 2001]. Farance, F. (2000). Draft standard for learning technology. Public and Private Information (PAPI) for Learners (PAPI Learner). Version 7.0. Technical report (June), Institute of Electrical an Electronics Engineers, Inc. Available at http://ltsc.ieee.org/wg2/papi_learner_06.doc. [Last access 16 November 2001]. Foster, P., Kraner, M., & Graziano, A. (2000). Couseware metadata design. Gestalt extensions to metadata standards for on-line education systems (GEMSTONES). Technical report, Fretwell-Downing Education. Available at http:// www.fdgroup.co.uk/gestalt/D0401_3.pdf. (Last access November 16th, 2001). GAIA Partners. (2000). GAIA, Generic Architecture for Information Availability. Technical report (March), FretwellDowning. Available at http://www.infowin.org/ACTS/RUS/PROJECTS/FINAL-REPORTS/fr-221.pdf. [Last access 16 November 2001].

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

373

GEM. (2001). Gateway to Educational Materials. WWW site. http://www.geminfo.org. [Last access 16 November 2001]. GEM Gateway. (2001). The Gateway to Educational Materials. WWW site. http://www.thegateway.org/. [Last access 16 November 2001]. GEMCat. (2001). GEM Cataloguing Module. WWW site. http://www.geminfo.org/Workbench/Cataloging/ index.html. [Last access 16 November 2001]. GESTALT. (2001). Getting educational system talking across leading. edge technologies project. WWW site. http:// www.fdgroup.co.uk/gestalt. [Last access 16 November 2001]. Hillmann, D. (2001). Using dublin core. DublinCore Metadata Initiative. Available at http://dublincore.org/documents/2001/04/12/usageguide/. [Last access 16 November 2001]. Hodgins, W. (2000). Draft standard for learning object metadata (LOM). Proposed draft 4.0. Technical report (February), Institute of Electrical and Electronics Engineers, Inc. Available at http://ltsc.ieee.org/p1484/doc/wg12/ LOM_WD4.htm. [Last access 16 November 2001]. Hodgins, W. (2001). Draft standard for learning object metadata (LOM) specification. Proposed Draft 6.0. Technical report (February), Institute of Electrical and Electronics Engineers, Inc. Available at http://ltsc.ieee.org/doc/wg12/ LOM_WD5.doc. [Last access 16 November 2001]. IETF. (2001). The Internet Engineering Task Force. WWW site. http://www.ietf.org. [Last access 16 November 2001]. Ilumina Project. (2001). Ilumina Digital Library. WWW site. http://www.ilumina-project.org. [Last access 16 November 2001]. IMS. (2001). IMS Global Learning Consortium. WWW site. http://www.imsproject.org. [Last access 16 November 2001]. ISO. (1996). Information technology. Programming languages, their environments and system software interfaces. Language-independent datatypes. Technical report, International Standards Organization. Available at http://www.iso.ch/cate/d19346.html. [Last access 16 November 2001]. ISO/IEC. (2001). International Standardization Organization/Institute Electrotechnical Commission Committee for Learning Technologies (ISO/IEC JTC1 SC36). WWW site. http://www.jtc1sc36.org. [Last access 16 November 2001]. LTSC. (2001). Learning Technologies Standardization Committee. WWW site. http://ltsc.ieee.org. [Last access 16 November 2001]. LTSC WG12. (1998a). IEEE Learning Technology Standards Committee (LTSC). Learning object metadata (LOM). Version 2.5. Technical report (December), IEEE, Inc. Available at http://ltsc.ieee.org/p1484/doc/wg12/LOMdoc2_5.doc. [Last access 16 November 2001]. LTSC WG12. (1998b). Learning object metadata (LOM): Base Scheme. Version 3.5. Technical report (December), IEEE, Inc. Available at http://ltsc.ieee.org/doc/wg12/scheme.html. [Last access 16 November 2001]. LTSC WG12. (1999). IEEE Learning Technology Standards Committee (LTSC). Learning object metadata (LOM). Working Draft 3.8. Technical report (November), IEEE, Inc. Available at http://ltsc.ieee.org/p1484/doc/wg12/ LOM3.8.doc. [Last access 16 November 2001]. Lynch, C. A. (2001). Metadata harvesting and the open archives initiative. ARL Bimonthly Report 217. Mason, T., & Sutton, S. (2000). Dublin Core Education Working Group: draft proposal. Technical report, DC Metadata Initiative. Available at http://dublincore.org/groups/education/. [Last access 16 November 2001]. Morgan, N. (1997). Proposal: DC-5 GEM project presentation. WWW site. http://www.geminfo.org/Genesis/1997/ Dublin_core/DC5_proposal.html. [Last access 16 November 2001]. Powell, J., & Fox, E. A. 1998. Multilingual federated searching across heterogeneous collections. D-Lib Magazine. PROMETEUS. (2001). PROmoting Multimedia access to Education and Training in EUropean Society. WWW site. http://prometeus.org. [Last access 16 November 2001]. Schwartz, C. (1998). Web search engines. Journal of the American Society for Information Science, 49(11), 973–982. Sharon, T., & Frank, A. J. 2000. Digital libraries on the internet. In 66th. IFLA Council and General Conference (August 2000). Siegel, J. (1999). CORBA 3 fundamentals and programming. Wiley. SMETE.ORG. (2001). SMETE Digital Library. WWW site. http://www.smete.org. [Last access 16 November 2001]. Smythe, C., & Shepherd, E. (2000a). IMS Learning Resource XML Binding Specification. Version 1.1. Technical report (June), IMS Global Learning Consortium, Inc. Available at http://www.imsproject.org/metadata/ mdbindv1p1.html. [Last access 16 November 2001].

374

L.E. Anido et al. / Computers & Education 38 (2002) 351–374

Smythe, C., & Shepherd, E. (2000b). IMS Meta-Data Best Practice and Implementation Guide. Version 1.1. Technical report (June), IMS Global Learning Consortium, Inc. Available at http://www.imsproject.org/metadata/ mdbestv1p1.html. [Last access 16 November 2001]. Sutton, S. A. (1998). Gateway to educational materials (GEM): metadata for networked information discovery and retrieval. Computer-Networks-and-ISDN-Systems 30, 1–7, 691–693. Sutton, S. A., Lankes, R. D., Small, R. V., & Eisenberg, M. B. (1998). Networked information discovery and retrieval for educational materials on the internet: metadata development, deployment, and evolution. In Proceedings of the 61st Annual Meeting of the American Society for Information Science, Volume 35 (1998), pp. 54–62. Inf. Today. Task Force on Metadata. (1999). Committee on cataloging. description and access. Technical report (June), Association for Library Collections and Technical Services. Available at http://www.ala.org/alcts/organization/ccs/ccda/tfmeta3.html. [Last access 16 November 2001]. Waldo. (2001). Waldo MetaSearch. WWW site. http://waldo.rtec.org/. [Last access 16 November 2001]. Wattenberg, F. 1998. A national digital library for science, mathematics, engineering, and technology education. D-Lib Magazine. Yahoo. (2001). Yahoo Search Engine. WWW site. http://www.yahoo.com. [Last access 16 November 2001]. Yeong, W., Howes, T., & Kille, S. (1995). Lightweight directory access protocol. Technical report (March), IETF. Available at http://www.ietf.org/rfc/rfc1777.txt. [Last access 16 November 2001]. Yoon, J. P., & Kim, S. (2000). Schema extraction for multimedia xml document retrieval. In Procs. of the 1st International Conference on Web Information Systems Engineering (WISE’00) (2000). IEEE Press.