From Multimedia Micro-University to Macro University and beyond

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

Contents lists available at ScienceDirect

Journal of Visual Languages and Computing journal homepage: www.elsevier.com/locate/jvlc

From Multimedia Micro-University to Macro University and beyond ⁎

Timothy Arndta, , Angela Guerciob a Department of Electrical Engineering and Computer Science and Department of Information Systems, Cleveland State University, Cleveland, OH 44115, USA b Department of Computer Science, Kent State University-Stark, North Canton, OH 44720, USA

A R T I C L E I N F O

A BS T RAC T

Keywords: Distance education Multimedia learning Educational materials transformations

For this special issue of the Journal of Visual Languages and Computing in honor of Prof. Shi-Kuo Chang we review Prof. Chang's research contributions to the field of education. Always visionary and inspirational, these works reflect Prof. Chang's student-centered approach and his interest in building relationships among international educators in order to bring the benefits of such collaborations among educators to the students while keeping in mind the motto “Small is beautiful”.

1. Introduction Prof. Shi-Kuo Chang is best known to the research community for his contributions to the fields of visual language and image database systems – including starting the first scientific conference on visual languages and founding the Journal of Visual Languages and Computing. He has made major contributions to many other areas however, and, in particular, as part of his long career as an educator – more than 35 years as a professor of computer science in U.S. universities including terms of service as Chair of department at both Illinois Institute of Technology and the University of Pittsburgh – education has long been an area of interest. He has performed research with education as the central topic as well as applying his research results from other areas in the education domain. Those of us who have been his students and collaborators know Prof. Chang's passion for education and we might point to his founding of the Knowledge Systems Institute (KSI) as an institution for teaching graduate level computer science using the principles he holds dear – the personal touch and the introduction of students to cutting-edge research, encouraging them to get involved in the same. In looking at the corpus of Prof. Chang's research in the field of education, a couple of themes shine through – first enabling the means to build international collaborations in teaching and research which will benefit students by exposing them to multiple points of view and emphases from international experts in their chosen field of specialization. This theme, which was evident more than 30 years ago when Prof. Chang brought together a group of researchers from many countries to collaborate in research in visual languages at the early visual languages conferences, is increasingly important in todays diverse and globalized education environments. Second, while wanting to leverage the possi-

⁎

bilities brought about by the rise of the Internet and distance education, Prof. Chang has always been concerned not to lose sight of the individual student in this online environment, thus providing a customized and personalized learning experience is necessary in order that the student not be overwhelmed and in order to stick to Prof. Chang's education motto “Small is beautiful”. Of course, as one of the pioneers of research in the area of visual languages, the themes of visual interfaces, multimedia, and visual languages and grammars are all interwoven in the research in learning carried out by Prof. Chang. In the rest of this paper we will provide an overview (not exhaustive) of Prof. Chang's research work in the education area, at times indicating how subsequent developments have vindicated the previous research, while at other times technology has driven practice in somewhat different directions, we will show how the parts fit together, and we will conclude with suggestions for further research in this area inspired by Prof. Chang's work. 2. Multimedia Micro-University From his earliest research, Prof. Chang has shown an interest in how computer technology can improve education (two of these early efforts are [1,2]). The first of his education research efforts in the modern Internet age concerned what he termed a Multimedia MicroUniversity (MMU) [3]. The Multimedia Micro-University is built-on the tele-action object (TAO) formalism [4]. TAOs in turn build on the active index formalism [5], where an active index consists of a number of index cells (ICs). The tele-action object formalism is introduced in more detail in Section 3. We will see the active index/index cells used again in the sentient map described in Section 3.1, and also in the research described in Section 6.

Corresponding author.

http://dx.doi.org/10.1016/j.jvlc.2016.10.002 Received 6 December 2015; Received in revised form 21 April 2016; Accepted 14 October 2016 Available online xxxx 1045-926X/ © 2016 Elsevier Ltd. All rights reserved.

Please cite this article as: Arndt, T., Journal of Visual Languages and Computing (2016), http://dx.doi.org/10.1016/j.jvlc.2016.10.002

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

different manner in existing LMSs. First, the administrative functions of the Micro-University are often incorporated in separate software tools, while in the research described they are developed on the same Micro-University paradigm. The research also incorporates knowledge through the TAO and knowledge table paradigms which allows for personalized learning experiences which supports the “Small is beautiful” paradigm which is often missing in today's gigantic online distance learning classes. The incorporated knowledge structures also support planning in a unified and organic manner.

A Micro-university is a teaching or learning environment for an individual teacher/student, designed to optimally serve the needs of that teacher or student. This contrasts with a virtual university which consists of one or more MMUs and embodies a collection of personalized learning environments for a single student interacting with a (virtual) teacher. In the extreme case a virtual university is the same as a single Micro-University, but usually a virtual university contains many Micro-Universities and performs many administrative functions supported by a virtual administration office. The administrative tools include software tools for managing users, curriculum, facilities, resources, plans, etc. Each of the above tools can be used by authorized users. MMUs can be small, medium or large ranging from a single software module to a complex configuration of software modules. Example MMUs include: Virtual Course-room, Virtual Collaboration Room, Virtual Laboratory, Virtual Library and Virtual Private Office, which operate on a dynamically changing collection of tele-action objects (TAOs) [4]. The Multimedia Micro-University was developed at the Knowledge Systems Institute (KSI) which was founded by Prof. Chang in 1978 to support research and professional education. KSI maintains a limited enrollment and offers courses on evenings and weekends and through distance learning. These characteristics influenced the design of the Micro-University to support KSI's education program and management operations. The components of the multimedia Micro-University consisted of a virtual library, an intelligent distance learning system and visualization and planning tools for the school's operations. The virtual library was implemented with a WYSIWYG paradigm and supported both keyword-based retrieval and visualization of the titles in the (physical) KSI library. The intelligent distance learning system is a complex finite-state machine which uses a knowledge base to model the educational process. It serves as an intelligent communication channel between the author of distance learning materials and the cooperative recipient. Different user roles have different interpretations of the system states, and the system states correspond to four application programs – the Tutor, the Advisor, the Publisher, and the Administrator. The Tutor program provides the course material to the student and then assesses mastery by means of online quizzes. The Advisor facilitates communication between student and teacher. The Administrator mediates and facilitates the logistics of the education process for administrators. The Publisher moves learning materials to a server, updating configuration tables along the way. The KSI distance learning system was supplemented with multimedia e-books incorporating multimedia elements such as audio and video as well as interactivity. The administration functions for KSI were facilitated by Multimedia Micro-Universities as well, in the form of visualization tools which give an overview of the school's current status. The whole system is developed using HTML, Java applets and CGI programs. Planning support is provided through a structured collection of knowledge tables [6] representing goals, plans, expected outcomes, status, and related plans. The knowledge tables are continuously created, updated, evaluated, and revised by key administrators. A tool called the Knowledge Table Editor supports knowledge-table-based planning and evaluation. The Micro-University was designed, implemented and evaluated by careful measurement of graduate student academic achievement. This achievement was measured quantitatively as part of a preliminary assessment which demonstrated that the Micro-University components already implemented contributed to the learning process at KSI. It can be seen that much of the functionality of the Multimedia Micro-University has been implemented in Learning Management Systems (LMSs) such as Blackboard Learn and Moodle which have been developed since the research described in this section was carried out. However, it can also be seen that several features of the multimedia Micro-University concept are either lacking or implemented in

3. Macro University Following the successful introduction of the Multimedia MicroUniversity, the vision was extended from a single institution to a federation of universities in order to provide the best opportunities for learning and research for students by exposing them to an international array of experts. Technical and financial considerations in the late 1990s led many academic institutions to set up a web site and start a distance learning program on the Internet. Coupled with a digital library, many academic institutions experimented with the concept of a virtual university. In some countries the local government takes the initiative to form a consortium of universities offering online courses from each institution. These consortia are sometimes called a virtual university or virtual campus. Thus the term “virtual university” sometimes means the distance learning courses of a single academic program, at other times to the distance learning program of an academic institution, and still others to a consortium of distance learning programs. In [7], the authors argue that establishment of such programs can give administrators of academic institutions unrealistic expectations. Some administrators believe once they have a virtual university, they can greatly reduce the number of instructors and thus become more cost effective. However, they fail to realize the importance of providing an effective learning environment on the Internet, which is not an easy task. Just hosting some courses on the Internet does not provide a learning environment. The students can easily get confused and disoriented, if left alone on the Internet. Beneath the virtual university, there needs to be another layer, offering a student a personalized learning environment. This is where the previously developed concept of the Multimedia Micro-University becomes important. At the same time, many visionaries have hinted at the possibility of combining the educational resources of a large number of academic institutions, thus creating a super rich learning environment. Without losing sight of the individual student's micro-universe, it is hoped that the coupling and coalition of numerous academic institutions will constitute a macro-universe for the students. Based on these concepts, Chang and his colleagues introduced a new framework for distance learning consisting of three layers: the Micro-University, the virtual university, and lastly the Macro University. A Micro-University, as described in the previous section, is a self-contained learning environment, usually on a single PC (or notebook, tablet, smartphone, etc.), for a single student interacting with a (virtual) teacher. A Micro-University is designed to serve a single student, but many instances of the same Micro-University can be instantiated to serve many students. A virtual university is a collection of personalized learning environments so that students can engage in learning activities from home, remote locations, etc. A virtual university is usually owned and operated by an academic institution and therefore has more administrative functions than a Micro-University. The Macro University was developed to be a framework such that multiple virtual universities could pool their resources together, thus creating a very rich learning environment for a large number of students worldwide [7]. Macro University was an international consortium of virtual universities. As such there was a wide variety of computational platforms used by students of Macro University. In order to minimize the problems associated with exchange of data in a 2

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

member institutions.

heterogeneous computing environment, an open standard for information exchange in the Macro University framework was required. An emerging standard (which has since become well established) for Electronic Data Interchange (EDI) — XML, eXtensible Markup Language was chosen. Virtual universities are structured as a collection of MMUs where each MMU is a collection of TAOs. TAOs are described by TAOML [8], and TAOML is itself an XML application. In Macro University the TAOs represent the salient objects for a distance learning environment: class slides; recorded and live lectures; exams; syllabi; recorded and live demos; etc. Each of these objects has its own characteristics which need to be described in order to present the TAO at the student's computer. The formal specification of objects is based upon tele-action objects (TAOs). Each tele-action object is a dual representation (G, K) consisting of a hypergraph structure G and a knowledge structure K [4]. The hypergraph G is used to describe the connections and relations between the sub-TAOs, and the knowledge structure K the actions and how to synchronize the actions. A formal model called the index cell that combines the desirable features of finite state diagram and Petri net is used to specify the knowledge structure. The presentation of the multimedia document, the hypergraph component and the knowledge component are specified using a Tele-action Object Markup Language (TAOML) [9]. A multimedia development environment called MICE (Multimedia IC developer's Environment) supports the prototyping of multimedia applications using formal specification based upon TAOs. The front-end to this environment is a visual specification tool that allows the user to specify a TAO in a user-friendly way and that automatically converts the specification to TAOML. TAOML was defined as an XML application. A DTD (Document Type Definition) contains the elements and the attributes necessary for the specification of a multimedia system described via TAOs. At least eleven universities and corporations participated in the Macro University experiment in distance education. It is interesting to note that since the Macro University was described around the year 2000, a standard for LMS materials based on XML has been developed – SCORM (Sharable Content Object Reference Model) – as an initiative of the Office of the United States Secretary of Defense [10]. This validates the choice of Macro University to use an XML for content interchange, although the TAOML described above is in many ways more advanced than the well known SCORM (Shared Content Object Reference Model) standard. The Experience API, also known as the Tin Can API, [11] is a more recent standard which is the successor to SCORM and which offers greater functionality. The Tin Can API is a web service that uses JavaScript Object Notation (JSON) for its data format. Tin Can has considerably more functionality than SCORM, including the ability to work outside of the web browser, ability to use mobile apps for learning, platform transition (e.g. computer to smart phone) and greater security. It would be interesting to see if a mapping of (at least a portion) of TAOML to and from Experience/Tin Can could be defined. An organization reminiscent of the Macro University – the Open Education Resources university (OERu) [12] - was founded in 2011 with support from UNESCO. OERu has a number of academic members who offer free courses online to students around the world. The goal of this project is expand educational opportunities to students who may have been excluded either because of their geographic location or because of the expense of attending university. The idea is to make use of free open educational resources (open courses and materials) from accredited universities. The only cost to the student is if/when he or she decides to be assessed for a course in order to gain academic credit. The academic credit could then be used at an OERu member institution or some other institution in order to gain academic credentials for the learner. The OERu emphasis may be seen to be more on affordability and accessibility while the Macro University project was more focused on providing personalized learning experiences for individual students by choosing from courses offered by experts at

3.1. Sentient map for Macro University One of the problems raised by distributed e-learning environments such as Macro University is that the user may be unaware of the location and/or existence of instructional resources and information items which may be stored in heterogeneous databases and data sources. This makes it difficult for a novice user to formulate spatial/ temporal queries in order to locate the required information items. To overcome those difficulties, the use of the sentient map is proposed [13]. The sentient map provides a uniform presentation of real space/ time or virtual space/time which allows the user to interact via the same set of gestures in either context [14]. The search requests communicated by those gestures are then transformed automatically into queries targeted at a specific information space. The retrieved data is then also automatically transformed according to the user's current interest. A sentient map has a type (geographical map, directory page, web page, etc.), a profile consisting of a set of creator-specified attributes, a visual appearance defined by a visual sentence defined by a visual grammar [15], and a collection of index cells [5]. The combination of index cell and visual appearance defines a tele-action object (TAO) [4]. A set of gestures is defined to create a gesture-based interface for interaction. The gestures are then transformed into an intermediate representation known as a σ-query for spatial/temporal information retrieval. The exact σ-query generated by a particular gesture query depends on the data model or models being used. Three basic gestures are used by the system: a finger gesture which is used to point at an object on the sentient map; a moving finger gesture which is used to indicate a time-line on the sentient map; a moving finger gesture which is used to indicate a boundary for a space on the sentient map. A gesture query is composed of these three basic gestures for point, time-line and space. The gestures are represented by mouse clicks and keystrokes called c-gestures. The σ-query is an intermediate representation in transforming gestures to queries. The σ-query has a SQL-like syntax, however it introduces a ‘CLUSTER’ keyword in the SELECT clause of the query. CLUSTER indicates that objects belonging to the same cluster must share some common characteristics. The result of a σ-query is clusterstring which describes the spatial/temporal relationships among the clusters [14]. Executing a σ-query results in a number of operators being executed to resolve the query against a distributed multimedia database or sensor inputs. The query engine applies input filtering, indexing and data transformation. The execution of a query can result in the modification of the multimedia database or it can provide useful feedback to the user by means of appropriate result visualizations. A prototype sentient map system for a virtual conference room was developed at Carnegie Mellon University. Users are represented by avatars in the virtual conference room which has a sentient map on the wall. There are three different possible maps that can be chosen. The avatars formulate gesture queries against the sentient map which are transformed to the intermediate and then final query results. The prototype system was also applied to the Macro University project. The avatars in this context represent students and instructors and gesture queries can be used to query e-learning courses offered anywhere in the world by a member of the Macro University project. A preliminary user study was carried out by means of questionnaires with a small group of users who confirmed the ease of use of the sentient map paradigm but requested more expressive gesture queries as well as better feedback in the form of SQL queries. This study was carried out before hardware for motion/gesture recognition such as Microsoft Kinect was widely available, so even better results might be found with a present-day implementation. Gesture-based interfaces for touch screen devices have also become commonly recently. These would provide another 3

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

administrative office. The objectives of developing the Growing Book were to: 1) share resources in developing distance learning materials, 2) share experiences through the teaching of a common distance learning course, 3) test and evaluate the distance learning and/or administrative tools, and 4) discover problems and possible solutions in distance learning. The Growing Book supported multi-level, multi-lingual and multi-modal usage of common learning materials.

possible platform for implementation of a system similar to the one presented in this section.

4. The Growing Book The Macro University project was a model for combining the educational resources from a large number of academic institutions, thus creating a rich learning environment. The project was built on the formerly proposed Multimedia Micro-University concept (Macro University was a collection of Micro-Universities). The next step in the evolution of this model was the Growing Book – a Growing Book is just a specialized type of Micro-University which will be described in this section. Without losing sight of the individual student’s needs, it was hoped that the coupling and coalition of academic institutions in the Macro University would constitute an ideal learning environment for the students. However, to provide an effective distributed learning environment through a consortium of institutions raises many technical problems. Just offering a set of courses on the Internet does not provide an effective distributed learning environment. The students can easily get confused and disoriented, if left alone on the Internet (some ideas on using the concept of design patterns from objectoriented software engineering in order to provide a standard way for students to interact with a distance learning course in order to avoid this disorientation are discussed in the study reported in [16]). Beneath the virtual university, there needs to be another layer, offering the students a personalized and personable learning environment. These considerations led to the Growing Book project [17,18]. A Growing Book is an electronic book co-developed by a group of international instructors who collaborate in teaching and research through the Macro University. Since the learning materials they use are constantly evolving, the Growing Book must be frequently updated and expanded, leading to its name. The Growing Book is used by each teacher both locally and in the distance learning environment. The various chapters of the Growing Book are owned by different instructors who may utilize and/or provide different tools for distance learning, self learning and assessment (note the difference between this ‘owner’ concept and the related Wiki concept in which any user may edit and/or modify contenet). The Growing Book was to be accessed in the distributed learning environment by learners with diverse linguistic skills, cultural backgrounds and perceptual preferences, and also can be used for teaching and research. The structure of the Macro University incorporating the Growing Book is shown in Fig. 1 which shows the various components of Macro University incorporating Growing Books and how they fit together. Macro University consists of several virtual universities, where each virtual university contains one or more Micro-Universities. The MicroUniversity can be a Growing Book since a Growing Book is just a particular type of Micro-University. A virtual university can also perform many complex administrative functions supported by a virtual

(a) Multi-level usage: The same learning materials can be used for courses at various levels (introductory, advanced undergraduate, graduate, etc.). Therefore, multi-level usage is necessary for the Growing Book. Multi-level usage of the Growing Book, is supported by a formal specification of the type of objects to be managed through the tele-action object formalism [4] – the same formalism used in developing the multimedia Micro-University. Part of the specification of the tele-action object can describe how advanced the learning materials specified by the TAO are (beginner, intermediate, advanced, etc.) and if a different level is required, it can be searched for using this same description (a type of meta-data). This also facilitates multi-lingual and multimodal usage of the Growing Book. (b) Multi-lingual usage: The same learning materials can be transformed into different languages so that the presentation of the Growing Book is multi-lingual. Language translation functions implemented by commercial language translators associated with the tele-action objects are used to transform the tele-action objects into different languages. (c) Multi-modal usage: The same learning materials can be used by physically challenged people or people with different perceptual preferences, so that the presentation of the Growing Book is multimodal. Perceptual translation functions transform the tele-action objects into different media. The focus is on the visualization of tele-action objects and the development of gesture-oriented interface because it can be used by people with hearing disabilities and shared by people with different languages and cultures. An example of a perceptual translation function is the function which translates gestures to queries described in Section 3.1. The prototype Growing Book was intended as a textbook for an undergraduate course on data structures and algorithms. The prototype Growing Book had the following characteristics: 1) learning materials accessible with a browser or customized interface with common look-and-feel and common buttons, 2) common assessment tools, 3) individual tools downloadable as plug-ins, 4) common programming examples, 5) adaptive learning with embedded audio/ video clips. In the Growing Book project, the student feels himself to be actively driving his course by requesting explanations, special documents, homework corrections, etc. In this sense interactivity is a basic issue

Fig. 1. – Architecture of the Macro University.

4

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

developed peer-reviewed textbooks for introductory college courses. The textbooks are open textbooks and are available in both hardcopy and digital formats (like the Growing Book's Data Structures text). The open textbook format is a book licensed under an open copyright license in order to increase access and affordability of the text. Affordability was not listed as the major concern of the Growing Book project, however the open copyright license does allow end-users to modify (personalize) the content, which would enable the type of personalization that the Growing Book foresaw. Even more closely aligned with the Growing Book goals is the FlexBook System [23] developed by the CK-12 Foundation. FlexBook is an authoring platform for creating, assembling and distributing interactive, multi-modal content for K-12 students (so the target audience is different, since Growing Book is aimed at university students). FlexBooks are fully customizable and incorporate interactive learning objects (multimedia objects, like the TAOs of the Growing Book). Flexbook provides a webbased collaboration model where users can combine and personalize learning content to produce customized textbooks which can be produced in multiple electronic formats.

in the Growing Book model. Interactivity is emphasized in managing all the different type of documents (images, text, video clips, audio, etc.), reflecting a teaching/learning communication model. Students utilize an interactive language, or a series of communication tools (triggered as events on specific parts of the multimedia documents) allowing them to interact and communicate in a variety of ways. There is a richer learning experience with respect to the standard series of documents as, for example, when a new view is obtained by the processing of a query in a relational database system. In order to support multi-level, multi-lingual and multi-modal usage of the Growing Book and to facilitate the exchange of information in a heterogeneous computing environment, a formal specification of objects and an open standard for information exchange are required. Chang [17] postulates that students, teachers and authors all need an interactive language built upon some basic operations (triggered as events on specific and specified parts of the multimedia documents) that simplify their response with respect to novel concepts in a given course, retrieving extra documents from virtual libraries, or simply communicating with other students and/or other teachers. In this way there may be an enrichment with respect to a standard series of documents as when a new view is obtained with respect to the processing of a query in a given database management system. The basic operations defined on the Growing Book are divided into several groups. a) Operations for Multi-level, Multimedia Customization: The first set of operations supports Matching, Abstraction, Weaving and Customization of multimedia documents. These are called MAWC operations [19]. The matching operation can be used, for example, to retrieve all documents from a virtual library that match a selected paragraph in the Growing Book by keyword matching. The abstraction operation could be used to create an abstraction of a chapter of the Growing Book at a certain level. The weave operation can be used to weave selected pieces in the Growing Book into a presentation stream. The customize operation can be used to create a customized version of the Growing Book from the objects which were abstracted and weaved using the previously described operations. b) Operations for Increasing/Updating Awareness: The user can specify an awareness vector, so that he/she can be informed about certain events. The awareness vector is a binary vector where each entry indicates the absence/presence of an awareness attribute. A user can also set privacy, so that he is not included in any awareness information activity. c) Operations for Communication: Communication operations are for sending messages to authors, instructors and students. A user may not know their exact names and/or e-mail addresses, but he can still send messages to the group of people he wants to communicate with. d) Operations for Watermarking: Watermarks can be added to or displayed for a multimedia document. e) Operations for Managing the Growing Book: There are many operations for gathering statistics about and managing the Growing Book. The Growing Book operations can be seen as commands for the customized IC (Index Cell) Manager (described in [5]) of the Growing Book. When the user submits a command consistsing of a name and its parameters to the Growing Book, the customized IC Manager processes this command. The name is treated as a message type by the IC Manager to be passed on with the parameters to the appropriate IC for processing. The Growing Book operations are implemented as “actions” (implemented as C programs) of the ICs managed by the IC Manager. The Growing Book project resulted in a collaboratively developed textbook on data structures and algorithms which was used by several members of the Macro University consortium. A hard copy of the resulting text was also published [20]. Given modern day technologies, it is easy to imagine incorporating Wiki technology [21] into the Growing Book project. Several related textbook initiatives have been developed since the Growing Book project. OpenStax College [22] initiated by OpenStax CNX at Rice University, with support from major foundations including the Bill and Melinda Gates Foundation, has

5. Transformation of distance learning materials The previous section described a Growing Book which is created by an international team or teachers and which has the goal of being multi-level, multi-lingual, and multi-modal – in other words, the Growing Book should support rich personalization for the needs of particular students. The MAWC operations on the Growing Book are the means to achieve this customization and can be implemented as commands for the customized IC manager. In this section we look at how those operations by the IC manager on the Tele-Action Objects (TAOs) which make up the Growing Book can be implemented [24]. Since Micro-Universities are just a generalization of the Growing Book, all of the transformation techniques described in this section apply to MMUs as well. Interoperability of diverse hardware and software platforms is one of the most pressing needs in computing today. This problem is of particular relevance in academic environments where the software systems and materials used for teaching are extremely varied. Almost every university or school nowadays has a distance learning platform and course materials developed by individual instructors. These may be regarded as “legacy” materials since, for the most part, they are unusable outside the context in which they were developed due to hardware and software dependencies. Federations such as Macro University using techniques such as the Growing Book require exchanged documents to be reformatted, different databases to be queried and the results of the queries sent over the network in a format independent of that of the databases queried, among other local differences. The instructor needs to be able to construct some virtual containers by composing resources that were not necessarily created by him but possibly by other colleagues and which are available on the network and reusable. The use of browsers and the Internet only appears to have furnished interoperability for legacy and newly created didactic software systems since on the one hand the limited flexibility of the HTML language (and in particular its tags) does not allow for the creation of easily manipulated structured documents and on the other hand there still exist various hardware platforms and operating systems. The student in the Macro University, for his part, often asks very simple questions of the type: “Who can offer me the most up-to-date C ++ course? ”, or “Where can I find an exhaustive explanation of Laplace transforms? ”, or “Who has solutions for exercises on the subject of cinematography? ”. It is true that sophisticated techniques to identify keywords in web documents have been developed for search engines, but none of these are able to satisfy the student's needs. For this reason, a new generation of markup languages with tags able to describe the contents of the document itself have been developed. 5

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

• •

The concept of information that describes some other information is known as metadata. In the rest of this section we will describe a methodology for distance learning based on the transformation of TAOs. The Growing Book is intended to support multi-level, multi-lingual and multi-modal usage of shared learning material. This goal is fully supported by the decision to use XML to represent the TAOs comprising the Growing Book since the XML approach separates content from presentation. The Growing Book can be customized using a Tele-action Object Transformer based on XML technology by applying a Component-Based Multimedia Software Engineering Approach resulting in a complete process of dataflow transformation that presents the Growing Book in different ways and at different levels of abstraction. The Tele-action Objects comprising the Growing Book can also be transformed to conform to summary and awareness information collected during a distance learning session. The Tele-action Object Transformer was implemented using the XSLT transformation language, XSL style sheets and the SAX API Java parser. In the context of this research, the output of the Tele-action Object Transformer is a Sentient Map (previously described in Section 3.1). The main idea of the sentient map [14] is to present all kinds of objects visually in a virtual map that senses the user's input gestures and reacts by retrieving and presenting the appropriate information. The term “map” is used here in the general sense. Geographical maps, directory pages, list of 3D models, web pages, documents, slides, images, video clips, etc. are all considered maps, as they all may serve as indexes and lead the user to more information. In practice, a sentient map is a gesture-enhanced interface for an information system. In advanced distance learning applications, students and instructors can use the sentient map environment in a virtual course-room. When the user, or the user's surrogate (the user's avatar), points at the sentient map on the wall, more information becomes available and is also visible to all the participants in the virtual course-room. Two scenarios have been identified corresponding to the usage mode of the dataflow transformation: stand-alone or distributed. In the stand-alone scenario, the main dataflow transformation process is local. The stand-alone compute platform loads the Tele-action Object Transformer and delivers a combination of the style sheet and the Growing Book information to be transformed on the same platform. The results are various media formatted as requested by the user (e.g. video format, audio format, text format, etc.) and transformed in accordance with the collected summary and awareness information. In the distributed scenario the dataflow transformation process is performed on a server which provides a transformation process to the clients that require it. The client forwards the TAOML document to the Tele-action Object Transformer or an equivalent generic dataflow; it also requests a data transformation and specifies the desired output data format. The server loads the Tele-action Object Transformer and delivers a combination of the style sheet and the source data to be transformed to the recipient's platform. In the distributed scenario it is possible to use a style sheet document and Tele-action Object Transformer transformation not only for receiving a desired presentation format, but also possibly to receive a data flow properly formatted for some other mobile device such as a tablet or smartphone. The dataflow transformation process is illustrated in Fig. 2. Note that the main components of the Tele-action Object Transformer are shown inside of the Transformer itself. Another advantage of the use of XML is also illustrated in the figure – the authors have adopted the IEEE LOM (Learning Object Metadata) [25] which is a standard for metadata for learning objects such as the metadata format for TAOs. This makes the approach interoperable with other distance learning systems and allows the reuse of learning resources. The process for generic TAOML-based data stream manipulation is principally composed of two main transformations:

Transformation from a generic data format (not necessarily hierarchically organized) to a well-formed XML/TAOML format. Transformation from XML/TAOML format to a document format (e.g. text, Microsoft Word, PDF, html etc.) or a media format (e.g. audio, video).

The prototype developed reflects this organization. The modular approach adopted allows one to obtain a standard document format in a modular prototype that can be reused or extended in a very easy way in other similar Java-based applications. The main driver of “Transformation from generic data format to well-formed XML/TAOML format” is the SAX API validating Java parser. It parses an input data stream and prints it out in XML format; it also generates events that correspond to different features found in the parsed XML document. The SAX API parser used in this prototype was the Java-based open-source tool called Xerces. The XSLT processor reads in both the XML document and the XSLT style sheet. The XSLT style sheet describes a set of patterns to match within the XML document and the transformations to apply when a match is found. Pattern matches are described in terms of the tags and attributes for elements found within an XML document. Transformations extract information from the XML document and format it into a desired format. Each match-transformation pair is called an XSLT template. The XSLT transformation process works in a way very analogous to the way scripting languages such as Python or Perl operate - applying regular expressions to an input stream and then transforming the elements that were found to an output stream. The XSLT processor used in this prototype was an excellent Java-based, open-source tool called Xalan. In a modern day implementation of the prototype system described in this section, the distributed option for dataflow transformation of the Growing Book could be performed in the Cloud. An example of a standard which performs somewhat similar functions is Learning Tools Interoperability (LTI) [26] which was created by IMS Global Learning Consortium. The main functionality of LI is to connect Learning Management Systems (LMS) with external tools in a standard way. The LMS is known as an LTI Tool Consumer and the connected tool as an LTI Tool Provider. Interoperability across digital learning environments is the emphasis, rather than data transformation, although IMS also identifies XML as a metalanguage which can be used for transformable, structured, text-based content where the XML documents can be transformed by XSLT and stylesheets [27]. A typical use of LTI would be the opening of an e-textbook from within an LMS. 6. Active index for e-Learning The active index has already been encountered in research described in the previous sections, both through the index cells which make up part of the TAOs (which in turn are the underlying formalism of the Micro-University/Growing Book) and as a standalone component. The research described in this section takes a more in depth look at how an active index can be integrated with an e-learning system incorporating mobile communications, sensor networks and search engines in order to form a distributed intelligence system, one of whose applications is in e-Learning [28]. The active index consists of a number of index cells (ICs). An IC is a kind of Finite State Machine which accepts input messages, executes operations and sends one or more output messages to one or more ICS or to the external environment. The Index Cell is an intelligent agent which perceives its environment through the messages received and executes actions in order to reach a desired goal. An e-learning system can be described as a distributed intelligence system with the instructors and students being intelligent beings. As part of such a system, a kind of intelligent behavior can be identified and synthesized into index cells. In the research described, two intelligent behaviors are discussed – proficiency and deficiency. 6

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

Fig. 2. – Dataflow Transformation for the Growing Book.

in the previous section can be used to achieve the multi-lingual and multi-modal characteristics of the Growing Book. Thus all of the pieces fit together into a harmonious whole. Adaptive learning of the type described in this section is now finding its way into commercial products. DreamBox Learning [29] is an online software provider for mathematical education at the elementary and middle school level. DreamBox Learning software makes use of an intelligent adaptive learning engine which identifies a student's skill levels and generates tailored lessons for each student. Other companies are active in this niche of the market as well, while adaptive learning systems for higher education (as envisioned in the work described in this section) have been slower to emerge.

These descriptions apply to students’ mastery of learning concepts described in multimedia documents which are each associated with index cells. Depending on the students’ understanding of the material, it will need to be adapted (made more difficult or easier) in order to adapt to the students. A proficiency-level index cell is associated with a particular multimedia document. When the level of proficiency of a group of students who are using that document reaches a certain predefined threshold, the IC will send a message to the instructor of the course informing him that a critical number of students have reached a high level of proficiency in the material covered in the document. The IC may also send messages to a set of multimedia documents, to increase the level of difficulty. Likewise, a deficiency-level index cell fires messages to the instructor and other multimedia documents with the aim of decreasing the level of difficulty. Levels of proficiency or deficiency are associated with particular multimedia documents. If a student understands the material, he or she will navigate (e.g.) to a document presenting more advanced material. Reaching this document may then be defined as the threshold for proficiency, and a proficiency-level index cell associated with it and triggering the IC it may present more challenging documents or send a message to the instructor requesting such material. Likewise, a document that the user navigates to in order to get remedial or background information may be associated with a deficiency-level index cell. There is a self-adjustment index cell associated with the multimedia documents which are in charge of transforming the document in order to increase or decrease its difficulty, depending on the message received (either by presenting more or less difficult documents or prompting the instructor to add those types of documents to the system). A prototype Multimedia Knowledge Environment for Index Cells, which is an Eclipse environment for modelling TAOML private knowledge was produced as part of this research. The tool captures both static and knowledge structure and produces an XML file containing that information. It can be seen that this approach can be used to achieve the multi-level usage goal for the Growing Books which are part of the Macro University, just as the TAOML transformations described

7. Chronobot The last of Prof. Chang's research projects to be surveyed – the Chronobot/Virtual Classroom (CVC) [30] – takes a somewhat different approach than the previous ones, though the techniques can be incorporated into the Macro University/Growing Book framework. The chronobot is a device for storing and borrowing time [31]. The underlying premise of the chronobot is that there is a device for exchanging knowledge and time. For example, one can spend some time now to acquire knowledge and later use that knowledge to save some time. The chronobot can also facilitate the exchange of time and knowledge among a group of people. E-learning is a natural application for the chronobot. As an example, a student who encounters a problem he is unable to solve on his own interacts with other students in the virtual classroom who may help him resolve his problem. The student who was helped must in turn lend time/efforts either in the past or future in order to reimburse those who helped him. The chronobot serves as the platform which facilitates time and knowledge exchanges. The virtual classroom is a communication tool which combines the functionality of a web browser, chat room, white board and multimedia display. The CVC integrates chronobot and virtual classroom into a single system. The CVC implements a client/server architecture. The clients include chronobots, teachers, and students which implement 7

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio Table 1 – Prof. Chang's research technologies used in his learning projects. Name

Description

Use

Reference

Knowledge Table Active Index

Unified knowledge representation. Indexing structure which can initiate action. Multimedia object with attached private knowledge. Visual display manipulated via gestures. Time management tool for storing and borrowing time.

MI GB

[6] [5]

MI, MA, GB MA

[4] [14]

CVC

[31]

Tele-action Object Sentient Map Chronobot

The relationship between some of the different entities introduced by Prof. Chang and used in the works surveyed in this paper are shown in the E-R diagram in Fig. 3. As is evident in this paper, Prof. Chang has been a pioneer in the area of education, distance learning, and virtual universities. We have surveyed most of his research in this field that we are aware of (his very early efforts reported in [1,2] are not included here). His research has inspired many of his colleagues as well as researchers around the world to continue his path in advancing the research in this particular field. The approach of the Micro- and Macro University can be considered the basis for the creation of an evolutionary model of smarter universities that adapts to the need of the student, the environment, and the community as described in [32]. A smarter university strives to offer rich, interactive and ever-changing learning environments by exploiting the suite of technologies and services available through the Internet, by empowering individuals’ abilities and attitudes, and by encouraging them to interact and collaborate in a framework in which people are co-responsible for raising and appraising the attitudes of everyone. Opinion mining and collection of needs guide the creation of a strategic vision, which thanks to a prioritization of selected goals and objectives guide the learning environment to a selection of proper contents, both domain specific and common shared. The selection of competencies in accordance to the standards and the policy of the educational environment must be identified and properly matched in order to achieve the planned objectives. The process must be iteratively monitored in an evolutionary approach [32]. Modern and advanced models today still explore and expand the original Growing Book idea of stimulating collaboration by supporting multi-level, multi-lingual and multi-modal usage of shared learning material as analyzed in [33,34] where newer tools such as modern platforms of collaboration, the Platform As A Service model, and cloud computing are integrated into the learning process. Ubiquitous eLearning via mobile computing platforms and incorporating social networks also require transformation of learning materials for personalized user experiences [35–37]. The educational technology landscape is vastly different today than it was twenty some years ago when the research described in this paper started. Mobile technology has had a large impact on what can be achieved in e-learning. OpenCourseWare such as that developed at MIT [38] along with the associated MOOCs [39] both greatly extend the amount of courseware which is freely available and greatly increase the reach of the institutions which provide the courseware through innovative new providers such as edX [40]. Other organizations, such

MI=Micro-University, MA=Macro University, GB=Growing Book, CVC=Chronobot/ Virtual Classroom.

user interfaces for different classes of users. A number of servers are part of the system: name server; communication server; storage server; KEM/TM server. The first three of these are fairly self-explanatory while the KEM/TM server manages time and knowledge exchange for the chronobot clients. Communication between the various components of the system are facilitated by the communication servers which are implemented using Java Messaging Service (JNS). The messages are formatted using XML and may be archived and retrieved on the basis of a time range. A customized testing tool was also implemented. Chronobot clients might, for example, belong to students in a graphics design course who are working on a group project together [30]. If one of the students is not able to complete his task, he interacts with his project mates in the virtual classroom using the above-mentioned servers. The student is able to complete his task with the help of his project mates, however he in turn must use his chronobot to put in effort to help his project mates. With the chronobot mechanism, this effort could have taken place in the past, or it might take place in the future. The Chronobot/Virtual Classroom implements technology for student interaction, as well as interaction between students and teachers, which could be usefully integrated into the Macro University/Growing Book project. Prof. Chang was granted a US Patent Office patent on the chronobot in 2010.

8. Conclusions and future directions The following table documents the use of several technologies developed by Prof. Chang and his colleagues and their usage in the major projects described in this paper (Table 1).

Fig. 3. Entity-Relationship Diagram of the Objects in the Works Surveyed.

8

Journal of Visual Languages and Computing xx (xxxx) xxxx–xxxx

T. Arndt, A. Guercio

as Khan Academy [41] are making multimedia lessons freely available today anyone, anywhere who has an Internet connection. While such research advancements are to be expected, they serve to confirm the visionary ideas of Prof. Chang. His research on multimedia learning materials and the worldwide reach of consortia of academic institutions offering personalized material are being realized, even if the means don’t directly follow the technology approaches pioneered by Prof. Chang. His thinking has been absorbed in the mainstream of distance learning thinking and has influenced where that thinking has moved. Those of us who have had the pleasure of collaborating Prof. Chang in research and teaching are grateful for his leadership in these areas.

[18]

[19]

[20] [21] [22] [23] [24]

References

[25] [26]

[1] S.-K. Chang, Computers and education - toward man-machine symbiosis, in: Proceedings of International Conference on Computers and Education, Taipei, Taiwan, Republic of China, July 5-7, 1978, pp. 5-15. [2] S.-K. Chang, D. Zhang, J. Stigler, A computerized abacus for office automation and computer-aided instruction, Int. J. Comput. Process. Chin. Orient. Lang., Publ. Chin. Lang. Comput. Soc. 1 (3) (1984) 205–210. [3] S.-K. Chang, E. Hassanein, C.Y. Hsieh, A Multimedia Micro-University, IEEE Multimed. Mag. 5 (1998) 60–68. [4] H.J. Chang, T.Y. Hou, A. Hsu, S.K. Chang, The management and application of teleaction objects, ACM Multimed. Syst. J. 3 (1995) 204–216. [5] S.K. Chang, Towards a theory of active index, J. Vis. Lang. Comput. 6 (1995) 101–118. [6] S.-K. Chang, C.Y. Hsieh, Knowledge table as a unified knowledge representation for office information system design, IEEE TC Off. Autom. Newsl. 3 (1989) 12–25. [7] S.-K. Chang, T. Arndt, F.R. Guo, S. Levialdi, A.C. Liu, J.H. Ma, T. Shih, G. Tortora, The Macro University – a framework for a consortium of virtual universities, Int. J. Comput. Process. Orient. Lang. 13 (2000) 205–222. [8] T. Arndt, A. Cafiero, and A. Guercio, Multimedia languages for teleaction objects, in: Proceedings of 1997 IEEE Symposium on Visual Languages, September 1997, pp. 318–327. [9] S.-K. Chang, Extending visual languages for multimedia, IEEE Multimed. Mag. 3 (3) (1996) 18–26. [10] SCORM website: 〈http://adlnet.gov/adl-research/scorm/〉 [11] API Experience website: 〈http://experienceapi.com〉 [12] OERu website: 〈https://oeru.org/〉 [13] S.-K. Chang, The sentient map and its application to the Macro University ELearning environment, Proc. VISUAL 2000, LNCS 1929 (2000) 1–13. [14] S.-K. Chang, The sentient map, J. Vis. Lang. Comput. 11 (4) (2000) 455–474. [15] T. Arndt, S.-K. Chang, A. Guercio, Formal specification and prototyping of multimedia applications, Int. J. Softw. Eng. Knowl. Eng. 10 (4) (2000) 377–409. [16] T.K. Shih, S.-K. Chang, and T. Arndt, Database support of web course development with design patterns, in: Proceedings of the Workshop on Web-based Education & Learning (WEL 2000), Hong Kong, June 18−20, 2000. [17] S.-K. Chang, A growing book for distance learning, advances in web-based learning,

[27] [28] [29]

[30]

[31]

[32] [33] [34] [35]

[36]

[37] [38] [39] [40] [41]

9

in: Proceedings of the First International Conference, ICWL 2002, LNCS 2436, 316. S.-K. Chang, S.Y. Shao, Management of the growing book as generalized objects, in: Proceedings of the 15th International Conference on Software Engineering and Knowledge Engineering, July 1-3, 2003, San Francisco, pp. 599–606. S.-K. Chang, M.J. Chen, L.Y. Deng, Y.J. Liu, MAWC Operations for the Growing Book,Proceedings of the Seventh International’l Conference on Distributed Multimedia Systems, Sept. 26-28, 2001, Taipei, Taiwan, 1-10. S.-K. Chang, Data Structures and Algorithms, World Scientific Publishing, 2003. M. Raman, Wiki technology as a ‘free’ collaborative tool within an organizational setting, Inf. Syst. Manag. 23 (2006) 59–66. OpenSTAX College website: 〈https://openstaxcollege.org/〉 CK-12 Flexbook website: 〈http://www.ck12.org/〉 T. Arndt, S.-K. Chang, A. Guercio, P. Maresca, An XML-based approach to multimedia software engineering for distance learning, J. Distance Educ. Technol. 1 (1) (2003) 21–45. IEEE Standard for Learning Object Metadata: 〈https://standards.ieee.org/ findstds/standard/1484.12.1-2002.html〉 Tools Learning Interoperability website: 〈https://www.imsglobal.org/activity/ learning-tools-interoperability〉 IMS Guidelines for Developing Accessible Learning Applications: 〈https://www. imsglobal.org/accessibility/accessiblevers/sec4.html〉 P. Maresca, S.-K. Chang, M. Pesce, Application of active index to the management of E-Learning, J. e-Learn. Knowl. Soc. (2006) 331–341. DreamBox Learning, White Paper, Intelligent Adaptive Learning: The Next Generation Technology, at 〈http://www.dreambox.com/wp-content/uploads/ downloads/pdf/DreamBox_Intelligent_Adaptive_Learning.pdf〉 S.-K. Chang, X. Li, R. Villamarin, D. Lyker and C. Bryant, The Design and Implementation of the Chronobot/Virtual Classroom (CVC) System, Proceedings of 2006 International Conference on Distributed Multimedia Systems (DMS06), Grand Canyon, USA, Aug 30–Sept 1, 2006. S.-K. Chang and G. Santhanakrishnan, Chronobot: A Time and Knowledge Exchange System for E-Learning and Distance Education, Proceedings of 2004 Distributed Multimedia Conference, September 8–10, 2004, pp. 443-450. M. Coccoli, A. Guercio, P. Maresca, L. Stanganelli, Smarter universities – a vision for the fast changing digital era, J. Vis. Lang. Comput. 25 (6) (2014) 1003–1011. P. Maresca, A. Guercio, L. Stanganelli, T. Arndt, Experiences in collaborative learning, J. e-Learn. Knowl. Soc. 10 (3) (2014) 121–145. M. Coccoli, P. Maresca, L. Stanganelli, A. Guercio, Experience of collaboration in PaaS for the smarter university model, J. Vis. Lang. Comput. 31 (2015) 275–282. T. Arndt and A. Guercio, XML-Based Course Material Transformation for Ubiquitous e-Learning Applications, Proceedings of KMIS, Madeira, Portugal, October 6-8, 2009. T. Arndt and A. Guercio, Social Network-Based Course Material Transformations For A Personalized And Shared Ubiquitous E-Learning Experience, Proceedings of UBICOMM 2011, Lisbon, Portugal, November 20–25, 2011. T. Arndt, A. Guercio, Course personalization for ubiquitous e-Learning, ISAST Trans. Comput. Intell. Syst. 2 (2) (2010) 1–11. MIT OpenCourseWare website: 〈http://ocw.mit.edu/index.htm〉 J. Baggaley, MOOC rampant, Distance Educ. 34 (3) (2013) 368–378. edX website: 〈https://www.edx.org/〉 Khan Academy website: 〈https://www.khanacademy.org/〉