Generalized metrics for the analysis of E-learning personalization strategies

Computers in Human Behavior 48 (2015) 310–322

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Generalized metrics for the analysis of E-learning personalization strategies Fathi Essalmi a, Leila Jemni Ben Ayed a, Mohamed Jemni a, Sabine Graf b,⇑, Kinshuk b a The Research Laboratory of Technologies of Information and Communication & Electrical Engineering (LaTICE), Higher School of Sciences and Technologies of Tunis (ESSTT), 5, Avenue Taha Hussein, B.P. 56, Bab Menara, 1008 Tunis, Tunisia b School of Computing and Information Systems, Athabasca University, 1200, 10011-109 Street, Edmonton, AB T5J3S8, Canada

a r t i c l e

i n f o

Article history: Available online 20 February 2015 Keywords: Personalization Personalization strategies Personalized E-learning systems Boolean logic Learners’ characteristics

a b s t r a c t For personalizing E-learning, several different strategies and characteristics can be used and considered by teachers and course authors/designers. In order to make appropriate decisions on how to best implement personalized E-learning, this paper focuses on the question: How to foresee personalization strategies that are appropriate for particular courses? To answer this question, we present an approach for recommending personalization strategies based on the learning objects included in the course as well as on how well they support particular combinations of learners’ characteristics. In particular, the paper presents generalized metrics which support teachers for analyzing and comparing personalization strategies, as well as deciding which one should be applied for personalizing each course. The approach was validated through experiments in order to test its feasibility and success when applied to a large number of learning objects and learners’ characteristics. Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction Advance personalized learning is one of the 14 most important challenges of the 21st Century (National Academy of Engineering, 2014). Personalization of E-learning considers the individual learners’ differences for adapting courses and learning scenarios. Concerning the individual learners’ differences, they are modeled in the form of learner profiles which include specific learners’ characteristics such as advanced according to the personalization parameter learner’s level of knowledge. Building the learner profile constitutes a fundamental step of the E-Learning personalization process. Several and different learner profiles/models are reported in the literature (Essalmi, Jemni Ben Ayed, Jemni, Kinshuk & Graf, 2010a). This difference constitutes a richness which should be exploited. An important question that needs to be studied before building the learner profile is: what are the personalization parameters (e.g., learner’s level of knowledge, motivation level, etc.) to consider in the learner profile? The motivation of this question is: There are several personalization parameters reported in the literature and it is very difficult to use all these parameters to personalize each course. For example, if we have to consider 19 ⇑ Corresponding author. Tel.: +1 780 752 6836. E-mail addresses: [email protected] (F. Essalmi), [email protected] (L.J.B. Ayed), [email protected] (M. Jemni), [email protected] (S. Graf), [email protected] ( Kinshuk). http://dx.doi.org/10.1016/j.chb.2014.12.050 0747-5632/Ó 2015 Elsevier Ltd. All rights reserved.

personalization parameters for personalizing a course and if we assume that each personalization parameters includes three different learners’ characteristics (e.g. the personalization parameter learner’s level of knowledge includes the learners’ characteristics: beginner, intermediate and advanced). In this case, the professor responsible for the course has to prepare 57 (=19  3) different learning scenarios. Furthermore, learners have to be evaluated with regard to 19 personalization parameters. In this case, the evaluation process will be time consuming and could decrease the learners’ motivation. As an answer to the question (what are the personalization parameters to consider in the learner profile), this paper presents generalized metrics analyzing combination of personalization parameters. These metrics allow selecting the combination of personalization parameters to use according to the specificities of courses. There has been an important academic revolution in E-learning systems. This revolution considers the leaner’s individual profiles in order to generate personalized courses. The whole aim of this revolution is the comfort of learners (learning the appropriate content in the appropriate way). There are many scientific papers which study this kind of personalization. For example, Brusilovsky and Millán (2007) presented different user profiling features. Furthermore, Brusilovsky and Millán (2007) present methods about how these features could be modeled in E-learning systems. In Table 1, we present 24 examples of E-learning personalization systems. For example, Despotovic´-Zrakic´, Markovic´, Bogdanovic´, Barac´, and

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

Krcˇo (2012) used the Felder–Silverman Learning Styles Model (FSLSM) (Felder & Silverman, 1988) to provide adaptivity in courses in Moodle (2014). In particular, Despotovic´-Zrakic´ et al. (2012) used data mining techniques to classify students into clusters with regards to Felder–Silverman learning styles model. Chookaew, Panjaburee, Wanichsan, and Laosinchai (2014) presented an Elearning environment which allows personalizing computer programming courses according to the learner’s level of knowledge and the Felder–Silverman learning style. As another example, we cite Lecomps5 (Limongelli, Sciarrone, Temperini, & Vaste, 2011), a web-based educational system which allows the production and adaptation of the course based on learner’s level of knowledge and Felder–Silverman learning styles. Another academic revolution has started in the new generation of E-learning personalization systems. The aim of this academic revolution is personalizing each course according to the appropriate personalization strategy (e.g., it is possible to personalize the course ‘‘Microsoft Word’’ by including personalization parameters different to the personalization parameters used to personalize the course ‘‘Programming Language Maple’’.). This second revolution is new and there are few works reporting approaches about the appropriateness of personalization strategies to courses. Essalmi et al. (2010a), Essalmi, Jemni Ben Ayed, Jemni, Kinshuk and Graf, (2010b) studied the architecture of personalization systems which allow personalizing courses with different personalization strategies. However, this previous work does not study generalized metrics analyzing personalization strategies. Furthermore, Kurilovas and Zilinskiene (2013) presented a method evaluating the suitability of learning scenarios to particular learning styles. However, this method does not evaluate or select the appropriate personalization strategy. It just evaluates the learning scenarios. Our approach is different since it proposes generalized metrics which support teachers to select the appropriate personalization strategy. For an operational decision on the appropriate personalization strategy, teachers need to have an idea about personalization strategies which could be easily used. Analyzing personalization strategies allows discovering information useful to select the appropriate one for a course. This analysis collects and discovers relations between metadata of learning objects and the learner characteristics to be included in the personalization strategy.

311

1.1. Technical terms This subsection explains technical terms and provides examples of them. The technical terms used in the paper are: personalization parameter, personalization strategy, combination operator, learning scenario and learning object. A personalization parameter could be considered as a set of complementary learners’ characteristics such as the learning style model or the learners’ level of knowledge. A set of complementary learners’ characteristics includes the opposite characteristics. For example, if the learners’ characteristic ‘‘active’’ of the active/reflective dimension of the Felder–Silverman learning style model is included then the opposite characteristic ‘‘reflective’’ has to be included too. Personalization parameters could be combined to have a personalization strategy. For example, it is possible to personalize a course by considering the two personalization parameters learner’s level of knowledge and active/reflective dimension of the Felder–Silverman learning style model. In this case, a combination operator such as ‘‘and’’ or ‘‘or’’ could be used. In particular, for the combination of the characteristics active of the active/reflective dimension of the Felder–Silverman learning style model ‘‘and’’ advanced of the parameter learner’s level of knowledge, a personalized learning scenario must contain the learning objects which are appropriate for both the characteristic active learning style and the characteristic advanced level of knowledge. But, when combining the same characteristics with the operator ‘‘or’’, personalized learning scenarios could be represented by the learning objects which are appropriate to the characteristic active learning style together with other learning objects which are appropriate to the characteristic advanced level of knowledge. A learning scenario can be represented by a graph of learning objects. ‘‘A learning object is any entity -digital or non-digital- that may be used for learning, education or training’’ (IEEE, 2002, p. 5). 1.2. Paradigm of generalized metrics The development of personalized learning scenarios requires more efforts and time than the development of a static course which uses a one-size-fits-all approach. These efforts and time depend on the number of divergent learning scenarios and on

Table 1 Examples of personalization systems classified according to personalization parameters. Personalized E-learning system

Personalization parameters

Interbook (Brusilovsky et al., 1996) KOD (Sampson, Karagiannidis, & Cardinali, 2002) SIMBAD (Bouzeghoub et al., 2003) MetaLinks (Murray, 2003) INSPIRE (Papanikolaou et al., 2003) MLTutor (Smith & Blandford, 2003) COLER (Constantino-González, Suthers, & Santos, 2003) SQL-Tutor (Mitrovic, 2003) EPSILON (Soller, 2004) SIETTE (Conejo et al., 2004) PERSO (Chorfi & Jemni, 2004) ELENA (Dolog, Henze, Nejdl, & Sintek, 2005) e-aula (Sancho et al., 2005) AHA! (Stash, Cristea, & de Bra, 2006) Milosevic et al. (2006) Graf et al. (2010) PASER (Kontopoulos et al., 2008) Isotani et al. (Isotani et al., 2009) Protus (Aleksandra et al., 2011 Lecomps5 (Limongelli et al., 2011) Despotovic´-Zrakic´ et al. (2012) Dwi and Basuki (2012) AMDPC (Yang et al., 2013) Chookaew et al. (2014)

Learner’s level of knowledge Learner’s level of knowledge, language preference, learning goals Learner’s level of knowledge, learning goals, media preferences Learner’s level of knowledge, learning goals, media preferences Learner’s level of knowledge, learning goals, learning style of Honey and Mumford Learning goals (based on user’s browsing history) Participation balance, progress on task, waiting for feedback Learner’s level of knowledge Learner’s level of knowledge Learner’s level of knowledge Learner’s level of knowledge, media preference Learner’s level of knowledge, language preference, learning goals Learner’s level of knowledge, learning goal, dimensions of the Felder–Silverman learning style Dimensions of the Felder–Silverman learning style, media preference, navigation preference Kolb learning cycle, motivation level Dimensions of the Felder–Silverman learning style Learner’s level of knowledge, learning goals Learning goals Learner’s level of knowledge, dimensions of the Felder–Silverman learning style Learner’s level of knowledge, dimensions of the Felder–Silverman learning style Dimensions of the Felder–Silverman learning style Learner’s level of knowledge Dimensions of the Felder–Silverman learning style, cognitive traits Learner’s level of knowledge, dimensions of the Felder–Silverman learning style

312

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

the number of learning objects required to fit the learners’ characteristics included in the personalization strategy. Therefore, in order to facilitate the personalization of a course and minimize the efforts and time required by teachers/authors to create additional learning objects, appropriate personalization strategies have to be selected according to two criteria:  The number of personalization parameters to be considered in the personalization strategy as well as the combination operator of the characteristics included in the personalization parameters.  The availability of learning objects which support particular learners’ characteristics. This availability could be verified automatically based on the metadata describing each learning object and the meaning coherence (semantic relations) of these metadata with the learners’ characteristics. This paper presents our approach for the analysis of personalization strategies by considering the above criteria. This new approach starts from the idea of selecting and applying an appropriate personalization strategy for each course. The analysis of personalization strategies is based on metrics which allow measurement of personalization strategies combining any set of personalization parameters. 1.3. Structure of the paper This paper is structured as follows. Section 2 discusses research on E-learning personalization and divergent characteristics of learners. Section 3 presents a proposed process for E-learning personalization based on the analysis of personalization strategies. Section 4 presents an example for the metrics analyzing personalization strategies. Section 5 presents an experimentation to evaluate our approach. Section 6 concludes the paper with a summary of the findings, the limits and potential research directions. 2. Literature review on E-learning personalization Any personalization process has to consider divergent characteristics of persons. In particular, E-learning personalization has to consider divergent learners’ characteristics. Each subset of divergent learners’ characteristics constitutes a personalization parameter. For example, the learners’ characteristics beginner, intermediate and advanced constitute the personalization parameter learner’s level of knowledge (Chorfi & Jemni, 2004). As another example, low motivation, moderate motivation and high motivation constitute the personalization parameter motivation level (Milosevic, Brkovic, & Bjekic, 2006). A comprehensive review by Essalmi et al. (2010a) resulted in the identification of 16 personalization parameters that have been most commonly used in the E-learning domain. In particular, the previous review considers the four dimensions of the Felder–Silverman learning style model as only one personalization parameter. This work considers each dimension of the Felder–Silverman learning style model (i.e., Sensing/Intuiting, Visual/Verbal, Active/Reflective, and Sequential/ Global) as one personalization parameter. In fact, each dimension includes complementary learners’ characteristics. Fig. 1 presents a taxonomy of 19 personalization parameters as the terminal nodes. These personalization parameters are the most commonly used ones in E-learning. They could be classified as parameters about why to learn, what to learn and how to learn?. The parameters about ‘‘why to learn?’’consider the motivation and the learning goal as learners’ individual differences. The parameters about ‘‘what to learn?’’ allow personalizing learning by considering the learners’ knowledge and the requested information. The parameters about

‘‘how to learn?’’ consider learners’ individual differences as manner in which they deal with learning scenarios. Parameters about ‘‘why to learn?’’ include the motivation level, and the learning goal. Parameters about ‘‘what to learn?’’ include the information seeking task and the learner’s level of knowledge. Parameters about ‘‘how to learn?’’ include the media preference, navigation preference, collaboration skills, pedagogical approach, learning styles model, cognitive traits and language. Several E-learning personalization systems use combinations of the cited personalization parameters (Essalmi et al., 2010a) and many of these systems are reported in the literature. Table 1 presents a significant review of such systems and enumerates for each system the applied personalization parameters. Most of these systems use the personalization parameter: learner’s level of knowledge. Many of them give importance to the learner’s media preference. For example, PERSO uses Case Based Reasoning approach to determine which course to propose to the students based on their levels of knowledge, and their media preferences (Chorfi & Jemni, 2004). SIMBAD allows building personalized courses by considering the learner’s level of knowledge, learning goals and media preferences. (Bouzeghoub, Carpentier, Defude, & Duitama, 2003). MetaLinks, an authoring tool for adaptive hyperbooks, has been used to personalize a geology hyperbook according to the learner’s level of knowledge, learning goals and media preferences (Murray, 2003). Dwi and Basuki (2012) reported a learning system that takes into consideration the learner’s level of knowledge for personalizing course. The system starts by a pretest to assess learners. Then, based on their level of knowledge, the system recommends the list of materials that they have to go through. Other works have attempted the integration of learning styles as a parameter for the personalization of learning scenarios. For example, INSPIRE adopts the learning style model of Honey and Mumford as the basis for determining the presentation of the educational material on each of the performance levels (Papanikolaou, Grigoriadou, Kornilakis, & Magoulas, 2003). e-aula (Sancho, Martínez, & Fernández-Manjón, 2005) allows the personalization of courses according to the learner’s level of knowledge, the learning goals and the Felder–Silverman learning style model. To do this, the work by Sancho et al. (2005) was based on a domain ontology to share knowledge on a specific domain and a pedagogical ontology to provide a description of a learning resource from an instructional perspective. Milosevic et al. (2006) used Kolb’s learning cycle for tailoring lessons. Their work also incorporated the learner motivation as a personalization parameter, which is used to determine the complexity and the semantic quantity of learning objects. Protus (Klasnja-Milicevic, Vesin, Ivanovic, & Budimac, 2011) is designed as a tutoring system which helps learners in learning essentials of programming languages. Protus considers the Felder–Silverman Learning Styles Model and the learner’s level of knowledge to recommend relevant links and activities for learners. Yang, Hwang, and Yang (2013) developed the Adaptation with Multi-Dimensional Personalization Criteria System (AMDPC) which considers the learners’ characteristics such as dimensions of the Felder–Silverman learning style and cognitive traits. The system identifies first the learning style of each learner then adjusts the learning content offered to him/her based on his/her needs. After that, the system identifies the learner’s cognitive traits (dependant/independent) and personalizes the course presentation. Furthermore, the personalization parameter learning goal is used by some systems. For example, PASER (Kontopoulos, Vrakas, Kokkoras, Bassiliades, & Vlahavas, 2008) has been developed for course planning according to learners’ goals and their level of knowledge, using a domain ontology which describes a hierarchy of the artificial intelligence area. Isotani, Inaba, Ikeda, and Mizoguchi (2009) proposed a method for group formation, according to

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

313

Fig. 1. A taxonomy of personalization parameters.

individual and group goals, based on a collaborative learning ontology, which represents relationships between group formation and interactions between individuals. The column ‘‘personalization parameters’’ of Table 1 contains different subsets of personalization parameters. For example, {learner’s level of knowledge} is used in the Interbook which is reported by Brusilovsky, Schwarz, and Weber (1996). As another example, {dimensions of the Felder–Silverman learning style} is used to provide adaptively in Moodle. In particular, Graf, Kinshuk, and Ives (2010) extended Moodle to allow personalization of courses with respect to dimensions of learning styles. Another work reported by Despotovic´-Zrakic´ et al. (2012) allows clustering learners and providing personalized courses by considering cluster of learners and the related learning styles.{learner’s level of knowledge, dimensions of the Felder–Silverman learning style} is used in Protus which is reported by Klasnja-Milicevic et al. (2011). Each of these subsets reflects personalization needs, and the strategies used to respond to these needs depend strongly on the selected subset of personalization parameters. Other personalization needs could continue to appear in the future given that several combinations of personalization parameters have not been reported in the literature (Essalmi et al., 2010a). However, there are several possible combinations of personalization paraP 19 1 19 19! meters. In fact, we have 524287 = 19 i¼1 C i , where C i ¼ i!ð19iÞ!, possible combinations when considering the subset of personalization parameters generated from the 19 personalization parameters presented above. Each combination is a personalization strategy which could be the most appropriate for the personalization of a given course. This paper responds to the question: How to select the best personalization strategy for personalizing a course? In this paper, we present an approach to foresee appropriate personalization strategies for a course based on mappings between learning objects and learners’ characteristics. In particular, these mappings are used to test the feasibility of generating personalized learning scenarios before starting the personalization process.

3. Process for the E-learning personalization When there are several personalization strategies which could be used, the selection of the appropriate strategy for personalizing a course becomes crucial. In this case, the personalization process starts by the analysis of personalization strategies to foresee the most appropriate of them. The process for the E-learning personalization based on the analysis of personalization strategies is presented as follows: Step1: Analysis of feasible personalization strategies In this step, rates representing the degree of appropriateness of a personalization strategy to personalize a course are computed. A rate allows teachers to foresee the feasibility of personalizing the course according to a certain personalization strategy. In particular, a rate near 1 expresses that the strategy is very appropriate to be used. A rate near 0 expresses that the strategy could not be applied for personalizing the course without significantly updating the set of learning objects forming the course. In fact, if the rate is close to 0, the learning objects do not fit the different learners’ characteristics included in the related personalization strategy. Updating the course may require a huge effort from the teacher/ author in order to make the course fitting all learners’ characteristics included in the related personalization strategy. The analysis of personalization strategies is automated based on the metadata describing the learning objects and the semantic relations between the metadata and learners’ characteristics. In particular, LOM (IEEE, 2002), as a metadata standard, and the semantic relations presented by Essalmi et al. (2010b) have been used. The analysis of personalization strategies supports a teacher’s decision about the appropriate personalization strategy. However, this analysis is not mandatory. If the teacher has an idea about the appropriate personalization strategy, he/she could apply it without conducting this analysis. Step2: Selection of personalization strategy

1

From 19 personalization parameters, it is possible to have personalization strategies which include 1, 2. . . or 19 personalization parameters. For example, there is 19 = C 19 1 (selecting one personalization parameter from 19 one) personalization strategies which include 1 personalization parameter. There is 171 = C 19 2 personalization strategies which include 2 personalization parameters. There is 969 = C 19 3 personalization strategies which include 3 personalization parameters.

In this step, a teacher/author has to select the personalization strategies to be applied for his/her course. If step1 is not included in this process, the teacher/author could select the personalization strategy according to his/her preference. However, he/she is not

314

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

guided to select the personalization strategy that fits his/her course. Including step1 allows the teacher to be supported to make an informed decision by referring to the calculated rates (calculated in step1). In particular, he/she could select the personalization strategy which has the best rate of appropriateness. The teacher/ author could also select another personalization strategy that he/ she prefers. To select a personalization strategy according to the teachers’ preferences, it is possible to use the system presented by Essalmi et al. (2010a). This system allows for the personalization in two levels. The first level allows for the selection (step 2) of personalization strategies according to the teachers’ preferences. The second level allows for the application (step 3) of the selected personalization strategy.

element is associated with a learners’ characteristic. A learners’ characteristic belongs to a personalization parameter. The second part investigates relations between the course and the personalization parameters which are based on the analysis of the relations in the first part. These investigated relations are as follows: (1) A course has an appropriate personalization parameter. (2) A course has an appropriate combination of personalization parameters. These two relations should help to answer the following two questions: (1) What is the appropriate parameter for personalizing the course? (2) What is the appropriate combination of parameters for personalizing the course? The first question has been investigated by Essalmi et al. (2010b), Essalmi, Jemni Ben Ayed, Jemni, Kinshuk and Graf, (2011). Accordingly, this paper focuses on the second question.

Step3: Application of personalization strategy This step allows for the application of the personalization strategy selected in step 2. As mentioned before, this could be done, for example, by using the system presented by Essalmi et al. (2010a). This paper focuses on step1 (analysis of personalization strategies) and shows its feasibility in general (when a personalization strategy includes combinations of personalization parameters). In particular, this paper defines the generalized metrics which are used to analyze and compare personalization strategies. 4. An example illustrating the proposed metrics In order to help teachers to select the most appropriate personalization strategy for each course, some metrics are needed for the analysis of feasible personalization strategies. This section presents an example explaining the approach used for the analysis of personalization strategies. The details of the example are presented in Sections 4.1 and 4.2. Furthermore, an overview of the example is presented in Fig. 2 which includes two parts. The first part deals with relations between the concepts of a course, learning objects, metadata, learners’ characteristics and personalization parameters. These relations are as follows: a concept could be represented by one or more learning objects. Each learning object is described by one or more metadata elements and their values. A metadata

4.1. Information used in the example In order to illustrate the proposed metrics an example is presented. This example is based on an extract of the course presented by Essalmi et al. (2010b) and titled Compilation Theory which contains three learning concepts: introduction to compilation, lexical analyzer, and syntactic analyzer. The concept introduction to compilation is represented by one learning object: communication between lexical and syntactic analyzer. The concept lexical analyzer is represented by two learning objects: overview of the lexical analyzer and detailed view of the lexical analyzer. The concept syntactic analyzer is represented by two learning objects: overview of the syntactic analyzer and detailed view of the syntactic analyzer. All learning objects are annotated by some optional metadata. For example, the learning object communication between lexical and syntactic analyzer is annotated by the data element Difficulty having the value very easy, and the data element Interactivity Type having the value active. Furthermore, the example refers to semantic relations between metadata and learners’ characteristics. A comprehensive collection of semantic relations is presented in (Essalmi et al., 2010b). Among these relations, we cite the relations between the learners’ characteristics included in the personalization parameter learner’s level of knowledge and the values of the data element difficulty. For

Fig. 2. Overview of the analyses.

315

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

instance, the level of difficulty easy is associated in this example with the learners’ characteristics beginner while the level of difficulty difficult is associated with the characteristic advanced. Table 2 presents a matrix of appropriate learning objects generated from the structure of the course and the semantic relations (Essalmi et al., 2010b). In this matrix, concepts are presented in rows, and personalization parameters are presented in columns. In addition, each column is divided in sub-columns presenting the learners’ characteristics included in the personalization parameter presented in the column. Each cell of the matrix contains the learning objects representing a specific concept that are appropriate to a specific learners’ characteristic. Concerning the automatic filling of the cells with appropriate learning objects, a subset of data elements describing learning objects is used as input for the semantic relations between data elements and learners’ characteristics. For example, the description of the learning object communication between lexical and syntactic analyzer with the data element (Difficulty, very easy) is an input for the semantic relation between the same data element and the learners’ characteristic beginner included in the personalization parameter learner’s level of knowledge. As a consequence, the learning object communication between lexical and syntactic analyzer is placed in the cell intersection of the row representing the concept introduction to compilation and the sub-column representing the learners’ characteristic beginner. However, some cells are empty. A cell is empty if there is no learning object representing the concept in the row and which is appropriate to the learners’ characteristic in the sub-column. For example, the cell intersection of the row introduction to compilation and the sub-column sequential is empty due to the fact that there is no learning object representing the concept introduction to compilation and that is appropriate to the learners’ characteristic sequential. Given that a personalization strategy is a combination of personalization parameters, the comparison of personalization strategies depends on the combined personalization parameters as well as the combination operators. The next section explains our approach to analyze and identify personalization strategies that are appropriate for personalizing courses. 4.2. Combination of two personalization parameters This section presents four generalized metrics: TolerantCRC2 (Tolerant Concept Representation for a combination of learners’ Characteristics included in 2 personalization parameters), ExigentCRC2 (Exigent Concept Representation for a combination of learners’ Characteristics included in 2 personalization parameters), TolerantCRP2 (Tolerant Concept Representation for a combination of 2 personalization parameters) and ExigentCRP2 (Exigent Concept Representation for a combination of 2 personalization parameters).

These metrics allow the analysis of personalization strategies which include two personalization parameters. The first difference which distinguish these metrics is that: ExigentCRC2 and TolerantCRC2 considers all concepts represented by learning objects which are appropriate to any combination of learners’ Characteristics (included in two personalization parameters); however TolerantCRP2 and ExigentCRP2 consider only fully personalized concepts. A concept is said fully personalized for a given parameter if it is represented by learning objects which are appropriate to all combination of learners’ Characteristics. The second difference between these metrics is that TolerantCRC2 and TolerantCRP2 use the combination operator or but ExigentCRC2 and ExigentCRP2 use the operator and. In the following paragraphs, these four metrics are described in detail. Considering the two personalization parameters learner’s level of knowledge and sequential/global dimension of Felder–Silverman learning style model, Table 3 presents the two metrics TolerantCRC2 and ExigentCRC2 based on the example presented before. TolerantCRC2 considers all concepts represented by learning objects which are appropriate to any combination of learners’ characteristics using the operator or. However, ExigentCRC2 considers all concepts represented by learning objects which are appropriate to any combination of learners’ characteristics using the operator and. The possible subsets of two personalization parameters based on the above example are: {learner’s level of knowledge, active/reflective dimension of Felder–Silverman learning style model}, {learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model}, and {active/reflective dimension of Felder–Silverman learning style model, sequential/global dimension of Felder–Silverman learning style model}. We explain in depth the metrics TolerantCRC2 and ExigentCRC2 with the subset {learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model}. By combining the two personalization parameters included in this subset, six couples of learners’ characteristics have to be considered. Each couple contains a learners’ characteristic from the personalization parameter learners’ level of knowledge and another one from the personalization parameter sequential/global dimension of Felder–Silverman learning style model. In each row of Table 3, a couple of learners’ characteristics is presented. The table contains two columns; one of them represents the metric TolerantCRC2 and the other one represents the metric ExigentCRC2. Each column is divided in three sub-columns representing the concepts included in the course. Furthermore, a cell is filled by the value 1 if there is a learning object representing a specific concept (in the sub-column) and this learning object is appropriate to a specific combination of learners’ characteristics (in the row). The values of TolerantCRC2 and ExigentCRC2 are determined by the average of the values filled in the cells of the column specific for each metric. In general, for every couple of personalization parameters, we can establish a similar table using the same approach as we did for the two parameters learner’s level of knowledge and

Table 2 Matrix for appropriate learning objects. Learner’s level of knowledge beginner Introduction to compilation Lexical analyzer

Syntactic analyzer

Communication between lexical and syntactic analyzer Overview of the lexical analyzer Overview of the syntactic analyzer

intermediate

advanced

Active/reflective dimension of Felder– Silverman learning style model

Sequential/global dimension of Felder–Silverman learning style model

active

sequential

global

Detailed view of the lexical analyzer Detailed view of the syntactic analyzer

Overview of the lexical analyzer

reflective

Communication between lexical and syntactic analyzer Detailed view of the lexical analyzer Detailed view of the syntactic analyzer

Overview of the syntactic analyzer

316

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

Table 3 Combining the two personalization parameters learner’s level of knowledge and sequential/global dimension of Felder–Silverman learning style model based on Boolean logic. Metric

TolerantCRC2

ExigentCRC2

Introduction to compilation

Lexical analyzer

Syntactic analyzer

Introduction to Compilation

Lexical analyzer

Syntactic analyzer

(Beginner, sequential) (Beginner, global) (Intermediate, sequential) (Intermediate, global) (Advanced, sequential) (Advanced, global)

1 1 0

1 1 1

1 1 1

0 0 0

0 1 0

0 1 0

0 0 0

1 1 1

1 1 1

0 0 0

0 1 0

0 1 0

Average

(1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)/18 = 0.77

(1 + 1 + 1 + 1)/18 = 0.22

Table 4 Comparing personalization strategies composed of two personalization parameters by considering the metrics TolerantCRC2 and ExigentCRC2. Metric (a, b)

Average of TolerantCRC2(a, b)

Average of ExigentCRC2(a, b)

(learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model) (learner’s level of knowledge, active/reflective dimension of Felder–Silverman learning style model) (active/reflective dimension of Felder–Silverman learning style, sequential/global dimension of Felder–Silverman learning style model)

0.77 0.66 0.83

0.22 0.05 0.00

Table 5 Comparing personalization strategies composed of two personalization parameters by considering the metrics TolerantCRP2 and ExigentCRP2. Metric (a, b)

Average of TolerantCRP2(a, b)

Average of ExigentCRP2(a, b)

(learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model) (learner’s level of knowledge, active/reflective dimension of Felder–Silverman learning style model) (active/reflective dimension of Felder–Silverman learning style, sequential/global dimension of Felder–Silverman learning style model)

0.66 0 0.66

0 0 0

sequential/global dimension of Felder–Silverman learning style model. For this reason, we can add the two parameters as an argument of the metrics TolerantCRC2 and ExigentCRC2. For instance, TolerantCRC2(a, b) represents the metric Tolerant Concept Representation for the combination of the two personalization parameters a and b. Table 4 represents the values of the metrics TolerantCRC2(a, b) and ExigentCRC2(a, b) for all possible combinations of two personalization parameters (a, b) in our example. The choice of personalization strategy depends on the used metric. For example, by considering the metric TolerantCRC2, the subset of personalization parameters {active/reflective dimension of Felder–Silverman learning style model, sequential/global dimension of Felder–Silverman learning style model} is the best choice, as shown in Table 4. On the other hand, by considering the metric ExigentCRC2, the subset of personalization parameters {learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model} would be the best choice. In the same manner, two generalized metrics could be defined by considering a combination of two personalization parameters: TolerantCRP2(a, b) and ExigentCRP2(a, b). TolerantCRP2 considers concepts represented by learning objects appropriate to tolerant combinations of personalization parameters. For example, two out of three (2/3 = 0.66) is the value of TolerantCRP2(learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model). Two concepts (Lexical analyzer and Syntactic analyzer) out of three could be fully personalized by considering a tolerant combination of the two personalization parameters learners’ level of knowledge and sequential/global dimension of Felder–Silverman learning style model. These two concepts are represented by learning objects which are appropriate to all tolerant combination of learners’ characteristics included in the considered

personalization parameters. Concerning ExigentCRP2, it considers the concepts represented by learning objects appropriate to exigent combinations of personalization parameters. For example, zero out of three concepts is the value of ExigentCRP2(learners’ level of knowledge, sequential/global dimension of Felder–Silverman learning style model). Table 5 presents the results of applying the metrics TolerantCRP2 and ExigentCRP2. If the number of personalization parameters (to include in the personalization strategy) is more than two, the principle of analyzing combination of them is the same. For example, for combining three personalization parameters, TolerantCRC3, ExigentCRC3, TolerantCRP3 and ExigentCRP3 can be used. In this case, combinations of three learners’ characteristics are considered. 5. Experimentation This section presents an experimentation conducted to test the feasibility and the significance of analyzing and comparing personalization strategies. Section 5.1 presents the procedure of the experimentation. After that, Section 5.2 presents results of the course analysis based on the proposed metrics. Section 5.3 presents the reliability of the metrics and Section 5.4 presents the results of the validation of the proposed metrics based on a comparative study. 5.1. Procedure A total of 571 learning objects representing 100 concepts included in six courses and semantic relations (between data elements and learners’ characteristics) were used for comparing

317

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322 Table 6 Combination of two personalization parameters for the six courses. Courses Parameters

Microsoft Word C1

C2

C3

Microsoft Excel Metrics C4

C1

C2

C3

Programming Language C C4

C1

Part 1. Combination of two personalization parameters for the courses: Microsoft Word, Microsoft Excel, and Programming Language C MP, ARFSLS 0.75 0 1 1 0.62 0 0.93 0.75 0.75 MP, SGFSLS 0 0 1 1 0 0 0.87 0.75 0 MP, VVFSLS 0.7 0 1 1 0.46 0 0.93 0.75 0.5 MP, HMLS 0.37 0 1 1 0.31 0 0.9 0.75 0.37 MP, LLK 0.57 0 1 1 0.31 0 0.91 0.75 0.63 MP, NP 0 0 1 1 0 0 0.87 0.75 0 MP, ML 0.66 0 1 1 0.52 0 0.93 0.75 0.6 ARFSLS, SGFSLS 0 0 1 1 0 0 0.87 0.75 0 ARFSLS, VVFSLS 0.7 0 1 1 0.46 0 0.93 0.75 0.55 ARFSLS, HMLS 0.25 0 1 1 0.21 0 0.9 0.75 0.25 ARFSLS, LLK 0.59 0 1 1 0.31 0 0.91 0.75 0.6 ARFSLS, NP 0 0 1 1 0 0 0.87 0.75 0 ARFSLS, ML 0.69 0.14 1 1 0.45 0 0.93 0.75 0.7 SGFSLS, VVFSLS 0 0 0.92 0.85 0 0 0.56 0.12 0 SGFSLS, HMLS 0 0 0.5 0 0 0 0.43 0 0 SGFSLS, LLK 0 0 0.76 0.42 0 0 0.37 0 0 SGFSLS, NP 0 0 0 0 0 0 0 0 0 SGFSLS, ML 0 0 0.9 0.71 0 0 0.79 0.5 0 VVFSLS, HMLS 0.35 0 0.96 0.85 0.23 0 0.75 0.12 0.27 VVFSLS, LLK 0.59 0 0.97 0.85 0.2 0 0.72 0.12 0.43 VVFSLS, NP 0 0 0.92 0.85 0 0 0.56 0.12 0 VVFSLS, ML 0.66 0 0.97 0.85 0.4 0 0.89 0.5 0.46 HMLS, LLK 0.29 0 0.88 0.42 0.15 0 0.64 0 0.3 HMLS, NP 0 0 0.5 0 0 0 0.43 0 0 HMLS, ML 0.34 0 0.95 0.71 0.22 0 0.86 0.5 0.35 LLK, NP 0 0 0.76 0.42 0 0 0.37 0 0 LLK, ML 0.46 0 0.95 0.71 0.29 0 0.86 0.5 0.55 NP, ML 0 0 0.9 0.71 0 0 0.79 0.5 0

C2

C3

C4

0 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 1 1 1 1 1 1 1 1 1 1 1 1 0.6 0.5 0.8 0 0.93 0.8 0.9 0.6 0.96 0.9 0.5 0.96 0.8 1 0.93

1 1 1 1 1 1 1 1 1 1 1 1 1 0.2 0 0.4 0 0.8 0.2 0.4 0.2 0.8 0.4 0 0.8 0.4 1 0.8

Part 2. Combination of two personalization parameters for the courses: Programming Language Maple, Compilation Theory, and Databases Courses Programming Language Maple Compilation Theory Parameters Metrics

MP, ARFSLS MP, SGFSLS MP, VVFSLS MP, HMLS MP, LLK MP, NP MP, ML ARFSLS, SGFSLS ARFSLS, VVFSLS ARFSLS, HMLS ARFSLS, LLK ARFSLS, NP ARFSLS, ML SGFSLS, VVFSLS SGFSLS, HMLS SGFSLS, LLK SGFSLS, NP SGFSLS, ML VVFSLS, HMLS VVFSLS, LLK VVFSLS, NP VVFSLS, ML HMLS, LLK HMLS, NP HMLS, ML LLK, NP LLK, ML NP, ML

Databases

C1

C2

C3

C4

C1

C2

C3

C4

C1

C2

C3

C4

0.47 0 0.25 0.23 0.35 0 0.32 0 0.26 0.22 0.33 0 0.3 0 0 0 0 0 0.13 0.19 0 0.17 0.16 0 0.15 0 0.25 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0.96 0.92 0.96 0.94 0.98 0.92 0.98 0.89 0.94 0.92 0.96 0.89 0.96 0.5 0.44 0.7 0 0.66 0.72 0.85 0.5 0.83 0.83 0.44 0.8 0.7 0.83 0.66

0.84 0.84 0.84 0.84 0.94 0.84 0.94 0.78 0.78 0.78 0.89 0.78 0.89 0 0 0.42 0 0.36 0 0.42 0 0.36 0.42 0 0.36 0.42 0.42 0.36

0.44 0 0.47 0.22 0.41 0 0.41 0 0.36 0.14 0.3 0 0.31 0 0 0 0 0 0.18 0.33 0 0.34 0.15 0 0.15 0 0.26 0

0.11 0 0.14 0 0.02 0 0.08 0 0 0 0 0 0.02 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0.88 0.76 0.91 0.82 0.85 0.76 0.86 0.57 0.84 0.68 0.79 0.57 0.79 0.66 0.28 0.56 0 0.56 0.75 0.82 0.66 0.81 0.68 0.28 0.68 0.56 0.75 0.56

0.52 0.52 0.67 0.52 0.52 0.52 0.52 0.14 0.38 0.14 0.23 0.14 0.23 0.32 0 0.17 0 0.2 0.32 0.35 0.32 0.38 0.17 0 0.2 0.17 0.2 0.2

0.44 0 0.46 0.22 0.38 0 0.35 0 0.49 0.17 0.38 0 0.35 0.01 0 0 0 0 0.24 0.42 0 0.38 0.19 0 0.17 0 0.31 0

0.07 0 0.07 0 0 0 0 0 0.11 0 0.03 0 0 0 0 0 0 0 0 0.03 0 0 0 0 0 0 0 0

0.88 0.64 0.9 0.76 0.85 0.64 0.82 0.71 0.9 0.77 0.85 0.7 0.85 0.74 0.36 0.63 0.01 0.58 0.82 0.88 0.74 0.9 0.74 0.35 0.71 0.62 0.81 0.58

0.55 0.29 0.62 0.29 0.4 0.29 0.37 0.4 0.62 0.4 0.4 0.4 0.48 0.48 0 0.18 0 0.18 0.48 0.59 0.48 0.62 0.18 0 0.18 0.18 0.29 0.18

MP = Media Preference. ARFSLS = Active/Reflective dimension of the Felder–Silverman learning style model. SGFSLS = Sequential/Global dimension of the Felder–Silverman learning style model. VVFSLS = Visual/Verbal dimension of the Felder–Silverman learning style model. HMLS = Honey–Mumford learning style model. NP = Navigation preference. LLK = Learner’s level of knowledge. ML = Motivation level.

318

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

Table 7 Measure of reliability. 28 personalization strategies for the course Microsoft Word

28 personalization strategies for the course Microsoft Excel

28 personalization strategies for the course Programming Language C

Part1. Combinations of two personalization parameters for the courses: Microsoft Word, Microsoft Excel, and Programming Language C 0.74 0.73 Cronbach’s a of C1, C2, 0.71 C3 and C4. Part2. Combinations of two personalization parameters for the courses: Programming Language Maple, Compilation Theory, and Databases 28 personalization strategies for the course 28 personalization strategies for the 28 personalization strategies for the course Programming Language Maple course Compilation Theory Databases Cronbach’s a of C1, C2, C3 and C4.

0.73

0.82

0.83

Table 8 Number of times a personalization parameter has been selected as one of the most significant ones. Personalization parameter

Active/Reflective dimension of the Felder–Silverman learning style model Visual/Verbal dimension of the Felder–Silverman learning style model Sequential/Global dimension of the Felder–Silverman learning style model Honey–Mumford learning style Learner’s level of knowledge Media preference

28 = C 82 personalization strategies (Each personalization strategy includes 2 personalization parameters selected from 8 parameters). The six courses are Microsoft Word (included 77 learning objects representing 7 concepts), Microsoft Excel (included 53 learning objects representing 8 concepts), Programming Language C (included 67 learning objects representing 5 concepts), Programming Language Maple (included 60 learning objects representing 19 concepts), Compilation Theory (included 174 learning objects representing 34 concepts), and Databases (included 140 learning objects representing 27 concepts). The 571 learning objects were annotated with metadata by using the tool Reload Editor2 and an IMS package3 was generated for each course with the same tool. Then, each IMS package was used as input to sort personalization strategies of the associated course based on the averages of ExigentCRC2, ExigentCRP2, TolerantCRC2, and TolerantCRP2. Three studies were conducted. First, these courses were analyzed based on the generalized metrics ExigentCRC2, ExigentCRP2, TolerantCRC2, and TolerantCRP2 to demonstrate that these metrics allows the automatic selection of appropriate personalization strategies. Second, the reliability of the metrics was calculated by using the Cronbach’s a in order to proof that the proposed metrics are reliable. Third, a comparative study was conducted to validate the metrics. The automatic selection of personalization strategies based on the metrics was compared with a previous manual selection of personalization strategies to demonstrate the accuracy and the efficiency of the metrics. 5.2. Results of the course analysis based on the metrics After the analysis of the courses based on the four metrics ExigentCRC2, ExigentCRP2, TolerantCRC2, and TolerantCRP2, the averages for each metrics were calculated. These averages are noted as follows: C1 is the average of ExigentCRC2. C2 is the average of ExigentCRP2. 2 3

http://www.reload.ac.uk/editor.html. http://www.imsglobal.org/content/packaging/.

Courses Programming Language C

Data Base

Microsoft Excel

17 2

10

3 7 3 2 5 11

3 15 2

13 8

C3 is the average of TolerantCRC2. C4 is the average of TolerantCRP2. Table 6 presents the combinations of personalization parameters for the analyzed courses. The rows represent the combined personalization parameters (28 combinations of personalization parameters). The sub-columns C1, C2, C3, and C4 of the table represent the average of ExigentCRC2, ExigentCRP2, TolerantCRC2 and TolerantCRP2 respectively. As a result of this experiment, we observed that the averages of the metrics have different values with respect to the availability of learning objects associated with learners’ characteristics. Accordingly, the metrics are useful to compare personalization strategies. We observed also that C1 6 C3. This observation is explained by the fact that ExigentCRC2 considers the combinations of learners’ characteristics using the operator and, while TolerantCRC2 considers the combinations of learners’ characteristics using the operator or. Similarly, we have C2 6 C4. 5.3. Reliability of metrics The reliability of C1, C2, C3 and C4 was calculated by using the Cronbach’s a which has been widely used in the literature, and it expresses significant reliability when the a value is near 1. In genPN 2 !! r i¼1 Y i N eral case, a ¼ N1 1  , where r2X is the variance of the r2 X

sum of items for the variable,

r2Y i is the variance of item i in the

variable, and N is the number of items in the variable. In our case, r2X is the variance of the sum C1 + C2 + C3 + C4 for each couple of P4 2 is equal to personalization parameters. i¼1 rY i

r2C1 þ r2C2 þ r2C3 þ r2C4 which is the sum of variance of C1, C2, C3, and C4. N is equal to four which is the number of metrics’ averages (C1, C2, C3 and C4). Hair, Anderson, Tatham, and Black (1998) recommended that Cronbach’s a values from 0.6 to 0.7 were deemed as the lower limit of acceptability. An a value of more than 0.7 would indicate that the metrics are coherent. By measuring Cronbach’s a, an acceptable rate of metrics reliability was found. Table 7 presents the metrics reliability calculated for the six courses and

319

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322 Table 9 Ranks of personalization strategies in the manual experimentation. Combination of personalization parameters

MP, SGFSLS ARFSLS, SGFSLS SGFSLS, VVFSLS SGFSLS, HMLS SGFSLS, LLK ARFSLS, HMLS HMLS, LLK VVFSLS, HMLS MP, HMLS MP, LLK ARFSLS, LLK VVFSLS, LLK MP, VVFSLS ARFSLS, VVFSLS MP, ARFSLS

Programming Language C

Data Base

Microsoft Excel

Sum

Rank

Sum

Rank

Sum

Rank

8 17 2 0 13 17 13 2 8 21 30 15 10 19 25

11 5 13 15 8 5 8 13 11 3 1 7 10 4 2

2 10 0 3 15 13 18 3 5 17 25 15 2 10 12

13 8 15 11 4 6 2 11 10 3 1 4 13 8 7

14 6 10 5 8 5 7 9 13 16 8 12 18 10 14

3 13 7 14 10 14 12 9 5 2 10 6 1 7 3

MP = Media preference. ARFSLS = Active/Reflective dimension of the Felder–Silverman learning style model. SGFSLS = Sequential/Global dimension of the Felder–Silverman learning style model. VVFSLS = Visual/Verbal dimension of the Felder–Silverman learning style model. HMLS = Honey–Mumford learning style model. LLK = Learner’s level of knowledge.

Table 10 Ranks of personalization strategies according to the metrics C1, C2, C3 and C4. Combination of personalizationparameters

Courses Programming Language C

MP. SGFSLS ARFSLS. SGFSLS SGFSLS. VVFSLS SGFSLS. HMLS SGFSLS. LLK ARFSLS. HMLS HMLS. LLK VVFSLS. HMLS MP. HMLS MP. LLK ARFSLS. LLK VVFSLS. LLK MP. VVFSLS ARFSLS. VVFSLS MP. ARFSLS RC1: RC2: RC3: RC4:

Rank Rank Rank Rank

of of of of

a a a a

personalization personalization personalization personalization

strategy strategy strategy strategy

according according according according

Data Base

Microsoft Excel

RC1

RC2

RC3

RC4

RC1

RC2

RC3

RC4

RC1

RC2

RC3

RC4

15 14 13 12 11 10 8 9 7 2 3 6 5 4 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 14 15 12 1 10 12 1 1 1 10 1 1 1

1 1 13 15 10 1 10 13 1 1 1 10 1 1 1

12 12 11 12 12 10 9 7 8 5 5 4 2 1 3

6 6 6 6 6 6 6 6 6 6 4 4 2 1 2

13 12 10 15 14 8 10 7 9 5 5 3 1 1 3

11 7 5 15 13 7 13 5 11 7 7 3 1 1 4

11 11 11 11 11 8 10 7 4 4 4 9 2 2 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

8 8 13 14 15 6 12 10 6 4 4 11 1 1 1

1 1 10 13 13 1 13 10 1 1 1 10 1 1 1

to to to to

C1. C2. C3. C4.

shows that the metrics reliability is significant for all the courses (>0.7). Consequently, the results demonstrate that our proposed metrics constitute reliable measurement instruments. 5.4. Comparative study In order to evaluate the generalized metrics, we compare the most significant personalization strategies (composed of two personalization parameters) selected manually by students (Essalmi et al., 2010a) with the most significant personalization strategies generated based on C1, C2, C3 and C4. Our previous study focused on the specification of personalization strategies (Essalmi et al., 2010a). The detailed procedure about this previous study and its reliability rates are presented in (Essalmi et al., 2010a). The participants have been 17 third year students (bachelor in computer science). In the previous experimentation, a

teacher had clarified an activity: The students had to determine manually the most significant personalization parameters for each course. Table 8 shows the total number of times that each personalization parameter has been selected as one of the two most significant personalization parameters for each course. As an example, for the course programming language C, the personalization parameter active/reflective dimension of the Felder–Silverman learning style model was selected most often (17 times). Since the set of personalization parameters included in our current study changed slightly from the set of personalization parameters included in the previous study, only personalization parameters are included in the comparison that are included in both studies. An important question which needs to be studied is: are the personalization strategies sorted in a good way (similar to the manual evaluation)? To test the defined metrics, the ranks of

320

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

Table 11 Distances between the ranks of personalization strategies selected manually and their ranks according to the automatic evaluation. Combination of personalization parameters

Courses Programming Language C

Data Base

Microsoft Excel

DC1

DC3

DC4

DC1

DC3

DC4

DC1

DC3

DC4

MP. SGFSLS ARFSLS. SGFSLS SGFSLS. VVFSLS SGFSLS. HMLS SGFSLS. LLK ARFSLS. HMLS HMLS. LLK VVFSLS. HMLS MP. HMLS MP. LLK ARFSLS. LLK VVFSLS. LLK MP. VVFSLS ARFSLS. VVFSLS MP. ARFSLS

4 9 0 3 3 5 0 4 4 1 2 1 5 0 1

10 4 1 0 4 4 2 1 10 2 0 3 9 3 1

10 4 0 0 2 4 2 0 10 2 0 3 9 3 1

1 4 4 1 8 4 7 4 2 2 4 0 11 7 4

0 4 5 4 10 2 8 4 1 2 4 1 12 7 4

2 1 10 4 9 1 11 6 1 4 6 1 12 7 3

8 2 4 3 1 3 2 2 1 2 6 3 1 5 2

5 5 6 0 5 8 0 1 1 2 6 5 0 6 2

2 12 3 1 3 13 1 1 4 1 9 4 0 6 2

Average Median

2.8 3

3.6 3

3.33 2

4.2 4

4.53 4

5.2 4

3 2

3.46 5

4.13 3

DC1: Distance between the rank of a personalization strategy selected manually and its rank according to C1. DC3: Distance between the rank of a personalization strategy selected manually and its rank according to C3. DC4: Distance between the rank of a personalization strategy selected manually and its rank according to C4.

personalization strategies (composed of two personalization parameters) selected manually by students in a past experiment are compared with the ranks of personalization strategies generated based on the defined metrics. The ranks of personalization strategies (which include two personalization parameters) in the manual experimentation are calculated. To do that, the sums of the numbers of times that the personalization parameters (included in each personalization strategy) were selected by students as most significant are calculated. For example, this sum is 30 (17 + 13) for the personalization strategy which includes the two parameters Active/Reflective dimension of the Felder–Silverman Learning Style Model (ARFSLS) and Learner’s Level of Knowledge (LLK) related to the course programming language C. Personalization strategies are sorted according to the calculated sums. Then, the ranks of the personalization strategies are generated. For example, the rank of the subset of personalization parameters {ARFSLS, LLK} for the course programming language C is 1 since there is no other personalization strategy with a sum higher than 30. Table 9 presents the calculated sums and the ranks of personalization strategies. In a next step, ranks of personalization strategies according to the metrics C1, C2, C3 and C4 are calculated. To do that, personalization strategies are sorted according to each metric. Accordingly, the rates of each personalization strategy (as presented in Table 6) are used to generate ranks of personalization strategies. Table 10 presents the ranks of personalization strategies according to the metrics C1, C2, C3 and C4. For example, in Table 10, the rank of the subset of personalization parameters {ARFSLS, LLK} selected automatically based on the metric C1 for the course programming language C is 3. The sub-columns RC1, RC3 and RC4 of Table 10 have different values while RC2 has the same values for two courses and only slightly different values for the third course. Accordingly, the ranks RC1, RC3 and RC4 could be used for comparing personalization strategies while RC2 seems to be less significant. In fact, for RC2, all the cells of its sub-columns for the courses Microsoft Excel and programming language C present the same value 1. As a consequence, RC2 could not be used for comparing personalization strategies. This could be explained by the fact that C2 is very

exigent and all its rates are very low. This study showed the practical limitation of C2. This limitation could be explained by the fact that there are two constraints to have a significant rate for C2. The first one is related to the exigent way of combining two learners’ characteristics. To consider a combination of two learners’ characteristics, the learning object must be appropriate to each of them. The second one is related to the exigent way of considering personalized concepts. A concept is considered as personalized if there are learning objects representing the same concept and appropriate for all combinations of two learners’ characteristics (included in the personalization parameters of the personalization strategy). Table 11 presents the distances between ranks of personalization strategies selected manually and their ranks according to the defined metrics (C1, C3 and C4). For example, the distance between the ranks of the personalization strategy {ARFSLS, LLK} in manual experimentation and its rank according to RC1 is 2 (=3  1). As can be seen in Table 11, the distances DC1, DC3 and DC4 have good averages and medians for each of the three courses compared to the maximal distance which is 14 = 15  1(In the conducted comparative study, 15 was the higher rank and 1 was the minimal rank.). These averages and medians represent significant similarity between the ranks of personalization strategies selected manually and their ranks according to C1, C3 and C4. Consequently, the metrics’ averages C1, C3 and C4 can be considered as efficient measurement instruments. 6. Conclusion, limits and potential future directions This paper presented a generalized approach for the analysis of personalization strategies. The approach is based on mapping between course and combinations of personalization parameters. This mapping allows for determining learning objects appropriate to learners’ characteristics and then using metrics for prevision of appropriate personalization strategies. The paper also presents an experiment conducted to prove the efficiency of the metrics analyzing personalization strategies. The analysis of personalization strategies has the potential to offer the following benefits to teachers:

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

 Reducing the efforts of developing personalized courses by selecting appropriate personalization strategies. This benefit is due to the possibility of analyzing personalization strategies composed of one or more personalization parameters based on the proposed metrics. In particular, teachers could be supported to select the appropriate personalization strategy which fits best to a particular course in terms that it has already most or all learning objects that can support rich personalization based on the respective personalization strategy and therefore, requires teachers to do only little or no modifications to the course. Furthermore, teachers are made aware of which personalization strategies would require significant modifications in the course.  Guidance for maintaining (enhancing) courses by considering specific combinations of learners’ characteristics which have not appropriate learning objects. Locating the non-fully personalized concepts (not represented by learning objects which are appropriate to all combination of learners’ Characteristics.) helps the teacher to select the parts of the course to reformulate and/or to enrich. Localization of non personalized concepts could be assured by ExigentCRPn and TolerantCRPn. In addition, ExigentCRCn and TolerantCRCn could be exploited to locate learning objects which do not consider some combinations of learners’ characteristics. The paper studies generalized metrics for the analysis of personalization strategies by considering the relations between the individual learners’ differences and the learning objects supporting these differences. This work could be extended by analyzing the following limits and potential future directions: Sorting the personalization strategies according to one of the defined metrics does not imply that it is necessary to select the personalization strategy which has the best rate. The teacher responsible of the course can decide to use another personalization strategy (such as the second one in the sorted list of the personalization strategies) when he/she does not prefer or has some constraints to use the first personalization strategy. This opens two potential future directions. The first one consists of modeling the teacher preferences of personalization strategies (for example, the teacher likes to include a specific personalization parameter in the personalization strategy to use). The second direction deals with considering general constraints of using personalization strategies. These constraints could be pedagogical (when a specific combination of personalization parameters is not recommended from pedagogical perspective) or technical (when a specific implementation related to a personalization parameter is not available). The experimentations are conducted in one university and show that the proposed approach is feasible, reliable and its automation generates significant results (similar to the manual result). This experimentation could be extended and generalized in several universities in order to confirm and/or have feedback about additional constraints to consider.

Acknowledgements The authors acknowledge Ms. Awatef Ben Arbia for the annotation of learning objects with metadata, and the students of the ESST Tunis who participated in the experimentation of the presented approach. Authors acknowledge also the support of NSERC, iCORE, Xerox, and the research related gift funding by Mr. A. Markin.

References Bouzeghoub, A., Carpentier, C., Defude, B., & Duitama, F. (2003). A model of reusable educational components for the generation of adaptive courses. In The first

321

international workshop on semantic web for web-based learning in conjunction with CAISE’03 conference (pp. 222–233), Klagenfurt, Austria. Brusilovsky, P., & Millán, E. (2007). User models for adaptive hypermedia and adaptive educational systems – the adaptive Web, LNCS 4321, Springer-Verlag. pp. 3–53. Brusilovsky, P., Schwarz, E., & Weber, G. (1996). A tool for developing adaptive electronic textbooks on WWW. In WebNet 96 conference proceedings (pp. 64– 69). San Francisco. Chookaew, S., Panjaburee, P., Wanichsan, D., & Laosinchai, P. (2014). A personalized E-learning environment to promote Students’ conceptual learning on basic computer programming. Procedia – Social and Behavioral Sciences, 116, 815–819. Chorfi, H., & Jemni, M. (2004). PERSO: Towards an adaptive E-learning system. Journal of Interactive Learning Research, 15(4), 433–447. Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-De-La-Cruz, L., & Ríos, A. (2004). SIETTE: A web-based tool for adaptive testing. International Journal of Artificial Intelligence in Education, 14, 1–33. Constantino-González, M., Suthers, D., & Santos, J. (2003). Coaching web-based collaborative learning based on problem solution differences and participation. International Journal of Artificial Intelligence in Education, 13, 261–297. Despotovic´-Zrakic´, M., Markovic´, A., Bogdanovic´, Z., Barac´, D., & Krcˇo, S. (2012). Providing adaptivity in moodle LMS courses. Educational Technology & Society, 15(1), 326–338. Dolog, P., Henze, N., Nejdl, W., & Sintek, M. (2005). Personalization in distributed Elearning environments. ACM, 170–179. Dwi, C. A., & Basuki, A. (2012). Personalized learning path of a web-based learning system. International Journal of Computer Applications., 53(7), 17–22. Essalmi, F., Jemni Ben Ayed, L., Jemni, M., Kinshuk & Graf, S. (2010a). A fully personalization strategy of E-learning scenarios. Computers in Human Behavior, 26(4), 581–591. Essalmi, F., Jemni Ben Ayed, L., Jemni, M., Kinshuk., & Graf., S. (2010b). Selection of appropriate E-learning personalization strategies from ontological perspectives. Special issue on the design centered and personalized learning in liquid and ubiquitous learning places. Interaction Design and Architecture(s) Journal, 9– 10, 65–84. Essalmi, F., Jemni Ben Ayed, L., Jemni, M., Kinshuk., & Graf., S. (2011). Evaluation of Personalization Strategies Based on Fuzzy Logic. In The 11th IEEE international conference on advanced learning technologies (pp.254–256). Georgia, USA. Felder, R. M., & Silverman, L. K. (1988). Learning and teaching styles in engineering education. Engineering Education, 78(7), 674–681. Graf, S., Kinshuk & Ives, C. (2010). A flexible mechanism for providing adaptivity based on learning styles in learning management systems. In Proceedings of the international conference on advanced learning technologies (pp. 30–34). IEEE. Hair, J. E., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis 5. NJ: Prentice-Hall. IEEE. (2002). Inc., Draf Standard for Learning Object Metadata. Isotani, S., Inaba, A., Ikeda, M., & Mizoguchi, R. (2009). An ontology engineering approach to the realization of theory-driven group formation. In International journal of computer supported collaborative learning, 4(4), Springer. http://dx.doi. org/10.1007/s11412-009-9072-x>. Klasnja-Milicevic, A., Vesin, B., Ivanovic, M., & Budimac, Z. (2011). E-Learning personalization based on hybrid recommendation strategy and learning style identification. Computers & Education, 56, 885–899. Kontopoulos, E., Vrakas, D., Kokkoras, F., Bassiliades, N., & Vlahavas, I. (2008). An ontology based planning system for e-course generation. Expert Systems with Applications, 35, 398–406. Kurilovas, E., & Zilinskiene, I. (2013). New MCEQLS AHP method for evaluating quality of learning scenarios. Technological and Economic Development of Economy, 19(1), 78–92. Limongelli, C., Sciarrone, F., Temperini, M., & Vaste, G. (2011). The Lecomps5 framework for personalized web-based learning: A teacher’s satisfaction perspective. Computers in Human Behavior, 27, 1310–1320. Milosevic, D., Brkovic, M., Bjekic, D. (2006). Designing lesson content in adaptive learning environments. International Journal of Emerging Technologies in Learning, 1(2). Retrieved 4 March, 2014, from www.i-jet.org. Mitrovic, A. (2003). An intelligent SQL tutor on the Web. International Journal of Artificial Intelligence in Education, 13, 171–195. Moodle. (2014). Retrieved 4 March, 2014, . Murray, T. (2003). MetaLinks: Authoring and affordances for conceptual and narrative flow in adaptive hyperbooks. International Journal of Artificial Intelligence in Education, 13, 199–233. National Academy of Engineering. (2014). Grand Challenges for Engineering for the 21st Century, retrieved September 19, 2014, from http://www. engineeringchallenges.org/. Papanikolaou, K. A., Grigoriadou, M., Kornilakis, H., & Magoulas, G. (2003). Personalizing the interaction in a web-based educational hypermedia system: The case of INSPIRE. In User modeling and user-adapted interaction (pp. 213–267). Kluwer Academic Publishers. Sampson, D., Karagiannidis, C., & Cardinali, F. (2002). An architecture for web-based e-learning promoting re-usable adaptive educational e-content. Journal of Educational Technology & Society, 5, 27–37. Sancho, P., Martínez, I., & Fernández-Manjón, B. (2005). Semantic Web technologies applied toe-learning personalization in . Journal of Universal Computer Science, 11(9), 1470–1481. Smith, A. S. G., & Blandford, A. (2003). MLTutor: An application of machine learning algorithms for an adaptive web-based information system. International Journal of Artificial Intelligence in Education, 13, 235–261.

322

F. Essalmi et al. / Computers in Human Behavior 48 (2015) 310–322

Soller, A. (2004). Computational modeling and analysis of knowledge sharing in collaborative distance learning. User Modeling and User-Adapted Interaction: The Journal of Personalization Research, 14, 351–381. Stash, N., Cristea, A., & de Bra, P. (2006). Adaptation to Learning Styles in ELearning: Approach evaluation. In T. Reeves & S. Yamashita (Eds.), Proceedings of world

conference on E-learning in corporate, government, healthcare, and higher education (pp. 284–291). AACE: Chesapeake, VA. Yang, T.-C., Hwang, G.-J., & Yang, S. J.-H. (2013). Development of an adaptive learning system with multiple perspectives based on students’ learning styles and cognitive styles. Educational Technology & Society, 16(4), 185–200.