Implementation and performance evaluation of parameter improvement mechanisms for intelligent e-learning systems

Computers & Education 49 (2007) 597–614 www.elsevier.com/locate/compedu

Implementation and performance evaluation of parameter improvement mechanisms for intelligent e-learning systems Chenn-Jung Huang *, San-Shine Chu, Chih-Tai Guan Institute of Learning Technology, National Hualien University of Education, 123 Huahsi Road, Hualien, Taiwan 970, Taiwan Received 28 July 2005; received in revised form 7 October 2005; accepted 7 November 2005

Abstract In recent years, designing useful learning diagnosis systems has become a hot research topic in the literature. In order to help teachers easily analyze studentsÕ profiles in intelligent tutoring system, it is essential that studentsÕ portfolios can be transformed into some useful information to reflect the extent of studentsÕ participation in the curriculum activity. It is observed that studentsÕ portfolios seldom reflect studentsÕ actual studying behaviors in the learning diagnosis systems given in the literature; we thus propose three kinds of learning parameter improvement mechanisms in this research to establish effective parameters that are frequently used in the learning platforms. The proposed learning parameter improvement mechanisms can calculate the studentsÕ effective online learning time, extract the portion of a message in discussion section which is strongly related to the learning topics, and detect plagiarism in studentsÕ homework, respectively. The derived numeric parameters are then fed into a Support Vector Machine (SVM) classifier to predict each learnerÕs performance in order to verify whether they mirror the studentÕs studying behaviors. The experimental results show that the prediction rate for the SVM classifier can be increased up to 35.7% in average after the inputs to the classifier are ‘‘purified’’ by the learning parameter improvement mechanisms. This splendid achievement reveals that the proposed algorithms indeed produce the effective learning parameters for commonly used e-learning platforms in the literature. Ó 2005 Elsevier Ltd. All rights reserved.

*

Corresponding author. Tel.: +886 3833 5657; fax: +886 3823 7408. E-mail address: [email protected] (C.-J. Huang).

0360-1315/$ - see front matter Ó 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2005.11.008

598

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

Keywords: Portfolio; Open source software; Machine learning; Support vector machines; Chinese knowledge and information processing

1. Introduction As the Internet gains wide popularity around the world, e-learning is taken by the learners as an important study aid. In the past few years, designing useful learning diagnosis systems has become a hot research topic in the literature (Cheng, Lin, Chen, & Heh, 2005; Cheung, Hui, Zhang, & Yiu, 2003; Depradine, 2003; Lo, Wang, & Yeh, 2004; Tsaganou, Grigoriadou, Cavoura, & Koutra, 2003; Zhang, Cheung, & Hui, 2001). Among them, the SmartTutor is an intelligent tutoring system built by the researchers at Hong Kong University. All behaviors of SmartTutor are decided by the rule-base which consists of expert rules that allow the Planner to determine the selection and ordering of materials that should be presented to the learners and allow the Advisor to generate the appropriate instructions and tests to the learners (Cheung et al., 2003; Zhang et al., 2001). Based on the embedded concept hierarchy of a test sheet, Cheng et al. (2005) explored the possibility of using a hierarchical coding and analytical procedure to diagnosis individual and class learning and misconceptions. In Tsaganou et al. (2003), fuzzy-case based reasoning technique is used to construct a system for diagnosis of studentsÕ cognitive profiles of historical text compression. The ability of the system to give similar results when using different historical texts holds potential for use in individualized history instruction in an intelligent tutoring system. Lo et al. (2004) developed a Hypermedia-based English Learning system for Prepositions (HELP) to provide EFL (English as a Foreign Language) students learning diagnosis and remedial instruction according to the confidence scores given by the students to indicate their confidence in their answers. They claimed the experimental results support the prediction that uses confidence scores helps the students become effective learners in adaptive hypermedia. In Depradine (2003), an expert system is used in a so-called Code Information Extractor (CITOR) to extract syntax and structural information from partially complete or incorrect Java code during the implementation phase of the software development cycle. The experimental results demonstrate the effectiveness of the CITOR in specialized software development tools such as integrated development environments, intelligent tutoring systems, an software engineering tools. It was observed that studentsÕ portfolios seldom reflect studentsÕ actual studying behaviors in the learning diagnosis systems found in the literature, such as the duration of online learning, the quality of posted articles in the discuss section, and so on (Althoff, 1995; Chen, 2002; Chen, Hsieh, & Chen, 2003; Tung, 2003; Wu & Leung, 2002). ChenÕs study showed that the support of diversified functionality in e-learning platform causes the learnersÕ profiles are full of data which are unrelated to the pupilsÕ learning behaviors (Chen, 2002). Wu and Leung (2002) reported that extraneous information is mixed with useful one increases the difficulty to obtain due results from the data. Althoff claimed that in a case based reasoning (CBR) diagnostic system the problem of noise, which means inherent ambiguities, can manifest itself as two or more cases with identical diagnosis but different inputs (Althoff, 1995). Via questionnaire survey and multi-variance analysis, Tung (2003) reported that several critical factors such as the information technology used in e-learning platform, the extent of the participation and interaction of the learners, and the

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

599

diversified content of e-learning platform, strongly affect the performance of e-learning. Meanwhile, the information technology used in e-learning platform also produces interference effects on other factors used in the e-learning platform. Chen et al. (2003) observed that studentÕs profile built for an intelligent diagnosis system can only present studentÕs learning status whereas it is ineffective to support the learning feedback and analysis mechanisms using studentÕs profile alone. Although a data analysis method was suggested in Chen et al. (2003) to do further analysis on the raw data in studentÕs profile, the prediction accuracy for the studentÕs performance is still poor. Based on the reports given above, we realize that studentÕs profiles in an intelligent tutoring system need to be further analyzed and transformed into some useful information to reflect the extent of studentsÕ participation in the curriculum activity. Otherwise the performance of the intelligent diagnosis systems might be degraded to certain extent owing to the noisy input data. To the best of our knowledge, there is no much research work regarding extraction of useful information from studentÕs profiles in the literature so far. We thereby propose three kinds of learning parameter improvement mechanisms in this work to establish effective parameters that can be adopted in the frequently used learning platforms. The proposed learning parameter improvement mechanisms are able to calculate the studentsÕ learning effectiveness based on the duration of their online learning, extract the portion of a message in discussion section which is strongly related to the learning topics, and detect the degree of plagiarism in studentsÕ homework reports, respectively. The derived numeric parameters are then used to predict the learnersÕ performance in order to verify whether they mirror the studentÕs studying behaviors. Recently, a lot of e-learning platforms can be used for free on the Internet due to the increasing availability of Open Source software. In this work, we incorporate learning parameter improvement mechanisms into an Open Software e-learning platform, Moodle, http://moodle.org/, to verify the feasibility of the proposed algorithms. The most important reason for the choice of Moodle as the pupilsÕ online learning platform is that Moodle is used for free and is designed to support a social constructionist framework of education, such as collaboration, activities, critical reflection, etc. Meanwhile,there are several striking features in Moodle that well suit the pupilsÕ online learning in this work:  It is suitable for 100% online classes as well as supplementing face-to-face learning.  It supports simple, lightweight, efficient, compatible, low-tech browser interface.  Most text entry areas, such as resources, forum postings, etc., can be edited using an embedded WYSIWYG HTML editor.  It allows flexible array of course activities – Forums, Quizzes, Resources, Choices, Surveys, Assignments, Chats, Workshops.  The teachers can define their own scales to be used for grading forums and assignments.  It is effortless to proceed with further study and analysis on the learnersÕ profiles as they are comprehensively established. The experiments were conducted in two fifth grade classes at an elementary school, and the original data for the experiments were obtained from the learnersÕ portfolio established during online learning activities. The above-mentioned effective learning parameters generated by the learning parameter improvement mechanisms are then fed into a Support Vector Machine (SVM) (Burges, 1998; Chang & Lin, 2001; Cherkassky & Mulier, 1999; Vapnik, 1995; Vapnik, 1998) classifier to

600

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

predict each learnerÕs achievement. The experimental results verify the effectiveness of the learning parameter improvement mechanisms and the proposed algorithms is proved to be capable of generating the effective learning parameters for commonly used e-learning platforms in the literature. The remainder of the paper is organized as follows. Section 2 shows the details of the learning parameter improvement mechanisms. In Section 3, we will give a brief description of the SVM classifier used in the experiment. Section 4 reviews and discusses the experimental results. Conclusions and the future work are made in Section 5.

2. Architecture of learning parameter improvement mechanism Fig. 1 shows the architecture of the e-learning platform used in this work. The three learning parameter improvement mechanisms are illustrated inside the blue dotted frame. As seen from the figure, we employed the Moodle e-learning platform to assist in teaching at a fifth grade elementary Natural Science course. The inputs to the three learning parameter improvement mechanisms were obtained from the learnersÕ portfolio established during online learning activities. The effective learning parameters obtained from the learning parameter improvement mechanisms are then fed into a Support Vector Machine (SVM) classifier to predict each learnerÕs achievement and check if it is consistent with his/her grade of midterm examination that was held at the end of online learning activities.

Learners

Teacher

User Interface

Moodle E-Learning Platform Database

Online Learning Effective ness Calculator

Learning Parameter Database

Experimental Natural Science Course

Learning Topic Relevance Identifier

Homework Plagiarism Detector

CKIP

Learning Parameter Improvement Mechanism

Fig. 1. Architecture of the e-learning platform.

Learning Performance Prediction Database

SVM Classifier

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

601

2.1. Online learning effectiveness calculator Fig. 2 shows the flow diagram of each learnerÕs online learning effectiveness calculation process. The motivation of using this scheme comes from the observed learning pattern of the pupils in browsing the web pages. It is found that the pupils often spend a regular amount of time in reading the contents of the attractive web pages; whereas skip through the web pages rapidly that do not interest them. Moreover, if the pupils spend unusual long time on a web page, it is very often that the pupils were distracted by something else like chatting with other students or doing something else such as playing multiplayer online game, etc. (Budny & Hein, 1999; Leuthold, 1999). We thus try to adopt statistics approach in this work to derive the learning effectiveness value for each learner based on the browsing time of each web page for the learning materials organized by the instructor when the pupils surf on the e-learning platform. When each web page has similar inherent complexity and difficulty level, the learning effectiveness value of a web page is larger if the learning time spent on browsing the web page is closer to the regular browsing time that each individual learner spent on the web page in average. Next we give a detailed description of the online learning effectiveness calculation algorithm. The input Ti, shown on the upper-left of Fig. 2 represents learner XÕs browsing time of the ith web page during his/her online learning activities. Notably, the browsing time measured is a single trip to the web page instead of a sum of trips to the page over time. In this work we first compute learner XÕs average browsing time of each web page, n P Ti T 1 þ T 2 þ . . . þ T n i¼1  ¼ ; ð1Þ T ¼ n n where n represents the total number of web pages that learner X browsed. We then compute the deviation of the effective learning time for browsing the ith web page, ð2Þ d i ¼ T i  T . The bias of the effective learning time period for browsing the ith web page is defined as, di bi ¼  . T

ð3Þ T1 T2 T3

Average learning time T =

…

Total learning time Total number of browsed web pages n

Tn

Learning effectiveness for Learner X

le total = le 1+ le 2+

+ le n

Learning effectiveness of the ith web page browsing 1 le i bi 2 1

Deviation of learning time for the ith web page di

= Ti − T

Bias of learning time for the ith web page d bi = i T

Fig. 2. Flow diagram of each learnerÕs online learning effectiveness calculation.

602

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

Next we compute the weight value of the ith web page that represents the learning effectiveness when learner X browsed the ith web page, 1 . ð4Þ lei ¼ 2 bi þ 1 Notably, the integer one is added to the denominator to resolve the infinity problem when the bias is zero. Accordingly, lei becomes one when the bias bi is zero. This also consists with the definition of the learning effectiveness in this work since the learner spent a regular learning time in browsing the ith web page when the bias bi is zero. Furthermore, all the web pages organized for the learning materials on the e-learning platform are assumed to have similar complexities and difficulty levels in this work. In case different pages have varied inherent complexities and difficulty levels, the instructor should specify a difficulty level for each web page that is proportional to the estimated web page browsing time for each pupil, and then the rectified average browsing time of each web page is given by: T1 w T ¼ 1

þ Tw22 þ . . . þ wT nn

; ð5Þ n where wi denotes the complexity and difficulty level of ith web page. The deviation of the effective learning time for browsing the ith web page as given by Eq. (2) should be updated as follows accordingly, di ¼

Ti   T. wi

ð6Þ

The learning effectiveness that learner X achieved after browsing n web pages can be cumulated as follows, n X lei . ð7Þ letotal ¼ i¼1

Fig. 3 shows a sample learning effectiveness list generated by the system. The measurement of effective learning is computed by Eqs. (1)–(5) and then displayed on the web site when requested by the teacher.

List of Learning Effectiveness ID

Nick Name

Name

Learning Effectiveness

1

Shine

Tiffany Chiang

21.5

2

Teddy

Teddy Chen

1.19418e-010

3

Sunnyboy

Sunnyboy Chu

2

4

Jesamine

Jesamine Lin

0.500006

5

Moodle

Guest

0.518303

Fig. 3. List of learning effectiveness generated by the system.

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

603

2.2. Learning topic relevance identifier Fig. 4 shows the flow diagram of learning topic relevance identification process. In this mechanism, we determine the learning topic relevance to the articles posted on the discussion board by establishing the norm of word occurrence frequency distribution. The teacher first picks up the sample articles posted by the learners from the discussion board and assigns a value of relevance degree between the learning topics and the sample articles. Then we employ the so-called Chinese Knowledge and Information Processing (CKIP) system (Chen & Bai, 1998; Chen & Liu, 1992; Chen & Ma, 2002; Ma & Chen, 2003) developed by Academia Sinica in Taiwan to separate the Chinese words in each sample article. The purpose of CKIP system is to establish a fundamental research environment for Chinese natural language processing. A Chinese sentence contains no delimiters, such as a space, to separate words. Accordingly, a typical word segmentation system tries to find the possible word compositions of a sentence by comparing it with a lexicon, which results in word segmentation ambiguities. Most Chinese word segmentation systems deal with the problem of resolving ambiguity, rather than identifying unknown words that might possess up to 5% of all the words in an article. The Chinese word segmentation method proposed by CKIP system is claimed to be able to tackle unknown word identification. Its word segmentation process is based on the 100,000 built-in lexicons, morphological rules for quantifier words and reduplicated words, online extracted words, and additional domain-specific lexicons specified by users. The combination of word segmentation result generated by CKIP system and the value of relevance degree generated for each sample article are formed as the norm of word occurrence frequency distribution. This mechanism then refers to the norm to automatically issue a relevance degree value for each newly posted article as an effective measurement for each learnerÕs learning status on the discussion board. A summary of learning topic-related word occurrence frequency distribution norm construction algorithm is given below:

Fig. 4. Flow diagram of learning topic relevance identification process.

604

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

(1) We assume there are n sample articles posted by the learners on the discussion board. They are fed into the CKIP system to extract each individual word in a Chinese sentence. The word segmentation result for the ith sample article is [Pi,1, Pi,2, . . ., Pi,ki]T, where ki denotes the index for the last extracted word in the ith sample article. Fig. 5 illustrates the distribution of word segmentation for the n sample articles. Notably, the norm can be expressed as [Pi,1, Pi,2, . . ., Pi,ki, . . ., Pi,1, Pi,2, . . ., Pi,ki, Pn,1, Pn,2, . . ., Pn,kn]T in mathematical form. (2) We compute occurrence frequency for each separated word in n sample articles as shown in Fig. 6. Here we can see that word Pij appears Cij times. Then the total occurrence frequency of all words in the ith sample article, TCi, can be expressed as, k1 X C i;j . ð8Þ TC i ¼ j¼1

Let Rij denote the ratio of occurrence frequency of word Cij, to total word counts in the ith sample article. Then Rij can then be expressed as, C i;j . ð9Þ Rij ¼ TC i The distribution of word occurrence frequency ratio for the ith sample article can then be expressed as, T ð10Þ RFi ¼ ½Ri;1 ; Ri;2 ; . . . Ri;ki  . P1,1

P1,2

P1,3

P1,k1

P2,1

P2,2

P2,3

P2,k2

Pn,1

Pn,2

Pn,3

Pn,kn

Fig. 5. Word segmentation result for n sample articles.

C1,1

C1,2

C1,3

C1,k1

C2,1

C2,2

C2,3

C2,k2

Cn,1

Cn,2

Cn,3

Cn,kn

Fig. 6. Word occurrence frequency for n sample articles.

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

605

(3) As shown in Fig. 7, word occurrence frequency ratio for sample article i can be further weighted by the relevance degree between learning topics and the ith sample article issued by the teacher, RDi, TUi ¼ ½Ri;1 ; Ri;2 ; . . . ; Ri;ki T  RDi ¼ ½URi;1 ; URi;2 URi;3 ; . . . ; URi;ki T .

ð11Þ

(4) The weighted word occurrence frequency ratios for the same words in n sample articles sum together so mat no duplicate words appear in the distribution norm. The distribution norm can be further revised when the teacher goes through the contents of distribution norm and discards the words that are unrelated to the learning topics. The final distribution norm of weighted word occurrence frequency ratio for n sample articles, which is illustrated in Fig. 8, is then expressed by a long vector, T

UR ¼ ½UR1;1 ; UR1;2 ;    ; UR1;k10 ; UR2;1 ; UR2;2 ;    ; UR2;k20 ;    ; URn;1 ; URn;2 ;    ; URn;kn0  ;

ð12Þ

where the index for the last segmented word ki in the ith sample article is replaced with the index kiÕ to reflect the decrement of the number of segmented words in each sample article due to the merge of identical words within n sample articles and the removal of the words that are unrelated to the learning topics. Learning topic relevance degree

TU1 RF1 =[R1, 1 ,R1, 2 ,R1, 3 ,

, R1,k1]T × of article 1 assigned by teacher

TU2 RF1 =[R1, 1 ,R1, 2 ,R1, 3 ,

topic relevance degree , R1,k1]T × Learning of article 2 assigned by teacher

TU3 RF2 =[R2, 1 ,R2, 2 ,R2, 3 ,

Learning topic relevance degree , R2,k2]T × of article 3 assigned by teacher

TUn RFn =[Rn, 1 ,Rn, 2 ,Rn, 3 ,

Learning topic relevance degree , Rn,kn]T × of article n assigned by teacher

Fig. 7. Word occurrence frequency ratio weighted by the relevance degree.

UR1,1

UR1,2

UR1,3

UR1,k1'

UR2,1

UR2,2

UR2,3

UR2,k2'

UR3,1

UR3,2

UR3,3

UR3,k3'

URn,1

URn,2

URn,3

URn,kn

Fig. 8. The final norm of weighted word occurrence frequency ratio distribution.

606

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

After the norm of weighted word occurrence frequency ratio distribution is constructed, all the words in each new article posted on the discussion board will be segmented first by the CKIP system again and the distribution of word occurrence frequency ratio for the new article RFnew, should be realign with the word sequence in the norm of weighted word occurrence frequency ratio distribution as shown in Fig. 8. The value of relevance degree between the learning topics and the new posted article issued by our algorithm is expressed as, RDnew ¼

UR  RFnew ; jURj  jRFnew j

ð13Þ

where the  operator in the numerator denotes the inner product operator between two vectors and the denominator is used for normalization of the two vectors. We assume that a learner X posted m new articles on the discussion board, then the cumulative relevance degree issued for learner X based on the m articles is given by, m X RDi ; ð14Þ CRDX ¼ i¼1

where RDi is given by Eq. (13). Fig. 9 shows an example web page that exhibits the value of teacher-given relevance degree between the learning topics and the sample articles, while Fig. 10 gives an example of derived cumulative relevance degree between the learning topics and all the articles posted by each learner.

Fig. 9. Teacher-given relevance degree between learning topics and the sample articles.

Learner

Cumulative relevance degree

Teddy Chen

12.026895355911

Jesamine Lin

0

Sunnyboy Chu

59.207348917957

Fig. 10. Cumulative relevance degree computed for all the articles posted by each learner.

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

607

2.3. Homework report plagiarism detector In this mechanism, the grade of each learnerÕs homework is obtained by taking the average of the measurement of each learnerÕs homework report relevant to the learning topics and the degree of non-plagiarism estimated for each learnerÕs homework report. The way of measuring each learnerÕs homework report relevant to the learning topics is similar to the approach taken in Section 2.2. The learnerÕs homework report is first fed into the CKIP system to separate the words, and then the word segmentation result is compared with the word occurrence frequency distribution norm built by the learning topic relevance identifier as presented in Section 2.2. If the content of the homework report is strongly related to the learning topics, the value of the issued learning topic relevance degree will be also large. As for the measurement of plagiarism for each learnerÕs homework, we will compare each learnerÕs homework report with othersÕ and establish the similarity degree value of each learnerÕs homework report. The flow diagram of homework plagiarism detection process is illustrated in Fig. 11. A summary of homework report plagiarism detection algorithm used in this work is listed below: (1) Separate words in each learnerÕs homework using the CKIP system and compute the word occurrence frequency for each homework report. (2) Sort the word segmentation result for each two learnerÕs homework reports and put word occurrence frequency for the two homework reports in two vectors TC1 and TC2 as follows, 3 2 Cj;1 6 Cj;2 7 7 6 7 ð15Þ TCj ¼ 6 6 .. 7; 4 . 5 Cj;g where j stands for the index for the two homework reports and g denotes the total counts of the sorted word segmentation for each of the two homework reports.

Fig. 11. Flow diagram of homework report plagiarism detection process.

608

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

(3) The similarity between two homework reports can be expressed as, S 1;2 ¼

TC1  TC2 ; jTC1 j  jTC2 j

ð16Þ

where the denominator is sorely used for normalization purpose. (4) The degree of non-plagiarism for each learnerÕs homework is then given by, m P 1  S ij NPi ¼

j¼1;j6¼i

. ð17Þ m1 where m denotes the number of the learners, and Eq. (17) computes the average of the nonplagiarism degree between learner iÕs homework report and each report submitted by rest of m  1 learners. Figs. 12 and 13 show two example web pages that measure the relevance degree between the learning topic and each learnerÕs homework report, and the non-plagiarism degree between the homework report of each learner and those of the other learners, respectively. Notably, the above-mentioned relevance degree and the non-plagiarism degree are scaled so as to fall within the commonly adopted range 0–100. Table 1 shows the final grade of each learnerÕs homework report, which is obtained by computing the average of the relevance degree between the learning topics and each learnerÕs homework report, and the non-plagiarism degree between each learnerÕs report and those of the rest learners as shown in Figs. 12 and 13, respectively.

Learner

Relevance degree

Tiffany Chiang

78.9

Teddy Chen

93.2

Sunnyboy Chu

93.9

Jesamine Lin

54.1

Guest

100

Fig. 12. Relevance degree between the learning topics and each learnerÕs homework report.

Learner

Degree of non-plagiarism

Tiffany Chiang

55.250647754082

Teddy Chen

92.646970309018

Sunnyboy Chu

64.389873685387

Jesamine Lin

78.130269880329

Guest

0

Fig. 13. Non-plagiarism degree between each learnerÕs report and those of rest of the learners.

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

609

Table 1 Final grade of each learnerÕs homework report Learner

Relevance degree

Non-plagiarism degree

Grade of homework report

Tiffany Chiang Teddy Chen Chu sunnyboy Lin jesamine Guest

78.9 93.2 93.9 54.1 100.0

55.25 92.65 64.39 78.13 0

67.1 92.9 79.2 66.1 50.0

3. SVM classifier Support vector machines (SVM) have recently gaining popularity due to its numerous attractive features and eminent empirical performance (Burges, 1998; Chang & Lin, 2001; Cherkassky & Mulier, 1999; Vapnik, 1995; Vapnik, 1998). The main difference between the SVM and conventional regression techniques is that it adopts the structural risk minimization (SRM) approach, as opposed to the empirical risk minimization (ERM) approach commonly used in statistical learning. The SRM tries to minimize an upper bound on the generalization rather than minimize the training error, and is expected to perform better than the traditional ERM approach. Moreover, the SVM is a convex optimization, which ensures that the local minimization is the unique minimization. To solve a non-linear regression or functional approximation problem, the SVM non-linearly map the input space into a high-dimensional feature space via a suitable kernel representation, such as polynomials and radial basis functions with Gaussian kernels. This approach is expected to construct a linear regression hyperplane in the feature space, which is non-linear in the original input space. Then the parameters can be found by solving a quadratic programming problem with linear equality and inequality constraints (Vapnik, 1995). It is assumed that a training data set D ¼ fðxi ; y i Þ 2 Rn  R; i ¼ 1; . . . ; lg which consists of l pair training data (xi,yi), i = 1,. . .l, is given. The inputs xiÕs are n-dimensional vectors, and the system responses yiÕs are continuous values. Based on the knowledge of data set D, the SVM attempts to approximate the following function: N X f ðx; wÞ ¼ wi  ui ðxÞ þ b; ð18Þ i¼1

where b is the bias term, and wiÕs are the subjects of learning. Moreover, a mapping z = U(x) is chosen in advance to map input vectors x into a higher-dimensional feature space F, which is spanned by a set of fixed functions ui(x)Õs. By defining a linear loss function with the following e-insensitivity zone as shown in Fig. 14,  0 if jy i  f ðxi ; wÞj 6 e ; ð19Þ jy i  f ðxi ; wÞje ¼ jy i  f ðxi ; wÞj  e otherwise wiÕs in Eq. (18) can be estimated by minimizing the risk: ! l X 1 2 jy i  f ðxi ; wÞje ; R ¼ kwk þ C 2 i¼1

ð20Þ

610

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614 e

y f(x,w)

Fig. 14. e-In sensitivity loss function.

where C is a user-chosen penalty parameter that determines the trade-off between the training error and VC dimension of the SVM model. Note that the VC dimension is a scalar value that measures the capacity of a set of functions (Vapnik, 1995). Eq. (20) can be further derived into the following constrained optimization problem: ! l l X X 1 2  ð21Þ nþ n ; Rðw; n; 1Þ ¼ kwk þ C 2 i¼1 i¼1 subject to constraints 8 T > < y i  w xi  b 6 e þ n; wT xi þ b  y i 6 e þ n ; > : n; n P 0;

ð22Þ

where n and n* represent the measurements above and below the zone with the radius e in VapnikÕs loss function as given in Eq. (19), respectively. It can be shown (Vapnik, 1995) that the above constrained optimization problem is solved by applying the Karush–Kuhn–Tucker (KKT) conditions (Taha, 1997) for regression, and maximizing the following Lagrangian: LðaÞ ¼ 0:5aT Ha þ f T a; under constraints 8 l l P P >  > > < ai ¼ ai ; i¼1

i¼1

0 6 ai 6 C; > > > : 0 6 ai 6 C;

i ¼ 1; . . . ; l;

ð23Þ

ð24Þ

i ¼ 1; . . . ; l

where f ¼ ½e  y 1 ; e  y 2 . . . e  y N e þ y 1 ; e þ y 2 . . . e þ y N ; ðai ; ai Þ denotes one of l Lagrange multiplier pairs, and the Hessian matrix H is given as

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

 H¼

G

G

G

G

 .

611

ð25Þ

G denotes the corresponding kernel matrix. The best non-linear regression hyperfunction is then given by f ðx; wÞ ¼ Gwo þ bo ;

ð26Þ

where wo and bo denote the optimal desired weights vector and the optimal bias, respectively. wo and bo can be derived by wo ¼ a  a; l 1X bo ¼ ðy  gi Þ; l i¼1 i

ð27Þ ð28Þ

where g = G wo.

4. Experimental results and analyses To verify the effectiveness of learning parameter improvement mechanisms proposed in this work, two fifth grade classes at an elementary school have been chosen to use the Moodle e-learning platform during classroom teaching activities in a Natural Science course. Since the data collected in the learnersÕ learning profiles tend to be noisy and inconsistent, the proposed learning parameter improvement mechanisms are incorporated into the Moodle e-learning platform to identify outliers and correct inconsistencies in the collected data. The ‘‘purified’’ data is then fed into a SVM classifier to predict each learnerÕs performance and compare it with each learnerÕs grade of midterm examination. The inputs to the SVM classifier include each learnerÕs learning effectiveness extracted from the browsing time for each web page that the learner surfs on the Internet as given by Eq. (5), cumulative relevance degree issued for all the articles posted by each learner as given by Eq. (14), and the grade of each learnerÕs homework report which is computed by taking the average of the relevance degree between the learning topics and each learnerÕs homework report, and the non-plagiarism degree between the report of each learner and those of the rest learners as illustrated by Eqs. (12) and (15), respectively. Twenty-six pupils in each of the two classes joined the learning activities offered on the online learning website during a span of seven weeks period. Then we ran a series of tests by using a socalled leaving-one-out cross validation technique (LOOCV) (Hand, Mannila, & Smyth, 2001) to examine the validity of the learning parameter improvement mechanisms by feeding the ‘‘contaminated’’ and ‘‘purified’’ input data into a SVM classifier in turn, and comparing the results of the two classification outputs. Notably, the so-called LOOCV method removes a single sample in each trial, trains on the rest, and then tests the classifier on the removed single sample. Besides, we employed the SVM library developed by Lin Chang and Lin (2001) in this work to do classification work. Table 2 shows the prediction accuracy comparison of the outcomes of the SVM classifier that are fed with the ‘‘contaminated’’ and ‘‘purified’’ input data in the experiment of two fifth grade classes at an elementary school, where the heading ‘‘LPIM’’ refers to the cases when the learning parameter improvement mechanisms are included in the e-learning platform, and ‘‘No LPIM’’

612

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

Table 2 Performance comparison of three learning parameter improvement mechanisms Class

IPIM (%)

No IPIM (%)

Improvement ratio (%)

5-A 5-B

69.23 69.23

50 52

38.46 33.13

Table 3 Performance comparison of each individual parameter improvement mechanism Input data

Online learning effectiveness calculation (%)

Learning topic relevance identification (%)

Homework report plagiarism detection (%)

Purified Contaminated

61.54 50

53.85 51

67.31 51

represents the cases that the learning parameter improvement mechanisms are excluded in the elearning platform, respectively. Table 2 reveals that the prediction result of the input data filtered by the learning parameter improvement mechanisms is more accurate up to 38.46% and 33.13% than that for the unfiltered input data in the two classes, respectively. The evidence given in the experimental results demonstrates that the learning parameter improvement mechanisms indeed effectively remove the noisy data and correct inconsistencies in the learnersÕ learning profiles. We further ran a round of tests to examine the effectiveness of each individual learning parameter improvement mechanism by using the data extracted by each of the three proposed mechanisms as the sole input parameter to the SVM classifier in turn. Table 3 compares their classification results with the corresponding outcomes predicted with each single unfiltered parameter. The outcomes manifest that the employment of three filtered input parameters combination as the inputs to the classifier is feasible since the three parameters combination performs better than each individual filtered parameter alone when compared with the result illustrated in Table 2. Meanwhile, the algorithms of the online learning effectiveness calculator and the homework report plagiarism detector achieve much better performance than the learning topic relevance identifier does. We believe it is mainly because the effectiveness of the learning topic relevance detection algorithm is still limited owing to the failure of recognizing the quality of the articles posted by the learners, even though they might cite lots of learning topic-related terms in the posted articles. We also investigated why the experimental results showed some discrepancy between the predicted and actual performance of several pupilsÕ grades of the midterm examination. Based on the pupilsÕ replies, we found out that the midterm examination was held a week after the end of the online learning activities, and the pupils that performed better in the midterm examination studied extraordinarily harder than usual during the week before the midterm examination. It is why that those pupilsÕ grades of midterm examination were not truly reflected by their learning profiles.

5. Conclusions and future work It is well known that studentsÕ portfolios seldom reflect studentsÕ actual studying behaviors in the learning diagnosis systems. We thus proposed three learning parameter improvement

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

613

mechanisms in this work to remove the noise and correct inconsistency in the learnersÕ learning profiles. The first parameter ‘‘purified’’ in this work is each learnerÕs learning effectiveness extracted from the browsing time for each web page that the learner surfs on the Internet; and the second one is the cumulative relevance degree issued for all the articles posted by each learner; and the third parameter is the grade of each learnerÕs homework report based on relevance degree between the learning topics and each learnerÕs homework report and the non-plagiarism degree between the homework report of each learner and those of the rest learners. We employed a SVM classifier to examine the performance of the three learning parameter improvement mechanisms. A series of leaving-one-out cross validation tests verify that our algorithms are very effective in extracting useful parameters from learnersÕ learning profiles and the selection of the three kinds of parameters as the inputs to the classifier is appropriate. The three types of ‘‘purified’’ parameters can also be adopted as the inputs to an intelligent learning diagnosis system to give the learners and the teacher appropriate feedback that reflects the learning status of the learners during learning activities on an e-learning platform. In the case of an inconsistency between the predicted and actual performance, for instance, if some specific learner usually achieved distinguished performance in the past whereas the personal profiles on the e-learning platform reflect that his/her achievement turns bad in the experiment, the teacher can then investigate whether the poor performance is affected by the unfamiliarity with the usage of the computer, or the learnerÕs learning style does not fit the collaborative learning adopted in the e-learning platform. In the future work, we plan to construct an intelligent learning diagnosis system by using fuzzy expert systems technique, wherein the three parameters extracted in this work will be adopted as the inputs to the learning diagnosis system. The reason to use the fuzzy expert system is that it can function more like human experts who explain the reasoning processes behind their recommendation. Besides, we will investigate the possibility of other choices of parameter combination from the learnersÕ learning profiles to be used as the input arguments to the learning diagnosis system. The corresponding parameter improvement mechanisms will also be designed to ‘‘purify’’ the chosen data. Moreover, we will set up an online testing system in the future work to give the pupils a quiz right after the end of learning activities on the e-learning platform to truly reflect the effectiveness of the ‘‘purification’’ process of the parameters extracted from learnersÕ learning profiles. Acknowledgement This research was partially supported by National Science Council under grant NSC 93-2213E-026-001.

References Althoff, K.-D. (1995). CBR: a new force in advanced systems development. In A. Aadmodt, K.-D. Althoff, R. Magaldi, & R. Mine (Eds.). London: Tutorial. Budny, D. D., & Hein, T. L. (1999). Teaching to studentsÕ learning styles: approaches that work. Frontiers in Education Conference, 2(Nov), 12C1/7–12C114.

614

C.-J. Huang et al. / Computers & Education 49 (2007) 597–614

Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167. Chang, C.-C., & Lin, C.-J. (2001). LIBSVM: a library for support vector machines, 2001. Software available from http://www.csie.ntu.edu.tw/~cjlin/Libsvm. Chen, S.-C. (2002). A study of learning behaviors of elementary students in the web project-based learning environment. Master Thesis, National Tainan University. Chen, K. J., & Bai, M.-H. (1998). Unknown word detection for Chinese by a corpus-based learning method. International Journal of Computational linguistics and Chinese Language Processing, 3(1), 27–44. Chen, K. J., & Liu, S. H. (1992). Word identification for mandarin Chinese sentences. In Proceedings of COLING 1992 (pp. 101–107). Chen, K. J., & Ma, W.-Y. (2002). Unknown word extraction for Chinese documents. In Proceedings of COLING 2002 (pp. 169–175). Chen, Y.-P., Hsieh, H.-W., & Chen, N.-H. (2003). Dynamic constructing decision rules from learning portfolio to support adaptive instruction. In Proceedings of National Computer Symposium, December. Cheng, S.-Y., Lin, C.-S., Chen, H.-H., & Heh, J.-S. (2005). Learning and diagnosis of individual and class conceptual perspectives: an intelligent systems approach using clustering techniques. Computers and Education, 44(3), 257–283. Cherkassky, V., & Mulier, F. (1999). Vapnik–Chervonenkis (VC) learning theory and its applications. IEEE Transactions on Neural Networks, 10(5), 985–987. Cheung, B., Hui, L., Zhang, J., & Yiu, S. M. (2003). SmartTutor: an intelligent tutoring system in web-based adult education. The Journal of Systems and Software, 68, 11–25. Depradine, C. (2003). Expert system for extracting syntactic information from Java code. Expert Systems with Applications, 25, 187–198. Hand, D., Mannila, H., & Smyth, P. (2001). Principles of data mining. MIT press. Available from http://moodle.org/. Leuthold, J. H. (1998). Is computer-based learning right for everyone? In Proceedings of the 32nd annual Hawaii international conference on system sciences (Vol. 1, p. 8). Lo, J.-J., Wang, H.-M., & Yeh, S.-W. (2004). Effects of confidence scores and remedial instruction on prepositions learning in adaptive hypermedia. Computers and Education, 42, 45–63. Ma, W.-Y., & Chen, K. J. (2003). A bottom-up merging algorithm for Chinese unknown word extraction. In Proceedings of ACL workshop on Chinese language processing 2003 (pp. 31–38). Taha, H. A. (1997). Operations research: an introduction. Prentice-Hall, Inc.. Tsaganou, G., Grigoriadou, M., Cavoura, T., & Koutra, D. (2003). Evaluating an intelligent diagnosis system of historical text comprehension. Expert Systems with Applications, 25, 493–502. Tung, H.-K. (2003). A study on the critical factors of the performance of e-learning application on education from the learning theory perspective. Master Thesis, Ming-Chuan University. Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer-Verlag. Vapnik, V. (1998). The support vector method of function estimation. In J. A. K. Suykens & J. Vandewalle (Eds.), Nonlinear modeling: advanced black-box techniques (pp. 55–85). Boston, MA: Kluwer. Wu, A. K. W., & Leung, C.-H. (2002). Evaluating learning behavior of web-based training (WBT) using Web Log. In Proceedings of the International Conference on Computers in Education, December. (pp. 736–737). Zhang, J., Cheung, B., & Hui, L. An intelligent tutoring system: SmartTutor. In Proceedings of 2001 world conference on educational multimedia, hypermedia and telecommunications, June 2001 (pp. 2130–2131).