A fuzzy inference system-based criterion-referenced assessment model

Expert Systems with Applications 38 (2011) 11129–11136

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

A fuzzy inference system-based criterion-referenced assessment model Kai Meng Tay a,⇑, Chee Peng Lim b a b

Electronic Engineering Department, Faculty of Engineering, Universiti Malaysia Sarawak, Malaysia School of Electrical and Electronic Engineering, University of Science, Malaysia

a r t i c l e

i n f o

Keywords: Criterion-referenced assessment Fuzzy inference system Monotonicity property Sub-additivity property Sufficient conditions Rule refinement

a b s t r a c t The main aim of criterion-referenced assessment (CRA) is to report students’ achievements in accordance with a set of references. In practice, a score is given to each test item (or task). The scores from different test items are added together and then projected or aggregated, usually linearly, to produce a total score. Each component score can be weighted before being added together in order to reflect the relative importance of each test item. In this paper, the use of a fuzzy inference system (FIS) as an alternative to the conventional addition or weighted addition in CRA is investigated. A novel FIS-based CRA model is presented, and two important properties, i.e., the monotonicity and sub-additivity properties, of the FIS-based CRA model are investigated. A case study relating to assessment of laboratory projects in a university is conducted. The results indicate the usefulness of the FIS-based CRA model in comparing and assessing students’ performances with human linguistic terms. Implications of the importance of the monotonicity and sub-additivity properties of the FIS-based CRA model in undertaking general assessment problems are discussed. Ó 2011 Elsevier Ltd. All rights reserved.

1. Introduction Assessment in education is defined as a process of forming judgement about quality and extent of students’ achievements or performances. Judgement usually is based on information obtained by requiring students to attempt a number of specified test items or tasks, and submit their work for an appraisal of the quality. In criterion-referenced assessment (CRA), students’ grades are determined by comparing their achievements with a set of clearly defined criteria of learning outcomes. The main aim of CRA is to evaluate students’ achievements with reference to a set of objective reference points, which can be a simple pass–fail grading schema, or a series of key criteria (Burton, 2007; Sadler, 2005). There is a possibility for all students within a particular group to obtain very high or very low grades depending on the individuals’ performances against the established criteria. From the literature, the use of CRA in essay writing (Patrick & Phan, 2005; White, 2002) and clinical performance (Patricia, Shelley, & Trisha, 2009) has been reported. It has been pointed out that scoring usually refers to test items/ tasks rather than to the overall achievement (Joughin, 2008; Sadler, 2005). To ease the process of assessment, in general, a score is given to each test item/task. The individual scores are then aggregated to produce a total score. Sadler (2005) pointed that the scores from different test items/tasks can be added together and ⇑ Corresponding author. E-mail address: [email protected] (K.M. Tay). 0957-4174/$ - see front matter Ó 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2011.02.158

then projected, usually in a linear manner. Each score can also be weighted to reflect the relative importance of each task (Sadler, 2005). The use of fuzzy set related techniques in education assessment models is not new. Biswas (1995) presented a fuzzy set related method to evaluate students’ answer scripts. The work was further enhanced by Chen and Lee (1999). Ma and Zhou (2000) investigated another fuzzy set related method for assessment of student-centered learning tasks. Saliu (2005) used the fuzzy inference system (FIS) in CRA for Constrained Qualitative Assessment (CQA) with a case study. Kwok, Zhau, Zhang, and Ma (2007) proposed a fuzzy group multi-criteria decision making model for CRA of student group projects. Cin and Baba (2008) employed the FIS for English proficiency assessment. Chang and Chen (2009) proposed a fuzzy peer assessment system to satisfy the requirements of cooperative learning in an e-learning environment. Bai and Chen (2008) presented a fuzzy membership function and fuzzy rule-based approach that takes into consideration the difficulty, importance, and complexity of the questions in evaluating answer scripts from students. The proposed approach is able to distinguish the ranking order of students with the same score. The applicability of the FIS to other assessment models, i.e., FIS-based Failure More and Effect Analysis (FMEA) (Bowles & Peláez, 1995; Tay & Lim, 2008a, 2008b), FIS-based groundwater vulnerability assessment (Afshar, Mariño, Ebtehaj, & Moosavi, 2007) has also been reported. In this paper, a new FIS-based CRA model that utilizes rubric as the scoring tool and that involves subjectivity in learning is

11130

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

proposed. The FIS-based CRA model acts as an alternative to aggregate the score of each test item/task, and to produce a total score. We examine a number of reasons to support the use of the FIS, instead of the conventional addition or weighted addition, in assessment models. Inspired by the theoretical properties of a length function (Inder, 2005), we further propose two properties, i.e., the monotonicity and sub-additivity properties, to be associated with the FIS-based CRA model in order to ensure that a useful and valid comparison among students’ performances can be made. In this paper, we investigate and propose the sufficient conditions for the FIS-based CRA model to preserve the monotonicity property (Kouikoglou & Phillis, 2009; Tay & Lim, 2008a, 2008b). A rulerefinement method to improve the sub-additivity property of the FIS-based CRA model is also suggested. The organization of this paper is as follows. Motivations of this work are presented in Section 2. Background of the FIS, the theoretical properties of monotonicity and sub-additivity, the sufficient conditions, and other complementary techniques is given in Section 3. Details of the proposed FIS-based CRA model and its application to a laboratory project assessment task are presented in Section 4. Concluding remarks are presented in Section 5.

2. Motivations In this paper, we focus on the CRA model that utilizes rubric as the scoring tool for the test items/tasks, where subjectivity is involved. A rubric provides a list of criteria, and it helps to grade the quality from bad to good for each criterion (Kurt & Izmirli, 2009). It enables students’ performances from various aspects to be assessed. Scoring usually refers to test items/tasks, rather than to the overall achievement (Joughin, 2008; Sadler, 2005). The score from each test item is aggregated to produce a total score. The scores from different assessments are then added and projected (Sadler, 2005). Each score can be weighted before being added to reflect the relative importance of each task (Sadler, 2005). As an example, weighted addition in essay writing assessment was employed in White (2002). In this paper, the FIS is adopted as an alternative to simple addition or weighted addition. The idea of replacing simple addition or weighted addition with a more complicated method is not new. Sadler (2005) pointed out that aggregation of scores can be done by some designed algorithm or mathematical equation. Chen, Cheng, and Liu (2010) employed an adaptive ordered weighted averaging operator and a K-nearest-neighbor classification method to simulate teachers’ evaluation behaviors. The FIS is adopted owing to several reasons. First, the rubric criteria can be qualitative (rather than quantitative), e.g., a rubric score of 4 does not mean two times better than that of 2. Saliu (2005) considered the FIS as a solution to qualitative assessment, with the aim to keep qualitative assessment accountable. Second, the relative importance of each test item can be different. Indeed, the importance of each test item depends on the learning outcome. The FIS is able to customize the relationship between the score of each test item and the aggregated score. Third, various combinations of the scores from different test items can produce the same aggregated score, but the performance of the student can be different. The FIS can be deployed to tackle this problem. In addition to the above reasons, the FIS has a good function approximation capability, as demonstrated in a variety of problems including control, modeling, and classification (Jang, Sun, & Mizutani, 1997). Another advantage of using the FIS is its capability in incorporating human/expert knowledge, whereby information can be described by using vague and imprecise statements. Besides, the behavior of an FIS can be easily interpreted by humans,

whereby the relationship between its input(s) and output(s) is described by a set of fuzzy If-Then rules. From the literature, FIS-based assessment models are normally initiated as an alternative to the conventional assessment models to allow modeling of nonlinear relationship between the input(s) and output(s) (Afshar et al., 2007; Bowles & Peláez, 1995). The surface plot in Saliu (2005) demonstrates a nonlinear relationship between the score associated with each test item (input) and the total score (output). Over the years, one of the research directions of FIS-based assessment models is on their theoretical properties. Kouikoglou and Phillis (2009) argued that monotonicity is a natural requirement in assessment models. Tay and Lim (2008a,2008b) explained the importance of the monotonicity and output resolution properties for an FIS-based FMEA model. Broekhoven and Baets (2008,2009) further pointed out that monotonicity property is a common property in evaluation and selection procedures. In this paper, we explain the importance of the monotonicity and subadditivity properties of the FIS-based CRA model from the theory of length function (Inder, 2005). Saliu (2005) considered the failure of an FIS-based CRA model to fulfill the monotonicity property as an anomaly, and efforts to construct a monotonicity-preserving FIS-based CRA is essential. In this work, the main aim is to develop a simple (which can be easily understood and visualized by users), easy-to-use (which can be easily incorporated into an FIS-based assessment model), and yet reliable (which can be interpreted from the theoretical foundation) method to preserve the monotonicity property of the FIS. One of the practical and effective solutions is to apply the sufficient conditions to the FIS. In this paper, we investigate the applicability of the sufficient conditions to the proposed FIS-based CRA model. A rule refinement method is also adopted to improve the subadditivity property of the FIS-based CRA model.

3. Fuzzy inference systems and the monotonicity and subadditivity properties To make this paper self-contained, the background information on the FIS is presented, with monotonicity and sub-additivity defined. The sufficient conditions and the rule refinement techniques are also examined. 3.1. Fuzzy inference systems Consider an FIS with n inputs. Let  x ¼ ðx1 ; x2 ; . . . ; xn Þ be the input vector in a rectangular region, U = U1  U2      Un, Ui = [Li, Hi] for M

1 6 i 6 n. Consider Mi terms at the ith input space, A1i ; A2i ; . . . ; Ai i , which are represented by fuzzy membership functions l1i ðxi Þ; l2i ðxi Þ; . . . ; lMi i ðxi Þ, respectively. The output of the FIS, y ¼ f ðxÞ, falls within the range of [LV, HV]. If a full grid partition is used, the total Q number of fuzzy rules is ni¼1 M i . The fuzzy production rules can be represented as follows:

Rj1 ;j2 ;...;jn ðRule#M0 Þ : j

j

IFðx1 is A11 Þ AND ðx2 is A22 Þ    AND ðxn is Ajnn Þ j1 ;j2 ;...;jn

THEN y is B

where 1 6 ji 6 Mi. The product function is the AND operator in the rule antecedent. The compatibility grade, or known as the firing j strength, of each fuzzy rule, i.e., Rj1 j2 jn , is defined as l11 ðx1 Þ j2 jn l2 ðx2 Þ      ln ðxn Þ. To simplify the notation,Qeach fuzzy rule is represented by an index, M0 , where 1 6 M 0 6 ni¼1 Mi . The output is obtained by using the weighted average of a representative real j j j value, b 1 2 n , with respect to its compatibility grade, as in Eq. (1)

11131

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

y ¼ f ðxÞ Pjn ¼Mn ¼

3.3. The sub-additivity property

Pj2 ¼M2 Pj1 ¼M1

j1 j2 jn j1 j2 jn 1 ðx1 Þ  2 ðx2 Þ      n ðxn Þ  b jn ¼1    j2 ¼2 j1 ¼1 Pj2 ¼M2 Pj1 ¼M1 j1 Pjn ¼Mn j2 jn 1 ðx1 Þ  2 ðx2 Þ      n ðxn Þ jn ¼1    j2 ¼2 j1 ¼1

l

l

l

l

l

l

ð1Þ j j j

where b 1 2 n is the representative value of membership function Bj1 j2 jn , as in Eq. (2) j j jn

b1 2

¼ repðBj1 j2 jn Þ

ð2Þ j1 j2 jn

The representative value (rep) of a fuzzy set, i.e., B , carries important information about the overall location, or the ‘‘most typical’’ location of a fuzzy set in its domain. There are several ways how this value can be derived. Defuzzification is one of the most popular methods to obtain a crisp representative value within the universe of discourse of the fuzzy membership function (Jang et al., 1997). Alternatively, it can be determined by the point whereby the fuzzy membership function assumes the value of 1 (Huang & Shen, 2006, 2008). 3.2. The monotonicity property and the sufficient conditions Consider an FIS that fulfills the condition of monotonicity between its output with respect to its ith input within the universe of discourse. The output of the FIS monotonically increases as xi increases, as in Eq. (3)

f ðx1 ; x2 ; . . . ; x1i ; . . . ; xn Þ 6 f ðx1 ; x2 ; . . . ; x2i ; . . . ; xn Þ

as x1i < x2i

ð3Þ

From the literature, many investigations on the monotonicity property of FIS have been reported. Zhao and Zhu (2000) examined the conditions for an FIS to be of monotonicity, and analyzed the FIS operations step-by-step. Wu (1997) introduced an FIS based on the mean-of-inversion defuzzification technique. Another study on the monotonicity property is to differentiate the output of an FIS model with respect to its ith input, and to determine the sufficient conditions of an FIS as its differentiation is always greater than or equal to zero, i.e., 0

f ðxi Þ P 0

ð4Þ

Won, Park, and Lee (2002) derived the sufficient conditions for the first-order Sugeno fuzzy model with this approach. The application and extension of the sufficient conditions to various FIS modeling techniques and applications have been reported, e.g., an FIS with evolutionary computation-based learning (Won, Seo, Hwang, & Lee, 2001), an FIS with least-square learning (Koo, Won, & Lee, 2004), an FIS with fuzzy rule interpolation techniques (Tay & Lim, 2009), an hierarchical FIS (Kouikoglou & Phillis, 2009), and a Type-2 FIS (Li, Yi, & Zhao, 2009). For an FIS to be of monotonicity, the sufficient conditions state that two conditions are required. Condition (1) is a method how the membership functions should be tuned in order to ensure that the FIS satisfies the monotonicity property. Assume both lp and lq are differentiate-able. For lp  lq (in the sense of Koczy and Hirota (1993)), the condition as in Eq. (5), which can be extended to Eq. (6), has to be fulfilled

lp0 ðxÞ  lq ðxÞ  lp ðxÞ  lq0 ðxÞ  0

ð5Þ

lp0 ðxÞ lq0 ðxÞ  lp ðxÞ lq ðxÞ

ð6Þ

Condition (2) highlights the importance of having a monotonic rule base in the FIS model.

The monotonicity property can be extended to the sub-additivity property (Tay & Lim, 2008a, 2008b). It is defined as the sensitivity of a model to produce an increasing output with respect to changes of the ith input. While obtaining the same estimated output for several input conditions, as in Eq. (7), can be considered as satisfying the monotonicity property, it does not fulfill the subadditivity principle.

f ðx1 ; x2 ; . . . x1i ; . . . ; xn Þ ¼ f ðx1 ; x2 ; . . . x2i ; . . . ; xn Þ

as x1i < x2i

ð7Þ

The sub-additivity property suggests to include a minimum threshold, i.e., a(a > 0), in Eq. (7) when xi increases by e(e > 0), as in Eq. (8), and the condition in Eq. (8) is satisfied when Eq. (9) is fulfilled, as follows:

f ðx1 ; x2 ; . . . xi ; . . . ; xn Þ þ a 6 f ðx1 ; x2 ; . . . xi þ e; . . . ; xn Þ; where a > 0; f 0 ðxi Þ P a;

and

e>0

where a > 0

ð8Þ ð9Þ

One of the solutions to increase the derivative of f(xi) is to refine the rule base (Tay & Lim, 2008a, 2008b). Consider a single-input FIS model with M fuzzy rules. Rules Rl and Rl+1 are two rules with neighboring input membership functions, as follows:

Rl : IF x is R

lþ1

ll THEN y is Bl : IF x is llþ1 THEN y is Blþ1

If the number of the output membership functions, N, is smaller than M, i.e., M > N, this implies that there are two or more rules that are mapped to the same output membership function. Without loss

Define learning objective(s) and learning outcome(s) Development of test item(s)/task(s) Development of an FIS-based CRA instruments for performance assessment Development of assessment criteria for each test item/task Development of scoring rubric for each test item/task

Development of fuzzy membership function for each for test item/task (Condition (1))

Expert knowledge collection (Condition (2))

Rules refinements (Condition (2)) Construction of the FIS-based assessment model

Assessment for each test item/task

Aggregating of assessment score(s) using the FIS-based CRA model Fig. 1. The proposed FIS-based CRA model.

11132

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

of generality, let O1, O2, . . . , ON be the output membership functions, where rep(ON) >    > rep(O2) > rep(O1). If both rules (Rl and Rl+1) are mapped to the same membership function, i.e., Ok, then Bl = Bl+1 = Ok, and the representative value, bl = bl+1 = rep(Ok). Using the rule refinement procedure, Rl is mapped to Ok with l W , and to Ok1 with 1  Wl, where 1 > Wl > 0, as follows:

located between Ok and Ok1. The representative value of the new membership function, ONew, is computed using Eq. (10)

repðONew Þ ¼ W l  repðOk Þ þ ð1  W l Þ  repðOk1 Þ

ð10Þ

4. The proposed FIS-based CRA model and its application

Rl :

ll THEN y is Ok with W l IF x is ll THEN y is Ok1 with ð1  W l Þ IF x is

This indicates that Rl is slightly lower in term of the estimate as compared with that of Rl+1. On the other hand, Ok with Wl and Ok1 with 1  Wl constitute a new membership function which is

Fig. 1 shows the flowchart of the proposed education assessment procedure with an FIS-based CRA model. The teaching and learning process usually begins with the definition of the learning objectives and outcomes. From there, test items and their assessment criteria are defined. In this paper, a case study on the use of the proposed FIS-based CRA model to assessment of a laboratory

Table 1 The scoring rubric for electronic circuit design. Score

Linguistic terms

Criteria

10

Excellent

9–8

Very good

7–6

Good

5–3

Satisfactory

2–1

Unsatisfactory

The circuit is complex (P10 necessary ICs). Able to apply knowledge in circuit design. Able to simulate and explain the operation of the designed circuit clearly The circuit is moderate (7–9 necessary ICs). Able to apply most of the learned knowledge. Able to simulate and explain the operation of the circuit clearly The circuit is moderate (5–6 necessary ICs). Some unnecessary components are included. Able to apply most of the learned knowledge. Able to simulate the circuit and explain circuit operation briefly The circuit is simple (3–4 necessary ICs). Some unnecessary components are included. Apply some of the learned knowledge. Simulate only parts of circuit and explain the circuit operation briefly The circuit is very simple (1–2 necessary ICs). Apply very little of the learned knowledge. Unable to simulate and explain the operation of the designed circuit

Table 2 The scoring rubric for electronic circuit development. Score

Linguistic terms

Criteria

10–9

Excellent

PCB: Demonstrate excellent soldering technique (no cold solder joints, no bridge joints; all components leads are soldered to the pad). The circuit fully operates as expected Project board: All components, jumpers, and cables are well-arranged and tidy. The circuit fully operates as expected

8–7

Very good

PCB: Demonstrate good soldering technique (some cold solder and/or bridge joints; some components leads are not soldered to the pad). The circuit operates as expected Project board: Most components, jumpers, and cables are not tidy. The circuit operates as expected

6–5

Good

PCB: Demonstrate good solder technique. (Some cold solder and/or bridge joints; some components lead are not soldered to the pad). Part of the circuit malfunctions Project board: The components are well-arranged but jumpers and/or cables are messy. Part of the circuit malfunctions

4–3

Satisfactory

PCB: Demonstrate poor solder technique (many cold solder and/or bridge joints; many components leads are not soldered to the pad). A major part of the circuit malfunctions Project board: The arrangements of components, jumpers and/or cables are messy. A major part of the circuit malfunctions

2–1

Unsatisfactory

PCB: Demonstrate poor soldering technique (many cold solder and/or bridge joints; many components leads are not soldered to the pad). The circuit totally malfunctions Project board: The arrangements of components, jumpers and/or cables are very messy. The circuit totally malfunctions

Table 3 The scoring rubric for project presentation. Score

Linguistic terms

Criteria

10

Excellent

9–8

Very good

7–6

Good

5–3

Satisfactory

2–1

Unsatisfactory

Information is presented in a logical and interesting sequence. Full knowledge is demonstrated by answering questions with explanation and elaboration. Graphics used are explained and with reinforced screen text and presentation. Clear voice and correct, precise pronunciation of terms Information is presented in a logical sequence. Answer all questions, but fail to elaborate. Graphics used are related to text and presentation. Voice is clear. Pronounced most of the words correctly. Most audience can hear presentation Information is presented in a logical sequence. Answer all simple questions, but fail to elaborate. Graphics used are related to text and presentation. Voice is low; audience have difficulty hearing presentation. Pronounced most of the words correctly Jump around, difficult to follow presentation. Uncomfortable with information, and able to answer only simple questions. Graphics used rarely support text and presentation. Voice is low. Pronounces terms incorrectly Presentation cannot be understood because there is no sequence of information. Do not have grasp of information, cannot answer questions. Use superfluous graphics or no graphics. Speak unclearly, incorrectly pronounced terms, and speaks too quietly for audience to hear

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

project by students in Universiti Malaysia Sarawak (UNIMAS) is conducted. Students are required to perform three test items, (1) to design a digital electronic system based on the knowledge learned at their digital system subject, and their creativity and technical skills (electronic circuit design); (2) to develop the system either using printed circuit board or on breadboard (electronic circuit development); (3) to present and demonstrate their work (project presentation). Tables 1–3 show the scoring rubrics of the test items, respectively. Note that the scoring rubrics as in Tables 1–3 are classified as the holistic rubrics (Mertler, 2001).

Fig. 2. The membership functions of electronic circuit design.

11133

Each partition of the scoring rubric can be represented by a fuzzy set. As an example, the membership function of electronic circuit design is depicted in Fig. 2. Each membership function is assigned a linguistic term, e.g., a score 3–5 is assigned to Satisfactory, which refers to ‘‘The circuit is simple (3–4 necessary ICs). Some unnecessary components are included. Apply some of the knowledge learned. Simulate only parts of circuit and explain the circuit operation briefly’’. The same applies to rubrics of electronic circuit development and project presentation. Figs. 2–4 depict the fuzzy membership functions of electronic circuit design (laDg ), electronic circuit development (lbDv ), and project presentation (lcPr ), respectively. For example, for electronic circuit design, a score 3–5 is represented by l2Dg , and is labeled as Satisfactory. For electronic circuit development, a score 5–6 is represented by l3Dv , and is labeled as Good. For project presentation, a score 3–5 is represented by l2Pr , and is labeled as Satisfactory. The total score varies from 1 to 100, and is represented by seven fuzzy membership functions, i.e., ‘‘Excellent’’, ‘‘Very good’’, ‘‘Good’’, ‘‘Fair’’, ‘‘Weak’’, ‘‘Very weak’’ and ‘‘Unsatisfactory’’. The corresponding b values are the points where the membership value of B is 1. The b values are 100, 80, 70, 50, 40, 20, 0 for ‘‘Excellent’’, ‘‘Very good’’, ‘‘Good’’, ‘‘Fair’’, ‘‘Weak’’, ‘‘Very weak’’ and ‘‘Unsatisfactory’’, respectively. A fuzzy rule base is a collection of knowledge from the experts in the If-Then format. It describes the relationship between the rubrics in electronic circuit design, electronic circuit development, and project presentation with the total score. As an example, Fig. 5 shows two fuzzy production rules in the rule base that have been established by the lecturer who is responsible for the assessment. The FIS is used to produce the total score, as follows:

PMDg PMDv PMPr Total score ¼

Fig. 3. The membership functions of electronic circuit development.

Fig. 4. The membership functions of project presentation.

laDg  lbDv  lcPr  ba;b;c PMDg PMDv PMPr a b c c¼1 lDg  lDv  lPr a¼1 b¼1

a¼1

b¼1

c¼1

ð11Þ

4.1. Monotonicity and sub-additivity The monotonicity and sub-additivity properties are important for the FIS-based CRA model to produce a valid and meaningful comparison among students’ performances. It describes the relationship between one test item and the aggregated total score. In general, if a student obtains a higher score in electronic circuit design, the student is expected to obtain a higher total score. As an example, for two students who obtain the same score in electronic circuit development and project presentation, the monotonicity property suggests that the student with a higher score in electronic circuit design should have a total score that is higher than or equal to that of the other. The sub-additivity property further suggests that the student with a higher score in electronic circuit design should have a higher total score than that of the other. In this case, it is suggested that a = 1. Following the sub-additivity

Rule 1 IF electronic circuit design is Excellent And electronic circuit development is Excellent And project presentation is Excellent, THEN total score is Excellent

Rule 2 IF electronic circuit design is Excellent And electronic circuit development is Excellent And project presentation is Very Good, THEN total score is Excellent Fig. 5. An example of two fuzzy production rules.

11134

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

property, the student who has a higher score (with one mark difference) in electronic circuit design should have at least one mark higher in the total score than that the other. To preserve the monotonicity property, the sufficient conditions are applied to the FIS-based CRA model. Condition (1) is used to generate the membership functions of electronic circuit design, electronic circuit development, and project presentation. As shown in Figs. 2–4, these membership functions obey this condition. It is

Fig. 6. Visualization of the membership functions of electronic circuit design.

possible to project and visualize these membership functions by using Eq. (6). Assume that the Gaussian membership function, 2 2 GðxÞ ¼ e½xc =2r , is used in the FIS-based CRA model. The derivative of G(x) is G0 (x) = ((x  c))/r2)G(x). Using Eq. (6), the ratio of the Gaussian membership function returns a linear function, i.e., E(x) = G’(x)/G(x) = (1/r2)x + (c/r2). Figs. 6–8 depict the projection of the membership functions of electronic circuit design, electronic circuit development, and project presentation, respectively, using Eq. (6). As an example, in Fig. 6, one can observe that EExcellent(x) > EVeryGood(x) > EGood(x) > ESatisfactory(x) > EUnsatisfactory(x). This shows that Condition (1) is fulfilled. The same applies to Figs. 7 and 8. Condition (2) can be used to check the validity of the collected rule base. To improve the sub-additivity property, a rule refinement method is used. For example, Rule 2 in Fig. 5 can be refined by adding a weight, as shown in Fig. 9. Rule 2 is considered to be of a lower consequence than that of Rule 1.

4.2. Results and discussion Table 4 summarizes the assessment results from the FIS-based CRA model. Column ‘‘No.’’ shows the label of each student’s project. Columns ‘‘Dg’’, ‘‘Dv’’ and ‘‘Pr’’ list the scores for each test item, respectively. Columns ‘‘Assessment with the FIS-based CRA model’’ and ‘‘Assessment with the FIS-based CRA model (after rule refinement)’’ show the results before and after rule refinement. Column

Table 4 Assessment with the FIS-based CRA model. No.

Fig. 7. Visualization of the membership functions of electronic circuit development.

Fig. 8. Visualization of the membership functions of project presentation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Scores

Assessment with the FIS-based CRA model

Dg

Dv

Pr

Fuzzy score (%)

Expert’s knowledge

4 5 5 7 5 6 5 7 7 8 7 7 8 10 10

4 4 4 4 5 8 7 6 7 6 7 9 8 8 9

6 6 7 6 7 5 7 7 6 6 8 8 10 8 8

40 40 40 40 42 49 49 50 50 51 63 75 78 93 94

Weak Weak Weak Weak Weak Fair Fair Fair Fair Fair Good Very good Very good Excellent Excellent

Rule 1 If electronic circuit design is Excellent and electronic circuit development is Excellent and project presentation is Excellent then total score is Excellent (1.0)

Rule 2 If electronic circuit design is Excellent and electronic circuit development is Excellent and project presentation is Very Good then total score is Excellent (0.75) If electronic circuit design is Excellent and electronic circuit development is Excellent and project presentation is Very Good then total score is Very good (0.25)

Fig. 9. An example of the fuzzy production rule after rule refinement.

Assessment with the FIS-based CRA model (after rule refinement) Fuzzy score (%) 39 40 42 44 45 50 53 52 60 59 67 76 78 88 89

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

‘‘Fuzzy score’’ shows the assessment results from the FIS-based CRA model. Column ‘‘Expert’s knowledge’’ shows the linguistic term of each project. Fig. 10 depicts one of the completed projects. This project was given a score of 10 for electronic circuit design because it consisted of ten integrated circuits (ICs), and the student was able to simulate and clearly explain the operations of the designed system. The student was given a score of 9 for electronic circuit development as the system worked well, and all electronic components were installed on the breadboard correctly. Besides, all the components, ICs, and jumpers were well-arranged on the breadboard. The circuit was fully operated as expected. The student was also awarded a score of 8 for project presentation. The total score obtained by the student was 94 (with the FIS-based CRA model). From the experiment, the FIS-based CRA is able to produce an aggregated total score in accordance with expert’s knowledge. This can be observed as Fuzzy score is in line with that of the expert. The importance of the monotonicity property can be explained by comparing the results of students #1 and #2, with scores of 4 4 6 and 5 4 6 (‘‘Dg’’, ‘‘Dv’’ and ‘‘Pr’’, respectively). Both the students obtained the same score for Dv’’ and ‘‘Pr. However, student #1 was awarded a lower score (Dg = 4) than that of student #2 (Dg = 5) for electronic circuit design. The monotonicity property suggests that the mark of student #2 should not be lower than that of student #1, in order to have a valid comparison of their performances. From the observation, the FIS-based CRA model is able to fulfill the monotonicity property. Fig. 11 depicts a monotonic

11135

Fig. 12. A surface plot of total score versus electronic circuit design and electronic circuit development when project presentation = 5.

Fig. 13. A surface plot of total score versus electronic circuit design and project presentation when electronic circuit development = 5 after rule refinement.

Fig. 10. A digital system built by student #15.

surface plot of total score versus electronic circuit design and project presentation when electronic circuit development = 5. Again, Fig. 12 shows a monotonic surface plot of total score versus electronic circuit design and electronic circuit development when project presentation = 5. In summary, as long as Condition (1) and Condition (2) are fulfilled, the results show that the monotonicity property is satisfied. The importance of sub-additivity can be observed from Table 4. Students #1, #2, and #3 with 4 4 6, 5 4 6, and 5 4 7 (‘‘Dg’’, ‘‘Dv’’ and ‘‘Pr’’, respectively) were awarded a total score of 40. This is logical, but not effective enough to allow a valid and useful comparison of the three students’ performances. After rule refinement, the FISbased CRA model produced 39, 40, and 42 for the three students. This is a better evaluation of the students’ performances. Fig. 13 shows the surface plot of total score versus electronic circuit design and project presentation when electronic circuit development = 5, after rule refinement. As can be seen in Fig. 10, Region A is an almost static region (for project presentation score varies from 5 to 10). This region is improved after rule refinement (highlighted as Region B in Fig. 13). 5. Summary

Fig. 11. A surface plot of total score versus electronic circuit design and project presentation when electronic circuit development = 5.

We have proposed a simple, easy-to-use, and reliable method to construct an FIS-based CRA model in this paper. The FIS-based CRA model encompasses some theoretical properties of a length function, i.e., the monotonicity and sub-additivity properties. As it is important to ensure the validity of the FIS-based CRA model in

11136

K.M. Tay, C.P. Lim / Expert Systems with Applications 38 (2011) 11129–11136

producing a valid and meaningful comparison of students’ performance, the sufficient conditions are included in the proposed FIS-based CRA model to achieve this aim. The sufficient conditions help to develop an FIS-based CRA that fulfills the monotonicity property. Besides, a rule refinement method is used to improve the sub-additivity of the FIS-based CRA model. A case study on the assessment of a laboratory project by students has been conducted. From the experiments, the importance of monotonicity and sub-additivity in the FIS-based CRA can be observed. The sufficient conditions provide an effective solution to ensure the monotonicity property. With rule refinement, subadditivity of the FIS-based CRA model is satisfied too. As future work, we suggest to adopt a fuzzy rule selection technique to systematically select a small set of If-Then rules, and to utilize a fuzzy rule interpolation technique (Kóczy & Hirota, 1997) or an analogical reasoning technique (Turksen & Zhao, 1988) to predict the unknown rules. This is important because to collect a complete set of fuzzy production rules for the FIS-based CRA model is a tedious task. This approach is preferred in practice as it reduces the time and effort to construct an FIS-based CRA model that comes with a small set of If-Then rules. References Afshar, A., Mariño, M. A., Ebtehaj, M., & Moosavi, J. (2007). Rule-based fuzzy system for assessing groundwater vulnerability. Environment Engineering, 133, 532–540. Bai, S. M., & Chen, S. M. (2008). Automatically constructing grade membership functions of fuzzy rules for students’ evaluation. Expert Systems with Applications, 35(3), 1408–1414. Biswas, R. (1995). An application of fuzzy sets in students’ evaluation. Fuzzy Sets and Systems, 74(2), 187–194. Bowles, J. B., & Peláez, C. E. (1995). Fuzzy logic prioritization of failures in a system failure mode, effects and criticality analysis. Reliability Engineering & System Safety, 50, 203–213. Broekhoven, E. V., & Baets, B. D. (2008). Monotone Mamdani–Assilian models under mean of maxima defuzzification. Fuzzy Sets and Systems, 159, 2819–2844. Broekhoven, E. V., & Baets, B. D. (2009). Only smooth rule bases can generate monotone Mamdani–Assilian models under center-of-gravity defuzzification. IEEE Transactions on Fuzzy Systems, 17, 1157–1174. Burton, K. (2007). Skills and criterion referenced assessment: Turning LWB236 real property a law into a subject for the real world. Journal of South Pacific Law, 11(1), 56–63. Chang, T. Y., & Chen, Y. T. (2009). Cooperative learning in e-learning: A peer assessment of student-centered using consistent fuzzy preference. Expert Systems with Applications, 36(4), 8342–8349. Chen, Y. H., Cheng, C. H., & Liu, J. W. (2010). Intelligent preference selection model based on NRE for evaluating student learning achievement. Computers & Education. doi:10.1016/j.compedu.2009.09.020. Chen, S. M., & Lee, C. H. (1999). New methods for students’ evaluation using fuzzy sets. Fuzzy Sets and Systems, 104(2), 209–218. Cin, F. M., & Baba, A. F. (2008). Assessment of English proficiency by fuzzy logic approach. International Educational Technology Conference, 355–359. Huang, Z. H., & Shen, Q. (2006). Fuzzy interpolation reasoning via scale and move transformation. IEEE Transactions on Fuzzy Systems, 14, 340–359. Huang, Z. H., & Shen, Q. (2008). Fuzzy interpolation and extrapolation: A practical approach. IEEE Transactions on Fuzzy Systems, 16, 13–28.

Inder, K. R. (2005). An introduction to measure and integration. Alpha Science international. Jang, J. S. R., Sun, C. T., & Mizutani, E. (1997). Neural-fuzzy and soft computing. Prentice-Hall. Joughin, G. (2008). Assessment, learning and judgement in higher education. London: Springer Netherlands. Koczy, L. T., & Hirota, K. (1993). Ordering, distance and closeness of fuzzy sets. Fuzzy Set and System, 59(3), 281–293. Kóczy, L. T., & Hirota, K. (1997). Size reduction by interpolation in fuzzy rule bases. IEEE Transactions on Systems, Man and Cybernetics, Part B: Cybernetic, 27(1), 25–41. Koo, K., Won, J. M., & Lee, J. S. (2004). Least squares identification of monotonic fuzzy systems. In IEEE annual meeting of the fuzzy information, 2004. Processing NAFIPS 2004 (Vol. 2, pp. 745–749). Kouikoglou, V. S., & Phillis, Y. A. (2009). On the monotonicity of hierarchical sumproduct fuzzy systems. Fuzzy Sets and Systems, 160(24), 3530–3538. Kurt, A. A., & Izmirli, S. (2009). The views of teacher candidates about the use of a scoring rubric for the evaluation of their products in the course of instructional technologies and material development. Procedia Social and Behavioral Sciences, 1, 988–992. Kwok, R. C .-W., Zhau, D., Zhang, Q., & Ma, J. (2007). A fuzzy multi-criteria decision making model for IS student group project assessment. Group Decision and Negotiation, 16(1), 25–42. Li, C., Yi, J., & Zhao, D. (2009). Analysis and design of monotonic type-2 fuzzy inference systems. Fuzz-IEEE, 2009, 1193–1198. Ma, J., & Zhou, D. (2000). Fuzzy set approach to the assessment of student-centered learning. IEEE Transactions on Education, 43(2), 237–241. Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25). Patricia, N., Shelley, G., & Trisha, D. A. M. (2009). The use of scoring rubrics to determine clinical performance in the operating suite. Nurse Education Today, 29(1), 73–82. Patrick, G., & Phan, N. A. (2005). Assessment of creative writing in Vietnamese primary education. Asia Pacific Education Review, 6(1), 72–86. Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175–194. Saliu, S. (2005). Constrained subjective assessment of student learning. Journal of Science Education and Technology, 14(3), 271–284. Tay, K. M., & Lim, C. P. (2008a). On the use of fuzzy inference techniques in assessment models: Part I: Theoretical properties. Fuzzy Optimization and Decision Making, 7(3), 269–281. Tay, K. M., & Lim, C. P. (2008b). On the use of fuzzy inference techniques in assessment models: Part II: Industrial applications. Fuzzy Optimization and Decision Making, 7(3), 283–302. Tay, K. M., & Lim, C. P. (2009). On the use of fuzzy rule interpolation techniques for monotonic multi-input fuzzy rule base models. Fuzz-IEEE, 2009, 1736–1740. Turksen, I. B., & Zhao, Z. (1988). An approximate analogical reasoning approach based on similarity measures. IEEE Transactions on Systems, Man and Cybernetics, 18(6), 1049–1056. White, D. C. (2002). Criterion based assessment using the support of a computer. In Proceedings of the 35th Hawaii international conference on system sciences (pp. 362–370). Won, J. M., Seo, K., Hwang, S. K., & Lee, J. S. (2001). Evolutionary computation based identification of a monotonic Takagi–Sugeno–Kang fuzzy system. In The 10th IEEE international conference on -fuzzy systems (Vol. 3, pp. 1140–1143). Won, J. M., Park, S. Y., & Lee, J. S. (2002). Parameter conditions for monotonic Takagi–Sugeno–Kang fuzzy system. Fuzzy Sets and Systems, 132, 135–146. Wu, C. J. (1997). Guaranteed accurate fuzzy controllers for monotone functions. Fuzzy Sets and Systems, 92, 71–82. Zhao, H. L., & Zhu, C. Q. (2000). Monotone fuzzy control method and its control performance. IEEE International Conference on Systems, Man, and Cybernetics, 5, 3740–3745.