Toward predicting the performance of novice CAD users based on their profiled technical attributes

Engineering Applications of Artificial Intelligence 25 (2012) 628–639

Contents lists available at SciVerse ScienceDirect

Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai

Toward predicting the performance of novice CAD users based on their profiled technical attributes R.F. Hamade a,n, A.H. Ammouri a, H. Artail b a b

Department of Mechanical Engineering, American University of Beirut (AUB), Beirut, Lebanon Department of Electrical and Computer Engineering, American University of Beirut (AUB), Beirut, Lebanon

a r t i c l e i n f o

a b s t r a c t

Article history: Received 10 December 2010 Received in revised form 4 July 2011 Accepted 5 January 2012 Available online 2 February 2012

In previously published research (Hamade et al., 2005, 2007, 2009; Hamade and Artail, 2008) the authors developed a framework for analyzing the technical profiles of novice computer-aided design (CAD) trainees as they set to start training in a formal setting. The research included conducting a questionnaire to establish the trainees’ CAD-relevant technical foundation which served as the basis to statistically correlate this data to other experimental data collected for measuring the trainees’ performance over the duration of training. In this paper, we build on that work and attempt to forecast the performance of these CAD users based on their technical profiled attributes. For this purpose, we utilize three Artificial Neural Networks, ANN, techniques: Feed-Forward Back propagation, Elman Back propagation, and Generalized Regression with their capabilities are compared to those of Simulated Annealing as well as to those of linear regression techniques. Based on their profiled technical attributes, the Generalized regression neural network (GRNN) method is found to be most successful in discriminating the trainees including their predicted initial performance as well as their progress. & 2012 Elsevier Ltd. All rights reserved.

Keywords: CAD expertise development Technical attributes Training Prediction Simulated annealing Neural networks.

1. Introduction and background Organizations and educational institutions alike seek innovative ways to improve the effectiveness and efficiency of CAD training (McDermott and Marucheck, 1995) so that trainees can acquire knowledge on their way to achieving CAD competence (Johnson and Diwakaran, 2011). One important step toward enhanced CAD training output is to pre-select potential trainees based on their profiled attributes. A substantial amount of research that deals with predicting human performance based on the person’s profile could be found under the topic of human factors and ergonomics (HFE). While some predictive approaches of human performance are based mainly on theoretical descriptions and models (Glenn et al., 2005), many more utilize intelligent computations (Boussemart and Cummings, in press), with inputs ranging from short term observations (Pentland and Liu, 1999; Jastrzembski et al., 2009) to fairly sophisticated questionnaire-based input for profiling academic, psychosocial, cognitive, and demographic predictors of academic performance (McKenzie and Schweitzer, 2001). Such an aforementioned framework for quantifying human CAD performance is rooted in the authors’ previous works. Therefore,

n

Corresponding author. Tel.: þ961 1 350 000. E-mail address: [email protected] (R.F. Hamade).

0952-1976/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. doi:10.1016/j.engappai.2012.01.004

in attempting to predict the performance of novice CAD users, the authors in this paper take advantage of three factors (two of which have been previously established): 1. An experimental framework for assessing the development of CAD skills (performance measures) as training progresses (Hamade et al. 2005, 2007, 2009). This assessment is based on the speed, T(t) in minutes, needed to construct 3-dimensional CAD test models at a recall time t (in weeks of training). The rate of improvement of this measure, bt, was determined as well. Another assessment measure is based on the numbers of features, F(t), used in constructing the models which denotes the sophistication or effectiveness by which trainees acquire CAD skills. The rate of improvement of this measure, bf, was also determined. Therefore, performance measures are defined here as metrics that gauge performance based on the construction speed and sophistication of CAD models built as well as rate of progress of trainees over the duration of training. A summary explanation of these assessment measures will be briefly given in Section 2 for clarity. 2. An experimental framework for quantifying the trainees’ technical attributes profile (relevant technical foundation) was outlined in Hamade and Artail (2008). This is a profiling framework which identifies many of the determinant factors of performance. The trainee’s technical attributes have been profiled according to this methodology where we devise a questionnaire that is composed

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

Fig. 1. (a)–(d). CAD solid models used for performance assessment: test part 1 (top-left), test part 2 (top-right), test part 3 (bottom-left), and test part 4 (bottomright).

70 Initial Performance Time (min)

In regards to the last point, for forecasting purposes, the authors in this paper employ in addition to traditional linear regression (LR), Simulated Annealing (SA) as well as three different ANN methods in order to predict/forecast the performance of novice CAD users based only on their technical profiles as devised in Hamade and Artail (2008). Namely, the three types of networks are: the Feed-forward back propagation network (FFBN), Elman back propagation neural networks (EBN; Elman, 1990), and Generalized regression neural network (GRNN). Simulated Annealing has been successfully used to find optimum solutions of many problems of practical significance. Detailed theoretical treatment of SA with a number of applications can be found in Pham and Karaboga (2000). Specifically regarding utilization of neural networks (NNs), using NNs for forecasting purposes has been practiced for many years (see the review article by Zhang et al. (1998)). More specifically, many instances of utilizing neural networks in educational research are recorded especially for the purpose of forecasting student performance. While many of these studies utilize neural networks in order to predict student grades (Lykourentzou et al., 2009), others deal with more challenging predictions applications such as students potential for e-learning (Wanga and Liao, 2011), early identification of students at risk of failing a chemistry course (Cooper and Pearson, in press), identifying learning disability in children at early stages (Julie and Kannan, in press), learning framework for handwritten character recognition (Dong et al., 2002), forecasting failure risk for pre-medical students (Alenezi et al., 2009), prediction of graduates’ professional choice (Gerasimovic et al., 2011), and analyzing creativity of students (Yu, 2010). The abundant body of literature on these methods in a multitude of forecasting applications from across a wide range of interests is testament to their forecasting power. However, to the author’s best knowledge, the ability of AI techniques to predict to a certain level of accuracy the trainee’s potential to develop CAD competency has not been reported in the open literature before. We believe this application will be novel and that such framework developed by the authors may be of great help to intelligent tutoring systems capable of catering to individual training needs and capabilities by providing a description of the trainees. This is the scientific issue addressed by this paper. Furthermore, such a methodology will provide capabilities toward predicting future performance of CAD trainees especially

in the increasingly popular e-learning offerings of such courses. Hereafter, Simulated Annealing and ANNs will be used in the prediction of the T and F performance values along with their rates of improvement of these measures, bt and bf, respectively. It will be demonstrated that the generalized regression neural network (GRNN) is most successful of the methods evaluated in forecasting the trainees’ performance and in discriminating the trainees based on their technical profile over the duration of the formal training course. Collectively, the framework as presented in this work possesses what amounts to a diagnostic tool capable of pre-profiling potential trainees. Such early classification will help not only in better channeling of resources and human capital but also in helping CAD instructors to design customized CAD training material and to make instructional modifications that suit trainees’ individual profiles.

LR1 Results Time (min), Exp. data Time (min) predicted, LR1

60 50 40 30 20 10 0 1

11

21

31

16 Initial No. of Features Used

of focused and CAD-specific questions to reveal the trainees’ technical attributes. This important development will be briefly summarized in Section 3 for clarity. 3. As a natural continuation of the work, this paper is concerned with trying to predict/forecast the performance of novice mechanical CAD trainees. The data collected in Hamade and Artail (2008), along with similar data collected afterward, forms the basis as input for the Artificial Intelligence, AI, (here, Simulated Annealing, SA, and Artificial Neural Networks, ANN) techniques utilized for predicting the capability of students based on their answers to given questionnaires. We utilize these intelligent tools as predictive techniques for forecasting the performance of novice CAD users based on their profiled technical attributes. Statistical correlations were established in Hamade and Artail (2008) between the trainees’ responses to the questionnaire and the measured CAD skills development in order to determine the relevance of individual questions as well as the categories. Moreover, the degree by which each of the trainees’ technical attributes influence CAD competence development was determined as well. In this paper, we take the natural step of utilizing intelligent tools to predict such relations between cause (technical attributes) and effect (performance).

629

41 51 Trainees

61

71

81

91

LR1 Results

14

F, Exp. data F predicted, LR1

12 10 8 6 4 2 0 1

11

21

31

51 41 Trainees

61

71

81

91

Fig. 2. Predictions vs. experimental data: (top) Performance time and (bottom) Number of features used; Linear regression method 1, LR1.

630

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

2. Explanation of training and CAD performance measures For the successful forecasting of trainees, perhaps the main challenge in this work has been the need for a methodology for describing human performance while learning CAD. The authors undertook a substantial amount of work toward establishing such a methodology. Such a validated framework for CAD performance measures has been described in detail in (Hamade et al. 2005, 2007, 2009). However, a brief summary of this assessment of CAD competence is presented below for clarity.

CAD language refers to a 3-dimensional geometric shape such as protrusion, cut, hole, etc (see Hamade (2009) for more details on model construction). Ninety-four mechanical engineering senior students were assessed as they underwent formal group training in Pro/Engineer. Collecting training data for this cohort extended over a 4-year period: classes of 23, 21, 30, and 20 in the first, second, third, and fourth year, respectively. Training was conducted twice a week for 2 h per session for a total of 4 h per week and lasted over 15 weeks. Students were assigned one homework exercise after each lecture, consistent with the taught material, and practiced on their own time.

2.1. Training 2.2. Performance measures The CAD tool utilized was Pro/Engineer (Parametric Technology Corporation), version Wildfire. This is a high-end 3D mechanical CAD software that utilizes parametric and associative characteristics to construct solid representations. A solid, 3-dimensional model is constructed of intermediate features-of-size (will further be referred to as features) that are relatively simple. Note that a feature in solid

70

CAD performance by human operators may be accounted for by such factors as cognitive, motor, and human perception. Such a framework has been established by the classic Fitts’s Law (Fitts, 1954) which is typically applied by designers of human– computer interfaces, HCI, where movement is anticipated.

1.6

LR1 Results (T)

60 Data (Target) Prediction

Data (Target) Prediction

1.2 1

40 bt

Time (min)

50

30

0.8 0.6

20

0.4

10

0.2 0

0 1 15 17 29 30 40 49 55 60 66 69 70 77 78 79 81 83 86 90 Trainees

2

LR1 Results (F)

12

8

5 10 14 24 36 38 55 57 60 66 69 73 75 87 88 89 90 93 Trainees

LR1 Results (bf)

0.7 Data (Target) Prediction

10

6

Data (Target) Prediction

0.6 0.5 bf

Features

LR1 Results (bt)

1.4

0.4 0.3

4

0.2

2

0.1 0

0 1 15 17 29 30 40 49 55 60 66 69 70 77 78 79 81 83 86 90

2

5 10 14 24 36 38 55 57 60 66 69 73 75 87 88 89 90 93

Trainees

Trainees

Fig. 3. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from linear regression method 1 (LR1). Predictions are shown compared with actual experimental data (only testing data points are shown).

Table 1 Summary of prediction error values for the total set of trainees T, F, bt, and bf for each method used in the study; also listed are the error values for the 20% test data set (suffixed by _test). Prediction method

T error%

T_test error%

F error%

F_test error%

bt error%

bt_test error%

bf error%

bf_test error%

Overall error

LR1 LR2 SA FFBN EBN GRNN

36 30 29 16 17 5

44 37 69 38 30 24

24 22 32 8 8 3

28 18 48 23 22 16

39 33 24 17 22 7

43 36 46 43 43 35

27 21 26 20 20 5

28 14 61 23 40 23

34 26 42 24 25 15

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

Initial Performance Time (min)

70

LR2 Results Time (min), Exp. data Time (min) predicted, LR2

60 50 40 30 20 10 0 1

11

21

31

Initial No. of Features Used

16

41 51 Trainees

61

71

81

91

F, Exp. data F predicted, LR2

12 10 8 6 4 2 1

11

21

31

41 51 Trainees

61

71

81

91

Fig. 4. Predictions vs. actual data: (top) Performance time and (bottom) Number of features used; linear regression method 2, LR2.

where T(t) is the performance time (speed) when measured at time t, T(1) is the theoretical production time at first repetition (here at week 1), and bt is the learning slope (where bt ¼  log f/log 2 and f (0r f r1) being the learning rate). Having been monitored over the duration of training, this performance measure evaluates both the initial performance as well as rate of improvement of trainees. This speed of CAD knowledge acquisition is an indication of the efficiency of using the CAD system (in this case, Pro/Engineer). For the data collected of all 94 trainees, values of 37 min and 0.55 were calculated for T(1) and bt, respectively, where the corresponding learning rate is indicative of a highly cognitive process. The arithmetic average of the class initial (at 2 weeks) performance time or T(2 weeks) was found to be about 25 min. This value may be contrasted with class average performance time of about 10 min at the end of training.

1.2

LR2 Results (T) Data (Target) Prediction

50

1 0.8

30

0.6

bt

40

20

0.4

10

0.2

12

Data (Target) Prediction

9 12 13 22 24 30 34 36 39 48 60 61 63 65 72 75 82 84 90

11 15 17 21 31 44 65 67 69 71 73 76 81 84 86 88 90 91 93

Trainees

Trainees

LR2 Results (F) Data (Target) Prediction

8 bf

Features

10

LR2 Results (bt)

0

0

14

ð1Þ

LR2 Results

0

Time (min)

Important and measurable human factors include speed of performance and time to learn. Both of these factors will be addressed here with the former being treated as a ‘performance measure’ while the latter (in number of weeks of training) is treated as the time dimension in this longitudinal study. For this purpose, four test models, Fig. 1(a)–(d), that are comparable in complexity, were constructed by all trainees at prescribed intervals. Consequently, the trainees’ actual behavior is monitored and quantified in terms of satisfying required tasks (i.e., performance objective). The speed measure represents the efficiency of CAD knowledge acquisition by the trainees. Time performance for the class is found to conform to a power expression that follows the classic Wright’s learning curve (Wright, 1936) TðtÞ ¼ Tð1Þt bt

14

60

631

6 4 2 0 9 12 13 22 24 30 34 36 39 48 60 61 63 65 72 75 82 84 90 Trainees

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

LR2 Results (bf) Data (Target) Prediction

11 15 17 21 31 44 65 67 69 71 73 76 81 84 86 88 90 91 93 Trainees

Fig. 5. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from linear regression method 2 (LR2). Predictions are shown compared with actual experimental data (only testing data points shown).

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

Similar to performance time learning curves, it was also found that the number of features utilized to build these models behave in a decreasing fashion over time according to FðtÞ ¼ Fð1Þt bf

ð2Þ

where F(t) is the number of features when measured at time t, F(1) is the initial performance time, and bf is the learning slope or the number of features improvement rate. This assessment measure denotes the effectiveness by which trainees go about acquiring CAD skills. For the data collected of all 94 trainees, values of 9.04 features and 0.43 were calculated for F(1) and bf, respectively. The value of 9.04 should be contrasted with a value of 2 which indicates the absolute minimum number of features needed to construct such a part (Hamade et al., 2009).

3. Explanation of profiling data: Questionnaire Previous work by Hamade and coworkers has amounted to developing a framework for profiling some of the salient technical attributes of novice CAD users. Readers are referred to Hamade and Artail (2008) for details of the technical profile questionnaire utilized and corresponding results. However, in this section we summarize this framework for added clarity. All participants in training were asked to reply to a deployed questionnaire made up of 42 written, CAD-specific questions. These individual questions can be found in Hamade and Artail (2008). The trainees’ responses to these questions are transformed from an initial alphabetical scale, to a numerical scale, and, finally, to a consistent (out of 5 max) numerical scale. The individual questions, when aggregated as follows, are used to describe each trainee’s attributes according to the following seven categories: 1. 2. 3. 4. 5. 6. 7.

Basic math, BM (10 questions) Advanced math, AM (3 questions) CAD mathematical foundation, CM (4 questions) Computer science & engineering foundation, CSE (7 questions) Methodologies related to CAD, MR (9 questions) Graphics foundation, GF (3 questions) Mechanical design skills, MD (6 questions)

Many of these technical categories are strongly related to CAD knowledge development and were found (Hamade and Artail, 2008) to have strong statistical correlations to the trainee’s measured initial performance, their progress, or both.

develop more accurate and achievable training objectives that can be tuned based on the feedback of the predictions framework. Prediction has long been used to try to infer future behaviors based on current ones. A traditional approach is based on regression techniques, but a relatively more recent and sophisticated framework employs AI techniques like ANN, Simulated Annealing, and Genetic Algorithms, to name a few. In this work, we will explore the use of linear regression, Simulated Annealing, and ANN techniques to try to infer the performance of CAD trainees.

4.1. Linear regression Linear regression was used to model the relation between the answers to the questions in the questionnaire and the performance and modeling sophistication. In linear regression, this system is assumed to be linear and thus the modeling process will find T and F values in terms of different answers multiplied by different weights. Two types of relations were studied: first T, bt, F, and bf were modeled in terms of ‘Questions Categories’, hereafter called LR1, where the measures were found in terms of the above seven aggregated technical categories outlined in Section 3, namely: basic math (BM), advanced math (AM), CAD mathematical foundation (CM), computer science & engineering foundation (CSE), Methodologies related to Table 2 Simulated annealing parameters. Parameter

Value

Annealing function Temperature update function Maximum iterations Function tolerance Initial temperature Reannealing interval

Boltzmann annealing Exponential update 100,000 0.1 100 100

70 Initial Performance Time (min)

632

Simulated Annealing Results Time (min), Exp. Data Time (min) predicted, SA

60 50 40 30 20 10 0 1

4. Application: prediction techniques and results

21

16 Initial No. of Features Used

The technical profiling data used in this paper as input consists of the experimental technical data reported in Hamade and Artail (2008) in addition to similar data collected afterwards for a total of 94 trainees. This data is used as input for linear regressions, Simulated Annealing, and ANN techniques. It was only normal to build on our previous work to study the reliability of predicting the performance of CAD trainees based on learning their attributes at the start of the training. Being able to predict performance with some tolerable error offers significant benefits such as customizing the training process and delivery in accordance with the ‘estimated’ competence levels of trainees. More specifically, a sufficientlyreliable prediction framework would allow the training organizations, or even in our case, the course instructor, to place trainees in groups based on predicted performance levels and design training material and a process that would result in maximizing the learning outcomes. Moreover, the training administrator will be able to

11

31

41 51 Trainees

61

71

81

91

Simulated Annealing Results F, Exp. Data F predicted, SA

14 12 10 8 6 4 2 0 1

11

21

31

51 41 Trainees

61

71

81

91

Fig. 6. Predictions vs. actual data: (top) Performance time and (bottom) Number of features used; Simulated Annealing.

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

CAD (MR), graphics foundation (GF), and mechanical design skills (MD). Second, another method was utilized where T, bt, F, and bf were modeled in terms of the answers to the 42 individual questions, hereafter denoted LR2, and thus a relation was built between T, bt, F, and bf using 94 answers (one per trainee) for each question.

Initial Perfromance Time (min)

70

4.1.1. Linear regression method 1, LR1 In the first linear regression (LR1) method, linear regression analysis was conducted according to the quantified answers (by category) according to Eqs. (3) through (6) below: T ¼ a1 BM þ a2 AM þ a3 CM þ a4 CSE þ a5 MR þa6 GF þ a7 MD

ð3Þ

F ¼ b1 BMþ b2 AM þb3 CM þb4 CSE þ b5 MR þ b6 GF þ b7 MD

ð4Þ

bt ¼ c1 BMþ c2 AM þ c3 CMþ c4 CSE þc5 MR þ c6 GFþ c7 MD

ð5Þ

bf ¼ d1 BM þ d2 AM þ d3 CMþ d4 CSE þ d5 MR þ d6 GFþ d7 MD

ð6Þ

FFBN Results Time (min), Exp. data Time (min) predicted, FFBN

60 50 40 30 20 10 0 1

11

21

31

Initial No. of Features Used

16

The terms a, b, c, and d represent the weights assigned by the linear relation. These are determined by applying linear regression to the available 94 data sets by using the least square method to minimize the errors of the linear fit of Eqs. (3)–(6) that are arranged in the form Ax ¼ b

ð7Þ

where A is a matrix containing the answers of the 94 students to the questions (7 categories for LR1 and 42 questions for LR2), b is a vector that contains the actual targets, and x is the vector that contains the unknown weights. The solution for this equation is normally found by inverting A and multiplying by b.

81

91

FFBN Results F, Exp. data F predicted, FFBN

12 10 8 6 4 2

1.6 Data (Target) Prediction

11

21

31

41 51 Trainees

61

71

81

91

SA Results (bt)

1.4

Data (Target) Prediction

1.2 1

40 bt

Time (min)

71

Fig. 8. Predictions vs. actual data: (top) Performance time and (bottom) Number of features used; Feed-Forward Back propagation Network, FFBN.

30

0.8 0.6

20

0.4

10

0.2

0

0 6 16 17 23 32 33 47 55 56 57 59 64 68 74 76 77 82 84 86 Trainees

14

1

SA Results (F)

1.2 Data (Target) Prediction

12 10

1

3 14 15 19 20 23 32 38 39 46 47 49 55 57 59 62 74 87 Trainees SA Results (bf) Data (Target) Prediction

0.8

8 bf

Features

61

14

1

SA Results (T) 50

51 41 Trainees

0

70 60

633

6

0.6

4

0.4

2

0.2

0

0 2

5

8 10 24 25 32 37 46 58 59 60 61 62 70 72 79 81 93 Trainees

6 11 15 22 32 35 40 46 48 49 51 66 67 74 75 77 85 92 94 Trainees

Fig. 7. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from the Simulated Annealing optimizer; shown compared with actual experimental data (only testing data points shown).

634

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

The data points of the 94 students were divided into two groups: one that included the training samples and comprised 80% of the population, while the other included the testing data points and represent 20% of the population (selected by the code from the master set at random). The training group was used for the generation of the weights while the testing group was used for the validation of the weights obtained. After applying the least squares method to this problem to minimize linear errors, the optimal solution weights were obtained. These weights led to the output for the entire data set as shown in Fig. 2: top figure for performance time and bottom figure for the number of features used. Fig. 3 plots only the results of the 20% test data and is intended as such to accentuate the goodness of fit. Fig. 3(a) compares the predicted values vs. the experimental target data points of T(1) for each trainee. The same applies for bt, F(1), and bf in Fig. 3(b)–(d), respectively. To evaluate the performance of the prediction methods being studied, the percentage average error shown in Eq. (8) was taken into consideration. This error is defined as the total average error divided by the mean value of the category being considered (i.e., T, bt, F, bf). Pn ðt  pi Þ ErrorX ¼ i i  100 ð8Þ nX where X represents the category under consideration, n is the total number of trainees, ti is the ith trainee target, pi is the ith trainee prediction and X is the mean value of the category under consideration. For the linear regression method 1 (LR1), the prediction errors, for T, bt, F, and bf, the values were 36%, 39%, 24%, and 27%, respectively, for the whole data set and 44%, 43%, 28%, and 28% for the test set only

(Table 1). This result was predicted since the linear performance of the regression is biased by the training data set while the test data points are naturally expected to have higher error values. 4.1.2. Linear regression method 2, LR2 A second linear regression method (LR2) was also utilized in which T, F, and their rates are related directly to individual questions. After applying the least squares method to this problem to minimize linear errors according to Eq. (8), the determined weights led to the results shown in Figs. 4 and 5. The output for the entire data set is shown in Fig. 4: top figure for performance time and bottom figure for the number of features used. Fig. 5 plots only the results of the 20% test data set. Fig. 5(a) compares the predicted values vs. the experimental target data points of T(1) for those trainees. The same applies for bt, F(1), and bf in Fig. 5(b)–(d), respectively. The figures show the predicted values for the four performance measures as determined from the linear regression method 2 (LR2) as compared with the experimental data (for the test data sets only). This method predicts the T, bt, F, and bf values with average errors of 30%, 33%, 22%, and 21%, respectively, for the whole data set and errors of 37%, 36%, 18%, and 14% (Table 1). 4.2. Simulated Annealing Local optimization algorithms such as gradient-based methods will converge to the local minimum closest to the starting solution and require the calculation of the first-order derivatives of the cost function and relevant constraints. Although not guaranteed, global search algorithms have a much better chance of converging to the global minimum, depending on the problem at hand, and on average require less computation time.

FFBN Results (T)

FFBN Results (bt)

60

1.4 1.2

Data (Target) Prediction

40

1 0.8

30

bt

Time (min)

50

0.6 20

0.4

10 0

0 6

2

9 24 25 29 32 33 38 39 46 48 50 57 63 74 86 93 94 Trainees FFBN Results (F)

10 9 8 7 6 5 4 3 2 1 0

3 8 9 10 13 15 19 23 26 36 38 51 52 75 76 81 90 Trainees FFBN Results (bf)

1.2 1

Data (Target) Prediction

0.8 bf

Features

Data (Target) Prediction

0.2

0.6 0.4

Data (Target) Prediction

0.2 0

5

9 17 20 24 26 49 56 57 59 60 61 71 74 87 88 91 93 Trainees

6 11 17 19 23 26 31 39 50 53 62 63 69 71 83 84 86 89 Trainees

Fig. 9. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from Feed-Forward Back propagation Network, FFBN; shown compared with actual experimental data (only testing data points shown).

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

is as shown in Fig. 6: top for performance time and bottom for the number of features used. Fig. 7 plots only the results of the test data. Fig. 7(a) compares the predicted values vs. the experimental target data points of T(1) for each trainee of the test set. The same applies for bt, F(1), and bf in Fig. 7(b)–(d), respectively.

Global search algorithms are loosely classified into two categories, deterministic methods such as trajectory and homotopy methods, and the probabilistic methods such as Simulated Annealing, Genetic Algorithms, Tabu search, Ant Colony optimization, Random Walk Algorithm, and many more. Simulated Annealing is used here to optimize the weights of the prediction fit with the error function being the summation of the squared difference between the actual experimental values and predicted values for T, F, bt, and bf according to Eq. (8). Similar to LR, the total data set is randomly divided in software into training and test groups with 80% and 20% of total population, respectively. The training set is used in Simulated Annealing to determine the best fit, while the test set is used to evaluate the prediction for the newly introduced 20% of trainees. The parameters utilized in the Simulated Annealing study are as listed in Table 2. A MATLAB code was custom-written to perform the above described weight optimizations. The SA method predicted the T, bt, F, and bf values with average errors of 29%, 24%, 32%, and 26%, respectively, for the whole data set and errors of 69%, 46%, 48%, and 61% for the test set (Table 1). Apparently, the results were close to those of LR2. This is not surprising since what was done actually is multiplying the input matrix by a weight to get the output values which is the same as fitting a straight line to the data, and thus LR. The SA predicted output for the entire data set

4.3. Artificial neural networks algorithms Three forms of Artificial Neural Networks (ANN) are used here for the forecasting and prediction of the T and F performance values along with their rates of improvement. The network was trained mostly using the Back propagation algorithm that utilizes gradient descent in minimizing the error. This supervised learning technique used for classification and pattern recognition can be safely applied in our case to generate accurate prediction results. Three techniques where explored for best predictions where each of these networks possesses different characteristics and uses different built-in error minimization algorithms, and thus yields different predictions. The neural networks toolbox in MATLAB was used to implement the above algorithms. The architecture of the Feed-forward Back propagation network (FBNN) is characterized by the unidirectional flow of information throughout the network. The Back propagation learning method is used in this network to update the weights (w) and biases (b) of the neurons in the network layers. The Elman Back propagation

EBN Results

70

Initial Perfrormance Time (min)

635

60

Time (min), Exp. data Time (min) predicted, EBN

50 40

30 20 10 0 1

11

21

31

41

51

61

71

81

91

Trainees EBN Results

16

Initial No. of Features Used

14

F, Exp. data F predicted, EBN

12 10 8 6 4 2 0 1

11

21

31

41

51

61

71

81

91

Trainees Fig. 10. Predictions vs. actual data: (top) Performance time and (bottom) Number of features used; Elman Back propagation Network, EBN.

636

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

neural network (EBN) is a recurrent network where the output of the first layer is used to update its weight. The main characteristic of the EBN is its ability to store information which makes it able to learn temporal patterns as well as spatial patterns. The generalized regression neural network (GRNN) is the third NN explored in this paper. This type of networks is often used for function approximation. The first layer of a GRNN is a radial basis layer while the second one is a special linear layer without bias.

4.3.1. Feed-forward back propagation network, FFBN This type of networks is a variation of the basic back propagation network (Rumelhart et al., 1986) and uses feed-forward neuron connections, and is trained with the supervised back propagation learning rule. The network is trained by providing it with input data (questionnaire answers) and the corresponding output (actual T and F along with their learning slopes, bt and bf). The weights and biases of the network are updated to minimize the error in the classification. The network used here consists of 2 layers with each layer containing 15 and 1 neurons, respectively. Matlab’s purelin activation functions were used for this purpose. Similar to LR and SA, 80% of the answers were used for training the network and the remaining 20% were used to validate the performance of the network with respect to the unlearned inputs. This method predicts the T, bt, F, and bf values with an average error of 16%, 17%, 8%, and 20%, respectively, for the whole data set and an error of 38%, 43%, 23%, and 23% for the test data set (Table 1). A script that translates the network described above was used along with the MATLAB ANN toolbox to generate the resulting output for the entire data set as shown in Fig. 8: top for performance time and bottom for the number of features used. The results for the 20% test data are plotted in Fig. 9 where

Fig. 9(a) compares the predicted values vs. the experimental target data points of T(1). The same applies for bt, F(1), and bf in Fig. 9(b)–(d), respectively. 4.3.2. Elman back propagation network, EBN The EBN is a recurrent neural network that can maintain a copy of the previous values of its hidden layer (Cruse, 2006). This will provide the network with a short term memory. This type of networks is more powerful than the standard multilayer perceptron in its sequence prediction behavior. This network was fed with the same input vector used in the previous methods. EBN predicts the T, bt, F, and bf values with average errors of 17%, 22%, 8%, and 20%, respectively, for the whole data set and errors of 30%, 43%, 22%, and 40% for the test data set (Table 1). EBN’s forecasts of the entire data set are as shown in Fig. 10: top for performance time and bottom for the number of features used. Fig. 11 plots only the results of the test data. Fig. 11(a) compares the predicted values vs. the experimental target data points of T(1) for each trainee. The same applies for bt, F(1), and bf in Fig. 11(b)–(d), respectively. 4.3.3. Generalized regression neural network, GRNN The last NN used in this paper is the probabilistic GRNN which is perceived as more powerful in the sense that it can be generally more accurate and can be trained faster than the multilayer perceptron networks. Using the built-in MATLAB toolbox, the input vector was fed to a generated GRNN. This method predicts the T, bt, F, and bf values with an average error of 5%, 7%, 3%, and 5%, respectively, for the whole data set and an error of 24%, 35%, 16%, and 23% for the test data set (Table 1). GRNN’s forecasted output for the entire data set is shown in Fig. 12: top for

EBN Results (T)

EBN Results (bt)

70

1.2

60

Data (Target) Prediction

1 0.8

40 bt

Time (min)

50

30

0.6 0.4

20

0.2

10

0

0 3

4

7

8 12 15 24 26 47 52 54 59 69 71 72 73 78 79 84 89 94 Trainees

8 10 11 18 20 21 36 39 44 49 55 61 65 68 69 Trainees EBN Results (F)

EBN Results (bf) 0.8

12

0.7

10

0.6

8

0.5 bf

Features

14

6

0.4 0.3

4 2

Data (Target) Prediction

Data (Target) Prediction

0.2 0.1

Data (Target) Prediction

0

0 9 10 20 23 27 33 34 62 67 68 72 76 80 86 90 91 92 94 Trainees

8 10 14 17 19 20 25 32 40 47 48 57 61 69 74 76 90 91 Trainees

Fig. 11. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from Elman Back propagation Network, EBN; shown compared with actual experimental data for the test data sets only.

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

improvement. To visualize the performance of each prediction method, the moving average prediction error was calculated for all trainees. The predicted average error values for all methods utilized are plotted in Fig. 14 for T(1) (a), bt (b), F(1) (c), and bf (d). Table 1 is a summary of the average prediction errors for all the methods used in this study with the corresponding prediction error of the 20% validation test data sets are shown suffixed as (_test). The results show that the GRNN yielded the least prediction error for practically all performance measures. Therefore, this method can be used to closely predict the performance of the novice CAD users according to input vectors based on the individual answers to the questionnaire found in Hamade and Artail (2008). The resulting error values in Table 1 are plotted on a radar graph in Fig. 15. The star-like behavior of the ANN techniques is due to the fact that ‘learned’ data sets are typically predicted with minimal error while errors associated with unlearned test sets tend to be greater. This is contrasted to the behavior of linear regression where the test and train sets error values are close due to the linearity of the relation conducted by linear regression. As a future follow-up study, we plan to implement a pilot study where we place students in groups in accordance with the predictions that a GRNN would produce given their answers to the

performance time and bottom for the number of features used. Fig. 13 plots only the results of the test data. Fig. 13(a) compares the predicted values vs. the experimental target data points of T(1) for each trainee. The same applies for bt, F(1), and bf in Fig. 13(b)–(d), respectively.

5. Summary and conclusions Recent and sophisticated frameworks employ AI techniques such as Artificial Neural Networks, Simulated Annealing, and Genetic Algorithms. In this work, we will explore the use of regression and ANN techniques to try to infer the future performance of CAD trainees (as originally reported in Hamade and Artail, 2008). Findings reveal that reasonable predictions can be made based solely on trainee’s technical profiles prior to the start of the CAD training course. However, as compared to LR, SA, and to the two other ANN methods utilized (Feed-Forward Back propagation and Elman Back propagation) it was found that the General Regression Neural Networks, GRNN, produced the best predictions of performance. These predictions included all four performance measures, CAD model construction speed and its rate of improvement, and model sophistication and its rate of

Initial Performance Time (min)

70

637

GRNN Results

60

Time (min), Exp. data Time (min) predicted, GRNN

50 40

30 20 10 0 1

11

21

31

41

51

61

71

81

91

Trainees 16

GRNN Results

Initial No. of Features Used

14

F, Exp. data

12

F predicted, GRNN

10 8 6 4 2 0 1

11

21

31

41

51

61

71

81

91

Trainees Fig. 12. Predictions vs. actual data: (top) performance time and (bottom) number of features used; Generalized Regression Neural Network, GRNN.

638

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

GRNN Results (T)

GRNN Results (bt)

45

1.2

35

1

30

0.8

25 bt

Time (min)

40

20 15

0.6 0.4

10

Data (Target) Prediction

5

Data (Target) Prediction

0.2 0

0 1

4 13 16 22 23 28 30 31 40 43 57 60 70 73 77 78 80 87 Trainees

4 11 13 14 16 22 45 46 54 55 58 60 67 69 70 73 89 91 94 Trainees

GRNN Results (F)

GRNN Results (bf)

12

0.8 0.7 0.6

8

0.5

6

bf

Features

10

0.4 0.3

4

0.2

Data (Target) Prediction

2

Data (Target) Prediction

0.1 0

0 2 4

8 12 15 16 17 21 25 34 39 47 54 57 61 69 71 72 79 86 87 Trainees

7 14 34 37 38 39 46 49 51 52 53 54 56 74 80 81 83 Trainees

Fig. 13. Predicted T(1) (a), bt (b), F(1) (c), and bf (d) values as determined from Generalized Regression Neural Network, GRNN; shown compared with actual experimental data (only testing data points shown).

Time Predictions Average Error

bt Predictions Average Error 70%

60% EBN LR2

GRNN SA

60% Percent Error

Percent Error

50%

FFBN LR1

40% 30% 20% 10%

50% 40% 30% 20%

0% 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93

Trainees

Trainees

Features Predictions Average Error

bf Predictions Average Error 100%

FFBN LR1

EBN LR2

GRNN SA

80% Percent Error

Percent Error

GRNN SA

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93

80%

60%

EBN LR2

10%

0%

70%

FFBN LR1

50% 40% 30% 20%

FFBN LR1

EBN LR2

GRNN SA

60% 40% 20%

10% 0%

0% 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93

Trainees

Trainees

Fig. 14. Predictions of average error for T(1) (a), bt (b), F(1) (c), and bf (d) values for all utilized methods.

R.F. Hamade et al. / Engineering Applications of Artificial Intelligence 25 (2012) 628–639

%T Error %bf_test Error

%bf Error

70% 60% 50% 40% 30% 20% 10% 0% -10%

%T_test Error LR1 LR2 %F Error

FFBP EBP GRNN SA

%F_test Error

%bt_test Error %bt Error

Fig. 15. Prediction error comparison of for T, F, bt, and bf for each method used in the study and the corresponding test data set of each method (suffixed with _test).

questionnaire we have been using. We will use closely designed delivery techniques in terms of the level of supplied details and applied supervision. Our strategy will involve giving additional attention to those students who we predict, according to the developed model, will perform relatively less efficiently. Our objective is to help these groups achieve similar learning outcomes as compared to their supposedly-higher performing classmates. If the results of this future trial are in agreement with our theoretical results (i.e., being able to predict performance from questionnaires), we will adopt this methodology on a regular basis in future offering of this course and perhaps in other similar hands-on courses. A related investigation will involve studying the additional cost that will be incurred (e.g., manpower and preparation of additional material) vs. the expected benefits, from the point of view of the training/educational organization.

Acknowledgments The authors are grateful for the financial support of the University Research Board (URB) of the American University of Beirut. The second author also wishes to acknowledge the support of Consolidated Contractors Company through the CCC Doctoral Fellowship in Manufacturing. References Alenezi, J.K., Awny, M.M., Fahmy, M.M.M., 2009. Effectiveness of artificial neural networks in forecasting failure risk for pre-medical students 2009 Proceedings – The 2009 International Conference on Computer Engineering and Systems, ICCES’09, Art. no. 5383294, pp. 135–138. Boussemart Y., Cummings, M.L., Predictive models of human supervisory control behavioral patterns using hidden semi-Markov models, Eng. Appl. Artif. Intell., in press, doi:10.1016/j.engappai.2011.04.008.

639

Cooper, C.I., Pearson, P.T., A genetically optimized predictive system for success in general chemistry using a diagnostic algebra test. J. Sci. Educ. Technol. in press, doi:10.1007/s10956-011-9318-z. Cruse, H., 2006. Neural Networks as Cybernetic Systems, 2nd edn. Brains, Minds and Media, Bielefeld, Germany. Dong, J.-X., Krzyzak, A., Suen, C.Y., 2002. Local learning framework for handwritten character recognition. Eng. Appl. Artif. Intell. 15 (2), 151–159. Elman, J.L., 1990. Finding structure in time. Cognit. Sci. 14 (2), 179–211. Fitts, P.M., 1954. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 381–391. Gerasimovic, M., Stanojevic, L., Bugaric, U., Miljkovic, Z., Veljovic, A., 2011. Using artificial neural networks for predictive modeling of graduates’ professional choice. New Educ. Rev. 23 (1), 175–188. Glenn, F., Stokes, J., Neville, K., Bracken, K., 2005. An Investigation of Human Performance Model Validation, AFRL-HE-WP-TR-2006-0002, Report Number: A488544, /http://www.stormingmedia.us/48/4885/A488544.htmlS, accessed December 11, 2010. Hamade, R.F., Artail, H.A., Jaber, M.Y., 2005. Learning theory as applied to mechanical CAD training of novices. Int. J. Hum. Comput. Interact. 19 (3), 305–322. Hamade, R.F., Artail, H.A., Jaber, M.Y., 2007. Evaluating the learning process of mechanical CAD students. Comput. Educ. 49 (3), 640–661. Hamade, R.F., Artail, H.A., 2008. A Study of the Influence of Technical Attributes of Beginner CAD Users on Their Performance. Computer-Aided Design 40, 262–272. Hamade, R.F., 2009. Profiling the desirable CAD trainee: technical background, personality attributes, and learning preferences. ASME J. Mech. Des. 131 (12), 121009. ¨ Hamade, R.F., Jaber, M.Y., Sikstrom, S., 2009. Using univariate, multivariate, and aggregate learning curve models to investigate CAD competence. Comput. Ind. Eng. 56 (4), 1510–1518. Jastrzembski, T.S. Gluck, K.A., Rodgers, S., Krusmark, M., 2009. The predictive performance optimizer, mathematical modeling for performance prediction, Proceedings of the 18th Conference on Behavior Representation in Modeling and Simulation, Sundance, UT, 31 March–2 April 2009, Paper 09-BRIMS-017, pp. 141–142. Johnson, M.D., Diwakaran, R.P., 2011. An educational exercise examining the role of model attributes on the creation and alteration of CAD models. Comput. Educ. 57 (2), 1749–1761. Julie, M.D., Kannan, B. Attribute reduction and missing value imputing with ANN: prediction of learning disabilities, Neural Comput. Appl., in press, doi:10.1007/ s10956-011-9318-z. Lykourentzou, I., Giannoukos, I., Mpardis, G., Nikolopoulos, V., Loumos, V., 2009. Early and dynamic student achievement prediction in e-learning courses using neural networks. J. Am. Soc. Inf. Sci. Technol. 60 (2), 372–380. McDermott, C.M., Marucheck, A., 1995. Training in CAD: an exploratory study of methods and benefits. Eng. Manage., IEEE Trans. 42, 410–418. McKenzie, K., Schweitzer, R., 2001. Who succeeds at university? Factors predicting academic performance in first year Australian university students. Higher Educ. Res. Dev. 20 (1), 21–33. Pentland, A., Liu, A., 1999. Modeling and prediction of human behavior. Neural Comput.. 11, 229–242. Pham, D.T., Karaboga, D., 2000. Intelligent Optimization Techniques. Springer Verlag, London. Rumelhart, D.E., Hinton, G.E., Williams, R.J., 1986. Learning representations by back-propagating errors. Nature 323 (6088), 533–536. Wanga, Y.-H., Liao, H.-C., 2011. Data mining for adaptive learning in a TESL-based e-learning system. Expert Sys. Appl. 38 (6), 6480–6485. Wright, T., 1936. Factors affecting the cost of airplanes. J. Aeronaut. Sci. 3, 122–128. Yu, J., 2010. Analyzing creativity of students with neural networks, 2010 Proceedings – 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems, ICIS 2010 1, art. no. 5658709, pp. 118–121. Zhang, G., Patuwo, B.E., Hu, M.Y., 1998. Forecasting with artificial neural networks: the state of the art. Int. J. Forecast. 14 (1), 35–62.