- Email: [email protected]
A comparison of screening methods: Selecting important design variables for modeling product usability
ARTICLE IN PRESS
International Journal of Industrial Ergonomics 32 (2003) 189–198
A comparison of screening methods: Selecting important design variables for modeling product usability Sung H. Han*, Jongseo Kim Division of Mechanical and Industrial Engineering, Department of Industrial Engineering, Pohang University of Science and Technology, San 31, Hyoja Dong, 790-784 Pohang, South Korea Received 21 February 2002; received in revised form 16 September 2002; accepted 1 April 2003
Abstract Product usability is affected by a large number of design variables. For example, to design an audio/visual product such as a video-cassette player, the design of controls, information displays, layout of controls, etc. could affect the user task performance. In addition, the product shape, color, material, etc. could also affect the subjective user satisfaction. Altogether, the number of design variables could easily go up to hundreds. To build a model describing the functional relationship between the product usability and the product design variables, it is very important to select only the important design variables for securing modeling efficiency and obtaining an effective model. Although some studies used expert opinions to select them, they have drawbacks such as lack of selection objectivity and expert availability. This study proposes statistical methods for screening important design variables to substitute the expert opinions. Three methods (i.e., principal component regression; cluster analysis; and a partial least squares) are applied to a set of design variables for audio/visual electronic products. Performances of the variable screening methods are examined by building usability models. That is, the variables screened by each method are used to build a usability model. The model performances are then compared to those screened by expert opinions to determine which method provides the best performance. The results show that all these three methods have better model performances than the expert screening in terms of R2 ; the number of variables in the model, and PRESS. Relevance to industry This paper provides the product designers with efficient ways to select important design variables to the product usability such as luxuriousness, salience, rigidity, etc. The guidelines for choosing an appropriate method suggested in this study are expected to help the product designers to build an efficient usability model and as a result, to understand the relationship between the product design variables and the product usability. r 2003 Elsevier Science B.V. All rights reserved. Keywords: Product usability; Variable screening; Principal component regression; Cluster analysis; Partial least squares
1. Introduction
*Corresponding author. Tel.: +82-54-279-2203; fax: +8254-279-2870. E-mail address: [email protected] (S.H. Han).
Product usability is considered one of the most important factors for a product to be successful in the market (Dumas and Redish, 1994). Products with a variety of functions often fail in the market
0169-8141/03/$ - see front matter r 2003 Elsevier Science B.V. All rights reserved. doi:10.1016/S0169-8141(03)00063-5
ARTICLE IN PRESS 190
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
because they have a low level of usability. Now, usability is not only a ‘must’ requirement, but also a business phenomenon in designing a product (Rubin, 1994). To cope with this trend, many manufacturers attempt to develop a product with enhanced usability (Myers and Rosson, 1992). Traditionally, usability has focused more on the user performance than the subjective satisfaction in the human–computer interaction research areas (Nielsen and Lavy, 1994). When it comes to consumer electronic products, however, the subjective satisfaction is as important as the performance. This is mainly because they are not only a tool to accomplish an intended task, but also a decoration in the living room. To reflect this new concept, Han et al. (2000, 2001) defined the product usability for consumer electronic products. They suggested that the usability consist of two aspects: performance and image/impression. The performance was further classified into several dimensions such as memorability, learnability, etc. Likewise, the image/impression was further divided into specific dimensions such as luxuriousness, elegance, etc. It is postulated that the product usability is affected by how a product is designed. If a product design could be decomposed into specific design variables, then it is possible to model the relationship between the product design variables and the product usability. That is, the product usability is a function of design variables as shown in Fig. 1. Then, a natural question arises: ‘‘How do we identify design variables that affect the product usability?’’
Traditional product design process is characterized by a creative leap (Dorst and Cross, 2001). Talented designers, who have insights about what design variables are significant to the user, manipulate them to create a new design. This approach, however, may not be reliable and systematic. The Kansei engineering approach (Nagamachi, 1995; Ishihara et al., 1996) attempts to identify design variables by conducting a user survey. It provides a few design variables that are most likely to affect the user’s impression. On the other hand, the approach suggested by Han et al. (2000) uses a systematic framework for decomposing the product design into specific elements. They assume that the human interface of a product consists of ‘physical’ and ‘logical’ components. These components are then further decomposed into specific design variables by their properties such as individual, interaction, and integration (for detailed descriptions, refer to Han et al., 2000). Common to all these methods is that they require experts (designers or usability engineers) to come up with important design variables. It is obvious that the results are subjective and having lack of validity. That is, it is not guaranteed that the same results are obtained by different groups of experts. Another problem is that the number of design variables can easily go up to tens or hundreds. For example, Han et al. (2000) reported that a total of 88 design variables could affect the usability of typical audio/visual consumer electronic products such as video-cassette players, CD players, audio amplifiers, etc. Of course, some of them would be very important to the usability and
Fig. 1. Concept of the usability evaluation model.
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
some would be negligible. When the number of design variables is large, the modeling process is complex and inefficient. In addition, it is quite common to have a list of design variables that are highly correlated to each other. When the input variables are correlated, building a relationship model is difficult. Worse yet, it is highly likely to end up with an ineffective or invalid model. In order to overcome these drawbacks, this study proposes statistical techniques to screen out unimportant design variables given an initial set of candidate design variables. Three techniques (principal component regression (PCR); cluster analysis; and partial least squares (PLS)) are introduced as the alternatives for substituting expert opinions. A case study is presented to demonstrate the effectiveness of these techniques. In the case study, a set of design variables for audio/visual consumer electronic products is screened by these techniques. The results are then compared to those screened by the expert opinions. 2. Variable screening methods 2.1. PCR screening When variables are correlated, the principal component analysis provides a way of eliminating correlated ones so that only the independent remains (Dunteman, 1989). Principal components are defined as the variables that are mutually independent. Since correlations among the variables are reduced, the number of principal components is usually smaller than that of the
191
variables initially considered. A regression model using the principal components as the independent variables is called PCR. The following is a brief description of the method for screening variables using the PCR. For detailed descriptions of the PCR, refer to Dunteman (1989). Suppose there are p principal components (Ti ) selected from n independent variables (Xj ) as shown in Table 1. Pij represents the loading vector between principal component Ti and independent variable Xj : SSRðti Þ is defined as the variance explained by the principal component Ti when a PCR model is developed. Here, the usability evaluation scores of audio/visual consumer electronic products are used as the dependent variable in the model (detailed descriptions of the dependent variable are presented in the case study section). A large value of SSRðti Þ implies that the corresponding principal component Ti explains a majority of the variance in the PCR model. This means that Ti has great influence on the usability. SSRðt1 ; y; tp Þ is the total variance explained by the principal components. Eq. (1) defines the variance explained by Xj : The ratio of the variance explained by Xj to the total variance is defined as the degree of influence of Xj (see Eq. (2)). The larger the ratio, the greater the influence on the usability: p X SSRðti Þp2ij ; ð1Þ i¼1
Pp
i¼1
SSRðti Þp2ij
ð2Þ
: SSRðt1 ; t2 ; y; tp Þ
Table 1 Mathematical principle of PCR screening PC
T1 T2 y Tp SSRðt1 ; t2 ; y; tp Þ Variable influence
Variables
SSRðt1 Þ SSRðt2 Þ y SSRðtp Þ
X1
X2
y
Xn
P11 P21 y P Pp1p
P12 P22 y Pp2 Pp
y y y y y
P1n P2n y Ppn Pp
y
Pp
i¼1
Pp
SSRðti Þp2i1
SSRðti Þp2i1 SSRðt1 ; t2 ; y; tp Þ i¼1
i¼1
Pp
SSRðti Þp2i2
SSRðti Þp2i2 SSRðt1 ; t2 ; y; tp Þ i¼1
i¼1
SSRðti Þp2in
SSRðti Þp2in SSRðt1 ; t2 ; y; tp Þ i¼1
ARTICLE IN PRESS 192
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
Once the degree of the influence of each variable is determined, selecting important variables is straightforward. The variables with large ratios are selected and those with small ones are eliminated.
using the SCCO as the criterion for selecting the representative can result in selecting a variable with a low SCC. This ratio is defined as the DI. The variable with the maximum SCC and the minimum DI value of each cluster is chosen as the representative variable.
2.2. Cluster screening Cluster analysis is characterized by examining a variety of characteristics of variables and clustering them that share common characteristics (SAS Institute Inc, 1985). The resulting groups are mutually exclusive. A variable from each cluster is selected representing the characteristics of the variables in the cluster. Therefore, the number of the representative variables is smaller than the number of the variables initially considered. The resulting variables can be used to build a usability model. The following is a brief description of the method for screening variables using the cluster analysis. For detailed descriptions of the cluster analysis, refer to Anderberg (1973) and SAS Institute Inc (1985). There are two different criteria for selecting a representative variable from a cluster: squared correlation coefficient (SCC) between the cluster component and the variables in the cluster, and distinction index (DI) between the variable and the other cluster components. The SCC is equal to the R2 of the regression model that uses a variable in a cluster component and the other variables in the cluster as the dependent and independent variables, respectively. The variable providing the maximum SCC becomes the representative of the cluster because it shows the best performance in describing the cluster component. On the other hand, determining a DI is more complex. First, the R2 of a regression model between the variable in one cluster and the other cluster components is calculated. Every variable in the cluster is examined to come up with a variable providing the maximum R2 : The variable with the maximum R2 is defined as the SCC of a variable and the outer cluster (SCCO). A small SCCO implies that the corresponding cluster is clearly separated from the other clusters. Next, both SCC and SCCO values are subtracted from 1 separately and the ratio between them is calculated, because
2.3. PLS screening The PLS method selects only significant variables based on a model that builds the relationship between independent and dependent variables (Geraldi and Kowalski, 1986). The selected variables are expected to explain a majority of the total variance of the model. This method is frequently used in the field of chemical engineering, because there are a large number of independent variables that may affect quality characteristics of chemical products. The mathematical properties of the PLS screening are similar to those of the PCR screening (see Table 2). Suppose there are p latent variables (Li ) selected from n independent variables (Xj ) and usability evaluation score, as shown in Table 2. A regression model based on PLS is built using the Li as the independent variables and the usability evaluation scores as the dependent variable (detailed descriptions of the dependent variable are presented in the case study section). Wij represents the loading vector between the latent variable Li and the independent variable Xj : SSi implies the variance explained by the latent variable Li of the model. A large value of SSi implies that the corresponding latent variable Li explains a majority of the variance in the PLS model. Eq. (3) shows the variance explained by Xj : The ratio of the variance explained by Xj to the total variance implies the relative influence of each independent variable (Xj ) on the total variance (see Eq. (4)): p X
SSi Wij2 ;
ð3Þ
i¼1
Pp
i¼1 P p
SSi Wij2
i¼1
SSi
:
ð4Þ
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
193
Table 2 Mathematical principle of PLS screening Variance explained L1 L2 y Lp
SS1 SS2 y SSp
Variable importance in projection
X1
X2
y
Xn
W11 W21 y Wp1
W12 W22
y y y y
W1n W2n
y
n
n
Wp2
Pp 2 i¼1 SSi Wi1 P p SS i i¼1
If n; the number of the independent variables, is multiplied by the influence of each independent variable, then the average of the influence becomes 1. This transformed value is called variable importance in projection (VIP) (Umetrics AB, 2001). The VIP value of greater than 1 means that the corresponding independent variable has a significant effect on the dependent variable.
3. Case study 3.1. Data collection To examine the performance of the variable screening methods, an existing data set was used. Han et al. (1998, 2000) reported a study that used expert opinions to select important design variables for building usability models. They assumed that the product usability could be determined by the human interface of a product. In their study, a total of 36 different audio/visual consumer electronic products, such as video-cassette players, CD players, audio amplifiers, were analyzed to come up with design variables. The visual display on a CD player and the menu structure for operating it are examples of the design variables. Based on a framework developed by Han et al. (1998, 2000), a typical audio/visual product was decomposed into a total of 88 design variables. Note that 61 out of the 88 design variables were continuous while the remaining were discrete. As stated in the introduction, it was assumed that product usability consisted of the user performance and the image/impression towards a
n
Pp 2 i¼1 SSi Wi2 P p SS i i¼1
Wpn Pp 2 i¼1 SSi Win P p SS i i¼1
product. These two groups were then further classified into detailed dimensions that specified different aspects of each group. The performance was classified into 23 dimensions such as memorability, learnability, etc. Likewise, the image/ impression was classified into 25 dimensions such as luxuriousness, elegance, etc. Once the usability dimensions and design variables of audio/visual products were obtained, experts and product designers selected important design variables for each dimension because all the variables were not equally important to each dimension. A usability evaluation experiment was then conducted to examine the usability of the products. A total of 30 subjects were instructed to evaluate the products. Evaluation scores (using 100 point scales) were obtained for each dimension of each product used in the experiment. A total of 33 usability models based on a multiple linear regression technique were then developed, which related the selected design variables to each usability dimension. For detailed descriptions of the design variables, usability dimensions, and the models, refer to Kwahk et al. (1997) and Han et al. (1998, 2000, 2001). The R2 of the 33 usability models reported by Han et al. (1998, 2000) ranges from 0.70 to 0.99. Since the primary purpose of this study is to compare the performances of the models between the expert opinions and the three statistical screening methods, all the models should be compared regardless of their R2 values. However, comparing all the 33 models using the four methods would be time consuming and require a large amount of efforts. In addition, it is not
ARTICLE IN PRESS 194
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
necessary to compare all the 33 models to get insights into the performance differences among the methods. Thus, this study selected a part of the usability dimensions. The R2 values of the models using the expert opinions were used as a selection criterion. The usability models were classified into three groups: low (R2 values ranging 0.7–0.8), middle (R2 values ranging 0.8–0.9), and high (R2 values ranging 0.9–1.0). Two usability models were arbitrarily chosen from each group, which were subject to comparison in this study. Table 3 presents the definitions of the selected usability models and their R2 values. 3.2. Comparison of screened variables Only continuous variables were subject to the screening process since the PCR, the PLS, and the cluster screening were not able to handle discrete variables. Table 4 presents the screening results for the six usability dimensions. The number of screened variables varied with the screening methods. Generally, the PCR screening selected the largest number of variables on the average, while the PLS screening resulted in the smallest number. Note that the expert screening selected only five variables for the ‘volume’ dimension.
This result may come from the fact that there are not many variables that would affect the dimension or the initial list of the design variables does not contain enough number of variables to explain it. As shown in Table 3, the ‘volume’ model using the variables selected by the experts has a good performance. Thus, it is concluded that there are not many design variables to explain the ‘volume’ dimension. Note that the PLS screening always ends up with a smaller number of selected variables than the PCR across the dimensions. Also note that since the cluster screening does not consider the usability dimensions in selecting important variables, the same set of variables is selected across the dimensions. As stated in the previous section, the PCR and PLS screening methods use a usability model to select important variables. It is interesting to compare the screened variables resulting from these two methods. Table 5 shows the variables that are commonly selected by them. The ‘rigidity’ dimension has the lowest proportion of the commonly selected variables, while the ‘volume’ dimension shares almost all of the selected variables between the two. These two methods share approximately 77% of the variables on the average. This result may imply that the PLS seems to provide a better screening result in terms of the
Table 3 Performance of the usability models using expert opinions (Han et al., 1998, 2000) Usability dimension
Definition
Model R2
Elegance Comfort Volume Salience Rigidity Granularity
Degree to which a product is elegant or graceful Degree to which the user feels easy and comfortable with a product Feeling that a product looks voluminous or slim Degree to which a product is outstanding, prominent, and catching one’s eye Feeling that a product looks stout, stable, and secure Degree to which a product is worked out with great care and in fine detail
0.99 0.99 0.88 0.83 0.72 0.71
Table 4 Number of screened variables Method
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
Mean
Expert screening PCR screening Cluster screening PLS screening
26 26 20 20
5 35 20 20
30 26 20 16
42 28 20 25
18 32 20 20
37 32 20 15
26.3 29.8 20.0 19.3
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
195
Table 5 Variables commonly selected by the PCR and the PLS screening methods
Common variables PLS screening Ratio of common variables (%)
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
11 20 55
18 20 90
11 16 69
22 25 88
16 20 80
12 15 80
Table 6 Proportion of the variables commonly selected by the expert screening and the other screening methods (%) Method
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
Mean
PCR screening PLS screening Cluster screening
38 31 31
100 100 0
50 43 30
40 36 33
39 28 17
57 32 24
54 45 22.5
number of the selected variables. However, the screening performance should be determined by the model performance that uses the screened variables as the independent variables. Table 6 presents the proportion of the variables commonly selected by the expert screening and the other methods. The average proportion is only 35% without considering the ‘volume’ dimension, which means there is a great difference in selecting variables between the expert screening and the other statistical screening methods. Again, it is necessary to build a usability model using the selected variables to compare the screening performances among the methods. The following section describes the procedure of building the usability models. 3.3. Developing usability models In order to compare the performances of the screening methods, a regression model was built for each usability dimension. Here, the screened variables were used as the independent variables while the usability evaluation scores were used as the dependent variable of the model. A secondorder multiple linear regression was used to model the relationship because it was known to be appropriate for modeling the subjective response of human users (Williges, 1981). That is, a main effect and a pure quadratic effect of each variable, and a linear-by-linear interaction of two variables
were included in the model. In building the secondorder model, it was necessary to further screen out unimportant variables so that only significant model items could remain in the model. For this purpose, either the backward elimination or the stepwise regression technique (Myers, 1990) was used depending upon the error degrees of freedom of the model. The resulting model was then subject to the examination of multicollinearity (Myers, 1990) since the model items were expected to be highly correlated to each other. The variance inflation factor (VIF) was used as the criterion of examining the multicollinearity. Model items having the VIF value higher than 10 were eliminated from the model since it was assumed that no significant multicollinearity would exist with the VIF value of less than 10 (Myers, 1990). The resulting model was further elaborated by eliminating a model item with the maximum VIF value one by one. In this process, several alternative models were generated and they were compared to each other in terms of R2 ; PRESS, and the number of model items. A model with the best performance in terms of these three criteria was selected as the final model. In this study, four different models for each usability dimension were built, each using the variables generated by the four different screening methods. More detailed explanations about the model building can be found in Kim and Han (2000) and Han et al. (2000).
ARTICLE IN PRESS 196
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
3.4. Comparison of the usability models 3.4.1. Coefficient of determination All the models show an R2 value of greater than 0.70 (see Fig. 2). All the models using the variables selected by the cluster screening have the greatest R2 (0.96) on the average. Although lower than the cluster, the PLS and the PCR screening have an average R2 value of greater than 0.90, which implies that the variations resulting from the data are explained almost completely. On the other hand, the expert screening shows the lowest average R2 value (0.86), although it is still considered a good model. Note that the expert screening has a large variation in R2 depending upon the usability dimensions, while the other methods show relatively a uniform distribution of the R2 values. As far as the R2 value is concerned, the statistical screening methods seem to provide a better performance than the expert screening. 3.4.2. Number of model items The number of model items in a model has an important practical implication. When a model has a large number of model items, it is difficult to interpret the effect of each item although it has a high value of R2 : When it comes to interactions, it is even more difficult, if not impossible, to interpret the result. Fig. 3 presents the number of model items for all four models. Both the PLS and the PCR screening have relatively a smaller number of items (6.7 and 8.8 items, respectively) on the average than the expert and the cluster screening (14.2 and 12.8 items, respectively). This result may stem from the fact that the PLS and the PCR used the dependent variable in the screening process, while the other two did not. Together with the result of R2 ; the models using the PCR and the PLS screening seem to explain the variation of the usability evaluation scores with a fewer number of variables. 3.4.3. PRESS PRESS is an index for estimating the prediction capability and stability of a regression model (Myers, 1990). A model with a low PRESS is considered more stable and reliable. Fig. 4 shows the PRESS of the models. The expert screening has
Fig. 2. R2 of the usability models. Average R2 : expert screening (0.86), PCR screening (0.91), PLS screening (0.90), cluster screening (0.96).
Fig. 3. The number of model items in the usability models. Average: expert screening (14.2), PCR screening (8.8), PLS screening (6.7), cluster screening (12.8).
the highest PRESS on the average. The cluster and the PLS screening have a relatively low value of PRESS than the other two. Note that the PRESS of the models using the expert screening (average value of 1706.4) is four times as large as the models using the other statistical screening methods (average value of 396.5). In addition, the PLS and the cluster screening show a lower variation in PRESS than the others. It can be concluded that the PLS and the cluster screening seem to provide a better prediction capability and stability. 3.4.4. Discussion The models using the expert screening have an R2 value of 0.86 and approximately 14 model items on the average. Note that the ‘comfort’ and ‘elegance’ models result in an R2 value of almost
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
197
Table 7 Variable screening methods appropriate to the purpose of modeling Purpose of modeling
Recommended screening methods
Accurate prediction
PCR screening, cluster screening, PLS screening Expert screening
Easy interpretation of model items Stable prediction Fig. 4. PRESS of the usability evaluation models. Average PRESS: expert screening (1706), PCR screening (518), PLS screening (353), cluster screening (319).
1.0, which implies that the models explain almost all of the variations of the usability evaluation scores. However, they may not be practical, because they have too many model items (18 and 16 items, respectively). The large number of model items makes it difficult to interpret the models. The expert screening seems to be a relatively poor method than the others since it shows the poorest performance in terms of the three criteria. The R2 of the models using the PCR and PLS screening have an average value of 0.91 and 0.90 with approximately 9 and 7 model items, respectively. In addition, they show a good model stability and reliability. The models using the cluster screening show the highest value of R2 (0.96) on the average and the lowest value of PRESS. However, they have too many model items (13 items on the average) and as a result, they are expected to be difficult to interpret. In sum, the models using the statistical screening methods show reliable and excellent model performances with a relatively small number of model items. However, they may have a drawback, too. For example, some variables included in the models are not always easy to interpret. This is because the variables were selected based only on the distributional characteristics of the product design variables. In other words, variables having a high variance had a more chance to be selected as important ones. When combined with high correlations among the variables, nonsense variables could be included in the models. The expert screening, on the other hand, does not have this
Small number of model items
Cluster screening, PLS screening PCR screening, PLS screening
drawback since it selected variables based on the know-how of the experts. That is, the model does not include nonsense variables, which makes the interpretation of their effects relatively easier. In consequence, all the four screening methods could be applied depending upon the purpose of the modeling (see Table 7). The PCR, the cluster, and the PLS screening are recommended to build a highly accurate model. To add a high level of reliability and stability, either the PLS or the cluster screening is recommended. In addition, either the PLS or the PCR screening is recommended to build a model with a relatively smaller number of items. On the other hand, the expert screening is useful for building a model that does not include nonsense variables.
4. Conclusion This study compared four different methods of screening important design variables for building usability models. Given a usability model developed by the multiple linear regression technique, the comparison was made on the basis of the model performances such as the model R2 ; PRESS, and the number of model items. The expert opinions appeared to provide a usability model that did not contain nonsense design variables. The PCR and PLS screening methods turned out to provide a model with a reasonable number of model items and high R2 : The cluster screening showed a good performance in building an accurate and reliable model. This result implies that no method was superior in all the three
ARTICLE IN PRESS 198
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
criteria. That is, any method could be applied depending upon the purpose of the modeling. Guidelines were provided for choosing an appropriate screening method. Obtaining design variables affecting the product usability is a difficult task because there is no concrete relationship between the variables and the usability. It would be a usual practice that the designers end up with manipulating a few variables to create a new design. This study provides a tool for screening important design variables in a systematic manner. Any variables that are expected to affect the usability (regardless of their magnitude of influence) can be included in an initial list of potential variables, and then the methods suggested in this study can be applied to come up with important variables. This study is an initial attempt to provide a systematic way of modeling the product usability using product design variables. It is necessary to have more case studies to demonstrate the effectiveness of this approach.
References Anderberg, M.R., 1973. Cluster Analysis for Applications: Probability and Mathematical Statistics. Academic Press, New York. Dorst, K., Cross, N., 2001. Creativity in the design process: coevolution of problem solution. Design Studies 22 (5), 425–437. Dumas, J.S., Redish, J.C., 1994. A Practical Guide to Usability Testing. Albex, Norwood, NJ. Dunteman, G.H., 1989. Principal Components Analysis. SAGE publication Inc, Newbury Park, CA. Geraldi, P., Kowalski, B.R., 1986. Partial least-squares regression: a tutorial. Analytica Chimica Acta 185, 1–17. Han, S.H., Yun, M.H., Kim, K., Cho, S., 1998. Development of a usability evaluation method. Technical Report (97-G-17-
01-A-15), Ministry of Science and Technology of South Korea. Han, S.H., Yun, M.H, Kim, K., Kwahk, J., 2000. Evaluation of product usability: development and validation of usability dimensions and design elements based on empirical models. International Journal of Industrial Ergonomics 26, 477–488. Han, S.H., Yun, M.H., Kwahk, J., Hong, S.W., 2001. Usability of consumer electronic products. International Journal of Industrial Ergonomics 28 (3–4), 143–151. Ishihara, S., Ishihara, K., Nagamachi, M., Matsubara, Y., 1996. Analysis of Kansei structure on women’s suit design by neural networks. In: Koubek, R., Karwowski, W. (Eds.), Manufacturing Agility and Hybrid Automation-I. IEA Press, Louisville, KY, pp. 85–88. Kim, J., Han, S.H., 2000. A statistical approach to screening product design variables for modeling the product usability. Journal of the Ergonomics Society of Korea 19 (3), 23–37. Kwahk, J., Han, S.H., Yun, M.H., Hong, S.W., Jung, M.K., Lee, K.S., 1997. Selection and classification of the usability attributes for evaluating customer electronic products. Proceedings of the Human Factors and Ergonomics society 41th Annual Meeting, Human Factors and Ergonomics Society, Santa Monica, CA, pp. 432–436. Myers, R.H., 1990. Classical and Modern Regression with Applications. PWS-KENT, Boston, MA. Myers, B.A., Rosson, M.B., 1992. Survey on user interface programming. Proceedings of the CHI Conference on Human Factors and Computing System, SIGCHI, ACM, New York, pp. 195–202. Nagamachi, M., 1995. Kansei engineering: a new ergonomic consumer-oriented technology for product development. International Journal of Industrial Ergonomics 15, 3–11. Nielsen, J., Lavy, J., 1994. Measuring usability: preference vs. performance. Communications of the ACM 37 (4), 66–75. Rubin, J., 1994. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, New York. SAS Institute Inc., 1985. SAS User’s Guide: Statistics. Version 5. SAS Institute Inc., Cary, NC. Umetrics AB, 2001. User’s Guide to SIMCA-P. Version 9.0. Umetrics, Ume(a, Sweden. Williges, R.C., 1981. Development and use of research methodologies for complex system/simulation experiment. In: Morral, M., Kraiss, K. (Eds.), Manned System Design. Plenum, New York, pp. 59–87.
International Journal of Industrial Ergonomics 32 (2003) 189–198
A comparison of screening methods: Selecting important design variables for modeling product usability Sung H. Han*, Jongseo Kim Division of Mechanical and Industrial Engineering, Department of Industrial Engineering, Pohang University of Science and Technology, San 31, Hyoja Dong, 790-784 Pohang, South Korea Received 21 February 2002; received in revised form 16 September 2002; accepted 1 April 2003
Abstract Product usability is affected by a large number of design variables. For example, to design an audio/visual product such as a video-cassette player, the design of controls, information displays, layout of controls, etc. could affect the user task performance. In addition, the product shape, color, material, etc. could also affect the subjective user satisfaction. Altogether, the number of design variables could easily go up to hundreds. To build a model describing the functional relationship between the product usability and the product design variables, it is very important to select only the important design variables for securing modeling efficiency and obtaining an effective model. Although some studies used expert opinions to select them, they have drawbacks such as lack of selection objectivity and expert availability. This study proposes statistical methods for screening important design variables to substitute the expert opinions. Three methods (i.e., principal component regression; cluster analysis; and a partial least squares) are applied to a set of design variables for audio/visual electronic products. Performances of the variable screening methods are examined by building usability models. That is, the variables screened by each method are used to build a usability model. The model performances are then compared to those screened by expert opinions to determine which method provides the best performance. The results show that all these three methods have better model performances than the expert screening in terms of R2 ; the number of variables in the model, and PRESS. Relevance to industry This paper provides the product designers with efficient ways to select important design variables to the product usability such as luxuriousness, salience, rigidity, etc. The guidelines for choosing an appropriate method suggested in this study are expected to help the product designers to build an efficient usability model and as a result, to understand the relationship between the product design variables and the product usability. r 2003 Elsevier Science B.V. All rights reserved. Keywords: Product usability; Variable screening; Principal component regression; Cluster analysis; Partial least squares
1. Introduction
*Corresponding author. Tel.: +82-54-279-2203; fax: +8254-279-2870. E-mail address: [email protected] (S.H. Han).
Product usability is considered one of the most important factors for a product to be successful in the market (Dumas and Redish, 1994). Products with a variety of functions often fail in the market
0169-8141/03/$ - see front matter r 2003 Elsevier Science B.V. All rights reserved. doi:10.1016/S0169-8141(03)00063-5
ARTICLE IN PRESS 190
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
because they have a low level of usability. Now, usability is not only a ‘must’ requirement, but also a business phenomenon in designing a product (Rubin, 1994). To cope with this trend, many manufacturers attempt to develop a product with enhanced usability (Myers and Rosson, 1992). Traditionally, usability has focused more on the user performance than the subjective satisfaction in the human–computer interaction research areas (Nielsen and Lavy, 1994). When it comes to consumer electronic products, however, the subjective satisfaction is as important as the performance. This is mainly because they are not only a tool to accomplish an intended task, but also a decoration in the living room. To reflect this new concept, Han et al. (2000, 2001) defined the product usability for consumer electronic products. They suggested that the usability consist of two aspects: performance and image/impression. The performance was further classified into several dimensions such as memorability, learnability, etc. Likewise, the image/impression was further divided into specific dimensions such as luxuriousness, elegance, etc. It is postulated that the product usability is affected by how a product is designed. If a product design could be decomposed into specific design variables, then it is possible to model the relationship between the product design variables and the product usability. That is, the product usability is a function of design variables as shown in Fig. 1. Then, a natural question arises: ‘‘How do we identify design variables that affect the product usability?’’
Traditional product design process is characterized by a creative leap (Dorst and Cross, 2001). Talented designers, who have insights about what design variables are significant to the user, manipulate them to create a new design. This approach, however, may not be reliable and systematic. The Kansei engineering approach (Nagamachi, 1995; Ishihara et al., 1996) attempts to identify design variables by conducting a user survey. It provides a few design variables that are most likely to affect the user’s impression. On the other hand, the approach suggested by Han et al. (2000) uses a systematic framework for decomposing the product design into specific elements. They assume that the human interface of a product consists of ‘physical’ and ‘logical’ components. These components are then further decomposed into specific design variables by their properties such as individual, interaction, and integration (for detailed descriptions, refer to Han et al., 2000). Common to all these methods is that they require experts (designers or usability engineers) to come up with important design variables. It is obvious that the results are subjective and having lack of validity. That is, it is not guaranteed that the same results are obtained by different groups of experts. Another problem is that the number of design variables can easily go up to tens or hundreds. For example, Han et al. (2000) reported that a total of 88 design variables could affect the usability of typical audio/visual consumer electronic products such as video-cassette players, CD players, audio amplifiers, etc. Of course, some of them would be very important to the usability and
Fig. 1. Concept of the usability evaluation model.
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
some would be negligible. When the number of design variables is large, the modeling process is complex and inefficient. In addition, it is quite common to have a list of design variables that are highly correlated to each other. When the input variables are correlated, building a relationship model is difficult. Worse yet, it is highly likely to end up with an ineffective or invalid model. In order to overcome these drawbacks, this study proposes statistical techniques to screen out unimportant design variables given an initial set of candidate design variables. Three techniques (principal component regression (PCR); cluster analysis; and partial least squares (PLS)) are introduced as the alternatives for substituting expert opinions. A case study is presented to demonstrate the effectiveness of these techniques. In the case study, a set of design variables for audio/visual consumer electronic products is screened by these techniques. The results are then compared to those screened by the expert opinions. 2. Variable screening methods 2.1. PCR screening When variables are correlated, the principal component analysis provides a way of eliminating correlated ones so that only the independent remains (Dunteman, 1989). Principal components are defined as the variables that are mutually independent. Since correlations among the variables are reduced, the number of principal components is usually smaller than that of the
191
variables initially considered. A regression model using the principal components as the independent variables is called PCR. The following is a brief description of the method for screening variables using the PCR. For detailed descriptions of the PCR, refer to Dunteman (1989). Suppose there are p principal components (Ti ) selected from n independent variables (Xj ) as shown in Table 1. Pij represents the loading vector between principal component Ti and independent variable Xj : SSRðti Þ is defined as the variance explained by the principal component Ti when a PCR model is developed. Here, the usability evaluation scores of audio/visual consumer electronic products are used as the dependent variable in the model (detailed descriptions of the dependent variable are presented in the case study section). A large value of SSRðti Þ implies that the corresponding principal component Ti explains a majority of the variance in the PCR model. This means that Ti has great influence on the usability. SSRðt1 ; y; tp Þ is the total variance explained by the principal components. Eq. (1) defines the variance explained by Xj : The ratio of the variance explained by Xj to the total variance is defined as the degree of influence of Xj (see Eq. (2)). The larger the ratio, the greater the influence on the usability: p X SSRðti Þp2ij ; ð1Þ i¼1
Pp
i¼1
SSRðti Þp2ij
ð2Þ
: SSRðt1 ; t2 ; y; tp Þ
Table 1 Mathematical principle of PCR screening PC
T1 T2 y Tp SSRðt1 ; t2 ; y; tp Þ Variable influence
Variables
SSRðt1 Þ SSRðt2 Þ y SSRðtp Þ
X1
X2
y
Xn
P11 P21 y P Pp1p
P12 P22 y Pp2 Pp
y y y y y
P1n P2n y Ppn Pp
y
Pp
i¼1
Pp
SSRðti Þp2i1
SSRðti Þp2i1 SSRðt1 ; t2 ; y; tp Þ i¼1
i¼1
Pp
SSRðti Þp2i2
SSRðti Þp2i2 SSRðt1 ; t2 ; y; tp Þ i¼1
i¼1
SSRðti Þp2in
SSRðti Þp2in SSRðt1 ; t2 ; y; tp Þ i¼1
ARTICLE IN PRESS 192
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
Once the degree of the influence of each variable is determined, selecting important variables is straightforward. The variables with large ratios are selected and those with small ones are eliminated.
using the SCCO as the criterion for selecting the representative can result in selecting a variable with a low SCC. This ratio is defined as the DI. The variable with the maximum SCC and the minimum DI value of each cluster is chosen as the representative variable.
2.2. Cluster screening Cluster analysis is characterized by examining a variety of characteristics of variables and clustering them that share common characteristics (SAS Institute Inc, 1985). The resulting groups are mutually exclusive. A variable from each cluster is selected representing the characteristics of the variables in the cluster. Therefore, the number of the representative variables is smaller than the number of the variables initially considered. The resulting variables can be used to build a usability model. The following is a brief description of the method for screening variables using the cluster analysis. For detailed descriptions of the cluster analysis, refer to Anderberg (1973) and SAS Institute Inc (1985). There are two different criteria for selecting a representative variable from a cluster: squared correlation coefficient (SCC) between the cluster component and the variables in the cluster, and distinction index (DI) between the variable and the other cluster components. The SCC is equal to the R2 of the regression model that uses a variable in a cluster component and the other variables in the cluster as the dependent and independent variables, respectively. The variable providing the maximum SCC becomes the representative of the cluster because it shows the best performance in describing the cluster component. On the other hand, determining a DI is more complex. First, the R2 of a regression model between the variable in one cluster and the other cluster components is calculated. Every variable in the cluster is examined to come up with a variable providing the maximum R2 : The variable with the maximum R2 is defined as the SCC of a variable and the outer cluster (SCCO). A small SCCO implies that the corresponding cluster is clearly separated from the other clusters. Next, both SCC and SCCO values are subtracted from 1 separately and the ratio between them is calculated, because
2.3. PLS screening The PLS method selects only significant variables based on a model that builds the relationship between independent and dependent variables (Geraldi and Kowalski, 1986). The selected variables are expected to explain a majority of the total variance of the model. This method is frequently used in the field of chemical engineering, because there are a large number of independent variables that may affect quality characteristics of chemical products. The mathematical properties of the PLS screening are similar to those of the PCR screening (see Table 2). Suppose there are p latent variables (Li ) selected from n independent variables (Xj ) and usability evaluation score, as shown in Table 2. A regression model based on PLS is built using the Li as the independent variables and the usability evaluation scores as the dependent variable (detailed descriptions of the dependent variable are presented in the case study section). Wij represents the loading vector between the latent variable Li and the independent variable Xj : SSi implies the variance explained by the latent variable Li of the model. A large value of SSi implies that the corresponding latent variable Li explains a majority of the variance in the PLS model. Eq. (3) shows the variance explained by Xj : The ratio of the variance explained by Xj to the total variance implies the relative influence of each independent variable (Xj ) on the total variance (see Eq. (4)): p X
SSi Wij2 ;
ð3Þ
i¼1
Pp
i¼1 P p
SSi Wij2
i¼1
SSi
:
ð4Þ
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
193
Table 2 Mathematical principle of PLS screening Variance explained L1 L2 y Lp
SS1 SS2 y SSp
Variable importance in projection
X1
X2
y
Xn
W11 W21 y Wp1
W12 W22
y y y y
W1n W2n
y
n
n
Wp2
Pp 2 i¼1 SSi Wi1 P p SS i i¼1
If n; the number of the independent variables, is multiplied by the influence of each independent variable, then the average of the influence becomes 1. This transformed value is called variable importance in projection (VIP) (Umetrics AB, 2001). The VIP value of greater than 1 means that the corresponding independent variable has a significant effect on the dependent variable.
3. Case study 3.1. Data collection To examine the performance of the variable screening methods, an existing data set was used. Han et al. (1998, 2000) reported a study that used expert opinions to select important design variables for building usability models. They assumed that the product usability could be determined by the human interface of a product. In their study, a total of 36 different audio/visual consumer electronic products, such as video-cassette players, CD players, audio amplifiers, were analyzed to come up with design variables. The visual display on a CD player and the menu structure for operating it are examples of the design variables. Based on a framework developed by Han et al. (1998, 2000), a typical audio/visual product was decomposed into a total of 88 design variables. Note that 61 out of the 88 design variables were continuous while the remaining were discrete. As stated in the introduction, it was assumed that product usability consisted of the user performance and the image/impression towards a
n
Pp 2 i¼1 SSi Wi2 P p SS i i¼1
Wpn Pp 2 i¼1 SSi Win P p SS i i¼1
product. These two groups were then further classified into detailed dimensions that specified different aspects of each group. The performance was classified into 23 dimensions such as memorability, learnability, etc. Likewise, the image/ impression was classified into 25 dimensions such as luxuriousness, elegance, etc. Once the usability dimensions and design variables of audio/visual products were obtained, experts and product designers selected important design variables for each dimension because all the variables were not equally important to each dimension. A usability evaluation experiment was then conducted to examine the usability of the products. A total of 30 subjects were instructed to evaluate the products. Evaluation scores (using 100 point scales) were obtained for each dimension of each product used in the experiment. A total of 33 usability models based on a multiple linear regression technique were then developed, which related the selected design variables to each usability dimension. For detailed descriptions of the design variables, usability dimensions, and the models, refer to Kwahk et al. (1997) and Han et al. (1998, 2000, 2001). The R2 of the 33 usability models reported by Han et al. (1998, 2000) ranges from 0.70 to 0.99. Since the primary purpose of this study is to compare the performances of the models between the expert opinions and the three statistical screening methods, all the models should be compared regardless of their R2 values. However, comparing all the 33 models using the four methods would be time consuming and require a large amount of efforts. In addition, it is not
ARTICLE IN PRESS 194
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
necessary to compare all the 33 models to get insights into the performance differences among the methods. Thus, this study selected a part of the usability dimensions. The R2 values of the models using the expert opinions were used as a selection criterion. The usability models were classified into three groups: low (R2 values ranging 0.7–0.8), middle (R2 values ranging 0.8–0.9), and high (R2 values ranging 0.9–1.0). Two usability models were arbitrarily chosen from each group, which were subject to comparison in this study. Table 3 presents the definitions of the selected usability models and their R2 values. 3.2. Comparison of screened variables Only continuous variables were subject to the screening process since the PCR, the PLS, and the cluster screening were not able to handle discrete variables. Table 4 presents the screening results for the six usability dimensions. The number of screened variables varied with the screening methods. Generally, the PCR screening selected the largest number of variables on the average, while the PLS screening resulted in the smallest number. Note that the expert screening selected only five variables for the ‘volume’ dimension.
This result may come from the fact that there are not many variables that would affect the dimension or the initial list of the design variables does not contain enough number of variables to explain it. As shown in Table 3, the ‘volume’ model using the variables selected by the experts has a good performance. Thus, it is concluded that there are not many design variables to explain the ‘volume’ dimension. Note that the PLS screening always ends up with a smaller number of selected variables than the PCR across the dimensions. Also note that since the cluster screening does not consider the usability dimensions in selecting important variables, the same set of variables is selected across the dimensions. As stated in the previous section, the PCR and PLS screening methods use a usability model to select important variables. It is interesting to compare the screened variables resulting from these two methods. Table 5 shows the variables that are commonly selected by them. The ‘rigidity’ dimension has the lowest proportion of the commonly selected variables, while the ‘volume’ dimension shares almost all of the selected variables between the two. These two methods share approximately 77% of the variables on the average. This result may imply that the PLS seems to provide a better screening result in terms of the
Table 3 Performance of the usability models using expert opinions (Han et al., 1998, 2000) Usability dimension
Definition
Model R2
Elegance Comfort Volume Salience Rigidity Granularity
Degree to which a product is elegant or graceful Degree to which the user feels easy and comfortable with a product Feeling that a product looks voluminous or slim Degree to which a product is outstanding, prominent, and catching one’s eye Feeling that a product looks stout, stable, and secure Degree to which a product is worked out with great care and in fine detail
0.99 0.99 0.88 0.83 0.72 0.71
Table 4 Number of screened variables Method
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
Mean
Expert screening PCR screening Cluster screening PLS screening
26 26 20 20
5 35 20 20
30 26 20 16
42 28 20 25
18 32 20 20
37 32 20 15
26.3 29.8 20.0 19.3
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
195
Table 5 Variables commonly selected by the PCR and the PLS screening methods
Common variables PLS screening Ratio of common variables (%)
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
11 20 55
18 20 90
11 16 69
22 25 88
16 20 80
12 15 80
Table 6 Proportion of the variables commonly selected by the expert screening and the other screening methods (%) Method
Rigidity
Volume
Comfort
Granularity
Salience
Elegance
Mean
PCR screening PLS screening Cluster screening
38 31 31
100 100 0
50 43 30
40 36 33
39 28 17
57 32 24
54 45 22.5
number of the selected variables. However, the screening performance should be determined by the model performance that uses the screened variables as the independent variables. Table 6 presents the proportion of the variables commonly selected by the expert screening and the other methods. The average proportion is only 35% without considering the ‘volume’ dimension, which means there is a great difference in selecting variables between the expert screening and the other statistical screening methods. Again, it is necessary to build a usability model using the selected variables to compare the screening performances among the methods. The following section describes the procedure of building the usability models. 3.3. Developing usability models In order to compare the performances of the screening methods, a regression model was built for each usability dimension. Here, the screened variables were used as the independent variables while the usability evaluation scores were used as the dependent variable of the model. A secondorder multiple linear regression was used to model the relationship because it was known to be appropriate for modeling the subjective response of human users (Williges, 1981). That is, a main effect and a pure quadratic effect of each variable, and a linear-by-linear interaction of two variables
were included in the model. In building the secondorder model, it was necessary to further screen out unimportant variables so that only significant model items could remain in the model. For this purpose, either the backward elimination or the stepwise regression technique (Myers, 1990) was used depending upon the error degrees of freedom of the model. The resulting model was then subject to the examination of multicollinearity (Myers, 1990) since the model items were expected to be highly correlated to each other. The variance inflation factor (VIF) was used as the criterion of examining the multicollinearity. Model items having the VIF value higher than 10 were eliminated from the model since it was assumed that no significant multicollinearity would exist with the VIF value of less than 10 (Myers, 1990). The resulting model was further elaborated by eliminating a model item with the maximum VIF value one by one. In this process, several alternative models were generated and they were compared to each other in terms of R2 ; PRESS, and the number of model items. A model with the best performance in terms of these three criteria was selected as the final model. In this study, four different models for each usability dimension were built, each using the variables generated by the four different screening methods. More detailed explanations about the model building can be found in Kim and Han (2000) and Han et al. (2000).
ARTICLE IN PRESS 196
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
3.4. Comparison of the usability models 3.4.1. Coefficient of determination All the models show an R2 value of greater than 0.70 (see Fig. 2). All the models using the variables selected by the cluster screening have the greatest R2 (0.96) on the average. Although lower than the cluster, the PLS and the PCR screening have an average R2 value of greater than 0.90, which implies that the variations resulting from the data are explained almost completely. On the other hand, the expert screening shows the lowest average R2 value (0.86), although it is still considered a good model. Note that the expert screening has a large variation in R2 depending upon the usability dimensions, while the other methods show relatively a uniform distribution of the R2 values. As far as the R2 value is concerned, the statistical screening methods seem to provide a better performance than the expert screening. 3.4.2. Number of model items The number of model items in a model has an important practical implication. When a model has a large number of model items, it is difficult to interpret the effect of each item although it has a high value of R2 : When it comes to interactions, it is even more difficult, if not impossible, to interpret the result. Fig. 3 presents the number of model items for all four models. Both the PLS and the PCR screening have relatively a smaller number of items (6.7 and 8.8 items, respectively) on the average than the expert and the cluster screening (14.2 and 12.8 items, respectively). This result may stem from the fact that the PLS and the PCR used the dependent variable in the screening process, while the other two did not. Together with the result of R2 ; the models using the PCR and the PLS screening seem to explain the variation of the usability evaluation scores with a fewer number of variables. 3.4.3. PRESS PRESS is an index for estimating the prediction capability and stability of a regression model (Myers, 1990). A model with a low PRESS is considered more stable and reliable. Fig. 4 shows the PRESS of the models. The expert screening has
Fig. 2. R2 of the usability models. Average R2 : expert screening (0.86), PCR screening (0.91), PLS screening (0.90), cluster screening (0.96).
Fig. 3. The number of model items in the usability models. Average: expert screening (14.2), PCR screening (8.8), PLS screening (6.7), cluster screening (12.8).
the highest PRESS on the average. The cluster and the PLS screening have a relatively low value of PRESS than the other two. Note that the PRESS of the models using the expert screening (average value of 1706.4) is four times as large as the models using the other statistical screening methods (average value of 396.5). In addition, the PLS and the cluster screening show a lower variation in PRESS than the others. It can be concluded that the PLS and the cluster screening seem to provide a better prediction capability and stability. 3.4.4. Discussion The models using the expert screening have an R2 value of 0.86 and approximately 14 model items on the average. Note that the ‘comfort’ and ‘elegance’ models result in an R2 value of almost
ARTICLE IN PRESS S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
197
Table 7 Variable screening methods appropriate to the purpose of modeling Purpose of modeling
Recommended screening methods
Accurate prediction
PCR screening, cluster screening, PLS screening Expert screening
Easy interpretation of model items Stable prediction Fig. 4. PRESS of the usability evaluation models. Average PRESS: expert screening (1706), PCR screening (518), PLS screening (353), cluster screening (319).
1.0, which implies that the models explain almost all of the variations of the usability evaluation scores. However, they may not be practical, because they have too many model items (18 and 16 items, respectively). The large number of model items makes it difficult to interpret the models. The expert screening seems to be a relatively poor method than the others since it shows the poorest performance in terms of the three criteria. The R2 of the models using the PCR and PLS screening have an average value of 0.91 and 0.90 with approximately 9 and 7 model items, respectively. In addition, they show a good model stability and reliability. The models using the cluster screening show the highest value of R2 (0.96) on the average and the lowest value of PRESS. However, they have too many model items (13 items on the average) and as a result, they are expected to be difficult to interpret. In sum, the models using the statistical screening methods show reliable and excellent model performances with a relatively small number of model items. However, they may have a drawback, too. For example, some variables included in the models are not always easy to interpret. This is because the variables were selected based only on the distributional characteristics of the product design variables. In other words, variables having a high variance had a more chance to be selected as important ones. When combined with high correlations among the variables, nonsense variables could be included in the models. The expert screening, on the other hand, does not have this
Small number of model items
Cluster screening, PLS screening PCR screening, PLS screening
drawback since it selected variables based on the know-how of the experts. That is, the model does not include nonsense variables, which makes the interpretation of their effects relatively easier. In consequence, all the four screening methods could be applied depending upon the purpose of the modeling (see Table 7). The PCR, the cluster, and the PLS screening are recommended to build a highly accurate model. To add a high level of reliability and stability, either the PLS or the cluster screening is recommended. In addition, either the PLS or the PCR screening is recommended to build a model with a relatively smaller number of items. On the other hand, the expert screening is useful for building a model that does not include nonsense variables.
4. Conclusion This study compared four different methods of screening important design variables for building usability models. Given a usability model developed by the multiple linear regression technique, the comparison was made on the basis of the model performances such as the model R2 ; PRESS, and the number of model items. The expert opinions appeared to provide a usability model that did not contain nonsense design variables. The PCR and PLS screening methods turned out to provide a model with a reasonable number of model items and high R2 : The cluster screening showed a good performance in building an accurate and reliable model. This result implies that no method was superior in all the three
ARTICLE IN PRESS 198
S.H. Han, J. Kim / International Journal of Industrial Ergonomics 32 (2003) 189–198
criteria. That is, any method could be applied depending upon the purpose of the modeling. Guidelines were provided for choosing an appropriate screening method. Obtaining design variables affecting the product usability is a difficult task because there is no concrete relationship between the variables and the usability. It would be a usual practice that the designers end up with manipulating a few variables to create a new design. This study provides a tool for screening important design variables in a systematic manner. Any variables that are expected to affect the usability (regardless of their magnitude of influence) can be included in an initial list of potential variables, and then the methods suggested in this study can be applied to come up with important variables. This study is an initial attempt to provide a systematic way of modeling the product usability using product design variables. It is necessary to have more case studies to demonstrate the effectiveness of this approach.
References Anderberg, M.R., 1973. Cluster Analysis for Applications: Probability and Mathematical Statistics. Academic Press, New York. Dorst, K., Cross, N., 2001. Creativity in the design process: coevolution of problem solution. Design Studies 22 (5), 425–437. Dumas, J.S., Redish, J.C., 1994. A Practical Guide to Usability Testing. Albex, Norwood, NJ. Dunteman, G.H., 1989. Principal Components Analysis. SAGE publication Inc, Newbury Park, CA. Geraldi, P., Kowalski, B.R., 1986. Partial least-squares regression: a tutorial. Analytica Chimica Acta 185, 1–17. Han, S.H., Yun, M.H., Kim, K., Cho, S., 1998. Development of a usability evaluation method. Technical Report (97-G-17-
01-A-15), Ministry of Science and Technology of South Korea. Han, S.H., Yun, M.H, Kim, K., Kwahk, J., 2000. Evaluation of product usability: development and validation of usability dimensions and design elements based on empirical models. International Journal of Industrial Ergonomics 26, 477–488. Han, S.H., Yun, M.H., Kwahk, J., Hong, S.W., 2001. Usability of consumer electronic products. International Journal of Industrial Ergonomics 28 (3–4), 143–151. Ishihara, S., Ishihara, K., Nagamachi, M., Matsubara, Y., 1996. Analysis of Kansei structure on women’s suit design by neural networks. In: Koubek, R., Karwowski, W. (Eds.), Manufacturing Agility and Hybrid Automation-I. IEA Press, Louisville, KY, pp. 85–88. Kim, J., Han, S.H., 2000. A statistical approach to screening product design variables for modeling the product usability. Journal of the Ergonomics Society of Korea 19 (3), 23–37. Kwahk, J., Han, S.H., Yun, M.H., Hong, S.W., Jung, M.K., Lee, K.S., 1997. Selection and classification of the usability attributes for evaluating customer electronic products. Proceedings of the Human Factors and Ergonomics society 41th Annual Meeting, Human Factors and Ergonomics Society, Santa Monica, CA, pp. 432–436. Myers, R.H., 1990. Classical and Modern Regression with Applications. PWS-KENT, Boston, MA. Myers, B.A., Rosson, M.B., 1992. Survey on user interface programming. Proceedings of the CHI Conference on Human Factors and Computing System, SIGCHI, ACM, New York, pp. 195–202. Nagamachi, M., 1995. Kansei engineering: a new ergonomic consumer-oriented technology for product development. International Journal of Industrial Ergonomics 15, 3–11. Nielsen, J., Lavy, J., 1994. Measuring usability: preference vs. performance. Communications of the ACM 37 (4), 66–75. Rubin, J., 1994. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, New York. SAS Institute Inc., 1985. SAS User’s Guide: Statistics. Version 5. SAS Institute Inc., Cary, NC. Umetrics AB, 2001. User’s Guide to SIMCA-P. Version 9.0. Umetrics, Ume(a, Sweden. Williges, R.C., 1981. Development and use of research methodologies for complex system/simulation experiment. In: Morral, M., Kraiss, K. (Eds.), Manned System Design. Plenum, New York, pp. 59–87.