- Email: [email protected]
A fear detection method based on palpebral fissure
Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Contents lists available at ScienceDirect
Journal of King Saud University – Computer and Information Sciences journal homepage: www.sciencedirect.com
A fear detection method based on palpebral fissure Rawinan Praditsangthong, Bhattarasiri Slakkham, Pattarasinee Bhattarakosol ⇑ Department of Mathematics and Computer Science, Chulalongkorn University, Thailand
a r t i c l e
i n f o
Article history: Received 15 December 2018 Revised 30 April 2019 Accepted 1 June 2019 Available online xxxx Keywords: Interpalpebral fissure Palpebral fissure region Eye Emotion Decision tree Classification
a b s t r a c t Human emotions, such as smiling or laughing, can be expressed in various forms through the face whenever there are stimuli. These changing faces can reflect the emotional states that are used to identify a normal or an abnormal behaviour. This research aims to study the patterns in human faces and identify the areas of interest (AOI), which is called Facial Landmark Detection (FLD). The investigation of the external elements of eyes is performed, and it consists of the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). These elements are applied to classify the emotional differences between neutral and fearful emotions. A method for emotional classification was designed according to the changing values of the IPF, PFL, and PFR. An ID3 algorithm was used to classify the emotions. Three hundred sixty images were derived from horror-thriller-murder movies based on IMDb. This data set was utilized to generate the proposed pattern. This pattern was used to classify the emotions using a decision tree technique that led to the development of an emotional classification model. The accuracy of the emotional classification model between neutral and fearful emotions was 92.50%, thus proving that the proposed model is efficient. Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
1. Introduction The expression of human emotion consists of positive emotions and negative emotions, such as happiness, sadness, pride, anger, and fear. Moreover, most psychologists (Manssuer et al., 2016; Eva et al., 2016; Trojano et al., 2012; Corey and Komogortsev, 2011) believe that human actions are defined by the rudimentary emotions. Generally, a person can express emotions through their eyes and various actions, such as gazes, fixations, saccades, pupil locations, pupil dilations, or blinking. These actions can be used to classify human emotions. Various approaches, such as the Haar cascade, convolutional neural network (CNN), corneal reflection, projection function, or edge analysis (EA) (Soleymani et al., 2016; Khan et al., 2016; Almudhahka et al., 2016), can be applied to detect and classify human emotions. In addition, some studies use a low cost high-performance camera to detect the eye move⇑ Corresponding author. E-mail addresses: [email protected] (R. Praditsangthong), [email protected] (B. Slakkham), [email protected] (P. Bhattarakosol).
ments (Fuhl et al., 2016; Polatsek and ‘‘Eye, 2013; Papernot et al., 2016; Devahasdin Na Ayudhya and Srinark, 2009; LopezBasterretxea et al., 2015). Nevertheless, there are some limitations. For example, when the front camera of tablet is used to detect the pupil dilation or pupil size, the distance from the tablet is only 30 cm (Polatsek and ‘‘Eye, 2013; Huabiao et al., 2013; Patnik et al., 2017). One interesting region of the eyes is the palpebral fissure (PF), which is the elliptical space between the eye lids (Lu et al., 2016). Emotional changes can affect to the distance between the top edge and the bottom edge of the eye lids, which is called the interpalpebral fissure (IPF). These changes can be used to detect human emotions. Fig. 1 shows the emotional flow where the causes of each emotion are important subjects since there is no emotion, without the cause. Additionally, positive stimuli result in positive emotions, but negative stimuli result in negative emotions. This research focus on an emotional detection using the palpebral fissure. The objective of the study was to investigate the interpalpebral fissure and the palpebral fissure regions in order to analyse and classify neutral and fearful emotions.
Peer review under responsibility of King Saud University.
2. Related work
Production and hosting by Elsevier
Human emotions refer to the immediate expression of feelings in a situation with stimulus (Izard, 2013). Human emotions can be
https://doi.org/10.1016/j.jksuci.2019.06.001 1319-1578/Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
2
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 1. The Emotional Flow.
expressed in various ways, but most of these expressions are reflected in the face; these are called facial expressions. A facial expression refers to changes in the eyebrows, eyes, nose, and mouth of the person who received the stimuli. This countenance is always followed by some action, such as self defence, unconsciousness, or screaming. However, under the same situation, different people will react differently. Therefore, detecting emotions should not be performed by detecting human reactions but emotions should be detected from the countenance because the beginnings of those actions come from the same issue. As mentioned above, emotions should be detected using facial expressions, as shown in Fig. 2. According to Fig. 2, it is clear that shapes of the eyebrows, eyes, nose, and mouth of the same person are dissimilar under different situations. Thus, (Izard, 2013) stated that the expression of emotions can be defined in two forms: emotional states and emotional traits. An emotional state refers to a particular emotion in a certain period of time, such as the happiness state, angry state, or fearful state, and these states may last from seconds to hours, depending on the situation. In term of emotional traits, these refer to the frequent trend of an emotional occurrence in day-to-day life (for example, the happiness trait, angry trait, or fearful trait). Moreover, the emotional states can be classified into two categories: positive
emotions and negative emotions. Positive emotions include happiness, surprise, and pride, while sadness, anxiousness, anger and fear can be classified as the negative emotions. In addition, the facial recognition of emotional states for emotional interpretation has been applied to studies in the life sciences area. For example, research has compared the feelings of apes and humans using the facial expressions of both species (Parr and Waller, 2006). Since there are various objects on a face, such as eyebrows, eyes, the nose, and the mouth, the changes in these objects can be used as a communication signal, such as pain, hunger, and fear. Therefore, many researchers have studied emotion recognition and detection using these changes. For examples, the eye movements have been studied under various objectives (Ciesla and Koziol, 2012; Schurgin et al., 2014; Peng et al., 2005). To classify the emotional states, there are processes that need to be performed, as shown in Fig. 3.
2.1. Process-1 The detection of a person’s sentiment over a face was separated into five regions by Schurgin et al. (2014), which are the eyes, upper nose, lower nose, upper lip, and nasion, as shown in Fig. 4. Additionally, the results from Schurgin et al. (2014) showed that the eyes are the most significant organ for expressing feelings as reflected in the quote ‘‘the eyes are the window of the soul”.
2.2. Process-2
Fig. 2. Faces with different emotions.
Even though several techniques have been implemented, there are two mechanisms that are well-known among researchers because of their accuracy. These mechanisms are the Hough transform mechanism (Huabiao et al., 2013) and the Haar cascade object detector (Asier et al., 2015). The Hough transform mechanism and the Haar cascade object detector are applied to detect the pupils in order to interpret the meaning of the countenance. The main concept of the Hough transform mechanism is based on the dilation of the pupils. Lines and points are applied to distinguish the changes in the pupils from normal state to a specific state. The experiment was performed on every sentiment: sadness, anger, happiness, pride, and fear. In the Haar cascade object detector mechanism (Asier et al., 2015), an image will be defined as a two-dimensional table of a matrix. Each dimension contains the sum of all the pixels related to the position. Then, the values in the table is used to mark the points in the pupils, the positions of the pupil centres are defined and the pupil size is calculated. This experiment was performed under two states: the normal state and specific states. The specific states are the same as Hough transform mechanism, including disgust. The results showed that the pupil sizes from normal state and all specific states are not equivalent. As a consequence, Haar cascade object detector mechanism can be applied to indicate the emotions of people in different situations.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
3
Fig. 3. Emotional classification processes.
2.3. Process-3 The next step is the pattern classification of the emotional detection using the defined techniques. Popular classification methods are the Support Vector Machine (SVM), the Bayesian classifier, and the Extreme Learning Machine (ELM) (Lu et al., 2016; Ryu et al., 2013; Peng et al., 2005). Fig. 5 shows results of these techniques. The Support Vector Machine (SVM) is one type of machine learning that is used to classify data groups in a linear model using planes or hyperplanes in order to find the best linear classifier model. However, the obtained data are always fitted in a nonlinear form rather than linear form; therefore, this problem is solved using a kernel function. This function separates the data into different groups according to their features. The differences among groups are classified using the distances between groups. On the other hand, the Bayesian classifier can also be applied to organize data into various groups based on their features’ probabilities. Some research (Manssuer et al., 2016; Eva et al., 2016; Schurgin et al., 2014) used this technique to recognize the expression of six basic emotions; anger, disgust, fear, happiness, sadness, and surprise. The Bayesian approach refers to the role of a class to predict the values of the features for members of the class. The learning of the Bayesian classifier constructs a probability model of the features and uses this model to predict of the classification.
Fig. 4. Five regions for the facial expression detection algorithm.
An additional interesting machine learning technique is the Extreme Learning Machine (ELM), which is a single-hidden layer feed forward neural networks (SLFNs) (Lu et al., 2016). The ELM consists of an input layer, a hidden layer, and an output layer. In the input layer, the data randomly weighted, including the bias values. These weighted values are obtained from the calculation of Moore-Penrose inverse matrix. The input layer randomly identifies weight values, including the bias value of hidden layer. Thus, this approach focuses on preparing data appropriately before they are input into the system. Referring to Fig. 5, in the Emotional classification processes, these techniques have been implemented by various researchers in studies using the pupil diameter, pupil location, pupil size, gaze distance, fixation, saccade, eye blinking, or eye movements to classify emotional states (Mathews, 2003; Soleymani et al., 2012; Li and Mao, 2012). Some studies measure the smallest and the largest pupil sizes, some measure the speed of a personal’s eye blinking, and some measure the gaze time. Nevertheless, these measurements need to be performed using specific equipment in order to detect the required values. Unfortunately, this specific equipment is very costly and not portable. Consider the eye movement detection mechanism. This mechanism needs to determine the defined area of interest (AOI) before the eye tracking equipment can find the fixation orders, fixation counts, fixation durations, and scan paths (Lu et al., 2016). Once the area of interest is identified, called the Facial Landmark Detection (FLD) or localization is performed to find its accurate location on the face. This process is responsible for locating the eyebrows, centres of the eye, nose, upper lip, lower lip, or centres of the mouth (Feng, 2015). Consequently, the automatic facial landmark detection process can be implemented in order to observe and monitor the emotion expressions. Additionally, if the pupil locations and their radii are needed, a high resolution and high definition camera must be used (Ciesla and Koziol, 2012); which increases for users, and researchers with limited funds cannot employ this method. As mentioned above, the characteristics of eyes can be used by people to express their feelings. These characteristics include the pupil diameters, pupil locations, pupil sizes, eye blinking, and eye movements. One similarity in these characteristics is that the measurement is performed inside the eyeball, and none of the existing studies use the rim of eyes to detect emotions.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
4
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 5. Classification.
3. Materials and methods
quality of life, especially paralyzed patients, will be increase because the PMS can continuously monitor patients.
3.1. Problem statement and solution 3.2. Proposed method To increase the quality of life of patient, a patient monitoring system must be provided as soon as their life is in danger. An alarm button is installed for people to press when their lives are in danger or they need some helps. Unfortunately, an alarm button cannot be pressed in all situations, such as for paralyzed patients. Therefore, IP cameras or closed-circuit television (CCTV) can be implemented to capture the events in every significant area, and monitors must be used in a control room to observe these areas. Therefore, this solution is costly and not suitable for a small organization or a Healthcare Centre. Thus, a solution that uses only cameras and an automatic alarm is proposed. Moreover, the implemented cameras in this system are inexpensive, small and lightweight with moderate resolution. As a result, it is suitable for capturing only a small and specific area, such as the eyes, nose, or mouth. Therefore, researchers use only the eyes of a person as the focal area to identify critical situations instead of the entire face or other body movements. The captured images will be sent as a stream to be interpreted as normal or abnormal states by an image classification system on a personal computer, as shown in Fig. 6. If the defined state is abnormal, the alarm system will be automatically activated. Based on Fig. 6, the patient monitoring system (PMS) consists of a camera, a computer and an alarm system. Thus, a significant part in this system is the emotional classification model that must be installed in the patient monitoring system. Therefore, this paper proposes an emotional classification model (ECM) that can classify the emotions of patients as either neutral or fearful. Therefore, the
Eyes are composed of internal elements and external elements. These elements require different equipment to measure their changes. Most researchers have considered the internal elements, such as the iris, pupil, and lens. Unlike other researchers, this research will focus on the movements of the external elements of eyes, which are the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). As mentioned in above, this research will focus on the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR), as shown in Fig. 7(a). From Fig. 7 (a), the vertical distance between the palpebrale superius (PS) point and the palpebrale inferius (PI) point is the IPF while the horizontal distance between the endocanthion (EN) and the exocanthion (EX) is the PFL. Moreover, the IPF and PFL are applied to calculate the PFR that is used to identify the emotional differences, and, the formula for the PFR is presented in Fig. 7(b). Hence, this research aims to implement an emotional classification system between the neutral emotion and the fearful emotion based on the ID3 Decision Tree algorithm using the IPF and the PFR. Fig. 8 shows a diagram of the proposed method. This method starts from selecting an image from a horror-thriller-murder movie. The movies are collected from IMDb website, and they are ranked by experts and leading critics. The collected movies must have high rankings, be in high definition (HD) and have quality actors/actresses. These movies are expected to express fear. Thus, all fearful images can be captured from these movies using
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
5
Fig. 6. Infrastructure of the proposed patient monitoring system (PMS).
Fig. 7. The external elements and the PFR calculation.
FFmpeg, which is multimedia framework and a cross-platform solution to record, convert, and stream audio and video. (See Fig. 9). After the fearful images are captured, these images are inserted into the developed application to detect and extract the responding faces. The application was developed using Qt creator editor version 5 based on OS X 10.11 EI Capitan with the C++ language. In addition, an open source library, called the DLIB library, was embedded into the Qt creator. When we consider an image, one image normally consists of one frontal face, background, and other objects. Among these components, the most important object is the frontal face, and this will be captured using the DLIB library. However, the size of this captured face has no standard size or is the real size of a human face, which is defined in (Poston, 2000). Therefore, the size of the frontal face is adjusted to be the real size and this resized face has 68 facial mark-points added on the face. As a result, the new facial image is normalized and adjusted to 800x800 pixels. After marking the face with 68 mark-points, the
PI, PS, EX and EN points are defined. Then, the IPF and the PFL can be calculated. Finally, the neutral or fearful emotion can be identified according to the value of the PFR. 3.3. Data collection and processing As mentioned previously, all images are captured from horrorthriller-murder movies and the frontal face is the only object to be used. The selected number of movies is 33. For each movie, there are only 2 states to be captured from an actor: the normal state face and the fearful state face. In addition, more than one actor from each movie are used in this research and no actor and actress are used in multiple movies. Therefore, there are 60 actors in the age range of 18–52 years that are used as the samples in this experiment. For each actor, three scenes with a neutral emotion and three scenes with a fearful emotion from the movies are collected. Therefore, total number of images used in this experiment is 360. The duration of data collection is two months.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
6
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
3.4. Data analysis processing The 360 images were divided into two data sets: the training data set and the testing data set. The testing data set is used for the classification and analysis processes. The number of actors in the training data set is 40 while the number of actors in the testing data set is 20. Moreover, all images were calculated to determine the differences between the two emotions using the IPF, the PFL, and the PFR. After completing the data capturing process, 240 images from the training data set were retrieved from the database to determine means of the IPF, the PFL, and the PFR. Within the training data set, the 40 actors are 47.5% male and 52.5% female, and the highest age range is between 18 and 24 years old, which encompasses 32.5% of the actors. The statistical test is the paired samples t-test, which is used to determine the differences of mean values of the neutral emotion and the fearful emotion for each indicator. This test uses a 95% confidence level or the 0.05 significance level (a). The results of the test can be summarized as follows. 3.4.1. Test of the differences in mean IPF There is a significant difference between the IPF means of the right eyes under neutral and fearful emotions with a pvalue = 0.003 < 0.05 = a. Similar to the right eyes, the left eyes also have significant mean differences under the two focusing emotions with a p-value = 0.001 < 0.05 = a. Moreover, the values of the IPF from both eyes in the neutral situation are smaller than the values of the IPF in the fearful situation. Furthermore, the change of the mean IPF in males is higher than the change of the mean IPF in females. Table 1 presents the mean IPFs with the standard deviations of both eyes in the samples. 3.4.2. Test of the differences in mean PFL The results of sample t-tests indicate that only the right eye can reflect the emotion of fear because the values of the mean PFLs of the right eyes under neutral and fearful emotions are significant different with a p-value = 0.046 < 0.05 = a. The mean PFLs with the standard deviations are given in Table 2.
Fig. 8. Diagram of the application.
3.4.3. Test of the differences in mean PFR The testing results of the mean PFRs are the same as the results obtained for the mean IPFs. Moreover, the mean PFRs in the neutral state are smaller than the mean PFRs in the fearful state. Additionally, the changes of males’ eyes are larger than the changes of females’ eyes. Table 3 shows the mean PFRs with the standard deviations.
Table 1 The mean IPFs with the standard deviations (mm). Gender
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
8.83 (1.22) 10.21 (0.99) 9.52
10.45 (1.24) 11.35 (1.05) 10.90
8.76 (1.24) 10.30 (1.09) 9.53
10.50 (1.33) 11.34 (1.16) 10.92
Table 2 The mean PFLs with the standard deviations (mm). Gender
Fig. 9. Relation among Gender, Emotion, IPF, PFL, and PFR.
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
29.80 (1.88) 30.98 (2.26) 30.39
30.50 (2.27) 30.90 (1.53) 30.70
30.75 (2.48) 31.21 (1.18) 30.98
30.37 (1.54) 30.16 (2.21) 30.27
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx Table 3 The mean PFRs with the standard deviations (mm). Gender
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
207.06 (33.79) 248.84 (33.12) 227.95
251.39 (41.85) 276.12 (34.62) 263.76
212.17 (37.95) 252.64 (30.47) 232.41
250.80 (37.01) 269.68 (41.65) 260.24
According to three simple t-tests, it can be summarized that the eyes in the fearful state are larger or wider than the eyes in normal state. In addition, males’ eyes usually change more than females’ eyes during fear. Therefore, it could be said that gender and emotion are related to the changes of the mean values of the IPF, the PFL, and the PFR. Thus, when the mean value of any indicator is larger than normal, it is possible that the person is in a scared state. The relationship among these three parameters can be drawn as in Fig. 5. 3.4.4. Choosing the parameters and creating the model 3.4.4.1. Choosing the parameters. According to the above conclusions, there are at least three important parameters that can reflect human emotions. These parameters are gender, the IPF, and the PFR. However, the IPF and PFR may relate to other parameters, such as age and eye-side. Therefore, in order to obtain a precise model to determine the emotions, all reasonable parameters should be considered. In this case, the independent parameters to be considered are gender, age, eye-side, the IPF, and the PFR. The dependent parameter of the model is the emotion. Table 4 presents possible values of all parameters in the emotional model. Furthermore, Fig. 10 shows the possible relationships
Table 4 Values of the variables in this experiment. Independent Variable
Possible values
gender age-range eye’s location IPF PFL PFR Dependent Variable status of emotion
Male, Female 18–24, 25–31, 32–38, 39–45, and greater than 45 Right, Left 1; 1 1; 1 1; 1 Possible values Neutral, Fear
7
among these parameters, where the brown dashed line represents the expected effect of the factor towards an emotion. In addition, the relationship between gender and emotion and the relationship between age and emotion can be determined by calculating their correlations. The next step is to determine relation between age and emotion. Unfortunately, there is no existing link from gender to emotion, or vice versa. Since the distributions of emotions under each independent variable defined above are not normal, a nonparametric method, Kendall’s rank correlation coefficient test, is applied using the 95% confident level. The test also concerns the effects from the interactions among independent variables and the effects from the individual variable. The results from the test indicated that there are some relationships among these parameters and the relationship diagram can be illustrated in Fig. 11. According to the relationship diagram in Fig. 11, there is only two direct links to emotion. One link is the correlation between change of the IPF/PFR and emotion, and the second link is the correlation between the emotion and the interaction of gender and age. However, there are direct correlations between some interactions with gender, eye-side, or age. Therefore, it can imply that there are some hidden correlations between gender and emotion, eye-side and emotion, and age and emotion by passing through the existing correlations. Thus, it is a fact to say that every independent parameter is related to emotion. 3.4.4.2. Creating model. Based on the conclusion from the previous section, all parameters were compiled using a data science software tool called RapidMiner Studio version 8.1. The changed pattern from the parameters was customized to a general model as an emotional classification model with a many-to-one relationship among the variables, as shown in Eq. (1).
Emotion ¼ M ðgender; age; eye side; IPF; PFRÞ
ð1Þ
Eq. (1) refers to ‘‘Emotion is the model of gender, age, eye-side, IPF, and PFR with possible interactions”, and the possible values of all parameters are defined in Table 4. Since the emotional classification model is many-to-one relation and the solution of the Emotional model can be either neutral or fearful, a decision tree can be implemented to express all possible paths for an emotion’s derivation. In this research, the ID3 decision tree algorithm is used to create a tree-like graph or model. Thus, the data set was imported into RapidMiner version 7.4 to create a classification model of the emotional expressions from the value of the target attribute, which is called data labelling. Since there are 40 training data sets, the attributes of the training data set consist of the results of the calculation the of the average IPF and PFR for both the right eye and the left eye. Furthermore, the structure of the ID3 decision tree algorithm is shown in Fig. 12. 4. Results To find the values of all IPFs and PFRs for both eyes, the 68 facial landmarks must be set, since the emotional classification system relies on these landmarks when identifying an emotion from an image. There are 3 steps of this classification system.
Fig. 10. Expected relationships among the independent and dependent variables.
Step 1: chip size of the images After the images are input into the system, their sizes are adjusted to 800 800 pixels, which is similar to the real human face. Step 2: 68 facial landmarks’ detection The frontal facial images led to identify 68 points using the DLIB library. These 68 points are used to locate the facial landmarks.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
8
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 11. Relationships among all variables.
Fig. 12. ID3 Decision Tree Algorithm of Emotional Classification.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Step 3: predict the emotion The IPF, and the PFR are used to classify and predict neutral emotions and fearful emotions. The test data set contains 20 records or 120 sub-images. This data set is input into the emotional classification model that can be represented as a directed graph, as in Fig. 12. The accuracy of this classification process is dependent on 4 values: the true positives (TPs), the true negatives (TNs), the false positives (FPs), and the false negatives (FNs). These values are used to find the Precision and Recall as follows.
Precision ¼
Recall ¼
TP TP þ FP
TP TP þ FN
Accuracy ¼
TP þ TN TP þ TN þ FP þ FN
9
6. Conclusions The aim of this research is to study the patterns of emotion expressions on human faces by focusing on the external elements of eyes. The external elements of eyes consist of the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). The changes in the IPF, PFL, and PFR were compiled with RapidMiner Studio version 8.1. These values are used to classify the emotional differences between the neutral emotion and the fearful emotion using a decision tree algorithm, ID3. Thus, the experimental result of the patterns led us to develop an emotional classification model. The accuracy of the emotional classification model was 92.50% for classifying neutral emotions and the fearful emotions. Furthermore, this research is published and granted by the Department of Intellectual Property (Bhattarakosol et al., 2018). The future work for this research is to find other features on the face that can be combined with the PF to clearly define the emotions of faces. This work will allow these methods to be applied to many areas to increase wellbeing, such as the medical and healthcare field. Declaration of Competing Interest
From the testing result, TP is 17, FP is 0, FN is 3, and TN is 20. Consequently, Precision value is 100%, which means that the probability that the model can present the right answer is 1. Additionally, the Recall value is 85%, which means that the probability that the solutions that is presented when the model is used is right is 0.85. Finally, the Accuracy of this model is 92.50%, which means that there is a small chance that this model can predict the wrong emotion.
Acknowledgement
5. Discussion
References
Once people were confronted by stimuli, their faces usually change to express their emotions. This change can be seen in the eyebrows, eyes, nose, and lips. Among these components, the eyes are the most interesting feature because it can clearly express the emotion. Thus, several researchers have used eyes as their research focus in many areas, such as security, telecommunication, and healthcare (Trojano et al., 2012). The anatomy of the eye includes the iris, pupil, lens, cornea, and retina (Doganay et al., 2017). Since these elements are very small, specific instruments are required. Therefore, the emotional interpretation using these components is very costly and the captured image may not be applicable as is needed due to the poor equipment. Nevertheless, there is a feature that is larger than those five elements and is easy to detect using any simple camera, but is infrequently addressed by researchers. This element is called palpebral fissure (PF). This PF is not a part of the eye’s anatomy, but it is the space between upper and lower eye lids. This research uses the PF as an important factor to reflect the emotions because the PF can be measured using the IPF and the PFR. The changes in the IPF and the PFR, along with gender, age, and eye-side, can be used to create an emotional classification model that can reflect either neutral or fearful emotions. The model is created based on the results from the correlation analysis process under a 95% confidence level. Moreover, the statistical testing has determined that the sizes of eyes in the neutral state are smaller than the sizes of eyes in the fearful state. Additionally, men have larger PFs than women when they are afraid. Though the model can distinguish between neutral and fearful emotions, it cannot find the differences between fear and anger. This finding is because when people lose their temper, their eyes also open wider than normal, which is the same as when they are fearful. In contrast, this emotional classification model would be able to identify happiness and sadness according to the size of the PF which would be smaller than normal.
Almudhahka, N., Nixon, M., Hare, J., 2016. Human face identification via comparative soft biometrics. In: IEEE international Conference on Identity, Security and Behavior Analysis (ISBA). Asier, L.B., Mendez-Zorrilla, A., Garcia-Zapirain, B., 2015. Eye/head tracking technology to improve HCI with iPad applications. Sensors 15 (2), 2244–2264. Bhattarakosol, P, Praditsangthong, R., Slakkham, B., Feb. 15, 2018. An Automatic Patient’s Face Aided Detection System (APFADS). Department of Intellectual Property. TH Patent 1801000895. Ciesla, M., Koziol, P., 2012. Eye pupil location using webcam. CoRR 1202, 6517. Corey, H., Komogortsev, O., 2011. Biometric identification via eye movement scanpaths in reading. In: IEEE International Joint Conference on Biometrics (IJCB). p. 1. Devahasdin Na Ayudhya, C., Srinark, T., 2009. A method for real-time eye blink detection and its application. In: 6th International Joint Conference on Computer Science and Software Engineering (JCSSE). Doganay, F., Dadcai, Z., Topcu-Yilmaz, P., Doganay Aydin, H., 2017. The association between ocular dominance and physiological palpebral fissure asymmetry. J. Laterality: Asymmetries Body Brain Cognit. 22, 412–418. Eva, P., Tobias, B., Sylvain, D., David, S., 2016. Attentional bias for positive emotional stimuli: a meta-analytic investigation. Psychol. Bull. 142, 79–106. Feng, Zhen-Hua et al., 2015. Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting. IEEE Trans. Image Process. 24 (11), 3425–3440. Fuhl, W., Santini, T., Kasneci, G., Kasneci, E., 2016. PupilNet: convolutional neural networks for robust pupil detection. Comput. Vision Pattern Recognit. 1602. Huabiao, Q., Wang, X., Liang, M., Yan, W., 2013. A novel pupil detection algorithm for infrared eye image. In: IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC). p. 1. Indumathy, N., Patil, K., 2014. Medical alert system for remote health monitoring using sensors and cloud computing. Int. J. Res. Eng. Technol. 3, 884–888. Izard, Carroll E., 2013. Human Emotions. Springer Science & Business Media. Khan, S.A., Hussain, A., Usman, M., 2016. Facial expression recognition on real world face images using intelligent techniques: a survey. Int. J. Light Elect. Opt. 127, 6195–6203. Li, Zheng, Mao, Xia, 2012. Emotional eye movement generation based on geneva emotion wheel for virtual agents. J. Visual Lang. Comput. 23 (5), 299–310. Lopez-Basterretxea, A., Mendez-Zorrilla, A., Garcia-Zapirain, B., 2015. Eye/head tracking technology to improve HCI with iPad applications. Sensors 15, 2244– 2264. Lu, B., Duan, X., Yuan, Y., 2016. Facial Expression Recognition Based on Ensemble Extreme Learning Machine with Eye Movements Information. In: Proceedings of ELM-2015. p. 295. Manssuer, L.R., Pawling, R., Hayes, A., Tripper, S., 2016. The role of emotion in learning trustworthiness from eye-gaze: evidence from facial electromyography. Cognit. Neurosci. 7, 82–102.
None.
This research was supported by the Asahi Glass Foundation from Japan.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
10
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Mathews, Andrew et al., 2003. The face of fear: effects of eye gaze and emotion on visual attention. Visual Cognit. 10 (7), 823–835. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Berkay Celik, Z., Swami, A., 2016. The Limitations of Deep Learning in Adversarial Settings. In: IEEE European Symposium on Security and Privacy (EuroS&P). Parr, L.A., Waller, B.M., 2006. Understanding chimpanzee facial expression: insights into the evolution of communication. J. Social Cognit. Affect. Neurosci. 1 (3), 221–228. Patnik, S., Moharkar, L., Chaudhari, A., 2017. Deep RNN learning for EEG based functional brain state inference. In: International Conference on Advances in Computing, Communication and Control (ICAC3). Peng, P., Green, M.B., Ji, Q., Wayman, J., 2015. Automatic eye detection and its validation. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. p. 164. Polatsek, P., 2013. Eye Blink Detection. In: IIT.SRC. p. 1. Poston, A., 2000. Human Engineering Design Data Digest. CreateSpace Independent Publishing Platform, Warshington DC, USA. Ryu, J.B., Hyun, S.Y., Seo, Y., 2013. Real time eye blinking detection using Local Ternary Pattern and SVM. In: Eighth International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA). p. 598.
Schurgin, M.W., Nelson, J., Lida, S., Ohira, H., Chiao, Y., Franconeri, S.L., 2014. Eye movements during emotion recognition in faces. J. Vision 14, 14. Soleymani, M., Pantic, M., Pun, T., 2012. Multimodal emotion recognition in response to videos. IEEE Trans. Affective Comput. 3 (2), 211–223. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M., 2016. Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affective Comput. 7, 17–28. Trojano, L., Moretta, P., Loreto, V., Cozzolino, A., Santoro, L., 2012. Quantitative assessment of visual behavior in disorders of consciousness. J. Neurol. 259, 1888–1895.
Further reading Indumathy, N., Patil, K., 2014. Medical alert system for remote health monitoring using sensors and cloud computing. Int. J. Res. Eng. Technol. 3, 884–888.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
Contents lists available at ScienceDirect
Journal of King Saud University – Computer and Information Sciences journal homepage: www.sciencedirect.com
A fear detection method based on palpebral fissure Rawinan Praditsangthong, Bhattarasiri Slakkham, Pattarasinee Bhattarakosol ⇑ Department of Mathematics and Computer Science, Chulalongkorn University, Thailand
a r t i c l e
i n f o
Article history: Received 15 December 2018 Revised 30 April 2019 Accepted 1 June 2019 Available online xxxx Keywords: Interpalpebral fissure Palpebral fissure region Eye Emotion Decision tree Classification
a b s t r a c t Human emotions, such as smiling or laughing, can be expressed in various forms through the face whenever there are stimuli. These changing faces can reflect the emotional states that are used to identify a normal or an abnormal behaviour. This research aims to study the patterns in human faces and identify the areas of interest (AOI), which is called Facial Landmark Detection (FLD). The investigation of the external elements of eyes is performed, and it consists of the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). These elements are applied to classify the emotional differences between neutral and fearful emotions. A method for emotional classification was designed according to the changing values of the IPF, PFL, and PFR. An ID3 algorithm was used to classify the emotions. Three hundred sixty images were derived from horror-thriller-murder movies based on IMDb. This data set was utilized to generate the proposed pattern. This pattern was used to classify the emotions using a decision tree technique that led to the development of an emotional classification model. The accuracy of the emotional classification model between neutral and fearful emotions was 92.50%, thus proving that the proposed model is efficient. Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
1. Introduction The expression of human emotion consists of positive emotions and negative emotions, such as happiness, sadness, pride, anger, and fear. Moreover, most psychologists (Manssuer et al., 2016; Eva et al., 2016; Trojano et al., 2012; Corey and Komogortsev, 2011) believe that human actions are defined by the rudimentary emotions. Generally, a person can express emotions through their eyes and various actions, such as gazes, fixations, saccades, pupil locations, pupil dilations, or blinking. These actions can be used to classify human emotions. Various approaches, such as the Haar cascade, convolutional neural network (CNN), corneal reflection, projection function, or edge analysis (EA) (Soleymani et al., 2016; Khan et al., 2016; Almudhahka et al., 2016), can be applied to detect and classify human emotions. In addition, some studies use a low cost high-performance camera to detect the eye move⇑ Corresponding author. E-mail addresses: [email protected] (R. Praditsangthong), [email protected] (B. Slakkham), [email protected] (P. Bhattarakosol).
ments (Fuhl et al., 2016; Polatsek and ‘‘Eye, 2013; Papernot et al., 2016; Devahasdin Na Ayudhya and Srinark, 2009; LopezBasterretxea et al., 2015). Nevertheless, there are some limitations. For example, when the front camera of tablet is used to detect the pupil dilation or pupil size, the distance from the tablet is only 30 cm (Polatsek and ‘‘Eye, 2013; Huabiao et al., 2013; Patnik et al., 2017). One interesting region of the eyes is the palpebral fissure (PF), which is the elliptical space between the eye lids (Lu et al., 2016). Emotional changes can affect to the distance between the top edge and the bottom edge of the eye lids, which is called the interpalpebral fissure (IPF). These changes can be used to detect human emotions. Fig. 1 shows the emotional flow where the causes of each emotion are important subjects since there is no emotion, without the cause. Additionally, positive stimuli result in positive emotions, but negative stimuli result in negative emotions. This research focus on an emotional detection using the palpebral fissure. The objective of the study was to investigate the interpalpebral fissure and the palpebral fissure regions in order to analyse and classify neutral and fearful emotions.
Peer review under responsibility of King Saud University.
2. Related work
Production and hosting by Elsevier
Human emotions refer to the immediate expression of feelings in a situation with stimulus (Izard, 2013). Human emotions can be
https://doi.org/10.1016/j.jksuci.2019.06.001 1319-1578/Ó 2019 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
2
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 1. The Emotional Flow.
expressed in various ways, but most of these expressions are reflected in the face; these are called facial expressions. A facial expression refers to changes in the eyebrows, eyes, nose, and mouth of the person who received the stimuli. This countenance is always followed by some action, such as self defence, unconsciousness, or screaming. However, under the same situation, different people will react differently. Therefore, detecting emotions should not be performed by detecting human reactions but emotions should be detected from the countenance because the beginnings of those actions come from the same issue. As mentioned above, emotions should be detected using facial expressions, as shown in Fig. 2. According to Fig. 2, it is clear that shapes of the eyebrows, eyes, nose, and mouth of the same person are dissimilar under different situations. Thus, (Izard, 2013) stated that the expression of emotions can be defined in two forms: emotional states and emotional traits. An emotional state refers to a particular emotion in a certain period of time, such as the happiness state, angry state, or fearful state, and these states may last from seconds to hours, depending on the situation. In term of emotional traits, these refer to the frequent trend of an emotional occurrence in day-to-day life (for example, the happiness trait, angry trait, or fearful trait). Moreover, the emotional states can be classified into two categories: positive
emotions and negative emotions. Positive emotions include happiness, surprise, and pride, while sadness, anxiousness, anger and fear can be classified as the negative emotions. In addition, the facial recognition of emotional states for emotional interpretation has been applied to studies in the life sciences area. For example, research has compared the feelings of apes and humans using the facial expressions of both species (Parr and Waller, 2006). Since there are various objects on a face, such as eyebrows, eyes, the nose, and the mouth, the changes in these objects can be used as a communication signal, such as pain, hunger, and fear. Therefore, many researchers have studied emotion recognition and detection using these changes. For examples, the eye movements have been studied under various objectives (Ciesla and Koziol, 2012; Schurgin et al., 2014; Peng et al., 2005). To classify the emotional states, there are processes that need to be performed, as shown in Fig. 3.
2.1. Process-1 The detection of a person’s sentiment over a face was separated into five regions by Schurgin et al. (2014), which are the eyes, upper nose, lower nose, upper lip, and nasion, as shown in Fig. 4. Additionally, the results from Schurgin et al. (2014) showed that the eyes are the most significant organ for expressing feelings as reflected in the quote ‘‘the eyes are the window of the soul”.
2.2. Process-2
Fig. 2. Faces with different emotions.
Even though several techniques have been implemented, there are two mechanisms that are well-known among researchers because of their accuracy. These mechanisms are the Hough transform mechanism (Huabiao et al., 2013) and the Haar cascade object detector (Asier et al., 2015). The Hough transform mechanism and the Haar cascade object detector are applied to detect the pupils in order to interpret the meaning of the countenance. The main concept of the Hough transform mechanism is based on the dilation of the pupils. Lines and points are applied to distinguish the changes in the pupils from normal state to a specific state. The experiment was performed on every sentiment: sadness, anger, happiness, pride, and fear. In the Haar cascade object detector mechanism (Asier et al., 2015), an image will be defined as a two-dimensional table of a matrix. Each dimension contains the sum of all the pixels related to the position. Then, the values in the table is used to mark the points in the pupils, the positions of the pupil centres are defined and the pupil size is calculated. This experiment was performed under two states: the normal state and specific states. The specific states are the same as Hough transform mechanism, including disgust. The results showed that the pupil sizes from normal state and all specific states are not equivalent. As a consequence, Haar cascade object detector mechanism can be applied to indicate the emotions of people in different situations.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
3
Fig. 3. Emotional classification processes.
2.3. Process-3 The next step is the pattern classification of the emotional detection using the defined techniques. Popular classification methods are the Support Vector Machine (SVM), the Bayesian classifier, and the Extreme Learning Machine (ELM) (Lu et al., 2016; Ryu et al., 2013; Peng et al., 2005). Fig. 5 shows results of these techniques. The Support Vector Machine (SVM) is one type of machine learning that is used to classify data groups in a linear model using planes or hyperplanes in order to find the best linear classifier model. However, the obtained data are always fitted in a nonlinear form rather than linear form; therefore, this problem is solved using a kernel function. This function separates the data into different groups according to their features. The differences among groups are classified using the distances between groups. On the other hand, the Bayesian classifier can also be applied to organize data into various groups based on their features’ probabilities. Some research (Manssuer et al., 2016; Eva et al., 2016; Schurgin et al., 2014) used this technique to recognize the expression of six basic emotions; anger, disgust, fear, happiness, sadness, and surprise. The Bayesian approach refers to the role of a class to predict the values of the features for members of the class. The learning of the Bayesian classifier constructs a probability model of the features and uses this model to predict of the classification.
Fig. 4. Five regions for the facial expression detection algorithm.
An additional interesting machine learning technique is the Extreme Learning Machine (ELM), which is a single-hidden layer feed forward neural networks (SLFNs) (Lu et al., 2016). The ELM consists of an input layer, a hidden layer, and an output layer. In the input layer, the data randomly weighted, including the bias values. These weighted values are obtained from the calculation of Moore-Penrose inverse matrix. The input layer randomly identifies weight values, including the bias value of hidden layer. Thus, this approach focuses on preparing data appropriately before they are input into the system. Referring to Fig. 5, in the Emotional classification processes, these techniques have been implemented by various researchers in studies using the pupil diameter, pupil location, pupil size, gaze distance, fixation, saccade, eye blinking, or eye movements to classify emotional states (Mathews, 2003; Soleymani et al., 2012; Li and Mao, 2012). Some studies measure the smallest and the largest pupil sizes, some measure the speed of a personal’s eye blinking, and some measure the gaze time. Nevertheless, these measurements need to be performed using specific equipment in order to detect the required values. Unfortunately, this specific equipment is very costly and not portable. Consider the eye movement detection mechanism. This mechanism needs to determine the defined area of interest (AOI) before the eye tracking equipment can find the fixation orders, fixation counts, fixation durations, and scan paths (Lu et al., 2016). Once the area of interest is identified, called the Facial Landmark Detection (FLD) or localization is performed to find its accurate location on the face. This process is responsible for locating the eyebrows, centres of the eye, nose, upper lip, lower lip, or centres of the mouth (Feng, 2015). Consequently, the automatic facial landmark detection process can be implemented in order to observe and monitor the emotion expressions. Additionally, if the pupil locations and their radii are needed, a high resolution and high definition camera must be used (Ciesla and Koziol, 2012); which increases for users, and researchers with limited funds cannot employ this method. As mentioned above, the characteristics of eyes can be used by people to express their feelings. These characteristics include the pupil diameters, pupil locations, pupil sizes, eye blinking, and eye movements. One similarity in these characteristics is that the measurement is performed inside the eyeball, and none of the existing studies use the rim of eyes to detect emotions.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
4
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 5. Classification.
3. Materials and methods
quality of life, especially paralyzed patients, will be increase because the PMS can continuously monitor patients.
3.1. Problem statement and solution 3.2. Proposed method To increase the quality of life of patient, a patient monitoring system must be provided as soon as their life is in danger. An alarm button is installed for people to press when their lives are in danger or they need some helps. Unfortunately, an alarm button cannot be pressed in all situations, such as for paralyzed patients. Therefore, IP cameras or closed-circuit television (CCTV) can be implemented to capture the events in every significant area, and monitors must be used in a control room to observe these areas. Therefore, this solution is costly and not suitable for a small organization or a Healthcare Centre. Thus, a solution that uses only cameras and an automatic alarm is proposed. Moreover, the implemented cameras in this system are inexpensive, small and lightweight with moderate resolution. As a result, it is suitable for capturing only a small and specific area, such as the eyes, nose, or mouth. Therefore, researchers use only the eyes of a person as the focal area to identify critical situations instead of the entire face or other body movements. The captured images will be sent as a stream to be interpreted as normal or abnormal states by an image classification system on a personal computer, as shown in Fig. 6. If the defined state is abnormal, the alarm system will be automatically activated. Based on Fig. 6, the patient monitoring system (PMS) consists of a camera, a computer and an alarm system. Thus, a significant part in this system is the emotional classification model that must be installed in the patient monitoring system. Therefore, this paper proposes an emotional classification model (ECM) that can classify the emotions of patients as either neutral or fearful. Therefore, the
Eyes are composed of internal elements and external elements. These elements require different equipment to measure their changes. Most researchers have considered the internal elements, such as the iris, pupil, and lens. Unlike other researchers, this research will focus on the movements of the external elements of eyes, which are the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). As mentioned in above, this research will focus on the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR), as shown in Fig. 7(a). From Fig. 7 (a), the vertical distance between the palpebrale superius (PS) point and the palpebrale inferius (PI) point is the IPF while the horizontal distance between the endocanthion (EN) and the exocanthion (EX) is the PFL. Moreover, the IPF and PFL are applied to calculate the PFR that is used to identify the emotional differences, and, the formula for the PFR is presented in Fig. 7(b). Hence, this research aims to implement an emotional classification system between the neutral emotion and the fearful emotion based on the ID3 Decision Tree algorithm using the IPF and the PFR. Fig. 8 shows a diagram of the proposed method. This method starts from selecting an image from a horror-thriller-murder movie. The movies are collected from IMDb website, and they are ranked by experts and leading critics. The collected movies must have high rankings, be in high definition (HD) and have quality actors/actresses. These movies are expected to express fear. Thus, all fearful images can be captured from these movies using
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
5
Fig. 6. Infrastructure of the proposed patient monitoring system (PMS).
Fig. 7. The external elements and the PFR calculation.
FFmpeg, which is multimedia framework and a cross-platform solution to record, convert, and stream audio and video. (See Fig. 9). After the fearful images are captured, these images are inserted into the developed application to detect and extract the responding faces. The application was developed using Qt creator editor version 5 based on OS X 10.11 EI Capitan with the C++ language. In addition, an open source library, called the DLIB library, was embedded into the Qt creator. When we consider an image, one image normally consists of one frontal face, background, and other objects. Among these components, the most important object is the frontal face, and this will be captured using the DLIB library. However, the size of this captured face has no standard size or is the real size of a human face, which is defined in (Poston, 2000). Therefore, the size of the frontal face is adjusted to be the real size and this resized face has 68 facial mark-points added on the face. As a result, the new facial image is normalized and adjusted to 800x800 pixels. After marking the face with 68 mark-points, the
PI, PS, EX and EN points are defined. Then, the IPF and the PFL can be calculated. Finally, the neutral or fearful emotion can be identified according to the value of the PFR. 3.3. Data collection and processing As mentioned previously, all images are captured from horrorthriller-murder movies and the frontal face is the only object to be used. The selected number of movies is 33. For each movie, there are only 2 states to be captured from an actor: the normal state face and the fearful state face. In addition, more than one actor from each movie are used in this research and no actor and actress are used in multiple movies. Therefore, there are 60 actors in the age range of 18–52 years that are used as the samples in this experiment. For each actor, three scenes with a neutral emotion and three scenes with a fearful emotion from the movies are collected. Therefore, total number of images used in this experiment is 360. The duration of data collection is two months.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
6
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
3.4. Data analysis processing The 360 images were divided into two data sets: the training data set and the testing data set. The testing data set is used for the classification and analysis processes. The number of actors in the training data set is 40 while the number of actors in the testing data set is 20. Moreover, all images were calculated to determine the differences between the two emotions using the IPF, the PFL, and the PFR. After completing the data capturing process, 240 images from the training data set were retrieved from the database to determine means of the IPF, the PFL, and the PFR. Within the training data set, the 40 actors are 47.5% male and 52.5% female, and the highest age range is between 18 and 24 years old, which encompasses 32.5% of the actors. The statistical test is the paired samples t-test, which is used to determine the differences of mean values of the neutral emotion and the fearful emotion for each indicator. This test uses a 95% confidence level or the 0.05 significance level (a). The results of the test can be summarized as follows. 3.4.1. Test of the differences in mean IPF There is a significant difference between the IPF means of the right eyes under neutral and fearful emotions with a pvalue = 0.003 < 0.05 = a. Similar to the right eyes, the left eyes also have significant mean differences under the two focusing emotions with a p-value = 0.001 < 0.05 = a. Moreover, the values of the IPF from both eyes in the neutral situation are smaller than the values of the IPF in the fearful situation. Furthermore, the change of the mean IPF in males is higher than the change of the mean IPF in females. Table 1 presents the mean IPFs with the standard deviations of both eyes in the samples. 3.4.2. Test of the differences in mean PFL The results of sample t-tests indicate that only the right eye can reflect the emotion of fear because the values of the mean PFLs of the right eyes under neutral and fearful emotions are significant different with a p-value = 0.046 < 0.05 = a. The mean PFLs with the standard deviations are given in Table 2.
Fig. 8. Diagram of the application.
3.4.3. Test of the differences in mean PFR The testing results of the mean PFRs are the same as the results obtained for the mean IPFs. Moreover, the mean PFRs in the neutral state are smaller than the mean PFRs in the fearful state. Additionally, the changes of males’ eyes are larger than the changes of females’ eyes. Table 3 shows the mean PFRs with the standard deviations.
Table 1 The mean IPFs with the standard deviations (mm). Gender
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
8.83 (1.22) 10.21 (0.99) 9.52
10.45 (1.24) 11.35 (1.05) 10.90
8.76 (1.24) 10.30 (1.09) 9.53
10.50 (1.33) 11.34 (1.16) 10.92
Table 2 The mean PFLs with the standard deviations (mm). Gender
Fig. 9. Relation among Gender, Emotion, IPF, PFL, and PFR.
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
29.80 (1.88) 30.98 (2.26) 30.39
30.50 (2.27) 30.90 (1.53) 30.70
30.75 (2.48) 31.21 (1.18) 30.98
30.37 (1.54) 30.16 (2.21) 30.27
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx Table 3 The mean PFRs with the standard deviations (mm). Gender
Male Female Total mean
Right Eye
Left Eye
Neutral
Fearful
Neutral
Fearful
207.06 (33.79) 248.84 (33.12) 227.95
251.39 (41.85) 276.12 (34.62) 263.76
212.17 (37.95) 252.64 (30.47) 232.41
250.80 (37.01) 269.68 (41.65) 260.24
According to three simple t-tests, it can be summarized that the eyes in the fearful state are larger or wider than the eyes in normal state. In addition, males’ eyes usually change more than females’ eyes during fear. Therefore, it could be said that gender and emotion are related to the changes of the mean values of the IPF, the PFL, and the PFR. Thus, when the mean value of any indicator is larger than normal, it is possible that the person is in a scared state. The relationship among these three parameters can be drawn as in Fig. 5. 3.4.4. Choosing the parameters and creating the model 3.4.4.1. Choosing the parameters. According to the above conclusions, there are at least three important parameters that can reflect human emotions. These parameters are gender, the IPF, and the PFR. However, the IPF and PFR may relate to other parameters, such as age and eye-side. Therefore, in order to obtain a precise model to determine the emotions, all reasonable parameters should be considered. In this case, the independent parameters to be considered are gender, age, eye-side, the IPF, and the PFR. The dependent parameter of the model is the emotion. Table 4 presents possible values of all parameters in the emotional model. Furthermore, Fig. 10 shows the possible relationships
Table 4 Values of the variables in this experiment. Independent Variable
Possible values
gender age-range eye’s location IPF PFL PFR Dependent Variable status of emotion
Male, Female 18–24, 25–31, 32–38, 39–45, and greater than 45 Right, Left 1; 1 1; 1 1; 1 Possible values Neutral, Fear
7
among these parameters, where the brown dashed line represents the expected effect of the factor towards an emotion. In addition, the relationship between gender and emotion and the relationship between age and emotion can be determined by calculating their correlations. The next step is to determine relation between age and emotion. Unfortunately, there is no existing link from gender to emotion, or vice versa. Since the distributions of emotions under each independent variable defined above are not normal, a nonparametric method, Kendall’s rank correlation coefficient test, is applied using the 95% confident level. The test also concerns the effects from the interactions among independent variables and the effects from the individual variable. The results from the test indicated that there are some relationships among these parameters and the relationship diagram can be illustrated in Fig. 11. According to the relationship diagram in Fig. 11, there is only two direct links to emotion. One link is the correlation between change of the IPF/PFR and emotion, and the second link is the correlation between the emotion and the interaction of gender and age. However, there are direct correlations between some interactions with gender, eye-side, or age. Therefore, it can imply that there are some hidden correlations between gender and emotion, eye-side and emotion, and age and emotion by passing through the existing correlations. Thus, it is a fact to say that every independent parameter is related to emotion. 3.4.4.2. Creating model. Based on the conclusion from the previous section, all parameters were compiled using a data science software tool called RapidMiner Studio version 8.1. The changed pattern from the parameters was customized to a general model as an emotional classification model with a many-to-one relationship among the variables, as shown in Eq. (1).
Emotion ¼ M ðgender; age; eye side; IPF; PFRÞ
ð1Þ
Eq. (1) refers to ‘‘Emotion is the model of gender, age, eye-side, IPF, and PFR with possible interactions”, and the possible values of all parameters are defined in Table 4. Since the emotional classification model is many-to-one relation and the solution of the Emotional model can be either neutral or fearful, a decision tree can be implemented to express all possible paths for an emotion’s derivation. In this research, the ID3 decision tree algorithm is used to create a tree-like graph or model. Thus, the data set was imported into RapidMiner version 7.4 to create a classification model of the emotional expressions from the value of the target attribute, which is called data labelling. Since there are 40 training data sets, the attributes of the training data set consist of the results of the calculation the of the average IPF and PFR for both the right eye and the left eye. Furthermore, the structure of the ID3 decision tree algorithm is shown in Fig. 12. 4. Results To find the values of all IPFs and PFRs for both eyes, the 68 facial landmarks must be set, since the emotional classification system relies on these landmarks when identifying an emotion from an image. There are 3 steps of this classification system.
Fig. 10. Expected relationships among the independent and dependent variables.
Step 1: chip size of the images After the images are input into the system, their sizes are adjusted to 800 800 pixels, which is similar to the real human face. Step 2: 68 facial landmarks’ detection The frontal facial images led to identify 68 points using the DLIB library. These 68 points are used to locate the facial landmarks.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
8
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Fig. 11. Relationships among all variables.
Fig. 12. ID3 Decision Tree Algorithm of Emotional Classification.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Step 3: predict the emotion The IPF, and the PFR are used to classify and predict neutral emotions and fearful emotions. The test data set contains 20 records or 120 sub-images. This data set is input into the emotional classification model that can be represented as a directed graph, as in Fig. 12. The accuracy of this classification process is dependent on 4 values: the true positives (TPs), the true negatives (TNs), the false positives (FPs), and the false negatives (FNs). These values are used to find the Precision and Recall as follows.
Precision ¼
Recall ¼
TP TP þ FP
TP TP þ FN
Accuracy ¼
TP þ TN TP þ TN þ FP þ FN
9
6. Conclusions The aim of this research is to study the patterns of emotion expressions on human faces by focusing on the external elements of eyes. The external elements of eyes consist of the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). The changes in the IPF, PFL, and PFR were compiled with RapidMiner Studio version 8.1. These values are used to classify the emotional differences between the neutral emotion and the fearful emotion using a decision tree algorithm, ID3. Thus, the experimental result of the patterns led us to develop an emotional classification model. The accuracy of the emotional classification model was 92.50% for classifying neutral emotions and the fearful emotions. Furthermore, this research is published and granted by the Department of Intellectual Property (Bhattarakosol et al., 2018). The future work for this research is to find other features on the face that can be combined with the PF to clearly define the emotions of faces. This work will allow these methods to be applied to many areas to increase wellbeing, such as the medical and healthcare field. Declaration of Competing Interest
From the testing result, TP is 17, FP is 0, FN is 3, and TN is 20. Consequently, Precision value is 100%, which means that the probability that the model can present the right answer is 1. Additionally, the Recall value is 85%, which means that the probability that the solutions that is presented when the model is used is right is 0.85. Finally, the Accuracy of this model is 92.50%, which means that there is a small chance that this model can predict the wrong emotion.
Acknowledgement
5. Discussion
References
Once people were confronted by stimuli, their faces usually change to express their emotions. This change can be seen in the eyebrows, eyes, nose, and lips. Among these components, the eyes are the most interesting feature because it can clearly express the emotion. Thus, several researchers have used eyes as their research focus in many areas, such as security, telecommunication, and healthcare (Trojano et al., 2012). The anatomy of the eye includes the iris, pupil, lens, cornea, and retina (Doganay et al., 2017). Since these elements are very small, specific instruments are required. Therefore, the emotional interpretation using these components is very costly and the captured image may not be applicable as is needed due to the poor equipment. Nevertheless, there is a feature that is larger than those five elements and is easy to detect using any simple camera, but is infrequently addressed by researchers. This element is called palpebral fissure (PF). This PF is not a part of the eye’s anatomy, but it is the space between upper and lower eye lids. This research uses the PF as an important factor to reflect the emotions because the PF can be measured using the IPF and the PFR. The changes in the IPF and the PFR, along with gender, age, and eye-side, can be used to create an emotional classification model that can reflect either neutral or fearful emotions. The model is created based on the results from the correlation analysis process under a 95% confidence level. Moreover, the statistical testing has determined that the sizes of eyes in the neutral state are smaller than the sizes of eyes in the fearful state. Additionally, men have larger PFs than women when they are afraid. Though the model can distinguish between neutral and fearful emotions, it cannot find the differences between fear and anger. This finding is because when people lose their temper, their eyes also open wider than normal, which is the same as when they are fearful. In contrast, this emotional classification model would be able to identify happiness and sadness according to the size of the PF which would be smaller than normal.
Almudhahka, N., Nixon, M., Hare, J., 2016. Human face identification via comparative soft biometrics. In: IEEE international Conference on Identity, Security and Behavior Analysis (ISBA). Asier, L.B., Mendez-Zorrilla, A., Garcia-Zapirain, B., 2015. Eye/head tracking technology to improve HCI with iPad applications. Sensors 15 (2), 2244–2264. Bhattarakosol, P, Praditsangthong, R., Slakkham, B., Feb. 15, 2018. An Automatic Patient’s Face Aided Detection System (APFADS). Department of Intellectual Property. TH Patent 1801000895. Ciesla, M., Koziol, P., 2012. Eye pupil location using webcam. CoRR 1202, 6517. Corey, H., Komogortsev, O., 2011. Biometric identification via eye movement scanpaths in reading. In: IEEE International Joint Conference on Biometrics (IJCB). p. 1. Devahasdin Na Ayudhya, C., Srinark, T., 2009. A method for real-time eye blink detection and its application. In: 6th International Joint Conference on Computer Science and Software Engineering (JCSSE). Doganay, F., Dadcai, Z., Topcu-Yilmaz, P., Doganay Aydin, H., 2017. The association between ocular dominance and physiological palpebral fissure asymmetry. J. Laterality: Asymmetries Body Brain Cognit. 22, 412–418. Eva, P., Tobias, B., Sylvain, D., David, S., 2016. Attentional bias for positive emotional stimuli: a meta-analytic investigation. Psychol. Bull. 142, 79–106. Feng, Zhen-Hua et al., 2015. Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting. IEEE Trans. Image Process. 24 (11), 3425–3440. Fuhl, W., Santini, T., Kasneci, G., Kasneci, E., 2016. PupilNet: convolutional neural networks for robust pupil detection. Comput. Vision Pattern Recognit. 1602. Huabiao, Q., Wang, X., Liang, M., Yan, W., 2013. A novel pupil detection algorithm for infrared eye image. In: IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC). p. 1. Indumathy, N., Patil, K., 2014. Medical alert system for remote health monitoring using sensors and cloud computing. Int. J. Res. Eng. Technol. 3, 884–888. Izard, Carroll E., 2013. Human Emotions. Springer Science & Business Media. Khan, S.A., Hussain, A., Usman, M., 2016. Facial expression recognition on real world face images using intelligent techniques: a survey. Int. J. Light Elect. Opt. 127, 6195–6203. Li, Zheng, Mao, Xia, 2012. Emotional eye movement generation based on geneva emotion wheel for virtual agents. J. Visual Lang. Comput. 23 (5), 299–310. Lopez-Basterretxea, A., Mendez-Zorrilla, A., Garcia-Zapirain, B., 2015. Eye/head tracking technology to improve HCI with iPad applications. Sensors 15, 2244– 2264. Lu, B., Duan, X., Yuan, Y., 2016. Facial Expression Recognition Based on Ensemble Extreme Learning Machine with Eye Movements Information. In: Proceedings of ELM-2015. p. 295. Manssuer, L.R., Pawling, R., Hayes, A., Tripper, S., 2016. The role of emotion in learning trustworthiness from eye-gaze: evidence from facial electromyography. Cognit. Neurosci. 7, 82–102.
None.
This research was supported by the Asahi Glass Foundation from Japan.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001
10
R. Praditsangthong et al. / Journal of King Saud University – Computer and Information Sciences xxx (xxxx) xxx
Mathews, Andrew et al., 2003. The face of fear: effects of eye gaze and emotion on visual attention. Visual Cognit. 10 (7), 823–835. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Berkay Celik, Z., Swami, A., 2016. The Limitations of Deep Learning in Adversarial Settings. In: IEEE European Symposium on Security and Privacy (EuroS&P). Parr, L.A., Waller, B.M., 2006. Understanding chimpanzee facial expression: insights into the evolution of communication. J. Social Cognit. Affect. Neurosci. 1 (3), 221–228. Patnik, S., Moharkar, L., Chaudhari, A., 2017. Deep RNN learning for EEG based functional brain state inference. In: International Conference on Advances in Computing, Communication and Control (ICAC3). Peng, P., Green, M.B., Ji, Q., Wayman, J., 2015. Automatic eye detection and its validation. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. p. 164. Polatsek, P., 2013. Eye Blink Detection. In: IIT.SRC. p. 1. Poston, A., 2000. Human Engineering Design Data Digest. CreateSpace Independent Publishing Platform, Warshington DC, USA. Ryu, J.B., Hyun, S.Y., Seo, Y., 2013. Real time eye blinking detection using Local Ternary Pattern and SVM. In: Eighth International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA). p. 598.
Schurgin, M.W., Nelson, J., Lida, S., Ohira, H., Chiao, Y., Franconeri, S.L., 2014. Eye movements during emotion recognition in faces. J. Vision 14, 14. Soleymani, M., Pantic, M., Pun, T., 2012. Multimodal emotion recognition in response to videos. IEEE Trans. Affective Comput. 3 (2), 211–223. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M., 2016. Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affective Comput. 7, 17–28. Trojano, L., Moretta, P., Loreto, V., Cozzolino, A., Santoro, L., 2012. Quantitative assessment of visual behavior in disorders of consciousness. J. Neurol. 259, 1888–1895.
Further reading Indumathy, N., Patil, K., 2014. Medical alert system for remote health monitoring using sensors and cloud computing. Int. J. Res. Eng. Technol. 3, 884–888.
Please cite this article as: R. Praditsangthong, B. Slakkham and P. Bhattarakosol, A fear detection method based on palpebral fissure, Journal of King Saud University – Computer and Information Sciences, https://doi.org/10.1016/j.jksuci.2019.06.001