A multimodal study to measure the cognitive demands of hazard recognition in construction workplaces

Safety Science 133 (2021) 105010

Contents lists available at ScienceDirect

Safety Science journal homepage: www.elsevier.com/locate/safety

A multimodal study to measure the cognitive demands of hazard recognition in construction workplaces Pin-Chao Liao a, Xinlu Sun a, Dan Zhang b, * a b

Department of Construction Management, Tsinghua University, No. 30, Shuangqing Rd., HaiDian District, Beijing 100084, PR China Department of Psychology, Tsinghua University, No. 30, Shuangqing Rd., HaiDian District, Beijing 100084, PR China

A R T I C L E I N F O

A B S T R A C T

Keywords: Hazard recognition Near-infrared spectrum Multimodal monitoring Occupational environment Safety management

Hazard recognition has been extensively explored in previous studies. However, deficits have arisen due to the neglect of task-specific effects, information distortion by image-based experimental tasks, and the exclusive use of eye trackers. This study aimed to explore how cognitive patterns vary in simulated construction worksites with different types of hazards using multimodal monitoring. A hazard recognition task was conducted in a hazardous civil laboratory using both an eye tracker and a near-infrared spectrum system to capture pupil responses and cerebral oxyhemoglobin signals. Cognitive responses were analyzed according to hazard type and scene complexity. The results showed that falling hazards induced the most cerebral and pupillary activation. Scene complexity triggers an increase in pupil diameter and impacts cerebral activities by interaction with hazard type. This study also reveals the complementary functions of pupillary responses and neural processes in hazardous simulated worksites and a ceiling effect of cognitive resources. We conclude that construction workplaces with different types of hazards can induce different cognitive demands and should thus be treated individually. This information is potentially useful for practical applications.

1. Introduction Hazard recognition is regarded as one of the most effective ap­ proaches to proactive accident prevention. While unsafe behavior has been shown to account for 70–80% of offsite accidents, it has been argued that failure to recognize hazards directly precedes accidents (Haslam et al., 2005; Rasmussen, 1997; Suraji et al., 2001). However, it has also been noted that exposure to hazardous environments does not necessarily result in accidents (Atefeh et al., 2018). Additionally, Tixier et al. (2014) found that unsafe behavior is more attributable to insuffi­ cient hazard recognition ability than to a deliberate violation of safety guidelines. Proper detection and reporting of hazards are considered to significantly improve workplace safety (McSween, 2003). Accordingly, hazard recognition is generally regarded as the most fundamental element to minimize job site risks for both managers and workers. Unfortunately, up to 57% of job site hazards remain unrecognized (Albert et al., 2014, Bahn, 2013, Carter and Smith 2006, Perlman et al., 2014). Previous studies have struggled to address the problem of inad­ equate onsite hazard recognition ability by developing numerous assistance tools. However, Jeelani (2016) contends that the weaknesses of traditional interventions constrain the enhancement of hazard

recognition ability. In fact, it has been noted that some interventions are designed without a proper understanding of the hazard recognition process (Rozenfeld et al., 2010). The exploration and deciphering of hazard recognition are, therefore, necessary to improve safety performance. Previous studies have attempted to measure cognitive processes to gain insights into the hazard recognition process. Since hazard recog­ nition is regarded as a visual search task, the primary method has been to equip participants with eye-tracking devices to record oculomotor features. Recently, several studies have attempted to measure the mental workload evoked by hazardous environments with quantitative psy­ chological monitoring technologies, such as electroencephalography (EEG) and near-infrared spectroscopy (NIRS) (Atefeh et al., 2018, Chen et al., 2016). While these attempts were innovative and meaningful for occupational safety research, they tend to treat different hazards equally and thus fail to reveal the cognitive patterns and demands evoked by various hazardous workplaces. This research, along with previous studies employing eye trackers, exposes the substantial vulnerability in several aspects of the experimental approach. First, studies utilize insufficient or qualitative hazard classification methods, whereas mental workload and neural activation are considered to be task-specific

* Corresponding author. E-mail addresses: [email protected] (P.-C. Liao), [email protected] (D. Zhang). https://doi.org/10.1016/j.ssci.2020.105010 Received 14 August 2019; Received in revised form 6 September 2020; Accepted 18 September 2020 Available online 30 September 2020 0925-7535/© 2020 Elsevier Ltd. All rights reserved.

P.-C. Liao et al.

Safety Science 133 (2021) 105010

(Bonetti et al., 2019). Second, image-based experimental settings lead to a biased understanding of visual search patterns due to information distortion of the planar stimulus. Finally, the exclusive use of eyetracking devices provides an illustration of the decision-making phase of hazard recognition, which is open to multiple interpretations. Accordingly, the present study aimed to measure cognitive demands in simulated construction sites that differ in scene complexity and the types of hazard present. In the present study, we sought to address the deficits of previous studies through innovations in the experimental protocol. Specifically, we arranged hazards in a civil laboratory and conducted a realistic hazard recognition task. Participants were equip­ ped with an eye tracker and a series of NIRS devices for multimodal monitoring to investigate the association between brain activity and visuomotor features while in a simulated workplace with diverse haz­ ards. Scene complexity triggers an increase in pupil diameter and im­ pacts cerebral activities by interaction with hazard type. This study also reveals the complementary functions of pupillary responses and neural processes in hazardous workplaces.

address this deficiency, Dzeng et al. (2016) categorized hazards into two groups: obvious and unobvious. A similar scheme was used by Bahn (2013). Despite the authors’ best efforts, these hazard typologies are qualitative and difficult to reproduce. Further, risk perception can be highly variable between individuals, making the determination of conspicuousness variable. A quantitative method directly related to cognitive processes is necessary as an additional hazard classification tool for the investigation of these phenomena. Liao et al. (2017) used visual clutter as a proxy of scene complexity to quantify background distractions. This proved to be effective to measure cognitive difficulty in the context of complex scenes in the construction workplace (Liao et al., 2017, Sun et al., 2018). 2.1.2. Biased understanding of the cognitive processes of hazard recognition due to experiments being conducted in confined scenes In empirical studies, a dynamic construction workplace poses a number of limitations for safety management and experimental simu­ lation that are difficult to overcome. Hence, several researchers use planar stimuli to simulate workplace scenarios and analyze hazard recognition processes. Some researchers have developed digital con­ struction sites to replicate hazards. For example, Dzeng et al. (2016) established a construction site model with the help of the Google SketchUp software to examine hazard search patterns of experienced and novice workers. The more commonly used approach is to display hazardous environment as still images. For instance, Pandit et al. (2019) used 16 case images illustrating various construction operations to study the influence of the safety climate on hazard recognition. Similarly, up to 24 images captured from real construction projects were employed in the research by Albert et al. (2018) to validate the relationship between visual search patterns and hazard recognition ability. Videos from job sites have also been utilized in some studies. In the research by Kushiro et al. (2017), participants were asked to watch videos and detect risks, in order to examine the effectiveness of a video-based training tool. The abovementioned experiments are all based on two-dimensional mate­ rials, offering the advantages of a controllable inspection process, fixed visual angle, and standardized eye movement measurement. However, the reduction in dimensionality could lead to an over-simplistic repre­ sentation of stimuli, resulting in distorted visual search patterns and the consequent misunderstanding of hazard recognition processes. Re­ searchers have commented on the substantial difference that exists be­ tween image-based tasks and how hazards are perceived on construction sites (Borys 2012). Subtle environmental factors related to weather, noise, tasks, and equipment are also difficult to replicate using planar stimuli (Kushiro et al., 2017, Rowlinson et al., 2014). In fact, the effect of information shrinkage has been previously reported. For instance, Perlman et al. (2014) found that risk perception is improved through the use of a virtual construction site as compared with pictures. Thus, an experiment conducted in a natural environment is vital for hazard recognition studies. Ergonomists have argued that studying be­ haviors in natural environments can provide higher ecological validity as well as a more concise understanding of the cognitive process (Zhu et al., 2019). In this study, we attempted to recreate a construction site—a workplace with many potential hazards—in a civil laboratory. In comparison with a real construction site, the laboratory is rather static, easy to control, and allows performing experiments extending over a period of several days.

2. Literature review 2.1. Current experimental designs create a biased understanding of the cognitive processes involved in hazard recognition 2.1.1. Insufficient and qualitative hazard classification methods for taskspecific analysis Since hazard recognition is usually regarded as a complex process involving sensory, perceptual, and cognitive components, psychological analysis has gained increasing attention as a means of studying it. Several researchers have attempted to assess low hazard recognition ability and identify at-risk workers by quantitatively and directly monitoring mental workload. For example, Chen et al. (2016) developed an EEG-embedded helmet system to identify individuals who had weak risk perception abilities or tended to misestimate risks. Their research was based on the hypothesis that mental demands evoked by task dif­ ficulty would impact safety performance. They assessed EEG recordings of workers’ brain activity in an attempt to quantify mental overload and thus provide a prediction of the likelihood of unsafe behavior. The approach taken in that research was innovative and pioneering. Nevertheless, there is also a weakness in that the researchers failed to take into consideration different types of hazards. Previous studies employing eye-tracking devices have tended to classify hazards ac­ cording to diverse classification methods while observing search pat­ terns. Specifically, some researchers have classified hazards into categories according to injury type (Dzeng et al., 2016, Hasanzadeh et al., 2017). Perlman et al. (2014) identified hazards in the form of hazardous objects such as small obstacles, dangerous materials, and electric wires. Others have attempted to classify hazards according to energy type (Namian et al., 2016, Pandit et al., 2019). In relation to this, hazards caused by radiation, gravity, chemicals, and so on are known to have an influence on visual search processes. Atefeh et al. (2018) designed an experiment whereby participants equipped with NIRS de­ vices encountered hazards of different types and severity levels. The results demonstrated that the time to the peak value of hemoglobin concentration varies according to hazard severity and energy source. These efforts highlight the need for proper hazard classification before exploring cognitive demands and processing patterns. There are some similarities among the abovementioned classification methods based on injury type, energy type, and type of hazardous objects. Haz­ ards are grouped according to the hazard contents, which involves contextual reasoning. Such methods have been proven to be effective for separating hazard recognition processes. However, monotonous classi­ fication methods might not be effective because they show an insuffi­ cient association with the hazard recognition process. For example, several hazards related to a certain type of injury may differ according to scene complexity and thus involve distinct cognitive demands. To

2.2. Limits of eye movement data recorded by eye trackers The literature has shown that there are deficiencies regarding the sole utilization of eye-tracking devices when monitoring cognitive pro­ cesses involved in hazard recognition. Since it is regarded as a type of visual search task, the hazard recognition process has most commonly been analyzed using eye-tracking devices. As a proxy of visual attention, the oculomotor features recorded with eye tracking devices, such as saccades and fixation, integrate ocular activities and cognitive processes 2

P.-C. Liao et al.

Safety Science 133 (2021) 105010

(Albert et al., 2018, Dzeng et al., 2016). Usually, spatial information about saccades and fixation reveals shifts in the allocation of visual attention, while temporal information about fixation duration and per­ centage is used to assess the observer’s focus of interest and gauge cognitive difficulty. Observation of fixation duration can lead to an inconclusive correlation between hazard recognition and fixation duration features, since a long fixation could suggest that either the observer is attracted by an object, or that he or she has experienced some difficulty discerning it. Conflicting results have been obtained in several studies. According to a study by Hasanzadeh et al. (2016), workers with better situation awareness spend less time fixated on areas of interest. By contrast, Albert et al. (2018) found that people can recognize more hazards when the workplace and the hazard are examined for a longer period of time. Some researchers have found no significant relationship between fixation duration and hazard recognition ability (Liao et al., 2017), or have found that the relationship is highly variable depending on diverse memory conditions (Indrarathne and Kormos 2017). Such paradoxical conclusions might imply that the deficiency is merely due to the employment of eye trackers in the investigation of hazard recogni­ tion processes. This study attempted to overcome this defect by employing multi­ modal monitoring devices, including an eye tracker and an NIRS system. Research in human error and ergonomics has emphasized the impor­ tance of using multimodal monitoring methods. Employing different types of measurement provides complementary datasets for cognitive activities. For example, brain and eye movement signals illustrate

important features about the participants’ social and emotional infor­ mation for context-aware environments (Giannakos et al., 2019). Accordingly, an NIRS system for recording cerebral activation and an eye-tracker for recording oculomotor features were employed in this study. 2.2.1. The use of NIRS technology in cognitive research NIRS has attracted increasing interest from researchers assessing factors of human behaviors, especially driving behaviors. As a tool for neuroergonomics, NIRS has been documented as an effective approach to understand, evaluate, and improve human performance (Zhu et al., 2019). It measures cerebral activation by calculating cerebral blood volume and oxygenation changes. The interpretation of signals is based on the principle that an increase in cerebral activity is associated with an increase in regional blood flow to transport glucose and oxygen to meet the increased metabolic demands (Bonetti et al., 2019). Ergonomists have reported that cerebral activities are significantly associated with driving performance (Liu et al., 2016). The use of NIRS is considered to be sufficient to explore underlying cognitive and motor processes, which are representative of workload, training, and fatigue (Zhu et al., 2019). Compared with other equipment such as functional magnetic reso­ nance imaging (fMRI) or positron emission tomography (PET), the portability of NIRS allows participants to perform cognitive tasks freely without any constraints on movement. EEG is another popular tech­ nology in the field of human cognition. However, EEG recordings are easily contaminated by motion and eye blink artifacts (Mehta and

Fig.1. Research framework. 3

P.-C. Liao et al.

Safety Science 133 (2021) 105010

Parasuraman, 2013), whereas NIRS allows the assessment of neural hemodynamics during activities like walking, running, and biking (Mitsuo et al., 2004, Miyai et al., 2001). NIRS also offers a higher tem­ poral resolution than fMRI and a higher spatial resolution than EEG.

Table 1 Hazard types and descriptions.

3. Methodology 3.1. Research framework Fig. 1 shows the research framework in this study. First, an experi­ ment based on a realistic hazard recognition task was performed in a civil laboratory. Multivariate analysis of variance was performed to detect differences in cerebral activation and pupil diameter between several hazard groups and clutter groups. We also assessed the effect of the interaction between hazard type and visual clutter on psychological signals. A correlation analysis was then employed to determine whether ce­ rebral activity and pupil responses function in parallel during hazard recognition processes. Details of the methodology are provided in the subsequent sections.

Type

Visual clutter

Description

Fire prevention and control

Low

Electrical safety

Low

Fall protection

High Low High

Erroneous storage and preservation process of acetone Inaccessible fire hydrant covered with sundry objects Unprotected and unclosed electrical compartment Multiple electrical sockets in series Obstacles in the passageway Uncovered grooves on the ground threatening a fall

High

hazards and thus safe. 3.4. Experimental design The hazard recognition experiment was conducted in four steps, as described in detail in Fig. 3 of the paper by Sun and Liao (2019). The experiment was completed in approximately 25 min. A key aspect of the protocol was that participants were equipped with a laser pointer to indicate when a hazard was detected. They were instructed to circle around the hazardous area or object only after they had completed a search and confirmed a judgement to report the hazard. The time logs were collected after the subject had used the laser pointer. For detailed information about the experimental procedures, please refer to the section “3.5 Experimental design” in Sun and Liao (2019).

3.2. Participants Forty-eight students (35 males, 13 females) from the department of civil engineering participated in the experiment. The average age of the participants was 22.17 years. All underwent laboratory safety training upon their entrance to the college. No participants reported prior history of injury relevant to the study. All participants had normal uncorrected or corrected visual acuities and good neurological and cardiovascular health. Participants were fully informed about the experimental protocol, and informed consent was obtained before participation. They were provided financial compensation after the experiment. The experiments were approved by the Ethical Review Board of the Department of Psy­ chology, Tsinghua University.

3.5. Measurements and devices 3.5.1. Measurement of pupil diameter We measured pupil diameter as a proxy of visual activity in this study, rather than the most commonly used measurement of saccades and fixation features, as recent research has shown that pupil diameter is closely related to information processing and understanding (Giannakos et al., 2019). Specifically, it reflects emotional arousal and alertness triggered by visual detection of sensory stimuli (Qian et al., 2009). An increase in pupil diameter is observed when people process emotionally engaging stimuli (Bradley et al., 2008). More importantly, task-evoked pupil dilation has been shown to be a function of the cognitive work­ load and attention required to perform the task (Qian et al., 2009, Tang et al., 2018, Zennifa et al., 2018). In addition, Privitera et al. (2010) reported a significant pupil response when detecting a visual target in their experiment. Thus, changes in pupil diameter were monitored to demonstrate visuomotor features during hazard recognition. Moreover, changes in pupil diameter might be a more adequate measurement of emotional and cognitive conditions than saccade and fixation variables, especially in tasks based on practical scenarios, because they do not rely on the manual preset of the area of interest.

3.3. Simulated job site settings Before the formal experiment, a pilot experiment was performed to finalize the arrangement of the simulated job site. Two laboratory safety experts and members of the laboratory staff were employed to identify and confirm the hazards and the hazard types as well as the search route. The search route was shaped as an upside-down “U” with the hazards distributed evenly on both sides of the passageway. The exact start and end were also provided to the participants according to the experts’ suggestion. A total of nine hazards were recognized by the participants. Finally, six hazards were considered in this study based on the following conditions: (1) more than 20 people noticed the hazard, based on measurable changes in pupil diameter and NIRS data; (2) the hazard can be grouped with another hazard, so that the corresponding hazard type contains two or more hazards; (3) visual clutter of the hazards in each group varied, allowing us to compare the responses to different visual clutters for the same type of hazard. The six hazards were divided into three groups according to type, with each group containing a high cluttered hazard and a low cluttered hazard. Detailed descriptions about the selected hazards can be found in Table 1. For a detailed description of the other experimental settings, see Sun and Liao (2019). Moreover, the indoor lightning equipment was employed to main­ tain a stable illumination condition with about 200 lx to minimize the illumination effect on pupil diameter. It is necessary to stress that although the participants were asked to detect hazards in a practical scenario, they were protected from the risk of injury. Electrical and fire hazards were difficult to trigger because the experiment was conducted during a holiday. The lab was not in operation, and not all the equip­ ment in the lab was powered. The participants were also protected from falling hazards because the area that they could access was far from the

3.5.1.1. The eye-tracking device. The wearable eye-tracking devices utilized in this experiment were the Tobii Glasses II (Sweden). We used six cameras equipped with a mechanism to establish 3D models for the eyeballs. Absolute measurements of pupil diameter were derived from the models. The scene camera recording angle (i.e., the visual angle) was 82◦ in the horizontal plane and 52◦ in the vertical plane. The wireless data transmitting technology allowed the subject to observe and move without restriction. 3.5.1.2. Data preprocessing. Pupil diameter was obtained directly from the eye-tracking devices. Using MATLAB 2017a, data were normalized by conversion to a Z-score to eliminate individual effects. The Tobii Pro Lab software was employed to mark the time of interest (TOI) when participants were observing a hazard. Videos of the experiment for each 4

P.-C. Liao et al.

Safety Science 133 (2021) 105010

participant recorded with the eye-tracking device were reviewed to identify the exact frame where a hazard appeared, which was marked as the beginning of a TOI. This was considered as the time point when the participant discovered the hazard as it appeared in their visual field. The moment at which the participant began to use the laser pointer to give a decision was marked as the end of an observation. This period was set as the TOI, and the average pupil diameter during the TOI was calculated in the subsequent data analysis.

3.5.2.1. NIRS device. A 20-channel NIRS instrument (NirSmart, Hui­ chuang Co. China), which generates near-infrared light at two wave­ lengths (760 and 850 nm), was utilized to monitor changes in optical density at a sampling rate of 12 Hz. These measurements were later converted into Oxy-Hb and Deoxy-Hb concentration changes. Six probes and ten emitters were configured evenly at a distance of 3 cm on the forehead (Fig. 2). During the placement of the probes, the real-time signal intensity of each channel connecting the light source and the receiver was assessed and displayed by the software. When the signal quality was acceptable, a zero baseline was set, and the experimental protocol was executed.

3.5.2. Collection of NIRS data NIRS is based on optical density technology and measures the dif­ ference between the light transmitted by the device and the light re­ flected back to the receiver optodes (Bonetti et al., 2019, Delpy and Cope, 1997). At certain wavelengths, most biological tissues are trans­ parent, so few photons scatter to adjacent tissues. This makes it possible to measure the concentrations of specific chromophores according to their wavelength-specific absorption of infrared light. In this study, the concentration of oxygenated hemoglobin in the prefrontal cortex (PFC) was selected as the proxy measure of brain activity. This decision was based on hazard perception studies of driving safety, which most commonly monitor the PFC. Ergonomists found that the PFC is involved in higher-order cognitive functions. Generally, executive functioning processes such as maintenance of attention, memory, and planning are correlated with activation of the PFC (Bonetti et al., 2019). Activation of the PFC has also been found to be associated with hazard perception and decision-making in research into driving. The application of NIRS in construction safety and hazard recognition research is relatively inno­ vative. Hu et al. (2018) recently found that response time and levels of activation in the PFC are significantly linked to hazard type on con­ struction sites. However, NIRS signals are sensitive to hair on the scalp because hair follicles absorb light at near-infrared wavelengths. Con­ trastingly, there are no hair follicles obscuring the NIRS recording sites for the PFC, making it an ideal area from which to measure brain pro­ cesses for cognitive studies. Concentrations of oxyhemoglobin (Oxy-Hb), deoxyhemoglobin (Deoxy-Hb), and total hemoglobin (tHb) can be measured using NIRS. Neural activity is most commonly evaluated by measuring oxygenated hemoglobin concentration. Oxy-Hb data have also been found to be consistent with fMRI findings (Gary et al., 2002). Oxy-Hb, Deoxy-Hb, and tHb concentrations were all calculated in this study, but experi­ mental outcomes were obtained using Oxy-Hb changes after preprocessing.

3.5.2.2. Data preprocessing. Optical signals were filtered using a digital fourth-order low-pass Butterworth filter, with a cut-off frequency of 0.1 Hz to remove contamination from heart rate and respiration. Subse­ quently, movement noise was removed by applying moving standard deviation and spline interpolation routines using MATLAB 2017a. Cal­ culations were subsequently performed to estimate the Oxy-Hb con­ centration from the signal recorded through the 20 channels. To eliminate individual effects, data were normalized by calculating the Zscores for each subject. NIRS data and experimental videos were then imported into the Observer XT (Noldus, Wageningen, the Netherlands) software so that the beginning and end of hazard observation periods could be marked. Specifically, the interval of the events was set in the same way as the TOIs marked in pupil data preprocessing. The average concentration over the search process of each hazard was then estimated. 3.5.3. Visual clutter Visual clutter (VC) of the background scenarios was calculated and used to classify the hazards. As the field of view varied according to different visual angles and height of participants, the hazardous scenes recorded by the eye-tracking camera also varied between participants. The most common perspective from all video snapshots was selected as the background image for VC calculation. Four indices (color, size, distinction, and orientation) were calcu­ lated to measure VC. A detailed illustration of the computational process is provided in the methodology of Liao et al. (2017). Depending on the exact value of VC, the hazards in each group type were labeled to have a high or low level of visual clutter.

Fig. 2. Locations of the emitters and probes. Yellow points denote emitters, green points denote probes. The full color version of this figure is available online. 5

P.-C. Liao et al.

Safety Science 133 (2021) 105010

Fig. 3. Effect of hazard type on Oxy-Hb. On each box, the bottom and top edges of the box indicate the first and third quartiles, respectively; the whiskers represent the lowest datum within the 1.5 interquartile range of the lower quartile, and the highest datum within the 1.5 interquartile range of the upper quartile; the scale of the x-axis is not consistent for each subplot; the full color version of this figure is available online.

3.6. Data analysis

Table 2 Differences in Oxy-Hb concentration across hazard types and visual clutter groups. avg = averaged Oxy-Hb among all the channels, Haz = hazard type (including fall protection, electrical protection, fire prevention, and control hazards), VC = visual clutter. .: Significant at 0.10 level, * Significant at 0.05 level.

Analysis of variance (ANOVA) was utilized to explore whether there was a significant discrepancy among the psychological and physiolog­ ical responses in different hazardous environments. ANOVAs were per­ formed to compare the three hazard types as well as the two visual clutter groups. We tested whether changes of cerebral activity are more sensitive to different hazard types or to visual clutter and explored the interactions between the impact of visual clutter and hazard type. Although parametric tests such as ANOVA are commonly used for data analysis, the sample size in this study was small and the statistical assumptions such as normal distribution and homogeneity of variance could not be guaranteed. Thus, a random permutation test is an alter­ native way to test the significance of the effects in the ANOVA model. Briefly, a permutation test is a type of statistical significance test in which the distribution of the test statistic under the null hypothesis is obtained by calculating all possible values after randomly shuffling data several times. With the advances in computing technology, permutation tests have become widely used. The outcomes of permutation tests are more robust in the presence of outliers and missing data. Moreover, this type of test is able to provide a considerable degree of statistical power to mitigate the risk of family-wise error (Klesel et al., 2019). In this study, permutation simulations were performed at a level of 95% and 90% statistical significance (p < 0.05 and p < 0.1). The maximum number of iterations was one million, and the iteration stopping criteria was set as 0.01 to enhance the robustness of the outcomes. The average missing rate in this study was about 30.31% for the different hazards, ranging from 10.42% to 56.25%. This should be considered acceptable as this study is based on realistic tasks and because a missing rate of <40% is considered as “high performance” for natural visual object recognition in an image-based experiment (Pinto et al., 2008). As attention was fixed on the search route, and the effect of exploring different paths was eliminated, the order of the identified hazards was consistent among the participants. Accordingly, a t-test was performed to test the order effect. The result showed no significant difference between the recognition rates of the first three and the last three hazards (p = 0.934, >0.05).

VC Haz

VC:Haz

Channel

Df

Sum Sq

Mean Sq

Pr (Prob)

6 8 4 13 15 19 avg 9 12 avg

1 1 1 1 1 1 1 1 1 1

1.493 1.572 1.849 1.366 2.779 3.164 0.353 4.394 2.172 0.261

1.49311 1.57211 1.84939 1.36641 2.77887 3.1638 0.35316 4.3942 2.17232 0.26148

0.050. 0.084. 0.084. 0.064. 0.088. 0.012* 0.070. 0.012* 0.072. 0.072.

(only observed on two channels). However, a significant two-way interaction was also visible on Channels 9 and 12 and on the average level (p = 0.012, 0.072, and 0.072, respectively). To maximize the insights from the interactions, data from fire haz­ ards and electrical hazards were extracted in order to test significant effects on more channels (Table 3). Overall, Oxy-Hb concentration changed significantly on channels 1, 2, 4, and 12 through interaction between effects of hazard type and VC (p = 0.045, 0.045, 0.020, and 0.046, respectively). Specifically, we found that Oxy-Hb concentration decreased during the identification of fire hazards if the background was more cluttered. However, if the hazard was related to electrical pro­ tection defects, Oxy-Hb increased more in cluttered scenes. 4.2. Cognitive arousal measured by pupil dilation in different workplace hazards The average pupil dilation was extracted from the eye movement data. Table 4 shows the statistical results, and Fig. 5 shows pupil dilation under different VC levels and hazard types. The permutation test-based ANOVA revealed significant main effects of both VC and hazard type (p < 0.001). In addition, the two-way interaction between VC and hazard type was also significant (p = 0.0016 and 0.0012 in the left and right eye, respectively). There was an interaction between hazard type and VC, which affected pupil responses (Fig. 5). The interaction did not affect the

4. Results 4.1. Cognitive demand required to identify different workplace hazards measured by NIRS Overall, changes of cerebral activity are more sensitive to different hazard types than to visual clutter. Oxy-Hb demands in several channels showed manifest changes in response to various hazard types (Table 2). The results showed that falling hazards induced higher cerebral and pupillary activation than all other types of hazards. A main effect of hazard type was observed on several optodes. The Oxy-Hb concentration changed significantly across different hazard types on channels 4, 13, 15, and 19 and across the average level of all channels (p = 0.084, 0.064, 0.088, 0.012, and 0.070, respectively; Fig. 3). VC showed minimal main effects on cerebral hemodynamics

Table 3 Interaction between hazard type and visual clutter on Oxy-Hb. Haz = hazard type (including electrical protection, and fire prevention and control hazards), VC = visual clutter. *: Significant at 0.05 level. VC:Haz

6

Channel

Df

Sum Sq

Mean Sq

Pr (Prob)

1 2 4 12

1 1 1 1

2.901 1.836 2.877 2.517

2.90054 1.83629 2.87747 2.51714

0.045* 0.045* 0.020* 0.046*

P.-C. Liao et al.

Safety Science 133 (2021) 105010

from channels 13 and 14 showed a positive relationship. Contrarily, in the falling hazard group, all significant relationships between Oxy-Hb and pupil dilation were negative, and the number of correlation pairs increased to a total of 8. Correlations were clearly dependent on the hazard type the workers encountered. Fig. 6 shows the correlation be­ tween pupil diameter and Oxy-Hb in different hazard types.

Table 4 Differences in pupil diameter across hazard type and visual clutter groups. Haz = hazard type (including fall protection, electrical protection, and fire preven­ tion and control hazards), VC = visual clutter. **: Significant at 0.01 level, ***: Significant at 0.001 level. VC Haz VC:Haz

Pupil

Df

Sum Sq

Mean Sq

Pr (Prob)

Right Left Right Left Right Left

1 1 2 2 2 2

6.1011 8.9815 12.6187 11.0067 7.3404 5.9404

6.1011 8.9815 6.3093 5.5034 3.6702 2.9702

0.0004 *** <2e-16 *** <2e-16 *** <2e-16 *** 0.0016 ** 0.0012 **

5. Discussion 5.1. Different cognitive sources are dependent on the variance of visual load and cognitive content in hazardous workplaces Our results revealed that cognitive activity during hazard recogni­ tion tasks varies depending on the occupational environment. Cerebral responses appear to be activated differently depending on hazard type. However, pupil diameter differs depending on both hazard type and visual clutter. In our study, only four channels showed statistically significant OxyHb changes. However, the same tendency of variation was visible in most channels (Fig. 4). Such a tendency was also observed in another study where Oxy-Hb changes were only significant in 1 out of 16 channels; however, the hemodynamics in other channels showed a similar tendency, especially in the dorsolateral PFC (Durantin et al., 2014). In this study, significant interaction effects were observed in the channels located in the right, left dorsolateral, and middle PFC. In the present study, Oxy-Hb concentration was seen to increase when encountering a falling hazard but decrease when encountering a

overall tendency of pupil responses. Thus, the effects of the interaction on pupil diameter are not discussed in subsequent sections of the article. 4.3. Hazard type-dependent variation in cognitive patterns The correlation between Oxy-Hb concentration and pupil diameter was computed during the hazard recognition process. According to the results (Table 5), the relationship was different depending on the hazard type. In the fire hazard group, Oxy-Hb concentration from channels 1, 2, and 20 showed a significant relationship with the left or right pupil diameter. Of these, the Oxy-Hb estimated from channels 1 and 2 increased as pupils dilated, whereas the Oxy-Hb from channel 2, decreased. In the case of electrical hazards, the Oxy-Hb estimated from channel 1 showed a negative relationship to pupil dilation, whereas that

Fig. 4. Effect of the interaction between hazard type and scene complexity on Oxy-Hb. Each subplot represents a channel, and the diagram distribution is consistent with the channel distribution in Fig. 2; the x-axis represents visual clutter, from high to low; the y-axis represents normalized oxygenation changes in each channel.

Fig. 5. Effect of hazard type and visual clutter on pupil diameter. For each box, the limit of the boxes indicate the first and third quartiles; whiskers represent the lowest datum within the 1.5 interquartile range of the lower quartile, and the highest datum within the 1.5 interquartile range of the upper quartile. The full color version of this figure is available online. 7

P.-C. Liao et al.

Safety Science 133 (2021) 105010

difficulty could induce activation around the anterior/medial PFC in air traffic control tasks. Similar results were obtained in the studies by Ayaz et al. (2012) and Hasan et al. (2012). Our results demonstrate that more cognitive resources were mobilized for the detection of falling hazards than for other type of hazards; we also showed that electrical hazards were associated with the average level of cognitive demand and that the lowest cognitive demand was induced by fire and control hazards. In other words, when determining whether a falling hazard exists, workers use more cognitive resources, but when they encounter a fire hazard, a decision can be made without much cognitive mediation. We measured the pupil diameter to identify different cognitive pat­ terns. Pupil diameter responded to both hazard type and visual clutter. Significant pupil dilation was visible when falling or fire hazards were encountered. By contrast, electrical hazards triggered little cognitive arousal and reduced pupil diameter. As for visual clutter, prominent effects on pupil diameter were observed. Pupil diameter was seen to significantly increase when observing a hazard in highly cluttered scenes, but to decrease in low clutter scenes. Our interpretation of this result is that pupil dilation varies as a function of the cognitive workload or attention required for the cognitive tasks (Qian et al., 2009) and emotional arousal (Bradley et al., 2008). Hence, for fall hazards or high clutter hazards, emotional or cognitive arousal might increase. In conclusion, fall hazards consume a considerable amount of

Table 5 Correlation between pupillary and cerebral responses. Ch. = channel, Coeff = coefficient, E = electrical protection hazards, F = fire prevention hazards, P = fall protection hazards, .: Significant at 0.10 level, *: Significant at 0.05 level, **: Significant at 0.01 level, ***: Significant at 0.001 level. Hazard type

Pupil-left Ch.

Coeff.

Pr (Prob)

Ch.

F

Ch2 Ch20 Ch13 Ch14

1 − 0.8318 0.3116 0.2541

0.0417 0.0673 0.0486 0.0977

Ch1 Ch5 Ch9

− 0.6964 − 0.8277 − 0.881

0.0574 . 0.0614 . 6e-04 ***

Ch1 Ch2 Ch1 Ch13 Ch14 Ch1 Ch5 Ch9

Ch17

− 0.6939

0.086 .

avg

E P

Pupil-right * . * .

Coeff.

−

− − −

0.6553 0.6461 0.2723 0.3751 0.296 0.6114 0.8361 0.7814

− 0.3378

Pr (Prob) 0.0064 ** 0.0288 * 0.0751 . 0.0078 ** 0.0445 * 0.0528 . 0.0244 * <2e-16 *** 0.0922 .

fire hazard. According to a consensus reached by several studies (Bonetti et al., 2019, Liu et al., 2016), an increase in Oxy-Hb results from an increased requirement of oxygen consumption and the resultant in­ crease in blood flow. Such oxygen consumption is thought to reflect the underlying neural activity. Durantin et al. (2014) showed that control

Fig. 6. Relationship between processing patterns of pupil diameter and cerebral activity. Subplots use normalized data without units; each point represents a participant. 8

P.-C. Liao et al.

Safety Science 133 (2021) 105010

cognitive resources, which is reflected by both Oxy-Hb concentration and pupil diameter. When detecting a fire or control hazard, participants employed less brain resources than for other hazard types and displayed specific pupillary responses. Electrical hazards contrast with fire hazards in that they induce less pupillary responses but more cerebral activation. This might suggest that falling hazards trigger a large cognitive demand as well as emotional arousal and alertness (Qian et al., 2009). Additionally, visual clutter only affected pupil size. Visual clutter has been shown to be a proxy of scene complexity (Liao et al., 2017). Pupil dilation occurs in complex and cluttered scenes, while in uncluttered and simple scenes, pupils constrict. This may result from excessive cognitive workload caused by complex workplaces (Liao et al., 2017). Such conclusions can be considered consistent with previous research claiming that neural activation is task-specific (Quaresima et al., 2005, Tupak et al., 2012). Hence, our research reveals that con­ struction workplaces with different types of hazards and scene complexity induce task-specific cognitive patterns, namely neural and pupillary responses.

Such results might arise from information in the cluttered background of electrical hazards. The high complexity of the scene would not only reduce the saliency of the target but also provide important information, thus promoting a top-down search process. The mechanism is also pre­ mised in other areas. For instance, Doyon-Poulin et al. (2014) found that medium-clutter primary flight displays induce better performance and lower workload than high-clutter or low-clutter scenarios because of the presence of visual cues in the environment. Similarly, Dowd and Mitroff (2013) reported that working memory could override salient cues dur­ ing a visual search, which implies that the processing of items in working memory takes less time, allowing for quick determination of the search target even if the background is cluttered. In conclusion, participants recognized electrical hazards with the help of information in the cluttered background, and saliency-based guidance of attention plays a more significant role in fire hazard recognition. 5.3. Neural activity and eye-movements are not correlated across different hazardous workplaces

5.2. Patterns reflecting cognitive demands induced by cluttered scenes differ depending on the hazard type

Changes in hemodynamic activity and pupil diameter revealed similar but distinct information about cognitive processes. We analyzed the correlations between the two measures to gain insight into how they respond and coordinate during hazard recognition. The results suggest that neural activity and eye-movement are not directly correlated. Positive relationships between pupil responses and neural activity from several NIRS channels were found for the detection of fire or electrical hazards. However, when dealing with fall protection hazards, the two types of cognitive activity functioned conversely, such that participants who employed a large amount of cerebral activity to iden­ tify a fire or electrical hazard would also show higher pupil dilation. This differed from the identification task of a falling hazard, in which the participants’ pupil diameter often decreased when the levels of neural activity were high. These results demonstrate that the relationship be­ tween neural activity and pupil response depends on hazard type and is not a constant occurrence. Particularly, it suggests that the cognitive response mechanisms of neural activities and visual search patterns differ substantially across different hazard types. Such findings are novel, as few studies have been conducted to decipher the relationship between oculomotor response and neural ac­ tivity in the context of occupational hazard recognition. Our results also differ from those in other areas such as facial identification and driving behaviors. Kita et al. (2010) found no correlation between scanning strategies and changes in Oxy-Hb levels for facial identification, which implies that the functions of information sampling and information processing do not occur in parallel. By contrast, Hosseini et al. (2017) found a significant association between cerebral responses in the right superior parietal lobule and pupil dilation in driving tasks. The authors suggest that neural activity in this region arises from increased atten­ tional effort and alertness for visuomotor control. By contrast, the results of the current study imply that neural activity in the PFC is not uncon­ ditionally related to visuomotor cognitive load when identifying haz­ ards. In other words, cognitive resources are utilized differently for neural activity and visuomotor control in hazard recognition tasks. The differences may arise from the demand for different cognitive resources in different tasks. Falling hazards imposed the highest cognitive demand and workload as measured by both pupillary responses and cerebral activity, sug­ gesting a negative relationship between the two types of psychological response that could stem from limited cognitive resources. In this case, the simultaneous response of pupil diameter and cerebral activities in the PFC could reflect the lower cognitive demands for the electrical and fire hazards. However, in case of cognitive overload during the recog­ nition of fall hazards, the limited cognitive resources have to be allo­ cated economically to find a balance between neural and pupillary responses. This celling effect is fairly consistent with other theories

We found significant interactions between hazard type and visual clutter when participants allocated cognitive resources for hazard recognition. This reveals that the cognitive demands induced by clut­ tered scenes differ depending on the hazard type. In other words, although the main effect from scene complexity was not significant for cerebral activity when detecting hazards, this does not mean that cere­ bral activation was the same with each scene complexity. The interac­ tion effect with hazard type was prominent, which means that complex scenes did not always induce a high cognitive demand. Cerebral acti­ vation measured by Oxy-Hb concentration also differed between hazard types (specifically between fire and electrical hazards). The interaction between hazard type and scene complexity on ce­ rebral activation indicated that during the recognition of fire hazards, cortical oxygenation intensified when the scene was more cluttered. However, with electrical hazards, cognitive demands decreased in complex scenes. This indicates that a large amount of cognitive energy was utilized in complex scenes containing fire hazards and simple scenes containing electrical hazards. This provides original information about the effect of scene complexity on hazard recognition. Rather than simply obstructing visual search processes by creating distractions and pre­ venting effective attention, the results of our study reveal that the effect of visual clutter on hazard recognition is dependent on hazard type. When detecting fire hazards, high visual clutter led to a high cognitive load, while for electrical hazards, high visual clutter required less cognitive energy than low visual clutter. The two hazard-related dependent variables (hazard type and visual clutter) essentially reflect two kinds of visual search strategies, namely saliency-based and scene context-based strategies. Variety in visual clutter leads to different levels of saliency; thus, bottom-up guidance for attention is different (Theeuwes, 2010). However, different hazard types create disparate contexts for participants and hence trigger different topdown attentional guidance (Hout and Goldinger, 2015, Treisman and Gelade, 1980). Both strategies induce cognitive load for visual search tasks. In the context of these cognitive strategies, the results of this study suggest that saliency may lead to increased, rather than decreased, cognitive load. In other words, background distractions do not always cause search difficulty. Sometimes, distracting scenes may lead to easy discovery due to the presence of more information and convenience for contextual cueing. A practical interpretation is that cluttered and com­ plex fire hazard scenes induce more cognitive resources for cerebral activity, leading to cognitive overload and task difficulty for the ob­ servers. By contrast, cluttered electrical hazard scenes induce a lower cognitive load compared with uncluttered scenes, suggesting that it is easier for observers to recognize electrical hazards in cluttered scenes. 9

P.-C. Liao et al.

Safety Science 133 (2021) 105010

recognition. The fixation features recorded by the eye trackers imply the allocation and shift of attention. Attention paid, however, does not unidirectionally correlate with cognitive processes and demands, as it can be interpreted from two aspects: interest and difficulty. Thus, employing NIRS devices could help to acquire knowledge about how different workplace hazards invoke cognitive demands for workers and how workers respond to the hazards. As there are few studies using NIRS to study occupational safety, this study provided a valid means of measuring cerebral activity in the PFC to reveal cognitive patterns during hazard recognition. The primary measurement of PFC regions in driving safety behaviors is an important basis for setting the target region in this study. The significant results confirmed the validity of this technique, pointing to the value of this approach for future research. This study also offered a new type of experimental setting that sim­ ulates a natural hazardous environment. Rather than displaying haz­ ardous scenarios through images or videos, this study was implemented in a civil laboratory, which guaranteed a semi-naturalistic environment that is both static and natural simultaneously. The civil laboratory ful­ filled the requirements of naturally hazardous workplaces and acted as a continuous setting for several days during the period of experimenta­ tion. A realistic setting in a lab makes it possible to explore the work­ place freely, constantly, and guardedly. Such experimental arrangement can serve as a reference for future research conducted in natural scenes to provide more concise and deep knowledge of human behavior.

related to limited cognitive resources. For example, according to the selective attention theory developed by Nilli (2005), the processing of distractors would be increased under a low perceptual load, because the low perceptual load does not consume all the available resources. However, the squandering of finite resources under high perceptual load would lead to poor performance. Similarly, Granholm et al. (1996) also reported the effect of limited processing resources. In their experiment, pupil responses increased with processing demands that were below the resource limits. When processing demands exceeded the available re­ sources, pupil responses declined. Thus, the results of this study could be explained by the celling effect of limited cognitive resources, and may also act as evidence of the complementary function of pupillary dilation and cerebral activity. 5.4. Theoretical significance on hazard recognition The main theoretical contribution of this study lies in the finding that different cognitive patterns are observed in the presence of different types of hazards in a simulated worksite. First, different hazard types induce different levels of cerebral acti­ vation. Generally, falling hazards proved to be the most cognitive demanding tasks in terms of cerebral activity. In addition, falling haz­ ards induced significant pupil dilation. These results illustrate that, compared with electrical and fire-related hazards, more cognitive re­ sources were allocated to recognizing falling hazards, possibly by increasing cognitive-workload demands and emotional arousal. Second, scene complexity measured by visual clutter had significant impact on cognitive demands, as indicated by the dilation of pupils when the background to hazards was complex. This might imply that cluttered scenes induce mental overload or alertness (Qian et al., 2009, Tang et al., 2018, Zennifa et al., 2018). Additionally, scene complexity triggered different levels of cerebral activity through its interaction with hazard type. This might have resulted from the different effects of background distraction in different hazard contexts. Backgrounds of electrical hazards trigger a large amount of cognitive resources, creating attentional distraction. By contrast, fire hazard background information helps to promote recognition and thus requires less cognitive resources. Moreover, the present findings indicated that pupillary response and cerebral activity in the PFC function simultaneously when a constrained amount of cognitive resources is utilized for electrical and fire hazards. Nonetheless, when more cognitive resources are used for falling hazards, only one sort of cognitive response can act, probably because of the constrained total amount of cognitive resources (Granholm et al., 1996; Nilli, 2005). This could provide evidence of the complementary function of pupillary responses and cerebral activity.

5.6. Limitations and future research This study aimed to expand the experimental methods for hazard recognition research and reveal different cognitive patterns in work­ places. Although different cognitive responses as measured by pupil diameter and cerebral Oxy-Hb signals were observed and discussed, more efforts and exploration are required in the future. First, participants were treated as one group, and the psychological signals were averaged among the participants. It is worthwhile to compare cognitive patterns between different groups, such as experi­ enced/novice or capable/incapable. Second, cerebral activities in the PFC were monitored in this pilot study because this region has been proven to be associated with hazard perception in driving behaviors. Furthermore, this region has previously been shown to be unaffected by signal interference from hair. Nevertheless, some researchers have found that other brain regions, such as the parietal lobule, are also significant for cognitive processes (Liu et al., 2016). Observing neural hemodynamics in more brain regions may also be useful. In addition, certain limitations were induced by the experimental settings. In this study, only one search route was admitted to avoid the impacts from the selection of the route and the order of hazards. Moreover, the hazards were rather specific to represent the entire category. The conclusions should be carefully interpreted and should not be immediately gener­ alized beyond the construction workplaces and hazard types employed in this study. The different cognitive patterns revealed in this study could be the basis for future studies aimed at identifying the psychological mecha­ nisms of the cooperation between different types of cognitive functions such as cerebral activity and pupillary responses.

5.5. Methodological innovation This study was based on an innovative experimental protocol. The understanding of hazard recognition processes revealed from the experiment confirms the value of the design. First, it utilized multimodal monitoring technology to measure cognitive demands during hazard recognition processes. An eye tracker was employed to record oculo­ motor responses, in conjunction with a series of NIRS devices to measure cerebral activation levels. The results proved that the utilization of NIRS technology was worthwhile as it could reveal more information than the use of eye-tracking alone. In fact, NIRS could be a competitive indicator for cognitive processes as it directly reflects the levels of neural activity. In addition, the interpretation of the measurement variables, namely the oxygenated hemoglobin concentration, has been acknowledged. An in­ crease in oxygenated hemoglobin concentration is usually regarded as a signal of activation in the brain (Bonetti et al., 2019). Furthermore, cerebral activation is directly associated with cognitive processes. However, the limited information revealed by eye-tracking devices could be attributed to the oblique correlation of eye-tracking results with the judgement and decision-making phases during hazard

6. Conclusions This study explored the cognitive demands of the recognition of different types of hazards in simulated worksites using multimodal monitoring technology. The contributions of this study lie in the meth­ odology and the results, which are complemented by theoretical knowledge on cognitive processes involved in hazard recognition in construction workplaces. The results of the study suggest that simulated construction worksites with different hazard types and levels of scene complexity induce different cognitive patterns and cognitive demands 10

P.-C. Liao et al.

Safety Science 133 (2021) 105010

and should thus be treated individually. In addition, the use of multi­ modal monitoring in experimental protocols conducted in realistic hazard scenarios helps bridge existing gaps in research methodology.

Hout, M.C., Goldinger, S.D., 2015. Target templates: the precision of mental representations affects attentional guidance and decision-making in visual search. Attention Perception & Psychophysics 77, 128–149. Hu, M., Shealy, T., Hallowell, M., Hardison, D., 2018. Advancing construction hazard recognition through neuroscience: measuring cognitive response to hazards using functional near infrared spectroscopy. Presented at Construction Research Congress 2018. Indrarathne, B., Kormos, J., 2017. The role of working memory in processing L2 input: Insights from eye-tracking. Bilingualism Language & Cognition 21, 20. Jeelani, I., 2016. Personalized Hazard Recognition Training for Construction workers. Kita, Y., Gunji, A., Sakihara, K., Inagaki, M., Kaga, M., et al., 2010. Scanning strategies do not modulate face identification: eye-tracking and near-infrared spectroscopy study. PLoS ONE 5 e11050. Klesel, M., Schuberth, F., Henseler, J., et al., 2019. A test for multigroup comparison using partial least squares path modeling. Internet Res. https://doi.org/10.1108/ IntR-11-2017-0418. Kushiro, N., Fujita, Y., Aoyama, Y., 2017. Extracting Field Oversees’ Features in Risk Recognition from Data of Eyes and Utterances, pp. 590–596. Liao, P.-C., Sun, X., Liu, M., Shih, Y.-N., 2017. Influence of visual clutter on the effect of navigated safety inspection: a case study on elevator installation. Int. J. Occup. Saf. Ergonomics 1–15. Liu, T., Pelowski, M., Pang, C., Zhou, Y., Cai, J., 2016. Near-infrared spectroscopy as a tool for driving research. Ergonomics 59, 368–379. McSween, T.E., 2003. Values-based safety process: improving your safety culture with behavior-based safety, 2nd Edition. Mehta, R.K., Parasuraman, R.J.F.i.H.N., 2013. Neuroergonomics: a review of applications to physical and cognitive work. 7, 889. Mitsuo, S., Ichiro, M., Takeshi, O., Ichiro, O., Ikuo, K., et al., 2004. Prefrontal and premotor cortices are involved in adapting walking and running speed on the treadmill: an optical imaging study. Neuroimage 23, 1020–1026. Miyai, I., Tanabe, H.C., Sase, I., Eda, H., Oda, I., et al., 2001. Cortical mapping of gait in humans: a near-infrared spectroscopic topography study. Neuroimage 14, 1186–1192. Namian, M., Albert, A., Zuluaga, C.M., Behm, M., 2016. Role of safety training: impact on hazard recognition and safety risk perception. J. Constr. Eng. Manage. 142, 04016073. Nilli, L., 2005. Distracted and confused?: selective attention under load. Trends in Cognitive Sci. 9, 75–82. Pandit, B., Albert, A., Patil, Y., Al-Bayati, A.J., 2019. Impact of safety climate on hazard recognition and safety risk perception. Saf. Sci. 113, 44–53. Perlman, A., Sacks, R., Barak, R., 2014. Hazard recognition and risk perception in construction. Saf. Sci. 64, 22–31. Pinto, N., Cox, D.D., Dicarlo, J.J., 2008. Why is real-world visual object recognition hard? PLoS Comput. Biol. 4 (1) e27. Privitera, C.M., Renninger, L.W., Thom, C., Stanley, K., Mario, A.J.J.o.V., 2010. Pupil dilation during visual target detection. 10, 3. Qian, M., Aguilar, M., Zachery, K.N., Privitera, C., Klein, S., et al., 2009. Decision-level fusion of EEG and pupil features for single-trial visual detection analysis. IEEE Trans. Biomed. Eng. 56, 1929–1937. Quaresima, V., Ferrari, M., Torricelli, A., Spinelli, L., Pifferi, A., Cubeddu, R., 2005. Bilateral prefrontal cortex oxygenation responses to a verbal fluency task: a multichannel time-resolved near-infrared topography study. J. Biomed. 10, 11012. Rasmussen, J., 1997. Risk management in a dynamic society: a modelling problem. Saf. Sci. 27 (2), 183–213. Rowlinson, S., Yunyanjia, A., Li, B., Chuanjingju, C., 2014. Management of climatic heat stress risk in construction: A review of practices, methodologies, and future research. Accid. Anal. Prev. 66, 187–198. Rozenfeld, O., Sacks, R., Rosenfeld, Y., Baum, H., 2010. Construction job safety analysis. Saf. Sci. 48, 491–498. Sun, X., Chong, H.Y., Liao, P.C., 2018. Efficiency improvement by navigated safety inspection involving visual clutter based on the random search model. Int. J. Occup. Saf. Ergonomics Jose 1. Sun, X., Liao, P.C., 2019. Re-assessing hazard recognition ability in occupational environment with microvascular function in the brain. Saf. Sci. 120, 67–78. Suraji, A., Duff, A.R., Peckitt, S.J., 2001. Development of causal model of construction accident causation. J. Constr. Eng. Manage. 127 (4), 337–344. Tang, R., Kim, J.H., Parker, R., Jeong, Y.J., 2018. Indicating severity of vehicle accidents using pupil diameter in a driving simulator environment. In: Digital Human Modeling. Applications in Health, Safety, Ergonomics, and Risk Management, pp. 647–656. Theeuwes, J., 2010. Top-down and bottom-up control of visual selection. Acta Psychol. 135, 77. Tixier, A.J.P., Hallowell, M.R., Albert, A., van Boven, L., Kleiner, B.M., 2014. Psychological antecedents of risk-taking behavior in construction. J. Constr. Eng. Manage. 140. Treisman, A.M., Gelade, G., 1980. A feature-integration theory of attention. Cogn. Psychol. 12, 97. Tupak, S.V., Badewien, M., Dresler, T., Hahn, T., Ernst, L.H., et al., 2012. Differential prefrontal and frontotemporal oxygenation patterns during phonemic and semantic verbal fluency. Neuropsychologia 50, 1565–1569. Zennifa, F., Ageno, S., Hatano, S., Iramina, K., 2018. Hybrid system for engagement recognition during cognitive tasks using a CFS + KNN algorithm. Sensors (Basel) 18. Zhu, Y., Rodriguez-Paras, C., Rhee, J., Mehta, R.K., 2019. Methodological approaches and recommendations for functional near-infrared spectroscopy applications in HF/ E research. Hum. Factors, 18720819845275.

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements Funding: This work was supported by the National Natural Science Foundation of China [grant number 51878382]; We would like to thank Prof. Pan Peng for his help in laboratory arrangement, and the safety experts, Wang Zonggang and Jin Tongle, for their participation and suggestions in the pre-experiment. References Albert, A., Hallowell, M.R., Kleiner, B.M., 2014. Enhancing construction hazard recognition and communication with energy-based cognitive mnemonics and safety meeting maturity model: multiple baseline study. J. Constr. Eng. Manage. 140. Albert, A., Jeelani, I., Han, K., Azevedo, R., 2018. Are Visual Search Patterns Predictive of Hazard Recognition Performance? Empirical Investigation Using Eye-Tracking Technology. Atefeh, M., Mehdi, K., Ebrahim, K., Michael, B., Ann, M., 2018. Measures of Mental Alertness of Construction Workers to Enhance Job Site Safety. Presented at Construction Research Congress. Ayaz, H., Cakir, M.P., Izzetoglu, K., Curtin, A., Onaral. IEEE Aeroscpace Conference2012. Bahn, S., 2013. Workplace hazard identification and management: The case of an underground mining operation. Saf. Sci. 57, 129–137. Bonetti, L.V., Hassan, S.A., Lau, S.-T., Melo, L.T., Tanaka, T., et al., 2019. Oxyhemoglobin changes in the prefrontal cortex in response to cognitive tasks: a systematic review. Int. J. Neurosci. 129, 195–203. Borys, D., 2012. The role of safe work method statements in the Australian construction industry. Saf. Sci. 50, 210–220. Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J., 2008. The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45, 602–607. Carter, G., Smith, S.D., 2006. Safety hazard identification on construction projects. J. Constr. Eng. Manage. 132, 197–205. Chen, J., Song, X., Lin, Z., 2016. Revealing the “Invisible Gorilla” in construction: Estimating construction safety through mental workload assessment. Autom. Constr. 63, 173–183. Delpy, D.T., 1997. Cope MJPTotRSBBS. Quantification in Tissue Near-Infrared Spectroscopy. 352, 649–659. Dowd, E.W., Mitroff, S.R., 2013. Attentional guidance by working memory overrides salience cues in visual search. J. Exp. Psychol. Hum. Percept. Perform. 39, 1786–1796. Doyon-Poulin, P., Ouellette, B., Robert, J.-M., 2014. Effects of visual clutter on pilot workload, flight performance and gaze pattern. Presented at Proceedings of the International Conference on Human-Computer Interaction in Aerospace - HCI-Aero ’14. Durantin, G., Gagnon, J.F., Tremblay, S., Dehais, F., 2014. Using near infrared spectroscopy and heart rate variability to detect mental overload. Behav. Brain Res. 259, 16–23. Dzeng, R.-J., Lin, C.-T., Fang, Y.-C., 2016. Using eye-tracker to compare search patterns between experienced and novice workers for site hazard identification. Saf. Sci. 82, 56–67. Gary, S., Culver, J.P., Thompson, J.H., Boas, D.A., 2002. A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. J. Neuroimage 17, 719–731. Giannakos, M.N., Sharma, K., Pappas, I.O., Kostakos, V., Velloso, E., 2019. Multimodal data as a means to understand the learning experience. Int. J. Inf. Manage. 48, 108–119. Granholm, E., Asarnow, R.F., Sarkin, A.J., Dykes, K.L.J.P., 1996. Pupillary responses index cognitive resource limitations. Psychophysiology 33, 457–461. Hasan, A., Shewokis, P.A., Scott, B., Kurtulus, I., Ben, W., Banu, O.J.N., 2012. Optical brain monitoring for operator training and mental workload assessment. 59, 36–47. Hasanzadeh, S., Esmaeili, B., Dodd, M.D., 2016. Measuring construction workers’ realtime situation awareness using mobile eye-tracking, pp. 2894–2904. Hasanzadeh, S., Esmaeili, B., Dodd, M.D., 2017. Impact of construction workers’ hazard identification skills on their visual attention. J. Constr. Eng. Manage. 143, 04017070. Haslam, R.A., Hide, S.A., Gibb, A.G., Gyi, D.E., Pavitt, T., Atkinson, S., et al., 2005. Contributing factors in construction accidents. Appl. Ergon. 36 (4), 401–415. Hosseini, S.M.H., Bruno, J.L., Baker, J.M., Gundran, A., Harbott, L.K., et al., 2017. Neural, physiological, and behavioral correlates of visuomotor cognitive load. Sci. Rep. 7, 8866.

11