Personalized thermal comfort inference using RGB video images for distributed HVAC control

Applied Energy xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Applied Energy journal homepage: www.elsevier.com/locate/apenergy

Personalized thermal comfort inference using RGB video images for distributed HVAC control ⁎

Farrokh Jazizadeh , Wooyoung Jung 200 Patton Hall, 750 Drillfield Drive, The Charles E. Via Department of Civil and Environmental Engineering, Virginia Tech, Blacksburg, VA 24061, United States

H I G H L I G H T S of a ubiquitous thermal comfort assessment for energy-efficient HVAC. • ATheframework infers human thermoregulation states using RGB video images. • The framework draws on thermoregulation mechanisms and Eulerian video magnification. • Subtleframework blood flow variations to facial skin due to thermoregulation are inferred. • The feasibility was evaluated for 21 participants under low and high temperatures. •

A R T I C L E I N F O

A B S T R A C T

Keywords: Energy efficiency HVAC system Personalized thermal comfort Thermoregulation mechanism Blood perfusion Eulerian video magnification

HVAC systems account for more than 40% of energy consumption in buildings to provide satisfactory indoor environments for occupants. The integration of personalized thermal comfort in the operation of HVAC systems has been shown to be highly effective in enhancing energy efficiency of buildings. To this end, research efforts have proposed personalized thermal comfort assessment through voting (i.e., occupant feedback) and profiling as well as physiological response measurement. In this study, we have proposed a novel approach for enabling RGB video cameras as sensors for measuring personalized thermoregulation states – an indicator of thermal comfort. If their feasibility for thermoregulation state inference could be established, optical cameras provide a cost-effective and omnipresent solution for distributed measurement of thermal comfort and consequently control of HVAC systems for energy saving. Accordingly, we have proposed a framework that draws on the concepts of thermoregulation mechanisms in the human body as well as the Eulerian video magnification approach. The framework is composed of several components including face detection, skin pixels isolation, image magnification. And calculation of detection index to infer subtle blood flow variations to the facial skin surface (i.e., blood perfusion), which is due to thermoregulation adjustments. In order to minimize the impact of variable illumination condition and the ambient noise on the results, different combinations of methods for framework components were taken into account. The feasibility assessments were conducted through an experimental study with 21 participants under low (20 °C) and high (30 °C) temperatures. In total, 16 positive cases out of 18 statistically significant cases were observed resulting in 89% of success rate using the most promising combinations of the methods. The results demonstrate that the proposed framework could contribute to realization of a non-intrusive, cost-effective, and ubiquitous distributed thermal comfort assessment that has been proven critical in increasing energy efficiency of the HVAC system through distributed control feedback.

1. Introduction Increasing energy efficiency in buildings, as the major consumer in the United States [1], is of critical importance in achieving sustainability goals in the built environment. Thermal conditioning in buildings account for 48% of annual energy consumption in the United States [2]. The control logic in buildings relies on measuring

⁎

temperature variations in a thermal zone (i.e., sub-spaces in buildings with independent control units). This measurement represents the indoor thermal condition that is compared against a control set point - a quantity that is determined through thermal comfort indexing. The main objective of thermal conditioning is to provide and maintain occupants’ health, comfort, and productivity in an indoor environment. Therefore, thermal comfort quantification is critical in determining the

Corresponding author. E-mail addresses: [email protected] (F. Jazizadeh), [email protected] (W. Jung).

https://doi.org/10.1016/j.apenergy.2018.02.049 Received 6 September 2017; Received in revised form 17 December 2017; Accepted 8 February 2018 0306-2619/ © 2018 Elsevier Ltd. All rights reserved.

Please cite this article as: Jazizadeh, F., Applied Energy (2018), https://doi.org/10.1016/j.apenergy.2018.02.049

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

energy consumption demands in buildings [3–7]. Conventionally, thermal comfort is measured by using heat-budget models. These models look at the balance of the human heat budget to variable environmental and metabolic heat loads. This balance is controlled by an efficient (for healthy people) autonomous thermoregulatory system [8]. The most widely adopted heat-budget model is the PMV-PPD model [9] that was developed through comprehensive empirical studies in controlled environments. The model quantifies thermal comfort and the percentage of satisifed occupants through predictive mean vote (PMV) and predicted percentage of dissatisfied (PPD) indices, respectively. Although the PMV-PPD model has been used as the main approach in thermal comfort quantification, studies have shown that it could result in underestimating or overestimating personalized thermal comfort [10,11]. Nonetheless, in the wake of difficulty of quantifying the driving factors of PMV-PPD model in operational stage, conservative operational set points are commonly considered for thermal conditioning operations, which consequently lead to inefficiency of building systems’ energy consumption and serviceability. The heat budget models, coupled with temperature sensing in thermal zones comprise the control logic of the heating, ventilation, and air conditioning (HVAC) systems. Accordingly, the control logic relies on average thermal satisfaction of a group of occupants in thermal zones that is communicated to building systems through a single sensing point as a temperature set point. This configuration could result in inefficient communication and temperature distribution in thermal zones and consequently, inefficient energy consumption. In order to overcome these limitations, distributed thermal comfort measurement has been introduced as an alternative solution [12], in which, through a field demonstration, we have shown that integration of personalized thermal comfort into HVAC control logic, could result in considerable energy consumption reduction (∼40% in the test bed). The ubiquity of smart and personalized computing devices and the wireless communication networks enabled the application of personalized metrics for thermal comfort assessment. Occupants’ feedback for thermal comfort profile learning [12,13] as well as wearable sensors [14,15] for measuring ambient and physiological variables were among the main techniques that have been adopted for cyber-physical systems with an emphasis on human-centered control techniques. As the effectiveness of the techniques for evaluating personalized thermal comfort increases, the intrusiveness of the methods could be increased. Moreover, specialized devices will be needed to improve the quality of thermal comfort quantification, which is a challenge for ubiquitous and pervasive measurements. This observation brought us to the question that whether we could leverage optical video images via computer webcams for thermal comfort quantification for control feedback. Building occupants (specifically in office/administrative buildings) commonly interact with personal computers with connected video devices, which provide a non-intrusive platform for thermal comfort assessment. The main question, therefore, is whether computers can quantify human thermal comfort through RGB video images. Although video image analysis has been previously used in medical applications [5], its application in thermal comfort assessment has not been explored. This paper describes our study, in which we leveraged the human body thermoregulation mechanism and the Eulerian video magnification algorithm to devise a framework for identification of human thermoregulation states and evaluate the feasibility of inferring these states through RGB video images. The assessments in this study have been made under the constraints of control loops for building energy management systems.

Fig. 1. Research spectrum of thermal comfort quantification.

expenditure) of the HVAC systems have led to numerous research efforts in optimizing the operation of these systems. As an active research area in the past few decades [16–22], among the recent research efforts, several studies have focused on advanced control techniques to improve the operation of HVAC systems with comfort and energy objective functions. Advanced fuzzy logic controllers (e.g., [20]), metaheuristic optimization algorithms (e.g., [23]), and model predictive control (MPC) strategies (e.g., [24]) were proposed and evaluated for increased energy efficiency of HVAC systems. However, in these studies, PMV has been widely used as the metric for evaluating the thermal comfort of the users in the environment. Regardless of the control mechanism, the thermal comfort quantification is a critical component of energy management frameworks. Fig. 1 illustrates the comparative spectrum of techniques for quantification of thermal comfort that have been developed over the past few decades. As also reflected in the aforementioned studies, the PMV-PPD model is the most notable and commonly used thermal comfort quantification model. Although it is a completely non-intrusive approach, comparing the results of PMV-PPD model and occupants’ reported thermal preferences shows the overestimating or underestimating of this model [25–27]. The main reason for this discrepancy is that the PMV-PDD model does not account for individual occupants’ characteristics [10,11] and it uses a collective measure of thermal satisfaction through one sensing point (i.e., thermostat). Estimating human factors including metabolic rates and clothing insulation is often difficult. Considering the importance of human related variables in accuracy of thermal comfort models [28], a wide range of research studies has been conducted to assess thermal perception for individuals. In general, these methods could be grouped into two categories as also depicted in Fig. 1. Methods in the first category directly ask occupants to express their thermal satisfaction through participatory data acquisition (i.e., occupant voting and profiling systems - OVPS). However, approaches in the second group seek to measure physiological variables and then obtain a correlation between these variables and thermal comfort perception (i.e., physiological sensing technologies – PST). In OVPS, participatory based solutions, in conventional form, include collecting feedback through surveys [29] or interviews [30] to understand occupants’ perception. These methods are periodic and are mainly used to evaluate the performance of building energy management systems (BEMS) in case of complaints. To account for contextual dynamics (including occupants’ dynamics and seasonal changes) and leverage occupants’ adaptive capacity, personalized participatory sensing solutions (including our prior research [12,31,32]), were introduced to enable continuous sensing solutions for quantification of thermal comfort. In these approaches, ubiquitous devices such as smartphones [13,31,33,34] and personal computers [35,36] are employed to enable context-aware feedback from occupants

2. Thermal comfort management background The main criterion in design and operation of air conditioning systems is to ensure that an acceptable indoor air condition is provided for the occupants. Provision of thermal comfort and ventilation under the constraints of procurement and operational costs (i.e. energy 2

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

question was explored by looking at the human body thermoregulation processes that could be used to quantify these processes.

and integrate them with adaptive fuzzy [12,37] or probabilistic models [36,38] to create context-aware personalized thermal comfort models through on-line learning algorithms. These information sources are commonly coupled with optimization techniques for energy and/or thermal comfort optimization. Although these methods could not be considered as intrusive methods, their application could be disruptive, as they require user-BEMS continuous interaction. This is specifically an important aspect, when we consider seasonal variations. An alternative approach in PST category, that has been also gaining attention in recent years, is the application of wearable sensors to infer thermal comfort by measuring physiological variables such as skin temperature and galvanic skin response through skin-sensor contact. Studies have shown that overall-body thermal sensation can be predicted based on subjects’ mean skin temperature [14,39,40]. A few studies measured more than one physiological variable, including skin temperature, heart rate, electroencephalograph (electrical activity of the brain) and blood pressure to test physiological signals’ efficacy on overall thermal comfort sensation and acceptability [41,42]. As the features (i.e., variables), used in these studies, imply, the measurement techniques are intrusive considering the fact that sensors need to be in contact with human body on the skin surface. Moreover, the optimum and reliable results could depend on measurements at several locations on human body [43], which could bring about further discomfort. As noted earlier and illustrated in Fig. 1, review of the literature shows that by leveraging more advanced and context-aware techniques for evaluating personalized thermal comfort, an increased trend of intrusiveness is observed. Intrusiveness in this context has two dimensions: (i) level of the need for user engagement and (ii) the need for skin contact. OVPS techniques call for continuous feedback and model updates from users. Seasonal changes and change in clothing level could affect the accuracy of the models that use OVPS techniques. On the other hand, commonly used PST techniques not only call for additional hardware, but also they commonly require skin contact. As a non-intrusive approach that takes both dimensions into account, some studies have used infrared images to measure skin temperature variations and associate that with the ambient thermal condition changes [44–47]. Although the approach is promising considering the high correlation between skin temperature and thermal comfort, this methodology still calls for additional/specialized hardware systems, which make its wide adoption challenging. Aforementioned novel techniques for thermal comfort quantification in energy management systems address the challenges associated with pre-defined set points. However, in majority of the research efforts, these techniques have been studied in the context of existing HVAC system configuration, namely, single sensing point and maximum capacity operation. Therefore, this study contributes by seeking to achieve the objectives of non-intrusive personalized and distributed thermal comfort measurement that directly receives human body thermoregulation feedback. Achieving these objectives could pave the way for a shift in HVAC control logic that relies directly on the thermoregulation process for energy management without the need for additional and specialized sensing techniques.

3.1. Human thermoregulation process Thermoregulation in human body is correlated with change in the ambient condition. Thermoregulation is the process that allows human body to maintain its internal temperature close to 37 °C [48]. Thermoregulation is a dynamic equilibrium (not a steady state) with the environment, in which body regulates its internal temperature in terms of heat generation and heat exchange with the environment. In other words, when body temperature increases, the thermoregulation system triggers the heat loss; in contrast, while the body temperature is falling, the body increases heat production. The general process of thermoregulation is as follows: (1) thermal sensors, distributed across the skin, receive cold or warm signals, (2) brain integrates these signals with internal body temperature, and triggers the thermal adjustment mechanism to maintain the internal body temperature. Body loses heat through skin by two mechanisms, first regulation of blood vessels, and, if required, sweating. Vasodilation is a process, during which the smallest blood vessels under skin are instructed to dilate, or open. During the vasodilation, the blood vessels are enlarged in order to enhance the blood flow at the skin surface. Increasing the blood flow results in loss of heat from the skin into the environment. Sweating is a process which lets body to be cooler by evaporation. On the other hand, the heat is preserved by vasoconstriction and, if necessary, shivering. Vasoconstriction constricts blood vessels under the skin in order to decrease blood flow, gain heat and return internal body temperature back to normal. Shivering, which can be voluntary or involuntary, moves the muscles in order to produce heat. In addition to mentioned mechanisms, hormonal regulation is another way to rise metabolism and the amount of heat in the body to increase internal temperature [48,49]. In the normal indoor temperature ranges, the process of heat adjustment control will be associated with variations of blood flow to the skin surface (i.e., blood perfusion). We hypothesized that we could benefit from blood perfusion process to infer the state of the thermoregulation in the body. The process of delivering blood to the capillary bed in a biological tissue is called blood perfusion process. As users work in the environment sitting in front of their computers, images of head area and facial skin could be captured through the connected cameras. Variations in ambient temperature conditions will result in blood perfusion variation. If the variation of blood flow to the skin temperature could be measured, the thermoregulation state could be inferred. In order to achieve this objective, the variation of the blood flow should be detected by using image-processing techniques that are capable of detecting subtle changes in blood flow from RGB video images. 3.2. Image magnification techniques Photoplethysmography (PPG) technique, often operates with a red or infrared wavelength, has been widely used in healthcare applications as an optical measurement of physiological processes such as blood pressure, oxygen saturation, and blood volume changes [50]. PPG measures the variations in transmitted or reflected light associated with changes in perfusion in the catchment volume so the aforementioned physiological processes can be obtained. Poh et al. [51] and Verkruysse et al. [52] extended the PPG’s application through identifying human heart rate by using a remotely observed facial video, recorded by a webcam under normal ambient light. In other words, they have established a non-intrusive approach of using the PPG technology . Wu et al. [53], motivated by the previous development of the PPG techniques, developed the Eulerian Video Magnification (EVM) algorithm that amplifies the subtle variations in color or movement and visualizes the hidden information such as the perfusion of blood to the skin or a subtle

3. Fundamental concepts and methodologies As noted, in proposing a non-intrusive approach by leveraging ubiquitous computing devices for energy management in buildings, we sought to explore the capacity of video acquisition on personal computing devices. Building users, especially in office buildings, spend majority of their time in front of a computer, which provides an opportunity for developing monitoring technologies that use processed video images as an input. Fig. 2 illustrates the conceptual design of the framework that integrates information from video images into the operational process of HVAC systems. Realization of this concept brought us to a fundamental research question: whether thermal comfort could be quantitatively measured by video images? The answer to this 3

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Fig. 2. Conceptual framework of video-image based thermal comfort indexing and its integration with HVAC operations.

movement, caused by heartbeat. The EVM algorithm amplifies captured signal from a physical or physiological process by spatial-temporal filtering to enable visualization and a clear demonstration for subtle events. Although, the main motivation behind the EVM algorithm development is to enable visualization, in this study we have focused on its capabilities for revealing the differences between subtle variations of blood perfusion in different temperature ranges in order to identify human body reaction to change in temperature. 3.2.1. Eulerian video magnification (EVM) algorithm The EVM algorithm [53] performs spatial and temporal image processing to accentuate subtle temporal changes in a video. For spatial processing, the Laplacian pyramid [54], representing an image as a series of images with successively sparser densities, is applied to increase temporal signal-to-noise ratio of a video and reduce the computational complexity. Then, in order to extract the frequency bands of interest, temporal processing is performed at each spatially sorted image. For example, in our application, we are interested in heart beat rate, which could be between 0.4 and 4 Hz (24 – 240 beats per minute) to be applied as a band-pass filter. After the temporal processing, the filtered signal is magnified by an amplification factor specified by a user. Then, the magnified signal is added to the original image to obtain the final output. Given that the color intensity of a pixel at position x and frame t is CI (x ,t ) and the displacement due to the motion within the band-pass filter D (t ) . Then, the CI (x ,t ) can be represented by

CI (x ,t ) = f (x + D (t ))

Fig. 3. The amplified color variance by the EVM algorithm compared to the original color variance, reproduced from [53]

of images within the Laplacian pyramid), and the amplification factor. We seek to infer the blood perfusion state through tracking of skin color variations, associated with the thermoregulation states. Accordingly, the parameter selection for the EVM framework should reflect the characteristics of cardiovascular system. Using EVM as a component, in this study, we have presented our proposed framework for facial video processing to assess the feasibility of inferring human thermoregulation states from RGB video images.

(1)

By using a first-order Taylor series expansion, the changes in color intensity can be approximated as,

CI (x ,t ) = f (x ) + D (t )

∂f (x ) ∂x

(2)

4. Thermoregulation state identification framework

Using the amplification factor α , the approximated color intensity can be magnified and added to the color intensity CI (x ,t ) . Then we have

CI (x ,t ) = f (x ) + (1 + α ) D (t )

∂f (x ) ∂x

As noted, our grand vision in this research focuses on a novel energy management paradigm for HVAC systems that uses humans as sensor proxies for feedback from the environment. Distributed feedback from an environment could lead to energy conservation by providing more realistic distribution of temperature. Moving towards this vision, by leveraging the video magnification algorithm as the base approach for optical-image-based thermal comfort assessment, thermoregulation state identification process calls for addressing multiple challenges and artifacts in the input videos. Hence, the proposed framework is composed of the processes that tackle obstacles step by step. Our envisioned

(3)

The above equations explain a one-dimensional signal case but it can be generalized in a two-dimensional space. Fig. 3 shows the original color intensity, f (x ) , changing in time domain and the amplified color intensity by the EVM algorithm. A user of the EVM algorithm decides the frequency bands of interest (temporal band-pass filtering), the spatial frequency cutoff (the number 4

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Fig. 4. Same human subject, under the same lighting configuration, with a clear illuminance difference.

framework uses off-the-shelf webcams as the sensor for monitoring human subjects’ thermoregulation state. Considering normal conditions of using a computer, the webcam is commonly faced towards the users’ face. Given this operating condition, in a captured video, multiple unwanted video input will be included, which contains objects from the background, as well as, facial features that not only do not contribute to identification of the thermoregulation state, but also could have a negative effect. Examples of these features include hair, facial hair, eyewear and similar objects. Depending on the selected color space for analyses, and the characteristics of the aforementioned features, they could dramatically affect the matrix of color values that are captured to represent the facial complexion before and after signal magnification. Another challenging factor is the original color intensity of the captured videos. Multiple factors could play a role in changing the original intensity of the video images including the change in illumination of the room or on the facial area. Even in case of keeping a constant lighting condition in the room, the content of the computer display and its reflection on the face could affect the original combination of color values on the users’ face (Fig. 4). Taking the aforementioned challenges into account, we have devised a generalized framework to move towards thermoregulation state inference through video image analysis. Fig. 5 illustrates this framework, which seeks to quantify the human body response in different states of thermoregulation by approximating the effect of blood perfusion. Considering our interest in evaluating the intensity of the skin color in different thermal conditions, we aim to remove the pixels that have color intensities that might interfere with our objective, including background and facial features other than skin. In other words, we are only interested in color intensity variations in pixels that represent human face and skin. Therefore, face recognition and skin isolation algorithms are important components of this framework. As shown in the framework flowchart, the steps in this framework could be carried out in different color spaces. In this study, we have explored and presented the components of the framework for two color spaces of RGB and YIQ considering the potential benefits of each of these spaces (which have been described in the following sections of the article). Regardless of the color space, the first step is face recognition to focus the analysis on facial area. Upon detection of the face, a rectangular facial boundary is cropped and passed to the next steps of the analysis. In each frame of the video, the cropped images of the face are then used for skin isolation and image magnification (using the EVM algorithm). The magnified values of color are extracted based on the pixels that represent skin. Post-processing is carried out to minimize the effect of initial color intensity of images. Thermoregulation state analysis calls for comparison of the images at different points in time, when the thermal condition of an environment has changed. A user could be

Fig. 5. Thermoregulation state identification framework.

looking at different content on the computer display at each point, which in turn could negatively affect the output values, and the quantification of the detection index. For example, a user might be looking at a document with a dominant white color when the environment is warmer (and higher color intensities are expected due to blood perfusion) while looking at a webpage with a dominant red color while the environment is cooler (and lower color intensities are expected). Finally, by using the detection index value, thermoregulation state can be identified. The detection index is a metric that represents a measure of color intensity variations due to a thermoregulation response in the human body. In the following subsections, we have elaborated our proposed approach for each component of the framework. 4.1. Skin pixels isolation process In order to segregate the region of interest (ROI), which include the skin pixels in the facial area, face recognition and skin isolation processes have been deployed. We have adopted the Viola-Jones face recognition algorithm [55] that is a rapid face recognition approach using AdaBoost as a learning algorithm, which has been widely used for face detection due to its high accuracy and speed [51,56]. Upon detection of 5

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Fig. 6. Thresholding for skin pixels isolation using Otsu method.

shade, the pixels with Hue values within a margin of the maximum frequency will be extracted. Fig. 7 illustrates the process. Original images are used in this process to avoid the effect of image magnification on the pixel values. This process is repeated for each frame of the image.

the bounding box of a face, additional margins are applied to ensure that a complete view of the face is obtained. As noted, to eliminate the impact of non-skin elements, skin isolation process is performed (two different methods were used in this study). We have explored image thresholding, which uses the color intensity of the skin pixels. Therefore, the efficacy of thresholding methods depends on the level of contrast between the skin and its surrounding objects/elements. Otsu method [57] as a clustering-based approach for automated thresholding assumes that the image is composed of two classes and automatically calculates the optimal threshold separating the two classes and it is best performed in gray-color images. Therefore, this method converts a grayscale image to a binary image. The approach uses exhaustive search for a threshold that minimizes the intra-class variance. This approach works best, when the color of background is in considerable contrast with the skin color. Fig. 6 shows an example of the threshold identification using the Otsu method. In an alternative method, the thresholding is carried out by converting the RGB image into Hue, Saturation, and Value (HSV) color space. The signal from the Hue channel of the original (i.e., observed) videos is used for specifying the boundary of the skin color class [58]. This step is carried out on the cropped face bounding boxes and thus, the Hue value with maximum frequency represents the skin pixel. Considering the variation in skin colors due to the illumination and

4.2. Color space selection 4.2.1. RGB space In the thermoregulation process, higher temperatures result in vasodilation, increasing the blood perfusion (flow of blood to the skin) and vice versa. We have hypothesized that elevated red channel intensity will be observed as the temperature increases considering that a higher skin blood flow occurs. Therefore, in this approach, the RGB color space is used as the source data. However, an emphasis will be put on the data in the magnified signal of the red channel, as the red channel could be a better representative of the blood flow. 4.2.2. YIQ space In the second approach, we have explored the application of YIQ color space. Y is the channel that represents the luminance information while I and Q channels contain the chrominance information. Considering the aforementioned challenges that variable luminance

Fig. 7. Thresholding for skin pixels isolation using Hue channel frequency analysis.

6

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

for the time series. This upper envelope (as shown in Fig. 8) will be the representation of the peak amplitudes of the magnified intensity on the post-processed signal.

could bring about, YIQ was selected. By eliminating the information in Y channel, we could focus on the chrominance data (the summation of data in I and Q channels), which reduces the effect of the light intensity and obviate the need for a post-processing step.

4.4.2. The second approach PPG senses the variations in transmitted or reflected light to measure cardiovascular pulse wave. Therefore, the thermoregulation process, which adjusts the blood flow to the skin tissue, should result in change in the variance of observed signal through PPG. Therefore, the second representative detection index is obtained by the variance of either the maximum pixel value or the spatially-averaged pixel values in post-processed signal.

4.3. Post-processing 4.3.1. For RGB space The color intensity of the magnified images could be affected by several unwanted sources that affect the RGB values of the captured videos. Having the exact same illuminance in different videos hardly happens in reality. Therefore, to account for variable ambient illumination, the original images are subtracted from the magnified images. The resultant image thus solely focuses on variations of the color intensity:

DIF (x ,y ) = MVF (x ,y )−OVM (x ,y )

4.5. Thermoregulation state identification The mean values of the upper envelope curve (UEM ) are calculated and compared for detection of the thermoregulation state. Therefore, an averaging window could be used for identifying the states. This parameter could be used for real-time control of the HVAC systems through a comparative analysis and creating a learning model. For comparison purposes, the following variable could be used:

(4)

where DIF represents the difference image frames, MVF is the magnified video frames, OVF is the original video frames and x and y represent the location of the pixels. 4.3.2. For YIQ space As noted, no post-processing was conducted as Y channel data was eliminated.

L H DS (H ,L) = RIUEM −RIUEM

(5)

where, DS is the detection statistic, RI is the representative index, H denotes the high temperature, and L denotes the low temperature, H hence RIUEM is the mean value of upper envelope curve for higher L is the mean value of upper envelope curve for temperatures and RIUEM lower temperatures. When the variance of the post-processed signal is used, the detection variable will be as follows:

4.4. Detection index Detection index presents a scalar value, which represents the state of thermoregulation process. It is obtained from a time series of representative indices over different frames of a video. In this context, representative index (RI) refers to a quantity in each frame that represents the cardiac cycle characteristics at a given time.

DS (H ,L) = RIVH /RIVL

(6)

RIVH

where, V denotes the variance, is the variance of the representative index time series for higher temperatures, and RIVL is the variance of representative index time series for lower temperatures. In the first case, a positive value for DS indicates increased blood flow to the skin surface resulting in deducing the state of thermoregulation mechanism for heat loss and vice versa, while in the second case, a DS value higher than one indicates increased blood flow.

4.4.1. The first approach The red RI, the maximum and spatially-averaged intensity on the post-processed images are extracted as possible representations for the most significant change due to the thermoregulation state, which is highly likely to happen on the skin above the vein area [59]. By extracting the RI values from each frame in a video, a time series of RIs will be obtained. Fig. 8 illustrates an example of the RI time series. The fluctuations in this figure, represent the variations due to cardiac cycles. Considering that we are interested in the peak amplitudes, which represent peak value of the blood volume, an upper envelope curve is fit to the time series using a spline interpolation with a smooth outlining

5. Experimental assessments In order to evaluate the feasibility of the proposed methodology an experimental study was conducted. We were specifically interested in investigating (1) whether the proposed method could evaluate

Fig. 8. Red channel representative index time series (over frames of a video) and the upper envelope.

7

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Table 1 Characteristics for human subjects, participated in the study. No.

Gender

Facial feature

No.

Gender

Facial feature

1 2 3 4 5 6 7 8 9 10

Female Male Female Female Female Male Male Male Male Male

None Facial hair None Black Glasses Facial hair Facial hair Glasses and Facial hair Glasses None

11 12 13 14 15 16 17 18 19 20 21

Male Male Male Male Male Female Male Male Male Male Male

None Glasses None None None None None Glasses Glasses None Glasses

Fig. 9. Experimental set-up in the test bed.

frequencies of the band-pass filter were 0.4 – 4 Hz via empirical observation and literature recommendations [53]. In the following subsections, the performance assessment for the framework is elaborated:

thermoregulation states by comparing the inferred states at distinguishable thermal conditions and (2) whether the inferred states could be used as indicators of thermal comfort perception. Positive answers to these questions will demonstrate the potential of the RGB cameras along with the proposed framework for quantifying thermal comfort. Therefore, in this experimental study, we collected RGB video images under two distinct thermal conditions, and analyzed them with the proposed framework. We set up a thermal chamber with an air-conditioning unit with heating and cooling capabilities. The ambient temperature was measured by a DHT22 temperature ( ± 0.5 °C accuracy) sensor connected to an Arduino microcontroller. A total of 21 participants were recruited for the experiment; 16 male and 5 female subjects from 20 to 40 years of age. Except one (an African american female participant), all participants had light skin complexion and some of male subject wore glasses or had facial hair. The details are given in Table 1. None of the subjects notified us of any cardiovascular problem history and declared being healthy. Subjects were asked to avoid activities that could influence their heart rates or cardiovascular states such as drinking coffee or alcohol, smoking, and intense physical activities for at least two hours prior to the experiments. Furthermore, participants were advised to wear short sleeve t-shirts to curtail clothing insulation effect and keep wearing the same clothes during the experiment. The experimental studies were conducted upon receiving the approval of Virginia Tech’ Internal Review Board (IRB) and informed consent was obtained. The experiments were planned to use two different thermal conditions: high (30 °C) and low (20 °C) temperatures, which, depending on the season, are considered as energy-intensive modes in operation. In addition, it was expected to be distinguishable in terms of physiological responses, presented in RGB video images, and thermal perceptions. Upon entering our thermally conditioned experimental test bed, a 20min acclimation time was used to ensure that the thermoregulation process has been triggered. An off-the-shelf webcam (i.e., Logitech HD Pro Webcam C920) was used for collecting the video data. This camera has a maximum sampling rate of 30 frames per second (FPS). For each thermal condition, after the acclimation time was completed, a twominute video was captured while participants were asked to stay still in front of the camera with no sudden movements. At the same time, participants provided their thermal perception by using a five-degree thermal perception scale. The scale asks whether subjects perceived the environment as cold, cool, neutral, warm, or hot. The experimental setup has been illustrated in Fig. 9.

5.1.1. Skin isolation evaluation The skin pixels’ isolation is used to ensure that likelihood of monitoring color intensities only on skin pixels is increased. Two methods of skin isolation have been adopted in this study. Otsu thresholding, although is completely automatic (i.e., it does not require any input thresholds), commonly calls for a clear contrast between two classes. Therefore, its application is not effective in all the cases and depends on the background color. On the other hand, the Hue channel thresholding, although requires threshold input by a user, will be more effective in the absence of a clear contrast between skin and its surrounding environment. Fig. 10 illustrates the performance of the adopted methods for different example conditions. As seen, the Otsu method, although performs very well for dark backgrounds, could remove considerable parts of skin pixels for lighter backgrounds. In this study, we have selectively used both methods depending on the background color. Therefore, the skin pixel isolation was a semi-automatic approach. The threshold range for the Hue channel thresholding was selected through empirical observations. Otsu thresholding was used for 13 videos (with a clear contrast between background and facial features), and the Hue channel thresholding was performed for the remaining videos. A visual assessment was conducted to ensure that the skin pixels were properly isolated. Through this assessment, it was observed that both methods effectively excluded the background, unwanted facial features, such as hair and glasses. Although minimal residual non-skin pixels were still observed, the purpose of skin isolation process is to minimize the noise in our analysis. As an effective overarching approach, the Hue channel thresholding could be generalized. In order to automate the process for identifying the thresholds for Hue intensity a probability distribution (e.g., a Gaussian distribution) could be fit to the histogram at each image frame. Then, the upper bound and lower bound values for skin pixel isolation are determined using the mean and standard deviation values of the distribution. In our analyses for this study, the selection of upper and lower thresholds was conducted manually. This skin isolation process results in indices of the pixels that should be monitored for temporal variations of the magnified color intensity. Therefore, these indices are used for extraction of the magnified signal values.

5.1. Results Upon acquiring the video images under the contrasting thermal conditions, the data were analyzed using the methods described in Section 3. Visual check of the automatically cropped videos revealed that the face detection algorithm performed with no errors. Parameters for the EVM algorithm were set as follows: the amplification factor was set to 50, the level of Laplacian pyramid was set to 4, and the cut-off

Fig. 10. Performance of the skin isolation methods.

8

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

(DS). The highlighted cells represent cases that showed positive results, meaning that an increased blood perfusion has been inferred. A t-test analysis was conducted between the time series of representative indices in two temperatures to examine whether the observed differences in inferred thermoregulation states are statistically significant. In Table 3, statistical significance was not observed for the cells, which were highlighted with dotted hatches. As this table shows, majority of the cases were statistically significance. An overview of the overall results shows that using the mean value of signals from all skin pixels demonstrated a better performance compared to the case, where the signal from pixel with maximum value is used. The mean values over all the skin pixels reduces the likelihood of erroneous data in the process of identifying the max values. For the detection index, the upper envelope curve appears to be a better detection index for RGB color space, while variance performs better in YIQ color space. Considering all the methods, combinations 5 and 8 (see Table 3) showed the best performance with 18 cases with statistical significance outcome. For each combination, 14 positive cases were observed. However, one of the cases for each combination did not show statistical significance. Therefore, each method rendered 13 statistically significant positive cases out of 18 cases, which is a 72% success rate. Taking the two promising combinations into account, Table 4 shows a closer look at the performance analysis and the collective outcome of the most promising methods. According to this table, for three human subjects, the outcome from two methods are contradicting. However, one of the methods was able to infer the triggered thermoregulation process. Considering these case, the analyses demonstrates 16 positive cases. The results for the two most promising combination of methods have been visually presented in Fig. 12. The detection index for each thermal condition for RGB and YIQ color are seen in Fig. 12 (b) and (c), respectively. The detection indices were presented to provide an insight on the order of magnitude for the indices and a benchmark for comparing the detection statistic with the indices’ original values. Considering the cases that showed conflicting results for the two aforementioned combinations, the total number of positive cases (i.e., inferring the increase in inferred blood perfusion) reached to 16 out of 18 statistically significant cases, which rendered an 89% rate of significantly positive cases. In order to provide a clear answer for selecting one of the methods over the other for real-time applications, a number of other factors should be taken into account. In this study, we did not collect ground truth information associated to actual blood perfusion of the facial skin in order to provide a concrete evaluation for negative cases. A negative case could be due to the fact that for individual human subjects the level of sensitivity to the thermal changes might vary. This concept has been also reported in other studies [60]. In other words, the thermoregulation process for each individual might appear with a different indicator. This is specifically important given the fact that we did not strictly control individuals’ attire. For example, some of the human subjects wore short pants during the experiments and we did not record their meta data as the objective was to investigate whether facial blood perfusion could be sufficiently quantified as it relates to thermoregulation. Given this fact, a possibility includes heat dissipation from other parts of the skin, which results in a negative outcome. In order to investigate these factors, correct measurement of blood perfusion on skin will be beneficial along with controlled experiments that limit the skin exposure. Non-intrusive measurement of blood perfusion is not a trivial procedure. However, methods have been proposed for estimation of the blood perfusion. We are planning to measure the actual (i.e., estimated actual) perfusion through heat flux sensors as they have been adopted in some medical studies [61,62].

Table 2 Combinations of methods used in analyzing the video data. Combo No.

Signal extraction

Color Space

Method for representative index selection

1 2 3 4

Max Value from Pixels

RGB

Upper Envelope Mean Variance Upper Envelope Mean Variance

5 6 7 8

Mean Value from Pixels

YIQ RGB YIQ

Upper Envelope Mean Variance Upper Envelope Mean Variance

5.1.2. Assessment of thermoregulation state identification In the previous sections, the proposed generalized framework as well as methods for its components was presented. Considering different color spaces and methods for detection index calculation, the analyses for thermoregulation states identification were carried out using different combinations of these methods (as presented in Table 2). As noted, we collected data in two thermal conditions of low and high temperatures. The performance assessment thus was carried out using the metrics presented in Eqs. ((5) and (6)). These metrics represent the subtraction of detection index in lower temperature from detection index for higher temperature in RGB color space and division of detection index for higher temperature over detection index for lower temperature in YIQ color space. For the former a positive value and for the latter a value greater than one shows an acceptable performance. In order to provide a tangible illustration for the assessment method, Fig. 11 presents examples of time series of representative index in the videos at low and high temperature conditions. Moreover, the corresponding upper envelope curves in RGB color space were also presented. This illustration sheds light on the rationale behind the proposed framework. Considering the combinations of the framework components in Table 2, the videos, collected from 21 participants, were analyzed to identify detection statistics. Table 3 presents the results of quantifications for detection statistics

5.1.3. Thermal perception assessment The participants’ thermal perception was also measured to ensure the compatibility between the inferred thermoregulation states and perceived level of comfort. Under the configured thermal conditions,

Fig. 11. Illustrative comparison between the videos of a human subject at the 20 °C versus 30 °C (a) YIQ color space and (b) RGB color space.

9

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Table 3 Detection statics for different combination of methods in the proposed framework.

5.1.4. Energy management implications As noted, the end goal for the proposed approach is to enable distributed feedback for context-aware and human-centered control of energy management systems. The outcome of this study provides the ground for system integration in real-time control of HVAC energy resources. Indirect measurements (using occupant voting) of thermal preferences have shown to be effective in energy conservation in

subjects revealed distinguishable thermal perceptions. None of the subjects declared either warm or hot states at the low-temperature condition (81% revealed cool and cold), and all subjects felt either warm or hot at the high temperature. Table 5 shows the perception of the human subjects at different thermal conditions.

Table 4 Collective outcome of the most promising combinations of methods.

10

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

Fig. 12. Detection indices and detection statistics for different human subjects using the most promising combination of methods in our proposed framework.

consumption while improving the thermal satisfaction of occupants. That study helped us accentuate the importance of distributed feedback from an environment in enhancing the efficiency of the HVAC systems. Capitalizing on the lessons learned from our previous studies, the outcome of this study will further facilitate the realization of achieving distributed and ubiquitous human response to thermal conditions, which is expected to bring about even higher energy conservation. Enabling sensor proxies by relying on thermophysiological response of

buildings [12]. In our previous research studies, we have developed an OVPS-based HVAC control framework, in which we used thermal comfort profiling by leveraging user feedback through smartphone interfaces as well as room level ambient sensor systems [12]. In that study, we demonstrated that utilizing personalized thermal comfort profiles in control of the HVAC systems with additional sensing nodes at room level could results in considerable reduction of average daily airflow (∼40%), which is highly correlated with the energy 11

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

evaluated its performance through experimental studies. The study seeks to move towards new methods for a paradigm shift from spacecentered HVAC control to human thermoregulation-based control logics, which has been shown to increase energy efficiency of building operations. Although in recent years a number of studies have focused on thermal comfort assessments using physiological variables, the methods commonly call for intrusive and costly solutions. In pursuit of creating a non-intrusive, cost-effective, and ubiquitous solution for thermal comfort assessment (as an input to energy management systems), we focused on the use of RGB video images, captured by commercially available, off-the-shelf webcam technologies. To achieve this objective, we proposed a framework that draws on the mechanisms of thermoregulation as well as the Eulerian image magnification to infer the state of thermoregulation associated with varied blood flow to the skin tissues as a critical mechanism for preserving core body temperature. Focusing on the facial skin, as it could be conveniently captured with cameras on mobile computers, the framework consists of components for face detection, skin isolation, image magnification and detection index calculation. Given the challenges that varied illumination could pose on the performance of the approach, the framework has been explored for RGB and YIQ color spaces with specialized pre- and post-processing. The former space calls for a subtraction of original image from the magnified one to account for variable original color intensities and the latter space calls for eliminating the luminance channel to reduce the impact of varied illuminations. The proposed framework was evaluated for different combinations of the methods for the framework components. Considering binary variables of color space (RGB vs. YIQ), representative index (max versus mean color intensities of skin pixels), and detection index (mean value of the upper envelope vs. variance of the representative index time series), eight combinations of the methods were taken into account. An experimental study was conducted, in which data was collected for low(20 °C) and high- (30 °C) temperature environments from 21 human subjects, while video images of their faces were captured after a 20-min acclimation time. The analysis of the results revealed that mean color intensities of skin pixels show a better performance for both color spaces. However, it appears that mean value of the upper envelope curve and variance of the representative index time series work better for RGB and YIQ color spaces, respectively. In total, 13 positive cases (out of 18 statistically significant cases) were observed, rendering a success rate of 72%. Positive in this case, refers to an increase in inferred blood perfusion in facial area due to increased ambient temperature. Moreover, in three cases, the two aforementioned combinations of methods resulted in significant but conflicting results – positive with one approach and negative with another. If the number of those positive cases are taken into account, the total success rate will be 89%. The outcome of this study could facilitate ubiquitous and distributed feedback from an environment for improved efficiency in control of energy management systems in buildings. Prior studies (including those of the authors) have shown that enabling robust distributed measurement of thermal response in an environment using cost-effective techniques could bring about improved energy efficiency (potentially up to 50% [64]).

Table 5 Thermal perceptions presented by subjects in two temperatures. Thermal perceptions

# Of subjects (20 ± 0.5 °C)

# Of subjects (30 ± 0.5 °C)

Cold Cool Neutral Warm Hot

7 10 4 0 0

0 0 0 9 12

human body will account for both occupancy and preferences in the energy management systems, which in turn should bring about improved energy performance. Therefore, developing technologies that enable the scalability of distributed thermal comfort feedback could pave the way for realization of energy conservation in future HVAC systems. 6. Limitations and future directions Although promising performance has been observed during the evaluation studies, there are a number of limitations that call for further investigations. This study did not cover the entire range of the typical indoor temperature (20 – 30 °C [63]). Across such temperature ranges, it is highly likely to have various individual thermal perceptions and physiological responses, thus this approach requires a further evaluation of those circumstances. In other words, the sensitivity of the approach for smaller temperature variations should be taken into account. Another important factor that is critical in evaluating the performance of such system is the applicability of the proposed method in transient conditions. Studies in the field of thermal comfort assessment and its correlation to the physiological response of human body commonly use an acclimation (as used in this study) time in quantifying physiological responses to ensure their stabilization [63]. Therefore, the sensitivity of the approach in transient condition needs to be further evaluated. Another important category of limiting factors include the contextual conditions that might affect the quality of images. These conditions include illumination and movements of the human subjects. Although we have proposed solutions to address part of the illumination challenges, there are remaining factors such as low lighting conditions that need to be addressed. Motion artifacts are another source of noise that could affect the quality of PPG signals. Reducing the effect of such artifacts calls for specialized signal processing algorithms such as blind source separation and in-band noise reduction. Another contextual aspect that needs to be further investigated includes the factors that affect human body thermoregulation response. Examples of such factors include the clothing condition at different occasions, the proximity of the occupants in a shared space, and activities that change the cardiopulmonary operations. Given the promising performance that was observed in this study, the future directions of this research include: (1) exploring more effective methods for reducing/eliminating the impact of variable illumination, (2) application of motion tracking algorithms to account for Eulerian nature of EVM algorithm that monitors fixed pixels, (3) increasing the sample size and adopting a proper approach for ground truth (real blood perfusion to the skin tissue) measurement, (4) exploring the sensitivity of the approach in transitional thermal conditions – where the temperature varies with a faster pace compared to our experimental set-up, (5) evaluating the impact of contextual conditions that affect thermoregulation processes, and (6) system–integration for real-time control of HVAC system for energy conservation.

Acknowledgement The authors would like to thank all the participants that take time to participate in this study. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. References

7. Conclusion [1] BED-Book, Building Energy Data Book. U.S. Department of Energy, 2011. [2] U.S. Energy Information Administration. Heating and cooling no longer majority of U.S. home energy use. 2013 [cited 2017 1.8.]; Available from: http://www.eia.gov/

In this study, we have proposed a novel approach for leveraging omnipresent RGB video cameras for thermal comfort assessment, and 12

Applied Energy xxx (xxxx) xxx–xxx

F. Jazizadeh, W. Jung

[3]

[4]

[5]

[6]

[7] [8] [9]

[10] [11] [12]

[13]

[14] [15] [16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25] [26] [27] [28] [29] [30] [31]

[32]

[33]

todayinenergy/detail.php?id=10271&src=%E2%80%B9%20Consumption %20%20%20%20%20%20Residential%20Energy%20Consumption%20Survey %20(RECS)-b1. Chowdhury AA, Rasul M, Khan MMK. Thermal-comfort analysis and simulation for various low-energy cooling-technologies applied to an office building in a subtropical climate. Appl Energy 2008;85(6):449–62. Kim W, Jeon Y, Kim Y. Simulation-based optimization of an integrated daylighting and HVAC system using the design of experiments method. Appl Energy 2016;162:666–74. Korkas CD, Baldi S, Michailidis I, Kosmatopoulos EB. Occupancy-based demand response and thermal comfort optimization in microgrids with renewable energy sources and energy storage. Appl Energy 2016;163:93–104. Korkas CD, Baldi S, Michailidis I, Kosmatopoulos EB. Intelligent energy and thermal comfort management in grid-connected microgrids with heterogeneous occupancy schedule. Appl Energy 2015;149:194–203. Yang L, Yan H, Lam JC. Thermal comfort and building energy consumption implications–a review. Appl Energy 2014;115:164–73. Jendritzky G. and Tinz B. The thermal environment of the human being on the global scale. Global Health Action, 2009. 2. Fanger PO. Thermal comfort. Analysis and applications in environmental engineering. Thermal comfort. Analysis and applications in environmental engineering., 1970. Becker R, Paciuk M. Thermal comfort in residential buildings – Failure to predict by Standard model. Build Environ 2009;44(5):948–60. Maiti R. Physiological and subjective thermal response from Indians. Build Environ 2013;70:306–17. Jazizadeh F, Ghahramani A, Becerik-Gerber B. User-led decentralized thermal comfort driven HVAC operations for improved efficiency in office buildings. Energy Build 2014;70:398–410. Jazizadeh F. and Becerik-Gerber B. Toward adaptive comfort management in office buildings using participatory sensing for end user driven control. in: Proceedings of the Fourth ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Buildings. 2012. ACM. Liu W, Lian Z, Deng Q, Liu Y. Evaluation of calculation methods of mean skin temperature for use in thermal comfort study. Build Environ 2011;46(2):478–88. Feldmeier M. and Paradiso JA. Personalized HVAC control system. in 2010 Internet of Things (IOT). 2010. Hamdi M. and G. Lachiver. A fuzzy control system based on the human sensation of thermal comfort. in Fuzzy Systems Proceedings, 1998. IEEE World Congress on Computational Intelligence., The 1998 IEEE International Conference on. 1998. IEEE. Bruant M, Guarracino G, Michel P. Design and tuning of a fuzzy controller for indoor air quality and thermal comfort management. Internat J Solar Energy 2001;21(2–3):81–109. Kolokotsa D, Tsiavos D, Stavrakakis GS, Kalaitzakis K, Antonidakis E. Advanced fuzzy logic controllers design and evaluation for buildings’ occupants thermal–visual comfort and indoor air quality satisfaction. Energy Build 2001;33(6):531–43. Atthajariyakul S, Leephakpreeda T. Real-time determination of optimal indoor-air condition for thermal comfort, air quality and efficient energy usage. Energy Build 2004;36(7):720–33. Calvino F, La Gennusa M, Rizzo G, Scaccianoce G. The control of indoor thermal comfort conditions: introducing a fuzzy adaptive controller. Energy Build 2004;36(2):97–102. Kolokotsa D, Saridakis G, Pouliezos A, Stavrakakis GS. Design and installation of an advanced EIB™ fuzzy indoor comfort controller using Matlab™. Energy Build 2006;38(9):1084–92. Dounis AI, Caraiscos C. Advanced control systems engineering for energy and comfort management in a building environment—A review. Renew Sustain Energy Rev 2009;13(6–7):1246–61. Zhou G. Ihm P. Krarti M. Liu S. Henze G.P. Integration of an internal optimization module within EnergyPlus. in Proceedings of 8th International IBPSA Building Simulation Conference. 2003. Ferreira PM, Ruano AE, Silva S, Conceicao EZE. Neural networks based predictive control for thermal comfort and energy savings in public buildings. Energy Build 2012;55:238–51. Wong NH, Khoo SS. Thermal comfort in classrooms in the tropics. Energy Build 2003;35(4):337–51. Becker R, Paciuk M. Thermal comfort in residential buildings–failure to predict by standard model. Build Environ 2009;44(5):948–60. Van Hoof J. Forty years of Fanger’s model of thermal comfort: comfort for all? Indoor air 2008;18(3):182–201. Luo M, Zhou X, Zhu Y, Sundell J. Revisiting an overlooked parameter in thermal comfort studies, the metabolic rate. Energy Build 2016;118:152–9. Zhang Y, Wang J, Chen H, Zhang J, Meng Q. Thermal comfort in naturally ventilated buildings in hot-humid area of China. Build Environ 2010;45(11):2562–70. Karjalainen S, Koistinen O. User problems with individual temperature control in offices. Build Environ 2007;42(8):2880–7. Jazizadeh F. Kavulya G. Klein L. Becerik-Gerber B. Continuous sensing of occupant perception of indoor ambient factors. in ASCE International Workshop on Computing in Civil Engineering. 2011. Jazizadeh F, Ghahramani A, Becerik-Gerber B, Kichikaylo T, Orosz M. Humanbuilding interaction framework for personalized thermal comfort-driven systems in office buildings. J Comput Civil Eng 2013. Jazizadeh F, Marin FM, Becerik-Gerber B. A thermal preference scale for

[34]

[35]

[36]

[37]

[38] [39]

[40]

[41] [42]

[43]

[44] [45]

[46]

[47] [48] [49] [50] [51]

[52] [53]

[54] [55]

[56]

[57] [58] [59] [60] [61] [62]

[63] [64]

13

personalized comfort profile identification via participatory sensing. Build Environ 2013;68:140–9. Erickson VL. Beltran A. Winkler DA. TOSS: Thermal Occupancy Sensing System. in: Proceedings of the 5th ACM Workshop on Embedded Systems For Energy-Efficient Buildings. 2013. ACM. Murakami Y. Terano M. Obayashi F. Honma M. Development of cooperative building controller for energy saving and comfortable environment, in Human Interface and the Management of Information. Interacting in Information Environments 2007, Springer. p. 1078–1087. Fay D, Brown KN. O'Toole L. Lies, damn lies and preferences: a Gaussian process model for ubiquitous thermal preference trials, in Proceedings of the Fourth ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Buildings. 2012, ACM: Toronto, Ontario, Canada. p. 184–191. Bermejo P, Redondo L, de la Ossa L, Rodriguez D, Flores J, Urea C, Gamez JA, Puerta JM. Design and simulation of a thermal comfort adaptive system based on fuzzy logic and on-line learning. Energy Build 2012;49:367–79. Daum D, Haldi F, Morel N. A personalized measure of thermal comfort for building controls. Build Environ 2011;46(1):3–11. Humphrey M. Nicol F. McCartney KJ. Raja I. An analysis of some observations of the finger temperature and thermal comfort of office workers. in Proceedings of the 8th International conference on Indoor Air Quality and Climate (Indoor Air 99), Edinburgh. 1999. Wang D, Zhang H, Arens E, Huizenga C. Observations of upper-extremity skin temperature and corresponding overall-body thermal sensations and comfort. Build Environ 2007;42(12):3933–43. Wang Z, Ning H, Ji Y, Hou J, He Y. Human thermal physiological and psychological responses under different heating environments. J Therm Biol 2015;52:177–86. Yao Y, Lian Z, Liu W, Jiang C, Liu Y, Lu H. Heart rate variation and electroencephalograph – the potential physiological factors for thermal comfort study. Indoor Air 2009;19(2):93–101. Liu W, Lian Z, Deng Q. Use of mean skin temperature in evaluation of individual thermal comfort for a person in a sleeping posture under steady thermal environment. Indoor and Built Environment, 2014: p. 1420326X14527975. Tanda G. Skin temperature measurements by infrared thermography during running exercise. Exp Therm Fluid Sci 2016;71:103–13. Yi, B. and J.-H. Choi, Facial Skin Temperature as a Proactive Variable in a Building Thermal Comfort Control System, in Sustainable Human–Building Ecosystems. 2015. p. 117-125. Sharifani, P. Direct Measurement of Occupants’ Skin Temperature and Human Thermal Comfort Sensation for Building Comfort Control, in Sustainable Human–Building Ecosystems. 2015. p. 141–149. Ranjan, J. and J. Scott. ThermalSense: Determining Dynamics Thermal Comfort Preferences using Thermographic Imaging. 2016. Parsons K. Human thermal environments: the effects of hot, moderate, and cold environments on human health, comfort, and performance. 2014: Crc Press. Holland, K. and G. Krunick, Thermoregulation. Available at, http://www. healthline.com/health/thermoregulation#Overview1, 2013. Allen J. Photoplethysmography and its application in clinical physiological measurement. Physiol Meas 2007;28(3):R1–39. Poh M-Z, McDuff DJ, Picard RW. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Exp 2010;18(10):10762–74. Verkruysse W, Svaasand LO, Nelson JS. Remote plethysmographic imaging using ambient light. Opt Express 2008;16(26):21434–45. Wu H-Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 2012;31(4):65. Burt P, Adelson E. The laplacian pyramid as a compact image code. IEEE Trans Commun 1983;31(4):532–40. Viola, P. and M. Jones. Rapid object detection using a boosted cascade of simple features. in Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. 2001. Poh MZ, McDuff DJ, Picard RW. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans Biomed Eng 2011;58(1):7–11. Hamedani K, Bahmani Z, Mohammadian A. Spatio-temporal filtering of thermal video sequences for heart rate estimation. Expert Syst Appl 2016;54:88–94. Elgammal A, Muang C, Hu D. Skin Detection - a Short Tutorial. USA: Rutgers University; 2009. Bouzida N, Bendada A, Maldague XP. Visualization of body thermoregulation by infrared imaging. J Therm Biol 2009;34(3):120–6. Zhao J, Lam KP, Loftness V, Ydsties BE. Occupant individual thermal comfort data analysis in an office, in sustainable human–building. Ecosystems 2015. Kreith F. The CRC Handbook of Mechanical Engineering. Second Edition The CRC handbook of mechanical engineering: Heat and Mass Transfer; 2005. Al-Khwaji A, Vick B, Diller. Modeling and Estimating Simulated Burn Depth Using the Perfusion and Thermal Resistance Probe. Journal of Medical Devices, 2013. 7(3): p. 031003–031003-9. Liu W, Lian Z, Liu Y. Heart rate variability at different thermal comfort levels. Europ J Appl Physiol 2008;103(3):361–6. Ghahramani A, Jazizadeh F, Becerik-Gerber B. A knowledge-based approach for selecting energy-aware and comfort-driven HVAC temperature set points. Energy Build 2014;85:536–48. http://dx.doi.org/10.1016/j.enbuild.2014.09.055. ISSN 0378-7788.