Automatic classification of common building materials from 3D terrestrial laser scan data

Automation in Construction 110 (2020) 103017

Contents lists available at ScienceDirect

Automation in Construction journal homepage: www.elsevier.com/locate/autcon

Automatic classification of common building materials from 3D terrestrial laser scan data Liang Yuana, Jingjing Guob, Qian Wangb, a b

T

⁎

School of Construction Management and Real Estate, Chongqing University, Chongqing 400045, China Department of Building, School of Design and Environment, National University of Singapore, 4 Architecture Drive, Singapore 117566, Singapore

ARTICLE INFO

ABSTRACT

Keywords: Building material classification Terrestrial laser scanning Machine learning

Automatic building material classification has been a popular research interest over the past decades because it is useful for construction management and facility management. Currently, the proposed methods for automatic material classification are mainly based on 2D images by using the visual features of building materials. A terrestrial laser scanner (TLS) with a built-in camera can generate a set of coloured laser scan data that contain the surface geometries of building materials. The laser scan data include not only the visual features of building materials but also other attributes such as material reflectance and surface roughness. With more attributes provided, laser scan data have the potential to improve the accuracy of building material classification. Therefore, this research aims to develop a TLS data-based classification method for common building materials using machine learning techniques. The developed technique uses material reflectance, HSV colour values, and surface roughness as the features for material classification. A database containing the laser scan data of ten common building materials was created and used for model training and validation with machine learning techniques. Different machine learning algorithms were compared, and the best algorithm showed an average classification accuracy of 96.7%, which demonstrated the feasibility of the developed method.

1. Introduction In the past decade, automatic building material classification based on the state-of-the-art information technologies has been a promising research direction in the architecture, engineering, and construction (AEC) industry. Automatic material classification can improve the efficiency of a variety of tasks including damage detection and onsite material management and tracking [1,2]. Moreover, building information modelling (BIM) has received extensive attention from both the academic and industrial communities and has been increasingly adopted in the design, construction, and operation stages of construction projects. It has been an important task to generate as-is building information models (BIMs) that reflect the as-is conditions of buildings, which can be applied for various applications such as operation and maintenance (O&M) of existing buildings and building performance analysis [3,4]. An as-is BIM contains not only the geometric information of a building but also non-geometric information of building elements including building materials [5]. The material information is essential for many BIM applications such as the three-dimensional representation of objects and building energy simulation. Therefore, there is a high demand for automatic building material classification in order to

⁎

generate semantically rich as-is BIMs containing material information. Applying machine learning techniques for automatic building material classification has been a popular approach in AEC industry over the past years. The proposed material classification methods can be categorized as image-based and laser scan data-based according to the types of the collected data. Currently, image-based material classification methods are extensively used. The core technique of image-based methods focuses on using the visual features of building materials such as colour, texture, roughness, and projection [3,6–8] for automatic classification. However, image-based methods are heavily influenced by illumination conditions. Different illumination conditions strongly affect the visual characteristics of materials, causing difficulty for image-based building material classification. Moreover, poor textures on objects and unknown viewpoints also negatively affect the robustness and accuracy of image-based material classification [8]. Terrestrial laser scanning (TLS) provides a new and promising perspective for automatic building material classification. A TLS with a built-in camera can capture not only the visual features but also intrinsic properties of building materials such as the material reflectance. Meanwhile, unlike passive imaging which is critically dependent on environmental lighting conditions, TLS uses an active measurement

Corresponding author. E-mail address: [email protected] (Q. Wang).

https://doi.org/10.1016/j.autcon.2019.103017 Received 4 July 2019; Received in revised form 17 October 2019; Accepted 16 November 2019 0926-5805/ © 2019 Elsevier B.V. All rights reserved.

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

technique with infrared lights, which is not affected by environmental illumination conditions [9]. Therefore, TLS has great potentials to achieve more accurate material classification considering the more types of information provided and higher robustness to changeable lighting conditions. Moreover, TLS has been extensively adopted for constructing as-is BIMs that represent the as-is conditions of buildings [10], due to its high measurement accuracy and speed. As a result, using TLS data for building material classification does not require extra data collection if TLS data are already collected for constructing as-is BIMs. Despite the advantages of TLS, few previous studies have adopted TLS data for building material classification. This research aims to develop an automatic classification method for common building materials based on TLS data. The developed method will be useful for various applications especially when using TLS data to construct semantically rich as-is BIMs where material information is needed. In the rest of this paper, Section 2 presents a comprehensive literature review about automatic material classification. Based on the review, the features for TLS data-based material classification are determined in Section 3. Then, Section 4 provides a validation experiment under real-world environments in order to validate the accuracy of the proposed method on ten common building materials, and the experimental results are presented and discussed in Section 5. The limitations of this study and potential future works are discussed in Section 6, and lastly Section 7 concludes this study.

et al. [13] processed laser scan data using digital image processing techniques to detect damages on stony materials of historical buildings. Unsupervised classification algorithms were used in this study for the classification of 2D reflected intensity images derived from 3D laser scan data. This work showed the potential of using the reflected intensity from TLS for the recognition and characterization of certain damages in building materials of historical buildings. Riveiro et al. [14] presented a novel segmentation algorithm for automatic segmentation of masonry blocks from 3D laser scan data based on the reflected intensity values. Moreover, Sánchez-Aparicio et al. [15] developed an approach to detecting and classifying different pathological processes commonly presented on the masonry material of cultural heritage by combining the reflected intensity values and point cloud coordinates acquired by TLS. In addition to the reflected intensity values, TLS is often equipped with a built-in camera to capture the colour information of the scanned targets. Some studies have utilized the colour information for material classification. For example, Hassan et al. [16] confirmed the availability of material identification using the reflected intensity and Red-GreenBlue (RGB) values from TLS. They obtained the scan data of structural concrete, light-weight concrete, and clay brick samples for experiments. The experimental results showed that the scanned materials had different reflected intensity distributions, and the recorded RGB colour values could be used as a secondary parameter for material classification. Valero et al. [17] achieved automatic segmentation of individual masonry units and mortar regions in digitized rubble stone constructions based on coloured laser scan data acquired by TLS. The scan data of the target surface were converted into 2D depth maps as a feature for automatic segmentation, and colour information was used as another feature. The experimental results demonstrated the effectiveness of the technique. Also, Valero et al. [18] achieved automatic material defect area detection and classification of ashlar masonry walls by using TLS data including point cloud coordinates, reflected intensity, and RGB colour values. A supervised machine learning technique was used in this study. Although image-based material classification methods have made great advancement and are more extensively adopted than TLS databased methods, their applications in real-world environments still face challenges due to the complex field conditions and their dependence on environmental illumination conditions. The ability to capture more types of information and the use of active measurement technique makes TLS data-based material classification more promising. However, the previous studies on TLS data-based material classification are either based on only laboratory experiments in a controlled environment or for the recognition of only one or two categories of materials. To tackle the limitations of previous studies, this study examines the feasibility of using TLS data for the classification of ten different common building materials in real-world environments.

2. Related works Building material is any material that can be used for construction purposes, and the term construction material is also used as a synonym in some papers. Some previous studies have contributed to automatic building material classification using image-based methods or TLS databased methods. Image-based material classification using machine learning techniques [3,6–8,11] has been the dominant non-destructive material classification method. The image-based material classification approaches can be classified into two types: pixel-based methods and object-based methods. For pixel-based methods, each pixel is classified into a material category. For example, using colour as a feature, Son et al. [6] explored the performances of single classifiers and ensemble classifiers for classifying concrete, steel, and wood. Zhu and Brilakis [11] used material colour and texture as features to extract concrete material regions from the construction site images based on different classification algorithms including support vector data description, C-support vector classification, and artificial neural network. On the other hand, object-based methods classify each image patch into a material category. For example, Han and Golparvar-Fard [3] used material colour and texture as features and multiple discriminative machine learning models as classifiers to classify squared-shape image patches into fifteen common construction materials. Dimitrov and Golparvar-Fard [7] used texture and colour as features and various support vector machine algorithms as classifiers to classify 200 × 200 pixel image patches into twenty typical construction materials under real-world construction site conditions. Similarly, Lu et al. [8] developed a method to recognize building objects from images, and classify objects to material categories using five features including projection results from two directions, ratio values, RGB colour values, roughness values, and hue values. On the other hand, some other studies have explored building material classification based on TLS data. The classification approaches used in these studies were point-based, which is similar to the pixelbased method for images. The reflected intensity values collected by TLS are first adopted for material classification. For instance, Franceschi et al. [12] focused on the recognition of rocks in simple sedimentary successions that mainly consisted of limestones and marls using the intensity values from TLS. The results of a series of experiments indicated that the intensity values could provide a reliable method to classify the rocks in outcrop conditions. Armesto-González

3. Determination of features This section aims to determine the features used for automatic building material classification. For each scan point, the TLS collects a set of attributes comprising of the reflected laser beam intensity, RGB colour values, and x-y-z coordinates. A set of features for material classification are calculated based on the collected attributes. The features include 1) material reflectance, 2) colour, and 3) surface roughness, as explained in the following subsections. 3.1. Material reflectance 3.1.1. Theoretical fundamentals For each scan point, the TLS provides a reflected laser intensity value (Ir), which is determined by the type of material and scanning parameters. Although Ir is recommended to be a feature for material classification in some studies [14,16], different materials are likely to 2

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

present similar Ir values because various scanning parameters can also significantly affect Ir. Instead, among all the factors that affect Ir, the material reflectance ρ is the only intrinsic property of a certain material. In other words, the same material always has the same ρ value even though other factors are varying. However, it is worth noting that, for a specific material, the estimated material reflectance ρ values from TLS data can differ within a certain range. Therefore, the material reflectance ρ is adopted as a feature for material classification in this paper, and the ρ value will be estimated from the Ir value. Previous studies have investigated how to obtain the ρ value from the Ir value. Tan and Cheng [19] proposed a polynomial calculation method to recover the absolute ρ value of the scanned surface by eliminating the effects of the scanning range R from the TLS to the target and the laser beam incident angle θ on Ir. However, this method requires a target with known ρ to be a reference in the scanning scene. Franceschi et al. [12] proposed a linear fitting method to obtain the ρ values of different stone materials. However, their study was limited to cases with R > 15 m, and the same reference target (with unknown ρ) was required in different scanning scenes. Because it is inconvenient to place a reference target in real-world environments, this study develops a reference target-free method for calculating the ρ values of different materials. Because the laser intensity is the laser power per unit area [20], Eq. (1) can be obtained:

Fig. 1. The relationship between the Spherical coordinates system and the Cartesian coordinate system for a scan point Pi.

Eq. (5).

(1)

Ir = Pr / S

Ir = K

where Pr is the received laser signal power and ∆S is the unit area. For each scan point, Pr is determined by the sensor, the target, and atmospheric parameters. Eq. (2) illustrates the specific factors that determine the received signal power Pr:

Pr =

Pt Dr 2 4 R4 t

sys atm

Pt Dr 2 4R2

sys

atm cos

(2)

x = R cos cos y = R sin cos z = R sin

It Dr 2 4R2 S

sys atm cos

(6)

where α is the horizontal angle of the laser beam in the TLS's coordinate system, and β is equivalent to θ. Combing Eqs. (5) and (6), Eq. (7) can be obtained as:

cos Ir = K 2 R

(3)

x 2 + y2

= K

x 2 + y2 + z2

x 2 + y2 + z2

(7)

According to Eq. (7), Ir presents proportional relationships with cosθ/R2. However, several previous studies show that Ir and cosθ/R2 present linear relationships instead of proportional relationships when obtaining scan data by TLS [23–25]. It may be because TLS normalizes the Ir value in some way such that the Ir value is not proportional to Pr, which makes Eq. (7) invalid. According to previous studies, these linear relationships can be expressed as:

where θ is the incident angle of the laser beam on the scanned object. Three conditions need to be met for simplifying Eq. (2) into Eq. (3): 1) the entire laser footprint is reflected on only one surface (extended target) and the laser footprint area is circular, 2) the laser footprint area has a solid angle Ω of π steradians (Ω = 4π for scattering into the whole sphere), and 3) the target surface presents Lambertian scattering characteristics [22]. In case of TLS measurement in the AEC industry, the size of the reflecting surface usually greatly exceeds the size of laser footprint, and the scanned object can be regarded as an extended target with a solid angle of π steradians. Meanwhile, the laser reflection on the target surface could be approximately treated as a Lambertian scattering for most building materials [23]. Eq. (4) can be obtained by combing Eqs. (1) and (3):

Ir =

(5)

In Eq. (5), the R and θ values of a scan point can be calculated from the x-y-z coordinates as follows. As shown in Fig. 1, a laser scan point Pi can be presented in both the Spherical coordinate system as Pi(R, α, β) and the Cartesian coordinate system as Pi(x, y, z). The relations between the two coordinate systems are shown in Eq. (6):

where Pt is the transmitted power, σ is the effective target cross-section, Dr is the receiver aperture diameter, R is the range from the TLS to the target, βt is the laser beam width, ηatm is the atmospheric transmission factor, and ηsys is the system transmission factor [21]. Eq. (2) is only valid under two assumptions: 1) the receiver field of view matches the beam divergence, and 2) the laser emitter and the laser detector have the same distance to the scanned target. The cases in this study fulfil the above-mentioned assumptions. According to [21], Eq. (2) can be simplified into Eq. (3)

Pr =

cos R2

Ir = K

cos R2

+b

(8)

where b is a constant term. 3.1.2. Experimental validation Experiments were conducted to validate Eq. (8) using real TLS data. Taking the realistic scanning scenes of buildings into consideration, this study assumes that the range R is < 10 m. Although previous studies show the linear relationships between Ir and cosθ/R2, these studies consider only long-distance scanning (with R larger than 10 m). When the R value is within 10 m, the relationship between Ir and cosθ/R2 has not been investigated in previous studies. Therefore, this study examined the relationship using a set of experiments as follows. In the first step, we examined the relationships between Ir and cosθ

(4)

In the AEC industry, the range R is relatively short (no > 20 m). In this case, the ηatm value within the scanned region can be regarded as a constant. In addition, the It, Dr, and ηsys values will also be fixed when using the same TLS for scanning. Hence, ηatm, It, Dr, ηsys, and ∆S can be substituted by a total coefficient K, and Eq. (4) can be simplified into 3

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Fig. 2. The relationship between Ir and cosθ and 1/R2: (a) the relationship between Ir and cosθ when R ix fixed, where ρm and ρp are the respective material reflectance of metal plate and latex painting, and (b) the relationship between Ir and 1/R2 when θ is fixed, where K1, K2 and K3 are the different values of K.

and 1/R2 separately. When one factor (e.g. cosθ) is fixed, Ir should show a linear relationship with the other factor (e.g. 1/R2). This study used the FARO FocusS70 laser scanner to collect scan data, and the scan data of white latex painting and white metal plate were taken as examples. Data points with different θ values but the same R value were selected firstly. The Ir and cosθ values of these selected points were plotted in Fig. 2(a). Two different lines were fitted into the scattered data points of the two building materials separately, as shown in Fig. 2(a). The result showed that Ir presented an approximately linear relationship with cosθ, which has a R-Square (r2) value larger than 0.9. Then, data points with different R values but the same θ value were selected. The Ir and 1/ R2 values of these selected points were plotted in Fig. 2(b). According to Fig. 2(b), the relationship between Ir and 1/R2 presented three different trends when 1/R2 < 0.06 (approximately 4 m < R < 10 m), 0.06 < 1/R2 < 0.25 (approximately 2 m < R < 4 m), and 1/ R2 > 0.25 (approximately R < 2 m). When 1/R2 < 0.06 or 0.06 < 1/R2 < 0.25, the relationship between Ir and 1/R2 was approximately linear with r2 > 0.9. When 1/R2 > 0.25, the r2 values of the linear relationship between Ir and 1/R2 were about 0.8. We tested the relationships between Ir and cosθ and 1/R2 for various common building materials. All test results demonstrated that Ir had a linear relationship with cosθ. Also, Ir had different linear relationships with 1/R2 when 1/R2 < 0.06 (approximately 4 m < R < 10 m), 0.06 < 1/R2 < 0.25 (approximately 2 m < R < 4 m), and 1/

R2 > 0.25 (approximately R < 2 m). The different linear relationships with different R ranges were also observed in other studies, and it is because the scanner is equipped with a brightness reducer to protect the scanner from extremely high received laser intensity [25,26]. In the second step, we examined the linear relationship between Ir and cosθ/R2. As discussed before, the relationships between Ir and 1/R2 showed three different linear functions when 4 m < R < 10 m, 2 m < R < 4 m, and R < 2 m. As a result, the relationship between Ir and cosθ/R2 can be written as three separate linear functions for different R values:

cos + b1, 4 m < R < 10 m R2 cos Ir = K2 2 + b2 , 2 m < R < 4 m R cos Ir = K3 2 + b3, R < 2 m R

Ir = K1

(9)

where K1, K3, and K3 are three different values of coefficient K, and b1, b2, and b3 are three different constant terms. We still chose the material of white metal plate and white latex painting as examples. Random samples of laser scan data of the two materials with different Ir and cosθ/R2 values were selected and plotted in Fig. 3. All the sampling data had R values within 10 m. As shown in Fig. 3, the relationship between Ir and cosθ/R2 of two materials showed 4

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Fig. 3. Relation between Ir and cosθ/R2, where LPi and MPi are the respective data points of white latex painting and white metal plate and li is the fitted line of each set of data points.

three different linear relations when 4 m < R < 10 m, 2 m < R < 4 m and R < 2 m. Using linear function Ir = ρK(cosθ/R2) + b to fit the six sets of scattered points, the obtained r2 values of the all fitted lines were larger than 0.9, showing high goodness of fit. We also tested using laser scan data of other materials, and similar conclusions were drawn. In conclusion, the experimental results showed that Eq. (9) was valid. The Ir value had three different linear relationships with cosθ/R2 when 4 m < R < 10 m, 2 m < R < 4 m, and R < 2 m. Moreover, Fig. 3 also demonstrated that the Ir values of different materials may be equivalent at certain R and θ values. However, the slope ρK values of fitted lines were always different because the ρ values of the two materials were different. Therefore, we can get the ρ value of a material based on linear regression fitting of scan data of this building material, and then use the ρ value as a feature for material classification. Considering that the scanning scenes with 2 m < R < 4 m or 4 m < R < 10 m are much more common than R < 2 m for buildings, this study chooses to limit the research scope to R values between 2 m and 10 m. Fig. 4. Calculation of ρ based on neighbouring points of a scan point

3.1.3. Calculation of material reflectance from laser scan data For each laser scan point, the ρ value should be calculated for material classification in the following three steps. First, the neighbouring points of a scan point are obtained by finding all points within a s1 × s1 × s1 bounding box that is centred at this scan point. As shown in Fig. 4, all the points within the blue bounding box become the neighbouring points of the blue point Pi. Second, the cosθ/R2 values of the all neighbouring points are calculated based on Eq. (7), and the Ir values of the neighbouring points are extracted from the laser scan data. Third, a linear function is fitted into the cosθ/R2 and Ir values of neighbouring points according to Eq. (9), and the coefficient ρK1 or ρK2 is obtained. Because K1 or K2 is fixed when using the same TLS, and it is difficult to estimate their specific values, ρK1 or ρK2 is used in this study to represent ρ as the material reflectance.

space is preferred than the RGB colour space because of its better robustness under variable illumination conditions. Therefore, this study also adopts the HSV colour space as the features for automatic building material classification. The translation function from RGB to HSV is described in [28], as follows: R G B ,g= ,b= 255 255 255 Ma = max (r , g, b), Mi = min (r , g, b), = Ma r=

Mi

0 o, if Ma = Mi 60o × o H = 60 ×

3.2. Colour

g g

b b

60 o ×

A TLS with a built-in camera can capture the colour information of each scan point and record as RGB colour space (RGB values from 0 to 255). The RGB values can potentially help in building material classification. Object colours have been extensively used for not only object and material classification using 2D images in the computer vision community, but also material classification using laser scan data [27]. According to the literature, the Hue-Saturation-Value (HSV) colour

60 o ×

+ 0o, if Ma = r and g

b

+ 360 o, if Ma = r and g < b

b

r

r

g

+ 120o , if Ma = g + 240o, if Ma = b

0, if Ma = 0 S=

5

, otherwise Ma V = Ma

(10)

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

3.3. Surface roughness

4.2. Data collection and processing

With the millimetre-level laser beam diameter, TLS can capture geometric characteristics of building materials, e.g. surface roughness (Ra) that measures the irregularity of an area. The feasibility of estimating surface roughness based on laser scan data has been proven by previous research efforts [18]. In general, each category of building material presents different surface roughness. Previous studies show that it is potential to utilize surface roughness estimated from laser scan data for material classification. Therefore, this research also uses surface roughness as a feature for automatic building material classification. In this study, the surface roughness Ra of a scan point is calculated in the following four steps. First, for each scan point Pi, its neighbouring points are obtained as the points within the s2 × s2 × s2 bounding box that is centred at this point, as shown in Fig. 5. Second, a plane is fitted into the neighbouring points using the M-estimator Sample Consensus (MSAC) algorithm [29]. The fitted plane M can be expressed as Ax + By + Cz + D = 0. Third, the orthogonal distance di from each neighbouring point (xi, yi, zi) to the fitted plane is calculated using Eq. (11):

A FARO FocusS70 TLS was used to collect laser scan data of the ten materials. As shown in Table 1, this TLS had a measurement range of 0.6 to 70 m, and a field of view of 300° vertically and 360° horizontally. The beam diameter at exit was 2.12 mm and the divergence was 0.3 mrad. The laser scan data of the ten building materials were collected from buildings in the National University of Singapore (NUS) in Singapore. The collected scan data included both building interiors (i.e. ceilings, walls, and floors) and exterior facades. Furthermore, for each material, scan data were collected from different sites under different illumination conditions to make sure that the collected data of each material were representative. The data processing was executed in MATLAB2019a [31] after the laser scan data were extracted from the TLS's software FARO SCENE [32]. To calculate the ρ value of a scan point, the bounding box size s1 to find neighbouring points was set as 0.5 m. It was to make sure that the cosθ/R2 and Ir values of all the neighbouring points were distributed in a wide range so that the linear fitting provided accurate results for the estimation of ρ. Because the coefficient K was different when 2 m < R < 4m and 4 m < R < 10 m, linear fitting was carried out for two separate datasets with different R values. To calculate surface roughness Ra, the bounding box size s2 to find neighbouring points was set as 0.2 m such that the number of neighbours would be proper to calculate the local surface roughness.

di =

|Axi + Byi + Cz i + D| (11)

A2 + B2 + C 2

Lastly, the surface roughness Ra at point Pi is calculated as the standard deviation of the distances di of the n neighbouring points to the fitted plane [18]:

Ra =

1 n

n

(di i=1

1 µ )2 , with µ = n

4.3. Model training

n

di i=1

After calculating the ρ, HSV, and Ra values, two datasets with 2 m < R < 4m and 4 m < R < 10 m were created for respective training. The dataset with 2 m < R < 4m contained 41,000 data points (approximately 4100 data points for each building material), and the dataset with 4 m < R < 10 m contained 53,000 data points (approximately 5300 data points for each building material). Each data point represents a sample, comprising a building material category label, ρ value, H, S, and V values in the HSV colour space, and Ra value. This paper mainly considered building material classification as a multi-class classification problem. The overall workflow of model training and validation is presented in Fig. 7. Two datasets were trained and validated respectively. We used 80% of each dataset to train the classification model and the rest 20% to test the trained model. To find the best combination of features, different combinations of features (ρ, HSV values, and Ra) were tested. Besides, the Ir value and RGB values were also considered in the comparisons. To find the best classification model, different types of supervised learning classifiers were explored including Decision Trees (DTs), Discriminant Analysis (DAs), Naive Bayes (NBs), Support Vector Machines (SVMs), K-Nearest Neighbours (KNNs), and Ensembles. Each type of the above-mentioned classifiers

(12)

4. Experiments 4.1. Selection of building materials It is necessary to determine a set of common building materials as the target materials in this study. Some construction material libraries have been created for image-based material classification [3,7]. Therefore, taking the existing construction material libraries as a reference, we created a common building material set. As shown in Fig. 6, this study considered ten different categories of materials, including concrete, mortar, stone, metal, painting, wood, plaster, plastic, pottery, and ceramic. Although being extensively used in buildings, glass is not chosen in this study because TLS has difficulty in capturing transparent objects. For the ten categories of materials, one specific commonly-used building material was selected as a sample from each category.

Fig. 5. Calculation of surface roughness Ra based on neighbouring points and fitted plane 6

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Fig. 6. Ten categories of common building materials used in this study Table 1 Parameters of the TLS used in this study [30]. Model S

FARO Focus 70

Range

Field of view

Beam diameter at exit

Beam divergence

0.6–70 m

Vertical: 300° Horizontal: 360°

2.12 mm

0.3 mrad

included multiple algorithms. For example, KNNs contains Fine KNN, Medium KNN, Coarse KNN, Cosine KNN, Cubic KNN, and Weighted KNN algorithm. Hence, the different algorithms were also compared, and the one producing the highest accuracy was selected.

We also used a confusion matrix to show the detailed classification accuracy for each category of materials. The confusion matrix returns the average accuracy per material class. In a confusion matrix, each column represents the predicted material class, and each row represents the true material class.

4.4. Accuracy of the classification model

5. Experimental results

Classification accuracy was used in this research to measure the performance of different models. The accuracy of a classification model can be quantified by Eq. (13).

Accuracy =

TP + TN TP + TN + FP + FN

5.1. Comparisons of features and classifiers We trained different classifiers with different feature combinations in Classification Learner of MATLAB2019a. Two accuracy performance matrixes with different ranges of R are obtained, as shown in Tables 2 and 3. According to Table 2, when 2 m < R < 4 m, using ρ, HSV, and Ra as features and Ensemble as the classifier produced the highest classification accuracy of 96.2%. According to Table 3, when 4 m < R < 10 m, using ρ, HSV, and Ra as features and ensemble as the classifier produced the highest classification accuracy of 97.1%. In conclusion, using ρ, HSV, and Ra as features and Ensemble as the classifier always had the highest classification accuracy, showing an average

(13)

where TP, TN, FP, and FN are the numbers of True Positives, True Negatives, False Positives, and False Negatives, respectively. For instance, if a pottery material is correctly classified as pottery, it will be a TP. If a concrete material is incorrectly classified as pottery, it will be a FP for the pottery class. When a pottery material is not classified as the pottery class, the instance will be a FN for the pottery class. When a material is not pottery material and is correctly classified as a nonpottery material, the instance will be a TN.

Fig. 7. The overall workflow of training and validating models 7

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Table 2 The classification accuracy of different feature combinations and classifiers when 2 m < R < 4 m.

ρ Ra HSV RGB Ir ρ + Ra ρ + HSV Ra + HSV ρ + Ra + HSV

DTs (%)

DAs (%)

NBs (%)

SVMs (%)

KNNs (%)

Ensembles (%)

61.0 28.0 85.7 71.9 40.3 69.3 89.0 86.2 91.0

54.7 26.5 73.4 78.1 38.8 53.8 86.2 73.5 88.7

59.0 28.8 78.5 53.5 40.4 65.9 89.1 82.5 91.2

60.3 30.1 82.7 80.6 38.9 63.8 88.1 89.1 93.1

61.8 28.1 88.1 87.5 35.4 72.7 94.2 90.0 95.2

58.6 31.4 88.6 88.6 40.2 70.4 94.8 92.6 96.2

Table 3 The classification accuracy of different feature combinations and classifiers when 4 m < R < 10 m

ρ Ra HSV RGB Ir ρ + Ra ρ + HSV Ra + HSV ρ + Ra + HSV

DTs (%)

DAs (%)

NBs (%)

SVMs (%)

KNNs (%)

Ensembles (%)

77.0 30.9 71.5 65.8 29.7 85.3 93.5 77.9 94.1

67.6 26.6 67.1 68.2 24.6 71.5 88.9 69.5 89.6

74.9 26.0 71.9 48.8 24.6 78.6 91.2 77.0 95.2

72.0 26.0 72.2 60.8 24.5 79.1 90.8 80.0 90.5

77.5 24.7 75.0 75.0 28.5 87.2 95.6 82.7 96.6

75.7 29.5 77.7 76.3 29.4 86.8 95.7 84.7 97.1

classification accuracy of 96.7% when 2 m < R < 10 m. As mentioned above, this study used ρ instead of Ir, and HSV instead of RGB as the features for material classification. The experiments also tested the classification accuracy when using ρ, Ir, HSV, or RGB as the only feature. According to Tables 2-3, using ρ as the only feature had accuracies of 61.8% and 77.5%, respectively, which were much higher than the accuracies of 40.4% and 29.7% when using Ir. The experimental results showed that ρ was a much better feature for material classification than Ir. For the comparisons between HSV and RGB, using HSV colours as the only features had accuracies of 88.6% and 77.7%, respectively. The accuracies became 88.6% and 76.3% when using RGB colours. Although the accuracies were very similar, HSV colours still had a better overall performance than RGB colours. The comparisons proved that selecting ρ and HSV colours as features was preferred. We further compared the performances when using any two of ρ, HSV, and Ra as the features (i.e. ρ + Ra, ρ + HSV, and Ra + HSV). As shown in Tables 2-3, using the combination of any two features performed better than using any single feature on classification accuracy. The results indicated that all the features were useful for improving classification accuracy. Regarding the comparisons of classifiers, the experimental results showed that the ensemble algorithm was the best classifier. The result is consistent with the conclusion introduced in a previous study [6]. According to the experimental results presented in both Tables 2 and 3, the algorithm with the highest classification accuracy was the Bootstrap-aggregated Decision Trees (namely Bagged Trees), which is one of the Ensemble algorithms.

showed the highest TP rate of 100%, indicating that all the data of the painting material were correctly classified. The mortar material presented the lowest TP rate of 92%. It is shown that 7% of mortar data were wrongly classified as stone. This phenomenon indicated that the features of stone material were similar to these of mortar material. To further analyse the reasons for the low TP rate of mortar material, a parallel coordinates plot was drawn to visualize the multivariate data. As shown in Fig. 9, we compared the distributions of features (ρ, H, S, V, or Ra) for stone, mortar, and painting. For a certain feature, if the data of one material have a higher dispersion degree or have large overlap with another material, this feature will be less useful on improving the classification accuracy of this material. Compared to painting material, it was found that the mortar material had a higher dispersion degree on multiple features (e.g. S, V, and Ra), which resulted in large overlaps with other materials and caused the low TP rate of mortar. Meanwhile, it is also found that the H and S values of mortar were heavily overlapping with those of stone material, which explained why 7% of mortar data were wrongly classified as stone. 5.3. Discussion Building material classification is typically a multi-class classification problem, aiming to classify an unknown object into one of several pre-defined categories. However, it can also be a one-class classification problem that distinguishes one class from all other classes, if one particular material needs to be extracted. In this study, to test how accurately one building material can be recognized, one-class classification method was also implemented. The training of a one-class classifier is to define a classification boundary for a specific class so that the classification model can distinguish this specific class from other classes. Currently, main one-class classification methods can be categorized into three types: 1) densitybased methods, 2) reconstruction-based methods, and 3) boundarybased methods [33]. Some researchers [33,34] compared the accuracy performance of different one-class classification methods and found that One-class Support Vector Machine (OC-SVM) method and Support Vector Data Description (SVDD) method have better accuracy than other methods. The OC-SVM method aims to find a maximum margin hyperplane in the feature space to best separate the mapped data from

5.2. Accuracy analysis of each building material To further understand the material classification results, we used the confusion matrix to analyse the classification performance for the case with the highest classification accuracy (i.e. using ρ + Ra + HSV as features and Bagged Trees algorithm as the classifier). The classification results for both 2 m < R < 4 m and 4 m < R < 10 m were averaged, and the confusion matrix is shown in Fig. 8. Each row of the confusion matrix shows the percentage of TP and FN rates for the true class of the building material sample. It is found that all ten categories of materials produced TP rate of at least 92%. The painting material 8

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Fig. 8. Confusion matrix of the case with the highest classification accuracy (i.e. using ρ + Ra + HSV as features and Bagged Trees algorithm as the classifier)

respectively. For the OC-SVM classifier, the classification accuracy for different materials was between 81.0% and 89.0%. For the SVDD classifier, the classification accuracy for different materials was between 83.4% and 90.4%. The averaged accuracy for OC-SVM and SVDD was 85.5% and 86.5%, respectively. Though the overall accuracy of treating material classification as a one-class classification problem is lower than treating it as a multi-class classification problem, the accuracy is still acceptable. This result further demonstrated the feasibility of the proposed method in practical applications. 6. Limitations and future works Although this study shows promising results, future works are needed to address a few limitations. First, this study only considers cases where the target surface is plane and consists of a single building material. For the calculation of material reflectance and surface roughness, the estimated values would be different if the neighbouring points contain other materials. In future works, one possible direction is to check the consistency of material using colours because colours of a point are totally determined by this point only and not affected by neighbouring points. In addition, when the target surface is not a plane, the calculation of surface roughness would be inaccurate. To handle this problem, future research is needed to first examine whether the surface is planar or curved before estimating the surface roughness. Second, material humidity has a negative impact on the accuracy of calculating material reflectance ρ. Previous studies have found that the existence of moisture can change the reflected intensity Ir from a surface [16], which makes the estimation of ρ inaccurate. Further research is needed to get rid of the effects of material humidity when calculating the value of material reflectance ρ. Third, this study considered only ten common building materials. The classification is therefore limited to these ten materials only. Future work is needed to further extend the database with data of other

Fig. 9. Parallel coordinates plot of mortar, stone, and painting materials with the features as horizontal axis and the dispersion degree of data as vertical axis.

the origin [35], and the SVDD method learns a closed boundary around the data in the form of a hypersphere characterized by a centre and radius [36]. We divided the scan data set of each building material into a training dataset (80%) and a testing dataset (20%). For each of the ten materials, OC-SVM and SVDD classifiers were trained separately using a training dataset containing only data of this material. After obtaining the trained classifier, the testing dataset of this material was used to test the accuracy performance of the trained classifier. Similar to multi-class classification, ρ, Ra, and HSV values were used as features in one-class classification. For each classifier, experiments were conducted to find the optimal model parameters, such as the kernel function, γ value of the kernel function and cost coefficient. All these steps were completed in MATLAB2019a with the LIBSVM toolbox [31]. Table 4 presents the accuracy of each building material in one-class classification when 2 m < R < 4 m and 4 m < R < 10 m, 9

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

Table 4 The classification accuracy of each building material in one-class classification. OC-SVM

Ceramic Concrete Stone Metal Mortar Painting Plaster Plastic Pottery Wood

SVDD

2m < R < 4m (%)

4m < R < 10m (%)

2m < R < 4m (%)

4m < R < 10m (%)

87.9 85.0 84.8 85.5 81.0 88.0 83.2 83.6 87.0 83.4

89.0 85.9 85.5 85.9 82.7 88.8 85.0 85.9 88.2 84.6

88.0 86.3 85.9 84.0 87.7 89.0 83.6 85.8 84.8 83.4

89.4 87.7 87.5 84.7 88.7 90.4 85.8 86.7 85.9 84.0

building materials. In addition, this study did not consider glass as a building material because TLS cannot well capture the laser scan data for transparent materials. However, some studies have suggested a variety of effective ways to extract glass windows from point clouds collected by TLS [37,38]. Future research is needed to include glass in the material classification method. Fourth, some studies have demonstrated that combing multiple features derived from 2D images can effectively improve classification accuracy (e.g. combing RGB colour with H of HSV colour and projection or combing HSV colour with texture). TLS with a built-in camera can capture not only the RGB colours themselves but also the material texture, projection, etc. Hence, TLS data-based building material classification method may be able to reach a higher accuracy if thoroughly adopting multiple 2D image features derived from scan data instead of using only HSV colour, which will be future work.

and Bagged Trees algorithm as the classifier. The painting material showed the highest TP rate of 100%. Meanwhile, the mortar material presented the lowest TP rate of 92%, and 7% of mortar data were wrongly classified as stone because the two materials share similar feature values. To further test how accurately one building material can be recognized, we also adopted one-class classification method to train and validate the classification models. OC-SVM and SVDD, which are verified to be the best classifiers for one-class classification problem in previous studies, were adopted in this paper. The experiments showed that the averaged accuracy for OC-SVM and SVDD was 85.5% and 86.5%, respectively, which further demonstrated the feasibility of the proposed method.

7. Conclusion

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of competing interest

This study proposes a TLS data-based method for classifying common building materials. In the proposed method, material reflectance ρ, HSV colours, and surface roughness Ra are used as classification features. The ρ value is an intrinsic property of a certain material, and it can be inferred from the reflected laser intensity Ir. It is found that Ir has linear relationships with cosθ/R2, and ρ is a part of the coefficient of the linear function. Therefore, the ρ value of each scan point is obtained by fitting a linear function into the Ir and cosθ/R2 values of the neighbouring points of the scan point. The HSV colours are used in this study instead of the RGB colours because the HSV colours show better robustness to varying lighting conditions. The HSV colours are calculated from RGB colours that are obtained from the raw laser scan data. The Ra value is calculated as the average orthogonal distance from the neighbouring points of a scan point to the fitted plane of the neighbouring points. To validate the proposed technique, this study selected ten common building materials, including concrete, mortar, stone, metal, painting, wood, plaster, plastic, pottery, and ceramic. A FARO FocusS70 TLS was used to collect laser scan data of the ten different materials. The laser scan data were processed in MATLAB2019a to calculate the abovementioned features. To find the best combination of features, the different combinations of features were tested. In addition, different supervised learning classifiers were explored including Decision Trees (DTs), Discriminant Analysis (DAs), Naive Bayes (NBs), Support Vector Machines (SVMs), KNearest Neighbours (KNNs), and Ensembles. The experimental results showed that using ρ, HSV, and Ra as features and Ensemble as the classifier realized the highest classification accuracy of 96.7%. Experimental results also validated that ρ was a much better feature for material classification than Ir, and HSV colour outperformed RGB colour. Further analyses showed that all the ten categories of materials produced TP rate of at least 92% when using ρ + Ra + HSV as features

References [1] J. DeGol, M. Golparvar-Fard, D. Hoiem, Geometry-informed material recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1554–1562, , https://doi.org/10.1109/cvpr.2016. 172. [2] J.E. Meroño, A.J. Perea, M.J. Aguilera, A.M. Laguna, Recognition of materials and damage on historical buildings using digital image classification, S. Afr. J. Sci. 111 (1–2) (2015) 1–9, https://doi.org/10.17159/sajs.2015/20140001. [3] K.K. Han, M. Golparvar-Fard, Appearance-based material classification for monitoring of operation-level construction progress using 4D BIM and site photologs, Autom. Constr. 53 (2015) 44–57, https://doi.org/10.1016/j.autcon.2015.02.007. [4] Q. Wang, M. Kim, Applications of 3D point cloud data in the construction industry: a fifteen-year review from 2004 to 2018, Adv. Eng. Inform. 39 (2019) 306–319, https://doi.org/10.1016/j.aei.2019.02.007. [5] Q. Lu, S. Lee, Image-based technologies for constructing as-is building information models for existing buildings, J. Comput. Civ. Eng. 31 (4) (2017) 4017005, , https://doi.org/10.1061/(asce)cp.1943-5487.0000652. [6] H. Son, C. Kim, N. Hwang, C. Kim, Y. Kang, Classification of major construction materials in construction environments using ensemble classifiers, Adv. Eng. Inform. 28 (1) (2014) 1–10, https://doi.org/10.1016/j.aei.2013.10.001. [7] A. Dimitrov, M. Golparvar-Fard, Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections, Adv. Eng. Inform. 28 (1) (2014) 37–49, https://doi.org/10.1016/j.aei.2013.11.002. [8] Q. Lu, S. Lee, L. Chen, Image-driven fuzzy-based system to construct as-is IFC BIM objects, Autom. Constr. 92 (2018) 68–87, https://doi.org/10.1016/j.autcon.2018. 03.034. [9] S.Y. Chen, Y.F. Li, W. Wang, J. Zhang, Active Sensor Planning for Multiview Vision Tasks, 1 Springer, 2008, p. 25. [10] E.B. Anil, P. Tang, B. Akinci, D. Huber, Deviation analysis method for the assessment of the quality of the as-is building information models generated from point cloud data, Autom. Constr. 35 (2013) 507–516, https://doi.org/10.1016/j.autcon. 2013.06.003. [11] Z. Zhu, I. Brilakis, Parameter optimization for automated concrete detection in image data, Autom. Constr. 19 (7) (2010) 944–953, https://doi.org/10.1016/j. autcon.2010.06.008. [12] M. Franceschi, G. Teza, N. Preto, A. Pesci, A. Galgaro, S. Girardi, Discrimination

10

Automation in Construction 110 (2020) 103017

L. Yuan, et al.

[13]

[14]

[15] [16]

[17]

[18] [19] [20] [21] [22] [23]

[24]

between marls and limestones using intensity data from terrestrial laser scanner, ISPRS J. Photogramm. Remote Sens. 64 (6) (2009) 522–528, https://doi.org/10. 1016/j.isprsjprs.2009.03.003. J. Armesto-González, B. Riveiro-Rodríguez, D. González-Aguilera, M.T. Rivas-Brea, Terrestrial laser scanning intensity data applied to damage detection for historical buildings, J. Archaeol. Sci. 37 (12) (2010) 3037–3047, https://doi.org/10.1016/j. jas.2010.06.031. B. Riveiro, P.B. Lourenço, D.V. Oliveira, H. González Jorge, P. Arias, Automatic morphologic analysis of quasi-periodic masonry walls from LiDAR, Computer-Aided Civil and Infrastructure Engineering 31 (4) (2016) 305–319, https://doi.org/10. 1111/mice.12145. L.J. Sánchez-Aparicio, S. Del Pozo, L.F. Ramos, A. Arce, F.M. Fernandes, Heritage site preservation with combined radiometric and geometric analysis of TLS data, Autom. Constr. 85 (2018) 24–39, https://doi.org/10.1016/j.autcon.2017.09.023. M.U. Hassan, A. Akcamete-Gungor, C. Meral, Investigation of terrestrial laser scanning reflectance intensity and RGB distributions to assist construction material identification, lean and computing in construction congress, Proceedings of the Joint Conference on Computing in Construction, Volume 1 2017. E. Valero, F. Bosché, A. Forster, Automatic segmentation of 3D point clouds of rubble masonry walls, and its application to building surveying, repair and maintenance, Autom. Constr. 96 (2018) 29–39, https://doi.org/10.1016/j.autcon.2018. 08.018. E. Valero, A. Forster, F. Bosché, E. Hyslop, L. Wilson, A. Turmel, Automated defect detection and classification in ashlar masonry walls using machine learning, Autom. Constr. 106 (2019) 102846, , https://doi.org/10.1016/j.autcon.2019.102846. K. Tan, X. Cheng, Surface reflectance retrieval from the intensity data of a terrestrial laser scanner, J. Opt. Soc. Am. A 33 (4) (2016) 771–778, https://doi.org/10.1364/ josaa.33.000771. O. Svelto, D.C. Hanna, Principles of Lasers, vol. 4, Springer, 9781441913012 (1998), (pp). B. Höfle, N. Pfeifer, Correction of laser scanning intensity data: data and modeldriven approaches, ISPRS J. Photogramm. Remote Sens. 62 (6) (2007) 415–433, https://doi.org/10.1016/j.isprsjprs.2007.05.008. M.E. Becker, Evaluation and characterization of display reflectance, Displays 19 (1) (1998) 35–54, https://doi.org/10.1016/s0141-9382(98)00029-8. S. Kaasalainen, A. Jaakkola, M. Kaasalainen, A. Krooks, A. Kukko, Analysis of incidence angle and distance effects on terrestrial laser scanner intensity: search for correction methods, Remote Sens. 3 (10) (2011) 2207–2221, https://doi.org/10. 3390/rs3102207. W. Fang, X. Huang, F. Zhang, D. Li, Intensity correction of terrestrial laser scanning data by estimating laser transmission function, IEEE Trans. Geosci. Remote Sens. 53

(2) (2015) 942–951, https://doi.org/10.1109/tgrs.2014.2330852. [25] S. Kaasalainen, A. Krooks, A. Kukko, H. Kaartinen, Radiometric calibration of terrestrial laser scanners with external reference targets, Remote Sens. 1 (3) (2009) 144–158, https://doi.org/10.3390/rs1030144. [26] S. Kaasalainen, A. Kukko, T. Lindroos, P. Litkey, H. Kaartinen, J. Hyyppa, E. Ahokas, Brightness measurements and calibration with airborne and terrestrial laser scanners, IEEE Trans. Geosci. Remote Sens. 46 (2) (2008) 528–534, https:// doi.org/10.1109/tgrs.2007.911366. [27] Q. Wang, J.C. Cheng, H. Sohn, Automated estimation of reinforced precast concrete rebar positions using colored laser scan data, Computer-Aided Civil and Infrastructure Engineering 32 (9) (2017) 787–802, https://doi.org/10.1111/mice. 12293. [28] A.R. Smith, Color gamut transform pairs, ACM Siggraph Computer Graphics 12 (3) (1978) 12–19, https://doi.org/10.1145/800248.807361. [29] P.H. Torr, A. Zisserman, MLESAC: a new robust estimator with application to estimating image geometry, Comput. Vis. Image Underst. 78 (1) (2000) 138–156, https://doi.org/10.1006/cviu.1999.0832. [30] FARO. FARO® LASER SCANNER FOCUS. Available from: https://www.faro.com/ en-sg/products/construction-bim/faro-laser-scanner-focus/. Retrieved June 18, 2019. [31] MathWorks. MATLAB R, Available from, 2019. https://www.mathworks.com/ products/new_products/latest_features.html?s_tid=hp_release_2019a>. [32] FARO, FARO SCENE, Available from https://www.faro.com/en-gb/products/ construction-bim-cim/faro-scene/software/. [33] B. Krawczyk, M. Galar, M. Woźniak, H. Bustince, F. Herrera, Dynamic ensemble selection for multi-class classification with one-class classifiers, Pattern Recogn. 83 (2018) 34–51, https://doi.org/10.1016/j.patcog.2018.05.015. [34] L. Swersky, H.O. Marques, J. Sander, R.J. Campello, A. Zimek, On the evaluation of outlier detection and one-class classification methods, IEEE (2016) 1–10, https:// doi.org/10.1109/dsaa.2016.8. [35] B. Schölkopf, J.C. Platt, J. Shawe-Taylor, A.J. Smola, R.C. Williamson, Estimating the support of a high-dimensional distribution, Neural Comput. 13 (7) (2001) 1443–1471, https://doi.org/10.1162/089976601750264965. [36] D.M. Tax, R.P. Duin, Support vector domain description, Pattern Recogn. Lett. 20 (11−13) (1999) 1191–1199, https://doi.org/10.1016/s0167-8655(99)00087-2. [37] S. Pu, G. Vosselman, Knowledge based reconstruction of building models from terrestrial laser scanning data, ISPRS J. Photogramm. Remote Sens. 64 (6) (2009) 575–584, https://doi.org/10.1016/j.isprsjprs.2009.04.001. [38] S. Pu, Automatic Building Modeling from Terrestrial Laser Scanning, Springer, 2008, pp. 147–160, https://doi.org/10.1007/978-3-540-72135-2_9 978-3-54072135-2.

11