Removing vibrations in 3D reconstruction using multiple laser stripes

Optics and Lasers in Engineering 53 (2014) 51–59

Contents lists available at ScienceDirect

Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng

Removing vibrations in 3D reconstruction using multiple laser stripes Rubén Usamentiaga n, Julio Molleda, Daniel F. Garcia, Francisco G. Bulnes Department of Computer Science and Engineering, University of Oviedo, Campus de Viesques, 33204 Gijón, Asturias, Spain

art ic l e i nf o

a b s t r a c t

Article history: Received 24 June 2013 Received in revised form 15 July 2013 Accepted 6 August 2013 Available online 13 September 2013

Vibrations present a major challenge for accurate 3D reconstruction of moving objects. When the movement of the object is affected by vibrations, the reconstruction system estimates height profiles incorrectly, which leads to major errors in the reconstructed shape of the object. This work proposes a laser-based 3D reconstruction method that uses multiple laser stripes and produces redundant information that can be used to estimate and remove vibrations from the final 3D reconstruction. The effects of vibrations on systems with a single laser stripe are analyzed, and the detailed mathematical procedure to model and to estimate vibrations from multiple stripes is presented. Vibrations are then removed from the reconstructed shape, providing a vibration free result. Different tests are carried out with synthetic and real data with excellent performance. The analysis of the results indicates that a laserbased 3D reconstruction method using two laser stripes has similar or even better performance compared with systems with more stripes. Furthermore, it presents a far more cost efficient solution. & 2013 Elsevier Ltd. All rights reserved.

Keywords: Vibrations 3D reconstruction Laser triangulation Geometric transformations

1. Introduction 3D reconstruction is one of the most widely studied and fundamental problems in computer vision [1]. Many different methods have been explored to acquire 3D information from objects. These methods can be classified as active or passive [2]. Active methods use an external light source, while passive methods are not based on any specific light source. Among passive methods, stereo vision [3] is the most commonly used method. Other passive methods are based on shape detection using the shade, texture or motion of the object [4]. Active methods are more commonly used in 3D reconstruction because they produce better results. Structured [5] and laser light [6] are two types of light sources used in active methods, with laser being the most common. In laser-based 3D reconstruction methods, a laser stripe is projected over an object. Then, the system acquires images of the projection while the object or the laser stripe is moved forward. The projection of the stripe is deformed according to the shape of the object it is illuminating. Thus, using optical triangulation it is possible to calculate 3D information about the shape of the object. Laser-based 3D reconstruction methods are used for many different applications, such as buildings [7], road surface [8], steel strips [9], train rails [10], or weld seams [11]. In these environments, where surrounding light and laser power is not fully

n

Corresponding author. Tel.: þ 34 985 182626; fax: þ 34 985 181986. E-mail addresses: [email protected], [email protected] (R. Usamentiaga). 0143-8166/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.optlaseng.2013.08.001

controlled, laser-based 3D reconstruction methods suffer from accuracy and robustness problems [12]. Accuracy is significantly affected by the extraction of the laser stripe from the images. Previous works have focused on this problem proposing different alternatives [13–16]. Another major issue affecting accuracy in laser-based 3D reconstruction methods is vibrations. These 3D reconstruction methods are based on the acquisition images of the laser projection while either the object or the laser is moved forward. This movement can be affected by vibrations, resulting in an incorrect estimation of height profiles that propagates to 3D reconstruction. When the environment cannot be totally controlled, for example the movement of a train rail on a roll path, the effects of vibrations lead to major errors. One possible approach to remove or reduce the effects of vibrations in 3D reconstruction is based on using previous knowledge about the object or about the types of vibrations affecting the movement of the object. If the shape of the object is known a priori, a set of points in one height profile and the correspondent points in the next profile can be used to estimate the type of vibration and to remove its effects on 3D reconstruction. For example, this approach has been used when the shape of the object consists of two approximately straight lines connected by a round corner [17]. Another option is based on applying knowledge about the type of vibrations. For example steel strips are known to be composed of fibers with different wavelengths. However, these fibers have a known minimum wavelength of approximately 0.5 m. Everything below this limit is considered the resulting effect from vibrations. Thus, a low-pass filter with an adequate cutoff frequency can be used to remove this kind of vibrations from steel strips [18].

52

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

When the object has a known shape, for example a perfect plane, 3D reconstruction is not really necessary. However, laserbased triangulation can be used to characterize vibrations [19,20]. These methods estimate vibrations from the relative depth variations. A similar approach can be used with laser Doppler vibrometry [21,22]. In this case, vibrations are estimated from the Doppler shift of the reflected laser beam frequency. All these methods assume a static shape of the object, i.e., the shape does not change over time. Thus, only vibrations are responsible for variations in measurements. When the shape of the object changes, these methods are not applicable. A known shape of the object or a known type of vibrations could be considered 3D reconstruction invariants. When these invariants do not exist these procedures to remove vibrations are not possible. Therefore, in order to remove vibrations new invariants need to be created. This work proposes a method to remove vibrations in 3D reconstruction based on using multiple laser stripes. When more than one laser stripe is used, a new invariant is created: the displacement from one laser stripe to the next is not dependent on vibrations, but only on the shape of the object. Therefore, the real shape of the object can be obtained without the interference of vibrations. However, the exact number of stripes needed depends on the complexity of the vibrations. When vibrations only consist of vertical translations, two laser stripes are required. On the other hand, when vibrations are a combination of vertical translations and rotations, three laser stripes are required. More than three laser stripes produce an over-determined system that delivers a more robust solution. In this work all these possibilities are analyzed and the mathematical procedures to remove vibrations in these cases are developed. Then, the proposed procedure is applied to synthetic and real data. The accuracy of the results is discussed with and without noisy data and compared with the benefits of using more than two laser stripes. Results indicate that a laser-based 3D reconstruction method using two laser stripes presents a far more cost efficient solution than other approaches, also with similar performance.

2. Modeling vibrations Vibrations are periodic or random motion from an equilibrium position. In the case of laser-based 3D reconstruction methods, vibrations are an undesirable phenomenon that produces movements of the object as it moves forward. The consequences of these vibrations are particularly harmful for 3D reconstruction because the projection of the stripe is not only deformed due to the shape of the object itself but also due to the movement of the object caused by vibrations. Therefore, the 3D reconstruction of an Laser projector

object with a flat surface could result in an object with a wavy surface, as can be seen in Fig. 1. In a different scenario, it is the laser projector that moves and is affected by vibrations, for example in a robot arm. However, the result is the same but from a different point of view. In this work, vibrations are considered from the point of view of a moving object, although it would be equivalent from the point of view of a moving laser projector. The two types of vibrations affecting laser-based 3D reconstruction methods are translations and rotations. Translations are vertical movements of the object similar to bouncing, while rotations are semi-circular movements of the object around a pivot position similar to rocking. These vibrations occur while the object moves forward. The two types of vibrations can be present at the same time, although translations are much more common than rotations. The type of translations depends on the way the object is moved. When the object is moved using a conveyor belt, translations can appear as random vertical movements without a pattern. On the other hand, semi-sinusoidal translations can appear if the object hits the conveyor belt periodically. When the object is not actually touching the ground, for example when moving a sheet of paper, vibrations can show a sinusoidal pattern while the object moves forward along a roll path, upwards and downwards. Fig. 2 shows the resulting effect of translations in 3D reconstruction when a single laser stripe is used. The figure shows a simplified example with one single point from a side view. In this case the object moves forward while it is affected by vibrations, in particular with translations. In t1 a new point is acquired, but its position not only depends on the shape of the object but also on the vertical translation from t0 to t1. In t2 and t3 new points are acquired that are used to create a reconstruction of the shape of the object. However, the resulting reconstruction is corrupted due to the vertical translations, as can be seen in the figure. Rotations are less common because they only occur when the object is held at two distant points along the length of the object, such as steel strips. In these cases, the movement of the object can be affected by rotations around a pivot position, perpendicular to the movement. The two types of vibrations can be described using a mathematical model based on geometric transformations. The mathematical principles are described next.

2.1. Geometric transformations There are four classes of geometric transformations. These classes can be described in terms of those elements or quantities that are preserved or invariant, such as for example the distance or the angle. Also, they form a hierarchy of transformations, where a class extends the previous class. Camera

3D reconstruction Rotations

Movement of the object

Translations

Fig. 1. General architecture for laser-based 3D reconstruction methods with one single laser stripe.

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

53

Height

t0

Length

t1 Height

Vertical translation

Movement

Length

t2 Height

Vertical translation

Movement

Length

t3 Height

Vertical translation

Reconstruction

Movement

Real shape Length

Fig. 2. Effects of vibrations in laser-based 3D reconstruction with one single laser stripe.

Isometry is the most basic class. This class of transformation preserves the Euclidean distance. When the isometry also preserves the orientation it is called Euclidean transformation. This type of isometry is a composition of translation and rotation. Euclidean transformations model the motion of rigid objects. The second class of transformation is similarities. A similarity transformation is an isometry composed with an isotropic scaling (uniform scaling in all axes). This class of transformation does not preserve the distance, but the ratio of two lengths. An affine transformation is an extended version of a similarity transformation with an anisotropic scaling added (different scaling for each axis). This class of transformation preserves the ratio of areas and parallel lines. Angles are not preserved, so that shapes are skewed under the transformation. The fourth class is projective transformations. This class generalizes an affine transformation by considering the non-linear effects of the projection. The projective invariant is the cross ratio of lengths on a line. These classes of geometric transformations are composed of simple transformations, such as translation, rotation, or reflection. Translation is not linear in 2D, i.e., it cannot be represented with 2  2 matrix. Thus, homogeneous coordinates are used by adding a third coordinate. A 2D point, P ¼ ðx; yÞ, is transformed to homogeneous coordinates using

P¼

x y

0 1 x B C -P ¼ @ y A

!

1

ð1Þ

In homogeneous coordinates, a translation can be expressed using (2), where tx and ty are the horizontal and vertical translations, respectively 0 1 1 0 tx B C T ¼ @ 0 1 ty A ð2Þ 0 0 1 Scaling can be expressed using (3), where s is the scaling factor, in this case equal for both axes 0 1 s 0 0 B C S¼@0 s 0A ð3Þ 0 0 1 Rotation can be expressed using (4), where θ is the rotation angle 0 1 cos ðθÞ  sin ðθÞ 0 B C R ¼ @ sin ðθÞ ð4Þ cos ðθÞ 0 A 0

0

1

These three translations can be combined into a single matrix, H, using matrix multiplication (5). H represents a similarity transformation. Removing the scale parameter would result in Euclidean transformation. These combined transformations are applied from right to left 0 1 s cos ðθÞ s sin ðθÞ t x B C H ¼ T  S  R ¼ @ s sin ðθÞ s cos ðθÞ t y A ð5Þ 0

0

1

54

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59 Laser projectors

Any point could be transformed using (6), which produces Eqs. (7) and (8) 10 1 0 1 0 x′ x s cos ðθÞ s sin ðθÞ t x CB C B C B P′ ¼ V  P-@ y′ A ¼ @ s sin ðθÞ s cos ðθÞ t y A@ y A ð6Þ 1 1 0 0 1 x′ ¼ x  s  cos ðθÞy  s  sin ðθÞ þ tx

ð7Þ

y′ ¼ x  s  sin ðθÞ þy  s  cos ðθÞ þt y

ð8Þ

Camera

3D reconstruction

Rotations

Movement of the object

Translations

Fig. 3. Improved architecture for laser-based 3D reconstruction methods.

2.2. Modeling vibrations based on geometric transformations Two types of vibrations affect objects while they are moved or transported under the laser stripe: translations and rotations. Translations are vertical movements of the object while it is being moved. This type of vibrations could be modeled using the translation matrix (9), where ty represents the vertical movement 0 1 1 0 0 B C T ¼ @ 0 1 ty A ð9Þ 0 0 1 The other type of vibrations considered is rotations. This type of vibrations occurs on flexible objects, such as steel strips or paper sheets. Rotations are semi-circular movements of the object around a pivot position. This type of vibrations could be modeled using the rotation matrix using (4), where θ represents the rotation angle. However, this transformation is only valid when the pivot position is at the origin. When the rotation is not carried out around the origin but around a pivot point ðxp ; yp Þ, it is necessary to perform previously a translation from the pivot point to the origin, a rotation, and finally a translation from the origin back to the pivot point. The process can be expressed with a combination of two translations and one rotation matrix using 10 0 10 1 1 0 xp 1 0 xp cos ðθÞ  sin ðθÞ 0 CB B CB C R ¼ @ 0 1 yp A@ sin ðθÞ ð10Þ cos ðθÞ 0 A@ 0 1 yp A 0 0 1 0 0 1 0 0 1 A particular type of vibration V composed of translation and rotation can be modeled as a combination of (9) and (10), using matrix multiplication 0 1 cos ðθÞ  sin ðθ Þ xp cos ðθ Þðt y yp Þ sin ðθ Þ þ xp B C V ¼ R  T ¼ @ sin ðθÞ cos ðθÞ xp sin ðθÞ þ ðt y yp Þ cos ðθÞ þ yp A 0

0

1

ð11Þ A point P over the surface of an object with vibrations V is transformed using (12), which produces Eqs. (13) and (14) P′ ¼ V  P

ð12Þ

x′ ¼ x cos ðθÞy sin ðθÞ þ xp cos ðθÞðt y yp Þ sin ðθÞ þ xp

ð13Þ

y′ ¼ x sin ðθÞ þy cos ðθÞxp sin ðθÞ þ ðt y yp Þ cos ðθÞ þyp

ð14Þ

These equations create a model for vibrations that determines the transformation of points in space. The model is parametrized by four values: θ, xp, yp, and ty.

3. Estimating vibrations The model for vibrations is based on four unknown values: θ, xp, yp, and ty. Without a priori information about the shape of the object or about the type of vibrations it is not possible to estimate these values. The proposed procedure in this work is to use

multiple laser stripes. This provides redundant information that can be used to estimate vibrations, and thus, to remove them. The proposed architecture can be seen in Fig. 3. The projections of the laser stripes on the object are very close. Thus, the movement of the object caused by vibrations affects the deformation of the laser stripes equally. When the object moves forward, a new image is acquired. In the new image some of the laser stripes are projected at the same position as the previous image. The shape of the object in these two images is the same; the difference is only caused by vibrations. Therefore, this redundant information can be used to estimate vibrations. Fig. 4 shows how two laser stripes can be used to remove vertical translations. Again, the figure shows a side view with one longitudinal line, but the idea is the same with more points. In t0 two points are acquired. The vertical distance between these two points, P 0 ðt 0 Þ and P 1 ðt 0 Þ, only depends on the shape of the object. Later in t1, after the object is moved, two new points are acquired, P 0 ðt 1 Þ and P 1 ðt 1 Þ. The point P 0 ðt 1 Þ should be in the same position as P 1 ðt 0 Þ, but a vertical translation has modified it. The vertical translation can be estimated by subtracting P 0 ðt 1 Þ from P 1 ðt 0 Þ. The resulting value, V ðt 1 Þ is the vertical translation from t0 to t1. This vertical translation has affected P 0 ðt 1 Þ and P 1 ðt 1 Þ equally, therefore vibrations can be removed in P 1 ðt 1 Þ by adding Vðt 1 Þ, which results in PP 1 ðt 1 Þ. In t2 a new vertical translation occurs. This new vertical translation, Vðt 2 Þ, is estimated as P 1 ðt 1 ÞP 0 ðt 2 Þ. Now, vibrations in P 1 ðt 2 Þ are removed by adding Vðt 2 Þ and also Vðt 1 Þ, i.e., PP 1 ðt 2 Þ ¼ P 1 ðt 2 Þ þ Vðt 1 Þ þ Vðt 2 Þ. If V ðt 1 Þ þ Vðt 2 Þ is called VVðt 2 Þ, then PP 1 ðt 3 Þ would be calculated as P 1 ðt 3 Þ þ VVðt 2 Þ þ Vðt 3 Þ. For a general case, the position of a point PP 1 ðt i Þ without vibrations is calculated as P 1 ðt i Þ þ VVðt i1 Þ þ Vðt i Þ. This procedure completely removes vertical translations, producing a perfect reconstruction of the shape of the object. For more complex vibrations that include rotations more than two laser stripes are required. The mathematical procedure is described next.

3.1. Estimating vibrations with three or more laser stripes In (12) there are four unknown values: θ, xp, yp, and ty. These values define the vibration. In order to calculate these four values four equations are needed. These four equations can be created from two points, P 1 ¼ ðx1 ; y1 Þ and P 2 ¼ ðx2 ; y2 Þ, and from the obtained values after the vibration has transformed their positions and they are acquired again, P ′1 ¼ ðx′1 ; y′1 Þ and P ′2 ¼ ðx′2 ; y′2 Þ. This means that at least three laser stripes are required, so that two points are measured twice, as can be seen in Fig. 3. Each pair of points produces two equations, one for x and one for y. Therefore, with the two pairs of points four equations are generated. Gathering all values, model (15) can be written as 0

x′1 B ′ @ y1 1

1 0 x1 x′2 C B y′2 A ¼ V @ y1 1 1

x2

1

C y2 A 1

ð15Þ

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

55

Height

t0

Length

t1 Height

Vertical translation

Movement

Length

t2 Height

Vertical translation

Movement

Length

t3 Height

Vertical translation

Real shape

Movement

Reconstruction

Length

Fig. 4. Removing vibrations in laser-based 3D reconstruction using multiple stripes.

The system of equations in model (15) is non-linear and cannot be solved analytically. Therefore, it is not an appropriate model for fast and accurate computation. However, it can be transformed into a linear system by creating a set a intermediate variables. Four variables are created, V1, V2, V3, and V4, which correspond to V 1 ¼ cos ðθÞ

ð16Þ

V 2 ¼ sin ðθÞ

ð17Þ

V 3 ¼ xp cos ðθÞðt y yp Þ sin ðθÞ þ xp

ð18Þ

V 4 ¼ xp sin ðθÞ þðt y yp Þ cos ðθÞ þ yp

ð19Þ

Substituting (16)–(19) in (11) produces (20). Using this definition of V, a point P ¼ ðx; yÞ over the surface of an object with vibrations V is transformed using (21), which produces Eqs. (22) and (23) 0 1 V 1 V 2 V 3 B C V1 V4 A V ¼ @ V2 ð20Þ

The unknown values in (22) and (23) are V1, V2, V3, and V4. Thus, these equations can be transformed into (24) and (25). These equations can be expressed in matrix form, as shown in (26) xV 1 yV 2 þ V 3 ¼ x′

ð24Þ

yV 1 þ xV 2 þ V 4 ¼ y′

ð25Þ

x

y

1

y

x

0

!

0

V1

1

C 0 B B V2 C B C¼ 1 @ V3 A V4

x′ y′

! ð26Þ

ð21Þ

The unknown values V1, V2, V3, and V4 can be calculated with two points, P 1 ¼ ðx1 ; y1 Þ and P 2 ¼ ðx2 ; y2 Þ, and from the obtained values after the vibration has transformed their positions, P ′1 ¼ ðx′1 ; y′1 Þ and P ′2 ¼ ðx′2 ; y′2 Þ. The final system of equations is shown 0 ′ 1 0 1 x1 y1 1 0 0 V 1 1 x1 B C ′ C C B x1 0 1 CB y V B B y1 2 C B 1C B CB ð27Þ C¼B ′ C C B x2 y2 1 0 CB @ A@ V 3 A @ x 2 A ′ x2 0 1 y2 y2 V4

x′ ¼ V 1 xV 2 y þ V 3

ð22Þ

Gathering all the unknown parameters into a column vector, model (27) can be rewritten as (28), where X ¼ ½V 1 ; V 2 ; V 3 ; V 4 T

y′ ¼ V 2 x þ V 1 y þ V 4

ð23Þ

AX ¼ B

0

0 0

x′

1 1

0

V1

B C B P′ ¼ V  P-@ y′ A ¼ @ V 2 0 1

V 2 V1 0

10 1 x CB C V 4 A@ y A 1 1 V3

ð28Þ

56

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

The system of equations in (28) is linear and can be solved as 1

X¼A

ð29Þ

B

When the system is solved, the unknown value θ can be determined using (30). The values of xp, yp, and ty cannot be calculated directly. Instead, global values for translation in both axes, t ′x and t ′y , are calculated as (31) and (32). Although the final values of xp, yp, and ty are unknown, the model for the vibrations V is perfectly defined. Thus, the transformation of the points in space due to vibrations is known, and it can be reversed

θ ¼ tan

1

V2 V1

ð30Þ

t ′x ¼ V 3

ð31Þ

t ′y ¼ V 4

ð32Þ

This procedure could also be applied to estimate a similarity transformation. In this case the scale parameter, s, could be calculated as qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi s ¼ V 11 þ V 22 ð33Þ To calculate the unknowns in the transformation matrix V, at least two pair of points are needed. However, more than two points could also be used. In this case, matrix A would have more rows than columns and the systems of equations in (28) cannot be directly solved. With the purpose of minimizing the distances, the least square solution for the parameter vector X can be computed by X ¼ ðAT AÞ1 AT B

ð34Þ

Solving the least square problem using (34) could lead to numerical instabilities [23]. In general, an orthogonal decomposition is preferred. These methods are slower but more numerically stable. An alternative decomposition for X is the singular value decomposition (SVD) [24]. This method is the most computationally intensive, but is particularly useful if round-off errors could create singularity problems in matrix X. 3.2. Estimating vibrations with only two laser stripes When only two laser stripes are used, translations and rotations cannot be estimated because there is only one point measured twice. A single point can be used to create two equations, which are not enough to calculate the four unknown values. However, this point can be used to estimate translations and assume that there are no rotations. In this case, vibrations can

be modeled with a single translation matrix as follows: 0 1 1 0 0 B C V ¼ T ¼ @ 0 1 ty A 0 0 1

ð35Þ

In order to estimate the translation with P 1 ¼ ðx1 ; y1 Þ and from the obtained value after the vibration has transformed its position, P ′1 ¼ ðx′1 ; y′1 Þ, (36) could be used t y ¼ y1 y′1

ð36Þ

When three or more laser stripes are used there is a possibility that the system of equations does not have a solution, that is, the coefficient matrix is singular, for example due to repeated points. In this case, vibrations could be estimated the same as when using only two laser stripes, only translations are taken into account.

4. Results and discussion In order to assess the proposed procedure, several experiments have been carried out. First, a synthetic object was created. The object contains a low frequency curve at the center and high frequency curves at the edges, as can be seen in Fig. 5(a). Different types of vibrations were added to the object. The resulting 3D reconstructions using a single laser stripe are shown in Fig. 5(b)– (d). Fig. 5(b) shows the object with uniformly distributed random translations of 3% of the object height (Rand). Fig. 5(c) shows the object with rotations of 0.51 around the center of each profile (Rot). Fig. 5(c) shows the object with a combination of sinusoidal translations and rotations with same amplitudes as before (SinþRot). These types of vibrations have been used because they are common in many applications. In order to test the proposed procedure, it is assumed that there are multiple laser stripes available for 3D reconstruction. When only two laser stripes are used, only translations are estimated and removed. The difference between the original and the reconstructed shape of the object is assessed with the mean absolute error (MAE) using (37), where n is the number of points, Hi is the height at point i, and Ri is the reconstructed height at point i. This value is normalized using (37), where Hmax is the maximum height. This error value is a percentage over the maximum height of the object, and it is independent from the specific height of the object MAE ¼

1 n ∑ jH R j ni¼1 i i

ð37Þ

Fig. 5. Synthetic vibrations used for tests. (a) Real shape of the object. (b) Shape of the object after adding random translations. (c) Shape of the object after adding rotations. (d) Shape of the object after adding sinusoidal translations and rotations.

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

MAE H max

ð38Þ

Table 1 shows the error when using two, three and four laser stripes with different types of vibrations. The error in all cases is very low. Most errors are around 1e14, which is very close to the representation error of real numbers and it can be considered as zero (no error). In the case of random translations, the error is null in all cases. When rotations are added and three or more stripes are used the error is also null. However, when only two stripes are used the error increases. The obtained errors with two stripes are still very low and can be considered negligible for most applications. The results show that there is no difference between only rotations and translations plus rotations. The obtained errors with two stripes are very low. The reason is that estimating only translations removes most of the vibrations, even when there are rotations. This can be clearly seen with an example. Let us suppose there are two points with these values: P 1 ¼ ð5; 5Þ and P 2 ¼ ð5; 5:1Þ. Then, the position of these points is measured after vibrations with ty ¼ 1 and θ ¼ 0:251 (around the origin) transform the positions of the points. The result is P ′1 ¼ ð4:973; 6:021Þ and P ′2 ¼ ð4:973; 6:121Þ. If only one point is used to estimate vibrations (P1 and P ′1 , two laser stripes), the result would be ty ¼1.021 and θ ¼ 0. Removing this vibration from P ′2 results in a height of 5.099, that is, the error compared to P2 is 1:8e05. This error depends on the rotation and also on the proximity of the height of P1 and P2. However, consecutive points tend to be very close and the rotations in vibrations are low. Thus, this indicates that for most applications two laser stripes could be enough. Fig. 6 shows the differences between the real object and the reconstructed object using two and three laser stripes when vibrations consisting of sinusoidal translations and rotations are added. As can be seen, with two stripes errors appear at the center and at the edges, where there are curves in the object. However, these errors are lower than 1e05, negligible in most cases. The error when using two laser stripes depends on the degrees of rotation. Fig. 7 shows the error for different degrees. As can be seen, the error increases, but the value for 601 is still low. Moreover, 601 is an unrealistic high value in real applications. The proposed procedure assumes that the same points in an image are acquired again in the next image in a different laser stripe with accuracy. However, repeatedly measuring points always incur in measurement errors due to acquisition uncertainties and noise. Table 2 shows the error when using two, three and four laser stripes with different types of vibrations and when there is a 1% randomly distributed measurement error. This measurement error is added to

the positions of the transformed points after vibrations. The reconstruction errors increase notably compared with the previous experiments. Also, there is a major difference: the results when using only two laser stripes are better than when three or four are used. The reason is that adding noise can provoke spurious estimations of the vibrations. This problem finally results in outliers in the reconstruction of the shape. Fig. 8 shows the resulting reconstruction of the object using two and three laser stripes when vibrations consisting of sinusoidal translations and rotations are added, and with a 1% randomly distributed measurement error. The added noise provokes errors in the reconstruction. When using three laser stripes, outliers can be clearly seen. An example can be used to understand the problem with three laser stripes and outliers. Let us suppose the same two points as in the previous example: P 1 ¼ ð5; 5Þ, P 2 ¼ ð5; 5:1Þ, P ′1 ¼ ð4:973; 6:021Þ and P ′2 ¼ ð4:973; 6:121Þ. Vibrations are also ty ¼1 and θ ¼ 0:251 (around the origin). Assuming 1% error in the repeated measurement, the values of the transformed points would be P ′1 ¼ ð4:973; 6:081Þ and P ′2 ¼ ð4:973; 6:060Þ. The first point is measured 1% above and the second point is measured 1% below (worst case scenario). These measurement errors will result in an estimation of vibrations ty ¼7.13 and θ ¼ 1:1661, very different from the real vibration. Let us suppose the third point affected by the same vibrations is P ′3 ¼ ð5; 5:2Þ. Removing the real vibration would provide a height of 4.178, but removing the estimated vibrations would provide a height of 9.11. Therefore, the result is an outlier. The estimation of vibrations with noisy data can be improved by increasing the number of laser stripes. Fig. 9 shows the

0.05 0.04 Error

Error ¼ 100

57

0.03 0.02 0.01 0

0

10

20

30 40 Rotation degree

50

60

Fig. 7. Error when using two laser stripes and vibrations with different degrees of rotation.

Table 2 Reconstruction error when using multiple stripes and there is measurement uncertainty.

Table 1 Reconstruction error when using multiple stripes. Stripes

Rand

Rot

Sin þRot

Stripes

Rand

Rot

Sin þ Rot

2 3 4

1:9e16 2:1e15 9:5e14

2:7e06 4:8e15 8:0e14

2:7e06 4:8e15 8:0e14

2 3 4

9:7e02 6:6e01 3:6e01

1:1e01 5:1e01 3:3e01

1:1e01 5:4e01 4:2e01

Fig. 6. Error maps with vibrations consisting of sinusoidal translations and rotations. (a) Using two laser stripes. (b) Using three laser stripes.

58

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

resulting error for a different number of stripes. The error increases sharply from two to three stripes, and it decreases slowly as the number of stripes is increased. When there are a large number of stripes the error is low due to the improved robustness of the estimation of vibrations. The proposed procedure has been applied to real data. In this case, steel strips have been used. Strips are generally 2 m wide and 10 km long. 3D reconstruction is required for quality control, such as flatness inspection [25]. The laser source is located at the center of the roll path making the laser plane perpendicular to the plane on which the strip lies. The camera is located at 451 with respect to

Fig. 8. Error maps with vibrations consisting in sinusoidal translations and rotations. (a) Using two laser stripes. (b) Using three laser stripes.

5. Conclusion

0.8

Error

0.6 0.4 0.2 0

the laser plane, also centered over the roll path of the production line. The camera used is a Basler A501k, which provides images of 1280  1024 pixels using a CMOS sensor at 74 frames per second with 8 bit depth. Vibrations in steel strips consist of high frequency vertical translations. They have a period around 20 cm with a magnitude of approximately 5 mm. This type of vibrations can be effectively removed using low-pass filters. Fig. 10(a) and (c) shows two examples of steel strips reconstructed using a single laser stripe. As can be appreciated, the shape of the strips is corrupted by vibrations. Shape quality metrics calculated from this reconstruction would result in large errors. Vibrations can be estimated from data, as they are the high frequency components of the shape. Then, the proposed procedure with two laser stripes can be applied simulating this particular type of vibrations. Fig. 10 (b) and (d) shows the results. The proposed procedure estimates and removes all vibrations, producing a 3D reconstruction with a smooth surface. Manual measurements of the shape of the strips confirm the success of the resulting 3D reconstruction. The proposed procedure to remove vibrations using multiple laser stripes is very likely to find potential applications in a number of different areas. Vibrations affect the movement of the objects to a greater or lesser extent in any 3D reconstruction application based on laser light. Thus, height profiles extracted with a single laser stripe will contain errors, as the reconstructed shape will be affected by these vibrations. Therefore, the proposed procedure provides the opportunity to calculate a much more accurate 3D reconstruction of the object, regardless of the particular application.

2

4

6

8 10 Number of stripes

12

14

Fig. 9. Error when using multiple laser stripes with noisy data.

16

Vibrations are a major issue for laser-based 3D reconstruction methods. The movement of the object caused by vibrations affects 3D reconstruction and leads to large errors. The proposed procedure in this work is to use multiple laser stripes and estimate and remove vibrations from 3D reconstruction. First, vibrations are modeled based on geometrical transformations. In particular, using translations and rotations, the most common types of vibrations. Then, a mathematical procedure to estimate vibrations from data is developed. Using two laser stripes, only translations can be estimated. Using three laser stripes, both the translations and the rotations can be estimated. Using more than three laser stripes produces an overdetermined system that delivers a more

Fig. 10. 3D reconstruction of steel strips using the proposed procedure. (a) First strip with a single laser stripe. (b) First strip with two laser stripes. (c) Second strip with a single laser stripe. (d) Second strip with two laser stripes.

R. Usamentiaga et al. / Optics and Lasers in Engineering 53 (2014) 51–59

robust solution. Multiple tests have been carried out to evaluate the performance of the proposed procedure with synthetic and real data. Experimental results show excellent performance for the proposed procedure. Using three or more laser stripes completely removes vibrations from the 3D reconstruction. When only two laser stripes are used, very low error appears in the results, negligible for most applications. Furthermore, when tests are carried out using noisy data, a laser-based 3D reconstruction system with only two stripes clearly outperforms systems with more stripes, unless many stripes are used. The reason is that adding noise can provoke an incorrect estimation of vibrations that can finally result in outliers in the reconstruction of the shape. A laser projector can significantly increase the cost of a 3D reconstruction system. Also, using more than two laser stripes is only a marginal advantage in some cases. When noise is present in data, more than two laser stripes is a clear disadvantage, as was demonstrated in the tests. Therefore, unless there are very specific requirements, a system with only two stripes is recommended. A laser-based 3D reconstruction method with two laser stripes presents a far more cost efficient solution than others with similar or even better performance. Final tests on real data with two laser stripes indicate that the obtained 3D reconstruction after applying the proposed procedure is comparable to manual measurements, effectively removing the unwanted effects of vibrations on the estimated shape. Acknowledgments This work has been partially funded by the Project TIN200124903 of the Spanish National Plan for Research, Development and Innovation. References [1] Szeliski R. Computer vision: algorithms and applications. Springer; 2010. [2] Frauel Y, Tajahuerce E, Matoba O, Castro A, Javidi B. Comparison of passive ranging integral imaging and active imaging digital holography for threedimensional object recognition. Appl Opt 2004;43(2):452–62. [3] Tombari F, Mattoccia S, Di Stefano L, Addimanda E. Classification and evaluation of cost aggregation methods for stereo correspondence. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008. IEEE; 2008. p. 1–8. [4] Ballan L, Cortelazzo GM. Multimodal 3D shape recovery from texture, silhouette and shadow information. In: Proceedings of the third international symposium on 3D data processing, visualization, and transmission (3DPVT'06), 2006. p. 924–30.

59

[5] Salvi J, Pagès J, Batlle J. Pattern codification strategies in structured light systems. Pattern Recognition 2004;37(4):827–49. [6] Forest J, Salvi J. A review of laser scanning three-dimensional digitisers. In: IEEE/RSJ international conference on intelligent robots and system, vol. 1, 2002. [7] Hongjian Y, Shiqiang Z. 3d building reconstruction from aerial CCD image and sparse laser sample data. Opt Lasers Eng 2006;44(6):555–66. [8] Yu S-J, Sukumar SR, Koschan AF, Page DL, Abidi MA. 3d reconstruction of road surfaces using an integrated multi-sensory approach. Opt Lasers Eng 2007;45 (7):808–18. [9] Molleda J, Usamentiaga R, Garcia DF, Bulnes FG, Ema L. Shape measurement of steel strips using a laser-based three-dimensional reconstruction technique. IEEE Trans Ind Appl 2011;47(4):1536–44. [10] Liu Z, Sun J, Wang H, Zhang G. Simple and fast rail wear measurement method based on structured light. Opt Lasers Eng 2011;49(11):1343–51. [11] Fernandez A, Garcia R, Alvarez E, Campos AM, Garcia D, Usamentiaga R, et al. Low cost system for weld tracking based on artificial vision. IEEE Trans Ind Appl 2011;47(3):1159–67. [12] Isheil A, Gonnet JP, Joannic D, Fontaine JF. Systematic error correction of a 3d laser scanning measurement device. Opt Lasers Eng 2011;49(1):16–24. [13] Forest J, Salvi J, Cabruja E, Pous C. Laser stripe peak detector for 3d scanners. a FIR filter approach. In: Proceedings of the 17th international conference on pattern recognition. ICPR 2004, vol. 3. IEEE; 2004. p. 646–9. [14] Schnee J, Futterlieb J. Laser line segmentation with dynamic line models. In: Computer analysis of images and patterns. Springer; 2011. p. 126–34. [15] Usamentiaga R, Molleda J, Garcia DF. Fast and robust laser stripe extraction for 3D reconstruction in industrial environments. Mach Vision Appl 2012;23: 179–96. [16] Molleda J, Usamentiaga R, Garcia D, Bulnes FG, Espina A, Dieye B, et al. An improved 3D imaging system for dimensional quality inspection of rolled products in the metal industry. Comput Ind, http://dx.doi.org/10.1016/j.com pind.2013.05.002, in press. [17] Pernkopf F. 3d surface acquisition and reconstruction for inspection of raw steel products. Comput Ind 2005;56(8):876–85. [18] Usamentiaga R, Garcia DF, Molleda J, Bulnes FG, Peregrina S. Vibrations in steel strips: effects on flatness measurement and filtering. In: 43st IAS annual meeting. Conference record of the 2013 IEEE industry applications conference, vol. 1. IEEE; 2013. p. 1–10. [19] Doignon C, Knittel D. A structured light vision system for out-of-plane vibration frequencies location of a moving web. Mach Vision Appl 2005;16 (5):289–97. [20] Maurice X, Doignon C, Knittel D. On-line vibrations characterization of a moving flexible web with a fast structured lighting stereovision system. In: IEEE Instrumentation and Measurement Technology Conference Proceedings, 2007. IMTC 2007. IEEE; 2007. p. 1–5. [21] Bell JR, Rothberg S. Rotational vibration measurements using laser doppler vibrometry: comprehensive theory and practical application. J Sound Vib 2000;238(4):673–90. [22] Castellini P, Martarelli M, Tomasini E. Laser doppler vibrometry: development of advanced solutions answering to technology's needs. Mech Syst Signal Process 2006;20(6):1265–85. [23] Bjorck A. Numerical methods for least squares problems. Soc Ind Appl Math 1996;51. [24] Lawson CL, Hanson RJ. Solving least squares problems, vol. 161. SIAM; 1974. [25] Molleda J, Usamentiaga R, Garcia D, Bulnes F. Real-time flatness inspection of rolled products based on optical laser triangulation and three-dimensional surface reconstruction. J Electron Imaging 2010;19:031206.