Object segmentation based on refractive index estimated by polarization of specular reflection

Journal Pre-proof Object Segmentation Based on Refractive Index Estimated by Polarization of Specular Reflection Zhiying Tan, Baolai Zhao, Xiaobin Xu, Zhongwen Fei, Minzhou Luo

PII:

S0030-4026(19)31816-9

DOI:

https://doi.org/10.1016/j.ijleo.2019.163918

Reference:

IJLEO 163918

To appear in:

Optik

Received Date:

26 September 2019

Accepted Date:

26 November 2019

Please cite this article as: Tan Z, Zhao B, Xu X, Fei Z, Luo M, Object Segmentation Based on Refractive Index Estimated by Polarization of Specular Reflection, Optik (2019), doi: https://doi.org/10.1016/j.ijleo.2019.163918

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier.

Object Segmentation Based on Refractive Index Estimated by Polarization of Specular Reflection

Zhiying Tana,b *, Baolai Zhaoa,b , Xiaobin Xua,b, Zhongwen Feia,b , Minzhou Luoa,b

a

College of Mechanical and Electrical Engineering, Hohai University,

b

ro of

Changzhou, 213022, China

Jiangsu Key Laboratory of Special Robot Technology, Hohai University,

Corresponding author, E-mail address: [email protected] (Zhiying Tan).

re

*

-p

Changzhou, 213022, China

lP

Abstract:

na

Polarization is steadily attracting attention in machine vision due to its ability to capture the information not readily available in standard color or greyscale camera.

ur

This paper aims to segmentation object by means of approximate value of refractive index. First, estimate the approximate refractive index by using relationships between

Jo

the degree of polarization for specular reflection and zenith angle and refractive index, and then the approximate refractive index data are clustered to achieve object segmentation. The experimental results show that the segmentation method has great speed and accuracy in calculation.

Keywords: polarization; refractive index; degree of polarization; object segmentation; clustering;

1 Introduction

The polarization of light is a fundamental physical property that describes how

ro of

light moves through space [1]. Animals exploit polarization information for visual signals, orientation and navigation in a variety of ways [2-5]. Bio-inspired

polarization vision techniques for sensory systems promise to deliver more abundant

-p

and stable information than current computational approaches. The exploitation of

re

polarization information in the field of computer vision has become progressively more popular during the last few decades. One camera was used to capture different

lP

polarization directions, detect the polarization pattern across the full sky in a single

na

snapshot for localization and navigation [6]. A bioinspired polarization-sensitive imager can determine the geolocation of an observer based on radial underwater

ur

polarization patterns [7]. The polarization characteristics of the object surface are mainly determined by the reflected radiation of light. When the incident light is

Jo

reflected, the characteristics of the light will change, which is related to the physical and chemical characteristics of the object surface. Therefore, the polarization information of the light can reflect the polarization characteristics of the object. Based on this principle, the 3D surface reconstruction of object using polarization was proposed by Atkinson, G [8-11]. A growing trend in the application of polarization

technology for machine vision. It mainly focuses on the measurements [12], distinguishing surfaces material [13], image understanding et. al [14-15]. However, few studies have been done on the application of polarization to object segmentation in the image. Traditional image segmentation technology is based on the gray value, so there must be a clear gray difference between the object to be segmentation and the

ro of

background [16-17]. Convolutional neural network has achieved remarkable results in image segmentation, but it is also based on the intensity information [18]. When the

color of the object is similar to that of the background, the object segmentation can be

-p

achieved quickly and accurately by utilizing the difference of polarization degree and

re

phase angle information of the surface reflected light between the object and the background.

lP

In this paper, the principle of polarization imaging and the mathematical model

na

of polarization depending on refractive index and zenith angle is introduced in Section 2. In Section 3, the refractive index equation of the reflectance polarization degree of

ur

the object mirror is analyzed and the solution formula is given. In Section 4, the refractive index is estimated by azimuth instead of zenith angle, and the flow diagram

Jo

of object segmentation method proposed in this paper is given. Finally, the experiments of refractive index estimation and clustering segmentation are verified the method performance.

2 Polarization Vision

In this section, the imaging principle of polarization camera and the standard background theory is given, which is the theoretical basis of the proposed algorithm. In the visible band, the polarization information produced by the reflection of light on the surface of the object can reflect the polarization characteristics of the object. In Fig. 1, a schematic diagram is given for collecting polarization information of objects

ro of

using MER-502-79U3M POL polarizing camera. According to Fresnel reflection theory [19], polarized reflected light can be parameterized by three values: the

vector

N 

, phase angle



and degree of polarization



[10]. And the normal

of the object surface can be determined by zenith angle

[9].



and azimuth

na

lP

re

angle

I

-p

intensity

ur

Fig. 1. Schematic of a polarization image acquisition system.

Jo

The polarization camera is equipped with a globally exposed Sony IMX250MZR CMOS polarization photosensitive chip, which can simultaneously collect images at four angles

90

o

, 4 5 o , 1 3 5 o and 0 o . A layer of polarizer is added to the top of the

photodiode. The chip adds a layer of polarizer to the photodiode. Four polarizers with different angles ( 9 0 o , 4 5 o , 1 3 5 o and 0 o ) are designed to be placed on a single pixel. Each of the four pixels is used as a computing unit, as shown in Fig. 2.

Fig. 2. Distribution of four polarizers on a computing unit. Unlike traditional intensity imaging, which only reflects the intensity and spatial information of light, polarization cameras can calculate the degree and direction of

ro of

polarization by correlating polarizers in different directions. The intensities collected by polarization camera is defined as measured are defined as I

45

,I

90

,I

135

and I 0 , then

the polarization data for each cell is calculated via the Stoke’s parameters

I

0

 I

45

 I

90

 I

135

0

S2  I

45

 I

(2)

90

 I

(3)

135

na

S1  I

(1)

lP

2

re

Stoke’s parameters: S0 

and

-p

S 2 [12]:

S 0 , S1

Polarization image data:  S0

 

ur

I

1

2

=

(5)

ar c t an 2( S 2 , S1 )

Jo

2

(4)

2

S1  S 2

(6)

S0

where  is the degree of polarization, and arctan2 is the four quadrant inverse tangent. The specular reflection of a ray of light from a surface point, as shown in Fig. 3.

Assume that the surface is a smooth interface between air and surface of object. The Fresnel equations give the ratios of the reflected wave amplitude to the incident wave I

amplitude. The angles

and

T

are defined in Fig. 3, and

and

nI

nT

are the

refractive indices of the incident and reflecting media, respectively. Since the incident medium is air, the approximation is made that

n I =1 .

The angle

T

can be obtained

nI s i n  I

 nT s i n  T

(7)

ro of

using Snell’s Law:

This means that the refractive indices of the reflecting media / s i n T ,  I

 T

.

 1,

lP

re

-p

nT = s i n  I

nT

na

Fig. 3. Reflection and refraction of light on the surface of media. Combined with Fresnel parameters and Fresnel equations, it is deduced that the

n

ur

degree of polarization of specular reflected light can be expressed by refractive index and zenith angle  . The formula is as follows: 2 s i n  c os 

Jo

2

=

n

2

 si n   n 2

2

n

2

 si n  2

si n   2 si n  2

4

(8)

(a) Degree of polarization surface

(b) Polarization curves under different refractive index

ro of

Fig. 4. Variation of degree of polarization with refractive index and zenith angle.

In Fig. 4, the curves of degree of polarization with refractive index and zenith angle are given. In the range of zenith angle

  [ 0,  / 2]

, there is a unique

means that within the range of zenith angle less than

re

 / 4 . This

greater than

-p

maximum of polarization, and the maximum of polarization is only at the zenith angle

 / 4,

there is a one-to-one relationship between polarization degree and refractive index

na

lP

and zenith angle.

ur

3 Calculation of refractive index of object

Equation (9) can be obtained from the formula of degree of polarization (8) 4

 Bn

2

 C  0

(9)

Jo

An

where

A   s c os  2

4

(10) (11)

B  2  s c os  ( 2 s i n   s i n  )  4 s i n  c os  2

2

4

2

C   s( 2 si n   si n  ) 2

4

2

2

4

 4 s i n  c os  6

2

2

(12)

Formula for finding roots by quadratic polynomial equation (9) B 

n 

B

2

 4 AC

(13)

2A

The degree of polarization is a parameter to measure the degree of polarization in electromagnetic wave, which is the proportion of polarized light in the total light intensity. Generally speaking, the degree of polarization varies between 0 (natural   [ 0, 1]

and formula (10) - (12), there

is the following relationship 2

 4 AC =1 6 s i n  ( 1   )  0 2

-B 

B

2

2

(14)

 4 AC

=2 s i n  c o s  ( ( 2   ) s i n    2

2

 4 si n 

2

1  

2

2

c os  ) 2

2

(15)

re

 0

-p

B

ro of

light) and 1 (full polarization). According to

lP

From formula (10), (14) - (15), we can see that the solution of equation (9) exists. This means the refractive index 90

,I

135

can be calculated approximately by intensities I

and I 0 . Fig. 5 shows two roots

na

I

n

n1

and

n2

( n1

 n2

45

) of refractive index

calculated by formula (13) with fixed degree of polarization. According to the fact of  1 , n1

ur

refractive index n

Jo

surface of the object.

is taken as the estimated value of refractive index on the

,

(a) Curve of

n1

versus



(b) Curve of

n2

versus



ro of

Fig. 5. Curve of refractive index estimation with zenith angle.

-p

4 Object segmentation

c os  s i n     si n  si n      c o s   

na

px     py      pz 

lP

normal vectors can be expressed as

re

In the case of only considering specular reflection, the components of the surface

(16)

where azimuth angle  takes values of either the phase angle  or  + 

/ 2

.

ur

When the surface of an object is plane, the normal vector at each point in the plane is a fixed value. According to formula (16), the product of sine value of zenith

Jo

angle and cosine value of azimuth angle is fixed. Although there is no absolute linear relationship between zenith angle  and azimuth angle  , different azimuth angles correspond to different normal directions, and two planes can be distinguished under certain circumstances. The goal of this paper is to use vision to segment several known objects from relatively fixed background. Therefore, when the refractive index

is estimated in this paper, the zenith angle  is taken as the value  . Although the calculated refractive index approximation is different from the surface refractive index of the object, it can improve the target difference to a certain extent.

ro of

Fig. 6. Flow diagram of the proposed object segmentation method.

The flow diagram of object segmentation using polarization is given in Fig. 6. First, the gray-scale image I

45

,I

90

,I

135

and I 0 of four directions were collected by

-p

polarization camera, and the polarization degree and phase angle were obtained by

re

Stoke's parameters. Then the refractive index was approximately calculated by the mirror reflection refractive index equation. Finally, the object in the image is

ur

5 Experiments

na

lP

segmented by clustering algorithm.

The images processed in the experiment were acquired by camera MER-502-

Jo

79U3M POL fitted with an industrial lens MVL-MF1220M-5MP. The camera uses a globally exposed Sony IMX250MZR CMOS polarization photosensitive chip, which can simultaneously collect images at four angles of 4 5 o , 9 0 o , 1 3 5 o and 0 o . Fig. 7 shows the gray-scale images of different materials and colors collected by MER-502-79U3M POL camera under natural light environment. The images are collected in different

time periods, so the brightness of red plastic blocks in Fig.7(c) is obviously different from that in Fig.7 (d), but the result of target segmentation was very little affected by illumination and can be neglected approximately. The background of the image is the surface blackened metal, whose material is similar to that of the square metal block in

-p

ro of

Fig. 7 (a).

(b) A block of wood and a ceramic cup

ur

na

lP

re

(a) A metal with black surface and a wood block

(c) Red and green plastic blocks

(d) Red plastic blocks and red paper

Jo

Fig. 7. Intensity images of objects with different materials used in experiments.

Using the approximate formula of refractive index given in section 3, the approximate value of refractive index is taken from the larger values of two roots. Fig.8 shows the approximate refractive index distribution from different perspectives.

In Fig. 8 (a), the refractive index approximation of cylindrical wood blocks differs greatly from that of the background material, while the material of square objects is similar to that of the background, and there is no significant difference in the refractive index approximation, except for some bare metal parts. In Fig. 8 (b), the refractive indices of wood blocks and ceramic cups were different from those of the background, so the calculated values of their approximate refractive indices were also

ro of

different. In Fig. 8 (c), the material of red and green plastic blocks was the same, so

the calculated values of their approximate refractive index were basically the same. In Fig. 8 (d), the approximation values of refractive index of plastic blocks and paper

na

lP

re

-p

with similar colors were obviously different.

(b) Approximate n 1 of Object in Fig.7(b)

Jo

ur

(a) Approximate n 1 of Object in Fig.7(a)

(c) Approximate n 1 of Object in Fig.7(c)

(d) Approximate n 1 of Object in Fig.7(d)

Fig. 8. Approximate results of refractive index for different objects.

The refractive index data calculated from each image is expressed as threedimensional data (x, y, z), where (x, y) belongs to [0, 1], which is obtained by standardization of image coordinates, and the value of z was 1, when the approximate value of refractive index was greater than the threshold value (110 in this paper),

ro of

otherwise it is 0. The clusterdata function in MATLAB was used to cluster Z with the

lP

re

-p

value of 1 (x, y), and good segmentation results were obtained, as shown in Fig. 9.

(b) Clustering results of Fig.7 (b)

Jo

ur

na

(a) Clustering results of Fig.7 (a)

(c) Clustering results of Fig.7 (c)

(d) Clustering results of Fig.7 (d)

Fig. 9. Clustering results of objects based on approximate refractive index.

6 Conclusions

In this paper, a method for calculating the approximate refractive index by using polarization and phase angle has been given. Influenced by the refractive index calculated by azimuth approximation zenith angle and noise introduced in data

ro of

acquisition, although the approximate refractive index value can’t truly reflect the refractive index of the surface material of the object, it can greatly increase the

discrimination between the background and the object in the image and provide a

-p

basis for target segmentation. Through a number of experimental analysis, it is found

re

that the method can effectively segment objects of the different material with same colors, and objects of the same material with different colors. This can provide fast,

lP

stable and accurate position information for robot visual servo grasping.

na

It is hoped that follow-on work will involve a more polarization numerical study under different lighting environments of combined specular-diffuse reflection; in

ur

particular where polarization camera is used for underwater target recognition. This will provide important visual feedback for underwater vehicle operation. Other

Jo

potentially useful areas of future work includes the fusion of polarization camera and TOF camera to achieve faster and more accurate target recognition and location in the scene.

Acknowledgements

This research was funded by the Fundamental Research Funds for the Central Universities (Grant No. 2018B03914), Changzhou Sci&Tech Program (Grant No. CJ20190044), and Opening Fund of Jiangsu Key Laboratory of Special Robot

Jo

ur

na

lP

re

-p

ro of

Technology (Grant No. 2017JSJQR02).

Reference

[1] Born, Max, and Emil Wolf. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light. Elsevier, (2013) 43-47. [2] Roberts, Nicholas W., et al. Animal polarization imaging and implications for

ro of

optical processing. Proceedings of the IEEE, 102.10 (2014) 1427-1434.

[3] Zeil, Jochen, Willi A. Ribi, and Ajay Narendra. Polarisation vision in ants, bees and wasps. Polarized light and polarization vision in animal sciences. Springer,

-p

Berlin, Heidelberg, (2014) 41-60.

re

[4] Weir, Peter T., et al. Anatomical reconstruction and functional imaging reveal an ordered array of skylight polarization detectors in Drosophila. Journal of

lP

Neuroscience, 36.19 (2016) 5397-5404.

na

[5] Daly, Ilse M., et al. Dynamic polarization vision in mantis shrimps. Nature communications, 7 (2016) 12140.

ur

[6] Zhang, Wenjing, et al. Sky light polarization detection with linear polarizer triplet in light field camera inspired by insect vision. Applied optics, 54.30 (2015) 8962-

Jo

8970.

[7] Powell, Samuel B., et al. Bioinspired polarization vision enables underwater geolocalization. Science advances, 4.4 (2018) eaao6841. [8] Atkinson, Gary A., and Edwin R. Hancock. Recovery of surface orientation from diffuse polarization. IEEE transactions on image processing, 15.6 (2006) 1653-1664.

[9] Atkinson, Gary A., and Edwin R. Hancock. Shape estimation using polarization and shading from two views. IEEE transactions on pattern analysis and machine intelligence, 29.11 (2007) 2001-2017. [10] Atkinson, Gary A. Two-source surface reconstruction using polarisation. In: Scandinavian Conference on Image Analysis. Springer, Cham (2017) 123-135. [11] Atkinson, Gary A. Polarisation photometric stereo. Computer Vision and Image

ro of

Understanding, 160 (2017) 158-167.

[12] Atkinson, Gary A., et al. High-precision polarization measurements and analysis

for machine vision applications. 2018 7th European Workshop on Visual Information

-p

Processing (EUVIP), IEEE, (2018) 1-6.

re

[13] Wolff, Lawrence B., and Terrance E. Boult. Constraining object features using a polarization reflectance model. IEEE Transactions on Pattern Analysis & Machine

lP

Intelligence, 7 (1991) 635-657.

na

[14] Wolff, Lawrence B. Polarization vision: a new sensory approach to image understanding. Image and Vision computing, 15.2 (1997) 81-93.

ur

[15] Abd El Rahman, Olivier Morel, and David Fofi. Bio-inspired polarization vision techniques for robotics applications. Computer Vision: Concepts, Methodologies,

Jo

Tools, and Applications. IGI Global, (2018) 421-457. [16] Shi, Jianbo, and Jitendra Malik. Normalized cuts and image segmentation. Departmental Papers (CIS) (2000) 888-905. [17] Felzenszwalb, Pedro F., and Daniel P. Huttenlocher. Efficient graph-based image segmentation. International journal of computer vision, 59.2 (2004) 167-181.

[18] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 3431-3440.

Jo

ur

na

lP

re

-p

ro of

[19] Hecht, E.: Optics, 3rd edn. Addison Wesley, Longman (1998)