- Email: [email protected]
Underwater image enhancement method using weighted guided trigonometric filtering and artificial light correction
Accepted Manuscript Underwater Image Enhancement Method Using Weighted Guided Trigonometric Filtering and Artificial Light Correction Huimin Lu, Yujie Li, Xing Xu, Jianru Li, Zhifei Liu, Xin Li, Jianmin Yang, Seiichi Serikawa PII: DOI: Reference:
S1047-3203(16)30033-5 http://dx.doi.org/10.1016/j.jvcir.2016.03.029 YJVCI 1726
To appear in:
J. Vis. Commun. Image R.
Received Date: Revised Date: Accepted Date:
4 June 2014 9 June 2015 30 March 2016
Please cite this article as: H. Lu, Y. Li, X. Xu, J. Li, Z. Liu, X. Li, J. Yang, S. Serikawa, Underwater Image Enhancement Method Using Weighted Guided Trigonometric Filtering and Artificial Light Correction, J. Vis. Commun. Image R. (2016), doi: http://dx.doi.org/10.1016/j.jvcir.2016.03.029
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Underwater Image Enhancement Method Using Weighted Guided Trigonometric Filtering and Artificial Light Correction Huimin Lua,b,c, Yujie Lia, Xing Xud, Jianru Lib, Zhifei Liub, Xin Lic, Jianmin Yangc, Seiichi Serikawaa a
Department of Electronics and Electrical Engineering, Kyushu Institute of Technology, Kitakyushu 804-8550, Japan.
b
State Key Laboratory of Marine Geology, Tongji University, Shanghai 200092, China.
c
State Key Laboratory of Ocean Engineering, Shanghai Jiaotong University, Shanghai 200240, China.
d
University of Electrical Science and Technology of China, Chengdu 611731, China
* Corresponding author: [email protected]; Tel.: +81-93-884-3282; Fax: +81-93-884-3203. Abstract: This paper describes a novel method for enhancing optical images using a weighted guided trigonometric filter and the camera’s spectral properties in turbid water. Absorption, scattering, and artificial lighting are three major distortion issues in underwater optical imaging. Absorption permanently removes photons from the imaging path. Scattering is caused by large suspended particles found in turbid water, which redirect the angle of the photon path. Artificial lighting results in footprint effects, which cause vignetting distortion in the captured image. Our contributions include a novel deep-sea imaging method that compensates for the attenuation discrepancy along the propagation path, and an effective underwater scene enhancement scheme. The recovered images are characterized by a reduced noise level, better exposure of dark regions, and improved global contrast such that the finest details and edges are significantly enhanced. Our experiments showed that the average Peak Signal to Noise Ratio (PSNR) improved by at least 1 dB when compared with state-of-the-art-methods.
2 Keywords: deep-sea imaging; inherent optical property; image enhancement
1. Introduction There have been increased developments in deep-sea exploration using autonomous underwater vehicles (AUVs) and unmanned underwater vehicles (UUVs). However, the contrast of underwater images is still a major issue for application. It is difficult to acquire clear underwater images around underwater vehicles. Since the 1960s, sonar sensors have been extensively used to detect and recognize objects in oceans. Due to the principles of acoustic imaging, sonar-imaged images have many shortcomings such as a low signal to noise ratio and a low resolution. Consequently, vision sensors must be used for short-range identification because sonars yield to low-quality images [1]. In contrast to natural images, underwater images suffer from poor visibility. Firstly, Light is absorbed when sunlight is reflected by a water surface. Additionally, absorption substantially reduces the ambient light energy. Random attenuation and scattering degrade the contrast of the scene. Objects at a distance of more than 10 meters are almost indistinguishable, because the colors are faded [2]. Furthermore, artificial lighting can cause a distinctive footprint on the seafloor. Over the past two decades, researchers have been focusing on improving the quality of underwater images. Recent research on underwater image enhancement can be classified into two major categories according to input types: multiple or single inputs. For multiple inputs, Schechner et al. used a polarization imaging method to compensate for visibility degradation [3]. Treibitz et al. proposed a multi-directional illumination fusion method for turbid scene enhancement [4]. Ouyang et al. proposed a multiple laser lighting and bilateral filtering-based image deconvolution method for underwater image enhancement [5]. For single-input image methods, Ancuti et al. proposed a Laplacian fusion method that reconstructs a clear image in turbid water [6]. Fattal et al. first used the underwater dehazing method for underwater image enhancement [7]. This technique estimates the scene radiance and derives the transmission map
3 using a single image. However, it cannot sufficiently process images with heavy haze and colors can be distorted. Lu et al. [8] proposed a dehazing and color correction method for recovering underwater scenes. He et al. [9] proposed a scene depth information-based dark channel priors dehazing algorithm using a matting Laplacian. However, this algorithm requires significant computation time. To overcome this disadvantage, they also proposed a guided image filter [10] with the foggy image used as a reference image. However, this method leads to incomplete haze removal. Although these approaches can enhance the image contrast, there are several drawbacks that reduce their practical applicability. First, the imaging equipment is difficult to use in practice (e.g., a rangegated laser imaging system is rarely applied in practice [5]). Second, multiple input images are difficult to obtain [3, 4]. Third, these methods cannot sufficiently alleviate color distortions [6–9]. As noted in previous publications [6, 9], it is impossible to capture the same scene at the same time in turbid water. Consequently, single-image enhancement methods can achieve better results. Single underwater-image enhancement is a challenging, but ill-posed problem. In underwater imaging, the captured images are significantly influenced by absorption, scattering, and inhomogeneous illumination. In this paper, we introduce a novel scheme that enhances underwater images using a single image as input. The proposed method overcomes the previously mentioned drawbacks of conventional methods. The organization of this paper is as follows. Section 2 explains the ocean imaging model. Section 3 describes the method for underwater image enhancement and proposes our weighted guided trigonometric filter. Section 4 applies the proposed method to underwater optical images. Finally, Section 5 concludes this paper. 2. Ocean Imaging Model Artificial light and ambient light traveling through the water is the source of illumination in an ocean environment. Suppose that the amount of light radiation (W) formed after wavelength attenuation can be derived using the energy attenuation model as
4
EcW ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer(c) L ( x ) , c {r, g , b}.
(1)
where EcW ( x ) is the amount of illumination at the scene point, EcA ( x ) is the amount of illumination from ambient light at the scene point, EcI ( x ) is the illumination from artificial light, and Nrer is the normalized residual energy ratio. In this imaging model, artificial light is reflected at a distance L(x) from the camera. D(x) is the underwater scene depth. Absorption and scattering occurs in this process. Suppose that the attenuation rate is c ( x ) , then the illumination of ambient light is
Ec ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ), c {r, g , b}.
(2)
Following the Nayar–Narasimhan dehazing model [11], the captured image I c ( x ) formed at the camera plane can be formulated as
I c ( x ) EcA ( x ) Nrer(c) D ( x ) EcI ( x ) Nrer(c) L ( x ) c ( x ) tc ( x ) 1 tc ( x ) Ac , c {r, g , b}.
(3)
where the background Ac represents the light reflected by the object. tc ( x ) can be represented in terms of a light beam of wavelength over a distance d(x) within the water, that is,
Eresidual ( x ) tc ( x ) initial 10 d ( x ) Nrer (c)d ( x ) , E ( x )
(4)
where is the extinction coefficient of the medium. Nrer is the normalized residual energy ratio [14]; in the Ocean Type I, it satisfies
0.8 ~ 0.85 if 650 750m( red ) Nrer (c) 0.93 ~ 0.97 if 490 550m( green ) . 0.95 ~ 0.99 if 400 490m(blue)
(5)
Consequently, substituting Eq. (4) into Eq. (3) we can obtain
I c ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ) Nrer (c) d ( x ) 1 Nrer (c)d ( x ) Ac , c {r, g , b}.
(6)
5 The above equation incorporates the light scattering during the course of propagation from object to the camera, and the wavelength attenuation along the object–light and water depth. When the object– light distance L(x), scene depth D(x), and camera–object transmission map d(x) are known, we can recover the final clean image.
Figure 1. Schematic of ocean optical imaging model. Figure 1 shows the diagrammatic sketch of the proposed model. To improve the image quality, we use the process shown in Fig. 2. In the first stage, we determine the artificial light (if it exists) and remove the footprint or vignetting. In the next stage, we calculate the transmission map of the scene using dual-channels priors, refine the transmission map using a weighted guided trigonometric filter, and remove the scatters using the dehazing method. Finally, we recover the scene color using the camera’s spectral properties.
6
Figure 2. Scheme for the underwater image enhancement process. 3. Underwater Image Enhancement Method 3.1. Artificial Light Correction In deep sea, we must use artificial light for imaging. However, this causes footprint effects. Sooknanan et al. [12] proposed a multi-frame vignetting correction method for removing the vignetting phenomenon, which involves estimating the light source’s footprint on the seafloor. This artificial light correction is effective, but is computationally time consuming. Moreover, it is difficult to capture multiple images of the same scene in turbid water, because of floating sediments. With this in mind, we propose a single-image based vignette removal method [13]. We are interested in the overall effect of light attenuation through the system and not all of the image formation details, so we derived an effective degradation model. That is,
Z (r, ) O(r, )V (r ) ,
(7)
7 where Z is the image with vignetting, O is the vignetting-free image, and V is the vignetting function. Our goal is to find the optimal vignetting function V that minimizes the asymmetry of the radial gradient distribution. Taking the log of Eq. (7), we get
ln(Z (r, ) ln O(r, ) ln V (r ) .
(8)
Let Z=lnZ, O=lnO, and V=lnV. We denote the radial gradients of Z, O, and V for each pixel (r, θ) by RrZ(r,θ), RrO(r,θ), and RrV(r,θ). Then,
R Zr (r, ) R Or (r, ) R Vr (r, ) .
(9)
Given an image with vignetting (Z), we can find the maximum a posterior (MAP) solution to V. Applying Bayes rule, we get
V arg max P(V | Z) arg max P(Z | V) P(V) . V
(10)
V
We consider the vignetting function at discrete, evenly sampled radii (V(rt), rt∈Sr), where Sr ={ r0 , r1 ,…, rn-1 }. Each pixel (r, θ) is associated with the sector that contains it. The vignetting function is generally smooth, so we obtain
P( V ) e
s
rtSr V "(rt )2
,
(11)
where λs is chosen to compensate for the noise level in the image, and V ''( rt ) is
V"(rt )
V(rt 1 ) 2V(rt ) V(rt 1 ) . ( r )2
(12)
Using the sparsely prior method on the vignetting-free image O, O
P(Ζ | V) P( RrO ) e |Rr | , α<1.
(13)
Substituting Eq.(13) and Eq.(9), we have
P( Z | V ) e
( r , )|R Zr ( r , )R Vr ( r )|
.
(14)
The overall energy function P(Z|V)P(V) can be written as E
| R
(r , )
Z r
(r , ) R rV (r ) | s V "(rt ) 2 . rt Sr
(15)
8 We can estimate V(rt) by minimizing E. Then, we use the Iteratively Reweighted Least Squares (IRLS) to estimate the vignetting function. Figure 3 shows the principle of the de-vignetting method, and Figure 4 shows the simulation results of the color chart in a water tank. In this simulation, we used an OLYMPUS Tough TG-2 underwater camera (OLYMPUS Co., Japan), the water depth was 0.3 meters, the camera–object distance was 0.8 meters, and the object–light distance was 0.5 meters. We added deep-sea soil to the water at a turbidity rate of 50 mg/L.
Figure 3. Underwater vignetting correction method.
(a)
(b)
(c)
Figure 4. Result of artificial light correction: (a) input image; (b) vignetting-corrected image; and (c) estimated vignetting. 3.2. Transmission Map Estimation In [14], the author found that the red color channel is the dark channel of underwater images. During our experiments, we found that the lowest RGB channel in turbid water was not always the red channel. The blue channel may be significantly attenuated [15, 16], because there can be certain
9 organic matter that absorbs blue more significantly in certain coastal zones. Although the red wavelength is easily absorbed by traveling in water, the distance between the camera and object is not enough to significantly absorb the red wavelength (see Fig. 5). Therefore, the blue channel may be the smallest. Consequently, we used the red–blue dual channels to estimate a rough transmission map.
(a)
(b)
Figure 5. RGB histograms of underwater images.
As shown in Eq. (6), scattered and absorbed images reflected from the scene point can be defined as
J c ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ),
(16)
c {r, b}.
We denote the minimum pixel of the red and blue dual-channels for the underwater image J c ( x ) as
J min ( x ) min min J c ( y ), c {r, b}. c
(17)
y ( x )
If an object belongs to the foreground, the minimum value of its pixels is very small. Minimizing the local patch (x) on the scattered image I c ( x ) in Eq. (6), we have
min I c ( y ) min J c ( y ) Nrer( c) d ( x ) 1 Nrer( c) d ( x ) Ac ,
y ( x )
y ( x )
c {r, b}.
(18)
10 Suppose that Ac is the homogeneous background in small local patch, and that the normalized residual energy ratio Nrer(c)d ( y ) on the small local patch (x) is essentially a constant. Then, the min value of Eq. (18) can be written as
min I c ( y ) min J c ( y ) Nrer( c) d ( x ) min 1 Nrer( c) d ( x ) Ac ,
y ( x )
y ( x )
y ( x )
c {r, b}.
(19)
We can rearrange the above equation to get min I c ( y ) min J c ( y ) y ( x ) min min y ( x ) Nrer ( c) d ( x ) c c Ac Ac
(20)
min 1 Nrer ( c) d ( x ) , c {r, b}. c
Therefore, the second term of the above equation is equal to 0. Consequently, the estimated distance map is min I c ( y ) max Nrer (c)d ( x ) 1 min y ( x ) , c {r, b} . c c Ac
(21)
Finally, the distance map can be obtained using
min I c ( y ) d ( x ) ln 1 min y ( x ) / ln max Nrer( c), c {r, b} . c A c
(22)
3.3. Global Optimization of Transmission Map In the previous section, we roughly estimated the transmission map. However, because it is a patch-based operation, the obtained transmission map contains some block artifacts. So, we propose a weighted guided trigonometric filter (WGTF) to remove the artifacts. The WGTF process is first performed under the guidance of image G, which can be the input image itself (i.e., I c ( x ) ). Let Ip, Iq, Gp, and Gq be the intensity values at pixels p and q of the transmission map and the guided image, respectively, and wk be the kernel window centered at pixel k. Then,
11 WGTF ( I ) p
W
qwk
1
GTFpq
W
(G ) qwk
GTFpq
(G ) I q ,
(23)
where the kernel weight function is WGTFpq (G )
Here,
k and k2 are
(G p k )(Gq k ) 1 1 . 2 | w | k:( p ,q )wk k2
(24)
the mean and variance of the guided image G in the local window wk, and |w| is the
number of pixels in window w. When Gp and Gq are on the same side of an edge, the weight assigned to pixel q is large. When Gp and Gq are on different sides, a small weight will be assigned to pixel q. 3.4. Recovering the Scene Radiance The previous sections derived the refined transmission map d(x). To remove the scatter, we also need to solve the attenuation rate, c ( x ) . We use the least squares solution to achieve this, as in [14]. That is,
c ( x ) J c ( x )T J c ( x ) J c ( x ) T 1
EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) ,
(25)
c {r, g , b}. After removing the artificial light, Eq. (6) can be written as
I c ( x ) EcA ( x ) Nrer (c) D ( x ) c ( x ) Nrer(c) d ( x ) 1 Nrer(c)d ( x ) Ac , c {r, g , b}.
(26)
According to the Nayar–Narasimhan dehazing model, we can obtain the descattered image using Jˆc ( x )
I c ( x ) 1 Nrer ( c) d ( x ) Ac Nrer (c )d ( x )
EcA ( x ) Nrer (c) D ( x ) c ( x ) Nrer (c) d ( x ) ,
(27)
c {r, g , b}.
Figure 6 shows the descattered result of the proposed algorithm. The haze-like artifacts were completely removed, and the contrast was increased.
12
(a)
(b)
Figure 6. Descattered result of the proposed method: (a) devignetted image; and (b) descattered image. 3.5. Spectral Properties-based Color Correction In [14], the author simply corrected the color according to the attenuation of the water depth. However, in practice, the spectral response function [17] of a camera represents its relative sensitivity. We use the chromatic transfer function to weight the light from the ocean surface to a given object depth using
Esurface , Eobject
(28)
where the transfer function τ at wavelength λ is derived from the irradiance of the surface ( Esurface ) and the irradiance of the object ( Eobject ). Based on the spectral response of the RGB camera, we convert the transfer function to the RGB domain, that is n
RGB Cc ( ) ,
(29)
where the weighted RGB transfer function is τRGB, Cb(λ) is the underwater spectral characteristic function of color band c {r, g , b} . n is the number of discrete bands of the camera’s spectral characteristic function. Finally, the corrected image is calculated from the weighted RGB transfer function using J c ( x ) Jˆc ( x ) RGB ,
(30)
where J c ( x ) and Jˆc ( x ) are the color-corrected and uncorrected images, respectively. Figure 7 shows the color correction result in a simulation. Compared with the color chart in Figure 7(b), the proposed
13 method recovered the correct scene colors.
(a)
(b)
Figure 7. Result of the color correction: (a) color-corrected image; (b) true color chart. 4. Experiments and Discussions We evaluated the performance of the proposed algorithm objectively and subjectively using groundtruth color patches. Both results demonstrate that the proposed method has superior haze removal and color enhancing capabilities when compared with existing techniques. In the experiment, we compared the proposed method with recent state-of-the-art methods. We selected the best parameters for each method. We used a computer running Windows XP with an Intel Core 2 (2.0 GHz) and 1 GB RAM.
4.1 Comparison of Conventional Methods Figure 8 shows the results using different descattering methods. The images contained 345×292 pixels. Schechner’s method produced blurring effects in the processed image, and Bazeille’s preprocessing method was seriously distorted [18]. Fattal’s method has the drawback that we must manually determine the background and objects (or foreground) in the image. This is difficult in practical applications. Nicholas’s graph-cut-based method was computationally time consuming and the processed image was blurred [19]. In comparison with He’s method, our approach performed better, and did not have visible mosaic artifacts caused by the soft matting process. Some of the regions were too dark (e.g., the right corner of the coral reefs), and the haze was not removed in other regions (e.g., the center of the image). Xiao’s method produced some halos around the objects (Fig.
14 8(g)) [20]. Additionally, there were some unresolved scatters around the coral reefs when using Ancuti’s method [6]. Ancuti’s method is very difficult to select the fusion parameters. Moreover, Chiang’s work [14] caused color distortions. The proposed method performed well in terms of descattering.
(a)
(c)
(e)
(b)
(d)
(f)
15
(g)
(h)
(i)
(j)
Figure 8. Different descattering methods applied to underwater coral reef images: (a) input image; (b) Schechner’s method [3]; (c) Bazeille’s method [18]; (d) Fattal’s method [7]; (e) Nicholas’s method [19]; (f) He’s method [9]; (g) Xiao’s method [20]; (h) Ancuti’s method [6]; (i) Chiang’s method [14]; and (j) the proposed method. We used the refined transmission map to clearly compare all the methods. Figure 9 shows the probability of detection map using HDR-VDP2-IQA measurements [21]. This metric is based on the human visual system and is therefore sensitive to three types of structural changes: loss of visible contrast (green), amplification of invisible contrast (blue), and reversal of visible contrast (red). The proposed method mainly amplified the contrast (blue). Some locations exhibited reverse contrast (red), and only a few locations exhibited a loss (green) of contrast. We can see that there was a large difference between the other methods and the proposed method (Fig. 9(j)).
16
(a)
(c)
(e)
(g)
(b)
(d)
(f)
(h)
Figure 9. Probability of detection map for coral reefs: (a) Bazeille’s method [18]; (b) Fattal’s method [7]; (c) Nicholas’s method [19]; (d) He’s method [9]; (e) Xiao’s method [20]; (f) Ancuti’s method [6]; (g) Chiang’s method [14]; and (h) the proposed method.
17 In addition to the visual analysis mentioned above, we conducted a quantitative analysis using some statistical values of the images (see Table 1). We considered the peak signal to noise ratio (PSNR) [22], the quality mean-opinion-score (Q-MOS) [21], the structural similarity (SSIM) [23], and the CPU time. Let Ai and Bi be the i-th pixels of the original image (A) and the distorted image (B), respectively. The MSE and PSNR values for the two images are given by
1 MSE N
N
( A B ) , 2
i 1
i
(31)
i
and
L2 PSNR 10log10 MSE ,
(32)
where PSNR means the peak signal to noise ratio (values are over 0 and a larger value is best). An multi-scale SSIM method was proposed in [23] for assessing image quality. For input images A and B, let μA, σA, and σAB denote the mean of A, the variance of A, and the covariance of A and B. The parameters of relative importance (α, β, γ) are equal to 1. Then,
SSIM ( x, y)
(2 A B C1 )(2 AB C 2 ) ( A2 B2 C1 )( A2 B2 C 2 )
,
(33)
where C1 and C2 are small constants. SSIM is called the structural similarity (values are between 0 (worst) and 1 (best)). The objective quality predictions do not map directly to the subjective mean opinion scores (MOS). There is a non-linear mapping function between subjective and objective predictions. A novel logistic function was proposed in [23, 27] to account for this mapping, that is,
Q MOS
100 , 1 exp( q1 (Q q2 ))
(34)
where Q is the pooling function that produced the strongest correction for the quality databases. The Q-MOS value is between 0 (worst) and 100 (best). Table 1 displays the numerical results of the QMOS, PSNR, and SSIM for several images. These results indicate that our approach effectively
18 removed scatter.
Table 1. Comparative Analysis of Different Descattering Methods (see Figure 8). Methods
PSNR
Q-MOS
SSIM
CPU Time [s]
Schechner [3]
15.7184
40.8985
0.3362
―
Bazeille [18]
18.4609
49.8972
0.6157
4.55
Fattal [7]
28.1155
91.9044
0.8328
20.05
Nicholas [19]
24.8454
78.0455
0.6184
90.09
He [9]
21.4759
92.5893
0.8191
30.85
Xiao [20]
24.6261
92.3767
0.8030
14.64
Ancuti [4]
21.7877
82.1602
0.7937
20.29
Chiang [14]
25.3353
90.3737
0.8258
11.76
Proposed
26.2918
91.5225
0.8293
10.83
Figure 10 shows other experimental results using different methods. The images contained 600×424 pixels. The proposed method performed the best. Table 2 shows the PSNR and SSIM values for Fig. 10. The proposed method performed better than the other methods. Table 2. Comparative Analysis of Different Descattering Methods (see Figure 10). Methods
PSNR
Q-MOS
SSIM
CPU Time [s]
Bazeille [18]
13.0844
19.8556
0.2061
1.98
Fattal [7]
11.7316
19.2204
0.4581
11.48
Nicholas [19]
18.9630
43.0762
0.7763
167.86
He [9]
23.2120
77.8366
0.9642
116.22
19 Xiao [20]
24.5089
66.0857
0.9493
104.98
Ancuti [4]
27.0414
39.8664
0.9074
62.76
Chiang [14]
27.3216
71.7899
0.9597
38.73
Proposed
27.7829
77.6780
0.9698
28.68
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
20
(i) Figure 10. Different methods for underwater coral image reconstruction: (a) input image; (b) Bazeille’s method [18]; (c) Fattal’s method [7]; (d) Nicholas’s method [19]; (e) He’s method [9]; (f) Xiao’s method [20]; (g) Ancuti’s method [6]; (h) Chiang’s method [14]; and (i) the proposed method. 4.2 Performance Analysis using Real-world Images We used a deep-sea image from KAIKO ROV [25], which was taken when surveying Muroto at the Nankai Trough, Japan. The images were provided by the Japan Agency for Marine-Earth Science and Technology (JAMSTEC), and were recorded on August 11, 1997. This image was captured at latitude 32.3515°N and longitude 134.5368°E. The water depth was 3627 meters (see Fig. 11 for details).
Figure 11. Location of captured deep-sea images.
21
(a)
(b)
(c)
(d)
(e)
(f)
22 (g)
(h)
(i) Figure 12. Different methods for deep-sea image reconstruction: (a) input image; (b) Bazeille’s method [18]; (c) Fattal’s method [7]; (d) Retinex [24]; (e) He’s method [9]; (f) Xiao’s method [20]; (g) Ancuti’s method [6]; (h) Chiang’s method [14]; and (i) the proposed method.
Figure 12 shows that Bazeille’s (Fig. 12(b)) and Fattal’s (Fig. 12(c)) methods give very distorted results. There were also some turbidity artefacts in the results of Retinex’s (Fig. 12(d)) [24], Xiao’s (Fig. 12(f)) and Ancuti’s (Fig. 12(g)) methods. The color was distorted in the result of Chiang’s method (Fig. 12(h)). He’s method (Fig. 12(e)) performed much better than the other techniques, but was computationally time consuming (it took approximately 30 seconds to process the 600×400 pixel input image). Nicolas’s method could not process the input image, because of the low contrast. The objects derived using the proposed method were much clearer than the other methods.
4.3 Performance Analysis for Color Correction Figure 13 illustrates the results obtained using the different methods for a color chart in a water tank. In this experiment, we added 200 mg/L deep-sea soil to the water. Bazeille’s, Fattal’s, Chiang’s, and Ancuti’s methods distorted the color. Some scatter remained in the results from He’s, Nicholas’, and Xiao’s methods. As shown in Fig. 13, the proposed method effectively removed haze and correctly
23 recovered the color. We can conclude that our method can correctly recover the colors of underwater scenes in suitable illumination.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
24
(i)
(j)
Figure 13. Simulation results of the color chart in turbid water: (a) turbid image; (b) Bazeille’s result [18]; (c) Fattal’s result [7]; (d) Nicholas’s result [19]; (e) He’s result [9]; (f) Xiao’s result [20]; (g) Ancuti’s result [6]; (h) Chiang’s result [14]; (i) the proposed result; and (j) true color chart.
Figure 14. Color accuracy metric ∆E for measuring the color relations of color chart. We measured the color accuracy with the International Commission on Illumination (CIE) distance metric ∆E [26]. The metric ∆E represents the Euclidean distance between two color images A, B in a Lab color space, calculated from their L, a, and b values by the following formula: E ( LA LB )2 (a A aB )2 (bA bB )2
(35) where the smaller of ∆E value, the more similar of two images. In this experiment, we cropped 42 color blocks of each method in Figure 13. Figure 14 shows the average ∆E value of different methods. Obviously, the proposed method can recover the distorted color well.
25 Figure 15 illustrates the PSNR and SSIM values of the different methods using nine images with known ground truths. The underwater images are were taken at turbidities of 1 mg/L, 2 mg/L, 10 mg/L, 20 mg/L, 100 mg/L, 200 mg/L, 300 mg/L, 400 mg/L, and 500 mg/L. Figure 14(a) shows that the average PSNR value of the proposed method was approximately 1 dB better than the other methods. In low-turbidity water (<100 mg/L), the average SSIM value was larger than the conventional methods.
(a)
(b) Figure 15. (a) Comparison of PSNR values for different methods, and (b) comparison of SSIM values for different methods using nine images with known ground truths.
26
However, when the turbidity increased, the SSIM values of Bazeille’s method became higher than the proposed method. A visual analysis suggests that although the structure of the image was improved, Bazeille’s method distorted the color. In contrast, the proposed method effectively removed the scatter and recovered the color information. 5. Conclusions In this paper, we developed and implemented a novel image enhancement technique for underwater optical image reconstruction. We proposed a de-vignetting method for removing the footprint in an underwater image. We also proposed a simple prior based on the difference between the attenuation of the different color channels, which inspired us to estimate the transmission map using the red and blue dual channels. We applied a weighted guided trigonometric filter, which preserves edges, removes noise, and reduces the computation time. Furthermore, the proposed spectral-based underwater image color correction method successfully created colorful underwater images that were better than the results of state-of-the-art methods. Limitations: During these experiments, we found that an increase in the turbid sediment reduces the image contrast. This may result in an inability to accurately estimate ambient light in highly turbid water. Another issue is that the proposed method cannot remove the highlights of LED flashes. Other problems such as camera reflection and lens distortion should be considered in future work. Acknowledgements This work was partially supported by the Grant in Aid for Japan Society for the Promotion of Science (No.15F15077), Research Fund of State Key Laboratory of Ocean Engineering in Shanghai Jiaotong University (OEK1315), Research Fund of State Key Laboratory of Marine Geology in Tongji University (MGK1407), and Grant in Aid for Japan Society for the Promotion of Science
27 (No.13J10713). The first author also wishes to thank the Japan Agency for Marine-Earth Science and Technology (JAMSTEC) for offering the deep-sea images and videos. Author Contributions The work presented here was a collaboration between all the authors. Conflicts of Interest The authors declare no conflict of interest. References and Notes 1.
D.M. Kocak; F.R. Dalgleish; F.M. Caimi; Y.Y. Schechner. A focus on recent developments and trends in underwater imaging. Mar. Technol. Soc. J., 42(1), (2008), 52–67.
2.
R. Schettini; S. Corchs. Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP J. Adv. Sig. Pr., 746052, (2010), 1-15.
3.
Y.Y. Schechner; Y. Averbuch. Regularized image recovery in scattering media. IEEE T. Pattern Anal., 29(9), (2007), 1655–1660.
4.
T. Treibitz; Y.Y. Schechner. Turbid scene enhancement using multi-directional illumination fusion. IEEE T. Image Process., 21(11), (2012), 4662–4667.
5.
B. Ouyang; F.R. Dalgleish; F.M. Caimi; A.K. Vuorenkoski; T.E. Giddings; J.J. Shirron. Image enhancement for underwater pulsed laser line scan imaging system. In Proceedings of SPIE, 8372, (2012), 1-12.
6.
C. Ancuti; C.O. Ancuti; T. Haber; P. Bekaert. Enhancing underwater images and videos by fusion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (2012), 81–88.
7.
R. Fattal. Signal image dehazing. In SIGGRAPH, (2008), 1–9.
8.
H. Lu; Y. Li; L. Zhang; S. Serikawa. Enhancing underwater image by dehazing and colorization. International Review on Computers & Software, 7(7), (2012), 3470–3474.
28 9.
K. He; J. Sun; X. Tang. Single image haze removal using dark channel prior. IEEE T. Pattern Anal., 33(12), (2011), 2341–2353.
10. K. He; J. Sun; X. Tang. Guided image filtering. In Proceedings of the 11th European Conference on Computer Vision, 1, (2010), 1–14. 11. S. Narasimhan; S. Nayar. Contrast restoration of weather degraded images. IEEE T. Pattern Anal., 25(6), 2003, 713–724. 12. K. Sooknanan; A. Kokaram; D. Corrigan. Improving underwater visibility using vignetting correction. In Proceedings of SPIE, 8305, (2012), 1-8. 13. Y. Zheng; S. Lin; S.B. Kang; R. Xiao; J.C. Gee; C. Kambhamettu. Single-image vignetting correction from gradient distribution symmetries. IEEE T. Pattern Anal., 35(6), (2013), 1480– 1494. 14. J.Y. Chiang; Y.C. Chen. Underwater image enhancement by wavelength compensation and dehazing. IEEE T. Image Process., 21(4), (2012), 1756–1769. 15. S. Serikawa; H. Lu. Underwater image dehazing using joint trilateral filter. Comput. Elec. En., 41(1), (2014), 41–50. 16. H. Lu; Y. Li; S. Serikawa. Underwater image enhancement using guided trigonometric bilateral filter and fast automatic color correction. In Proceedings of IEEE International Conference on Image Processing, (2013), 3412–3416. 17. D.L. Bongiorno; M. Bryson; D.G. Dansereau; S.B. Williams. Spectral characterization of COTS RGB cameras using a linear variable edge filter. In Proceedings of SPIE-IS&T, 8660, (2013), 1-8. 18. S. Bazeille; I. Quidu; L. Jaulin; J.P. Malkasse. Automatic underwater image pre-processing. In Proceedings of Caracterisation Du Milieu Marin, (2006), 1–8. 19. C-B Nicholas; M. Anush; R.M. Eustice. Initial results in underwater single image dehazing. In Proceedings of IEEE OCEANS, (2010), 1–8. 20. C. Xiao; J. Gan. Fast image dehazing using guided joint bilateral filter. Visual Comput., 28(6–8), (2012), 713–721.
29 21. R. Mantiuk; K.J. Kim; A.G. Rempel; W. Heidrich. HDR-VDP2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM T. Graphic., 30(4), (2011), 40– 52. 22. M. Chambah; D. Semani; A. Renouf; P. Courtellemont; A. Rizzi. Underwater color constancy: enhancement of automatic live fish recognition. In Proceedings of SPIE, 5293, (2004), 157–168. 23. Z. Wang; A. C. Bovik; H. R. Sheikh; E. P. Simoncelli. Image quality assessment: from error measurement to structural similarity. IEEE T. Image Process., 13(1), (2004), 600–612. 24. M. Bertalmio; V. Caselles; E. Provenzi; A. Rizzi. Perceptual color correction through variational techniques. IEEE T. Image Process., 16(4), (2007), 1058–1072. 25. Japan Agency for Marine-Earth Science and Technology, http://www.jamstec.go.jp/e/ 26. O. Pele; M. Werman. Improving perceptual color difference using basic color terms. (2012), http://arXiv:1211:5556, 1-15.
30
Highlights: We propose the guided trigonometric filter to refine the coarse depth transmission map. The single de-vignetting method is introduced to remove the footprint. The spectral properties-based color correction method is proposed to recover the scene color.
S1047-3203(16)30033-5 http://dx.doi.org/10.1016/j.jvcir.2016.03.029 YJVCI 1726
To appear in:
J. Vis. Commun. Image R.
Received Date: Revised Date: Accepted Date:
4 June 2014 9 June 2015 30 March 2016
Please cite this article as: H. Lu, Y. Li, X. Xu, J. Li, Z. Liu, X. Li, J. Yang, S. Serikawa, Underwater Image Enhancement Method Using Weighted Guided Trigonometric Filtering and Artificial Light Correction, J. Vis. Commun. Image R. (2016), doi: http://dx.doi.org/10.1016/j.jvcir.2016.03.029
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Underwater Image Enhancement Method Using Weighted Guided Trigonometric Filtering and Artificial Light Correction Huimin Lua,b,c, Yujie Lia, Xing Xud, Jianru Lib, Zhifei Liub, Xin Lic, Jianmin Yangc, Seiichi Serikawaa a
Department of Electronics and Electrical Engineering, Kyushu Institute of Technology, Kitakyushu 804-8550, Japan.
b
State Key Laboratory of Marine Geology, Tongji University, Shanghai 200092, China.
c
State Key Laboratory of Ocean Engineering, Shanghai Jiaotong University, Shanghai 200240, China.
d
University of Electrical Science and Technology of China, Chengdu 611731, China
* Corresponding author: [email protected]; Tel.: +81-93-884-3282; Fax: +81-93-884-3203. Abstract: This paper describes a novel method for enhancing optical images using a weighted guided trigonometric filter and the camera’s spectral properties in turbid water. Absorption, scattering, and artificial lighting are three major distortion issues in underwater optical imaging. Absorption permanently removes photons from the imaging path. Scattering is caused by large suspended particles found in turbid water, which redirect the angle of the photon path. Artificial lighting results in footprint effects, which cause vignetting distortion in the captured image. Our contributions include a novel deep-sea imaging method that compensates for the attenuation discrepancy along the propagation path, and an effective underwater scene enhancement scheme. The recovered images are characterized by a reduced noise level, better exposure of dark regions, and improved global contrast such that the finest details and edges are significantly enhanced. Our experiments showed that the average Peak Signal to Noise Ratio (PSNR) improved by at least 1 dB when compared with state-of-the-art-methods.
2 Keywords: deep-sea imaging; inherent optical property; image enhancement
1. Introduction There have been increased developments in deep-sea exploration using autonomous underwater vehicles (AUVs) and unmanned underwater vehicles (UUVs). However, the contrast of underwater images is still a major issue for application. It is difficult to acquire clear underwater images around underwater vehicles. Since the 1960s, sonar sensors have been extensively used to detect and recognize objects in oceans. Due to the principles of acoustic imaging, sonar-imaged images have many shortcomings such as a low signal to noise ratio and a low resolution. Consequently, vision sensors must be used for short-range identification because sonars yield to low-quality images [1]. In contrast to natural images, underwater images suffer from poor visibility. Firstly, Light is absorbed when sunlight is reflected by a water surface. Additionally, absorption substantially reduces the ambient light energy. Random attenuation and scattering degrade the contrast of the scene. Objects at a distance of more than 10 meters are almost indistinguishable, because the colors are faded [2]. Furthermore, artificial lighting can cause a distinctive footprint on the seafloor. Over the past two decades, researchers have been focusing on improving the quality of underwater images. Recent research on underwater image enhancement can be classified into two major categories according to input types: multiple or single inputs. For multiple inputs, Schechner et al. used a polarization imaging method to compensate for visibility degradation [3]. Treibitz et al. proposed a multi-directional illumination fusion method for turbid scene enhancement [4]. Ouyang et al. proposed a multiple laser lighting and bilateral filtering-based image deconvolution method for underwater image enhancement [5]. For single-input image methods, Ancuti et al. proposed a Laplacian fusion method that reconstructs a clear image in turbid water [6]. Fattal et al. first used the underwater dehazing method for underwater image enhancement [7]. This technique estimates the scene radiance and derives the transmission map
3 using a single image. However, it cannot sufficiently process images with heavy haze and colors can be distorted. Lu et al. [8] proposed a dehazing and color correction method for recovering underwater scenes. He et al. [9] proposed a scene depth information-based dark channel priors dehazing algorithm using a matting Laplacian. However, this algorithm requires significant computation time. To overcome this disadvantage, they also proposed a guided image filter [10] with the foggy image used as a reference image. However, this method leads to incomplete haze removal. Although these approaches can enhance the image contrast, there are several drawbacks that reduce their practical applicability. First, the imaging equipment is difficult to use in practice (e.g., a rangegated laser imaging system is rarely applied in practice [5]). Second, multiple input images are difficult to obtain [3, 4]. Third, these methods cannot sufficiently alleviate color distortions [6–9]. As noted in previous publications [6, 9], it is impossible to capture the same scene at the same time in turbid water. Consequently, single-image enhancement methods can achieve better results. Single underwater-image enhancement is a challenging, but ill-posed problem. In underwater imaging, the captured images are significantly influenced by absorption, scattering, and inhomogeneous illumination. In this paper, we introduce a novel scheme that enhances underwater images using a single image as input. The proposed method overcomes the previously mentioned drawbacks of conventional methods. The organization of this paper is as follows. Section 2 explains the ocean imaging model. Section 3 describes the method for underwater image enhancement and proposes our weighted guided trigonometric filter. Section 4 applies the proposed method to underwater optical images. Finally, Section 5 concludes this paper. 2. Ocean Imaging Model Artificial light and ambient light traveling through the water is the source of illumination in an ocean environment. Suppose that the amount of light radiation (W) formed after wavelength attenuation can be derived using the energy attenuation model as
4
EcW ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer(c) L ( x ) , c {r, g , b}.
(1)
where EcW ( x ) is the amount of illumination at the scene point, EcA ( x ) is the amount of illumination from ambient light at the scene point, EcI ( x ) is the illumination from artificial light, and Nrer is the normalized residual energy ratio. In this imaging model, artificial light is reflected at a distance L(x) from the camera. D(x) is the underwater scene depth. Absorption and scattering occurs in this process. Suppose that the attenuation rate is c ( x ) , then the illumination of ambient light is
Ec ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ), c {r, g , b}.
(2)
Following the Nayar–Narasimhan dehazing model [11], the captured image I c ( x ) formed at the camera plane can be formulated as
I c ( x ) EcA ( x ) Nrer(c) D ( x ) EcI ( x ) Nrer(c) L ( x ) c ( x ) tc ( x ) 1 tc ( x ) Ac , c {r, g , b}.
(3)
where the background Ac represents the light reflected by the object. tc ( x ) can be represented in terms of a light beam of wavelength over a distance d(x) within the water, that is,
Eresidual ( x ) tc ( x ) initial 10 d ( x ) Nrer (c)d ( x ) , E ( x )
(4)
where is the extinction coefficient of the medium. Nrer is the normalized residual energy ratio [14]; in the Ocean Type I, it satisfies
0.8 ~ 0.85 if 650 750m( red ) Nrer (c) 0.93 ~ 0.97 if 490 550m( green ) . 0.95 ~ 0.99 if 400 490m(blue)
(5)
Consequently, substituting Eq. (4) into Eq. (3) we can obtain
I c ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ) Nrer (c) d ( x ) 1 Nrer (c)d ( x ) Ac , c {r, g , b}.
(6)
5 The above equation incorporates the light scattering during the course of propagation from object to the camera, and the wavelength attenuation along the object–light and water depth. When the object– light distance L(x), scene depth D(x), and camera–object transmission map d(x) are known, we can recover the final clean image.
Figure 1. Schematic of ocean optical imaging model. Figure 1 shows the diagrammatic sketch of the proposed model. To improve the image quality, we use the process shown in Fig. 2. In the first stage, we determine the artificial light (if it exists) and remove the footprint or vignetting. In the next stage, we calculate the transmission map of the scene using dual-channels priors, refine the transmission map using a weighted guided trigonometric filter, and remove the scatters using the dehazing method. Finally, we recover the scene color using the camera’s spectral properties.
6
Figure 2. Scheme for the underwater image enhancement process. 3. Underwater Image Enhancement Method 3.1. Artificial Light Correction In deep sea, we must use artificial light for imaging. However, this causes footprint effects. Sooknanan et al. [12] proposed a multi-frame vignetting correction method for removing the vignetting phenomenon, which involves estimating the light source’s footprint on the seafloor. This artificial light correction is effective, but is computationally time consuming. Moreover, it is difficult to capture multiple images of the same scene in turbid water, because of floating sediments. With this in mind, we propose a single-image based vignette removal method [13]. We are interested in the overall effect of light attenuation through the system and not all of the image formation details, so we derived an effective degradation model. That is,
Z (r, ) O(r, )V (r ) ,
(7)
7 where Z is the image with vignetting, O is the vignetting-free image, and V is the vignetting function. Our goal is to find the optimal vignetting function V that minimizes the asymmetry of the radial gradient distribution. Taking the log of Eq. (7), we get
ln(Z (r, ) ln O(r, ) ln V (r ) .
(8)
Let Z=lnZ, O=lnO, and V=lnV. We denote the radial gradients of Z, O, and V for each pixel (r, θ) by RrZ(r,θ), RrO(r,θ), and RrV(r,θ). Then,
R Zr (r, ) R Or (r, ) R Vr (r, ) .
(9)
Given an image with vignetting (Z), we can find the maximum a posterior (MAP) solution to V. Applying Bayes rule, we get
V arg max P(V | Z) arg max P(Z | V) P(V) . V
(10)
V
We consider the vignetting function at discrete, evenly sampled radii (V(rt), rt∈Sr), where Sr ={ r0 , r1 ,…, rn-1 }. Each pixel (r, θ) is associated with the sector that contains it. The vignetting function is generally smooth, so we obtain
P( V ) e
s
rtSr V "(rt )2
,
(11)
where λs is chosen to compensate for the noise level in the image, and V ''( rt ) is
V"(rt )
V(rt 1 ) 2V(rt ) V(rt 1 ) . ( r )2
(12)
Using the sparsely prior method on the vignetting-free image O, O
P(Ζ | V) P( RrO ) e |Rr | , α<1.
(13)
Substituting Eq.(13) and Eq.(9), we have
P( Z | V ) e
( r , )|R Zr ( r , )R Vr ( r )|
.
(14)
The overall energy function P(Z|V)P(V) can be written as E
| R
(r , )
Z r
(r , ) R rV (r ) | s V "(rt ) 2 . rt Sr
(15)
8 We can estimate V(rt) by minimizing E. Then, we use the Iteratively Reweighted Least Squares (IRLS) to estimate the vignetting function. Figure 3 shows the principle of the de-vignetting method, and Figure 4 shows the simulation results of the color chart in a water tank. In this simulation, we used an OLYMPUS Tough TG-2 underwater camera (OLYMPUS Co., Japan), the water depth was 0.3 meters, the camera–object distance was 0.8 meters, and the object–light distance was 0.5 meters. We added deep-sea soil to the water at a turbidity rate of 50 mg/L.
Figure 3. Underwater vignetting correction method.
(a)
(b)
(c)
Figure 4. Result of artificial light correction: (a) input image; (b) vignetting-corrected image; and (c) estimated vignetting. 3.2. Transmission Map Estimation In [14], the author found that the red color channel is the dark channel of underwater images. During our experiments, we found that the lowest RGB channel in turbid water was not always the red channel. The blue channel may be significantly attenuated [15, 16], because there can be certain
9 organic matter that absorbs blue more significantly in certain coastal zones. Although the red wavelength is easily absorbed by traveling in water, the distance between the camera and object is not enough to significantly absorb the red wavelength (see Fig. 5). Therefore, the blue channel may be the smallest. Consequently, we used the red–blue dual channels to estimate a rough transmission map.
(a)
(b)
Figure 5. RGB histograms of underwater images.
As shown in Eq. (6), scattered and absorbed images reflected from the scene point can be defined as
J c ( x ) EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) c ( x ),
(16)
c {r, b}.
We denote the minimum pixel of the red and blue dual-channels for the underwater image J c ( x ) as
J min ( x ) min min J c ( y ), c {r, b}. c
(17)
y ( x )
If an object belongs to the foreground, the minimum value of its pixels is very small. Minimizing the local patch (x) on the scattered image I c ( x ) in Eq. (6), we have
min I c ( y ) min J c ( y ) Nrer( c) d ( x ) 1 Nrer( c) d ( x ) Ac ,
y ( x )
y ( x )
c {r, b}.
(18)
10 Suppose that Ac is the homogeneous background in small local patch, and that the normalized residual energy ratio Nrer(c)d ( y ) on the small local patch (x) is essentially a constant. Then, the min value of Eq. (18) can be written as
min I c ( y ) min J c ( y ) Nrer( c) d ( x ) min 1 Nrer( c) d ( x ) Ac ,
y ( x )
y ( x )
y ( x )
c {r, b}.
(19)
We can rearrange the above equation to get min I c ( y ) min J c ( y ) y ( x ) min min y ( x ) Nrer ( c) d ( x ) c c Ac Ac
(20)
min 1 Nrer ( c) d ( x ) , c {r, b}. c
Therefore, the second term of the above equation is equal to 0. Consequently, the estimated distance map is min I c ( y ) max Nrer (c)d ( x ) 1 min y ( x ) , c {r, b} . c c Ac
(21)
Finally, the distance map can be obtained using
min I c ( y ) d ( x ) ln 1 min y ( x ) / ln max Nrer( c), c {r, b} . c A c
(22)
3.3. Global Optimization of Transmission Map In the previous section, we roughly estimated the transmission map. However, because it is a patch-based operation, the obtained transmission map contains some block artifacts. So, we propose a weighted guided trigonometric filter (WGTF) to remove the artifacts. The WGTF process is first performed under the guidance of image G, which can be the input image itself (i.e., I c ( x ) ). Let Ip, Iq, Gp, and Gq be the intensity values at pixels p and q of the transmission map and the guided image, respectively, and wk be the kernel window centered at pixel k. Then,
11 WGTF ( I ) p
W
qwk
1
GTFpq
W
(G ) qwk
GTFpq
(G ) I q ,
(23)
where the kernel weight function is WGTFpq (G )
Here,
k and k2 are
(G p k )(Gq k ) 1 1 . 2 | w | k:( p ,q )wk k2
(24)
the mean and variance of the guided image G in the local window wk, and |w| is the
number of pixels in window w. When Gp and Gq are on the same side of an edge, the weight assigned to pixel q is large. When Gp and Gq are on different sides, a small weight will be assigned to pixel q. 3.4. Recovering the Scene Radiance The previous sections derived the refined transmission map d(x). To remove the scatter, we also need to solve the attenuation rate, c ( x ) . We use the least squares solution to achieve this, as in [14]. That is,
c ( x ) J c ( x )T J c ( x ) J c ( x ) T 1
EcA ( x ) Nrer (c) D ( x ) EcI ( x ) Nrer (c) L ( x ) ,
(25)
c {r, g , b}. After removing the artificial light, Eq. (6) can be written as
I c ( x ) EcA ( x ) Nrer (c) D ( x ) c ( x ) Nrer(c) d ( x ) 1 Nrer(c)d ( x ) Ac , c {r, g , b}.
(26)
According to the Nayar–Narasimhan dehazing model, we can obtain the descattered image using Jˆc ( x )
I c ( x ) 1 Nrer ( c) d ( x ) Ac Nrer (c )d ( x )
EcA ( x ) Nrer (c) D ( x ) c ( x ) Nrer (c) d ( x ) ,
(27)
c {r, g , b}.
Figure 6 shows the descattered result of the proposed algorithm. The haze-like artifacts were completely removed, and the contrast was increased.
12
(a)
(b)
Figure 6. Descattered result of the proposed method: (a) devignetted image; and (b) descattered image. 3.5. Spectral Properties-based Color Correction In [14], the author simply corrected the color according to the attenuation of the water depth. However, in practice, the spectral response function [17] of a camera represents its relative sensitivity. We use the chromatic transfer function to weight the light from the ocean surface to a given object depth using
Esurface , Eobject
(28)
where the transfer function τ at wavelength λ is derived from the irradiance of the surface ( Esurface ) and the irradiance of the object ( Eobject ). Based on the spectral response of the RGB camera, we convert the transfer function to the RGB domain, that is n
RGB Cc ( ) ,
(29)
where the weighted RGB transfer function is τRGB, Cb(λ) is the underwater spectral characteristic function of color band c {r, g , b} . n is the number of discrete bands of the camera’s spectral characteristic function. Finally, the corrected image is calculated from the weighted RGB transfer function using J c ( x ) Jˆc ( x ) RGB ,
(30)
where J c ( x ) and Jˆc ( x ) are the color-corrected and uncorrected images, respectively. Figure 7 shows the color correction result in a simulation. Compared with the color chart in Figure 7(b), the proposed
13 method recovered the correct scene colors.
(a)
(b)
Figure 7. Result of the color correction: (a) color-corrected image; (b) true color chart. 4. Experiments and Discussions We evaluated the performance of the proposed algorithm objectively and subjectively using groundtruth color patches. Both results demonstrate that the proposed method has superior haze removal and color enhancing capabilities when compared with existing techniques. In the experiment, we compared the proposed method with recent state-of-the-art methods. We selected the best parameters for each method. We used a computer running Windows XP with an Intel Core 2 (2.0 GHz) and 1 GB RAM.
4.1 Comparison of Conventional Methods Figure 8 shows the results using different descattering methods. The images contained 345×292 pixels. Schechner’s method produced blurring effects in the processed image, and Bazeille’s preprocessing method was seriously distorted [18]. Fattal’s method has the drawback that we must manually determine the background and objects (or foreground) in the image. This is difficult in practical applications. Nicholas’s graph-cut-based method was computationally time consuming and the processed image was blurred [19]. In comparison with He’s method, our approach performed better, and did not have visible mosaic artifacts caused by the soft matting process. Some of the regions were too dark (e.g., the right corner of the coral reefs), and the haze was not removed in other regions (e.g., the center of the image). Xiao’s method produced some halos around the objects (Fig.
14 8(g)) [20]. Additionally, there were some unresolved scatters around the coral reefs when using Ancuti’s method [6]. Ancuti’s method is very difficult to select the fusion parameters. Moreover, Chiang’s work [14] caused color distortions. The proposed method performed well in terms of descattering.
(a)
(c)
(e)
(b)
(d)
(f)
15
(g)
(h)
(i)
(j)
Figure 8. Different descattering methods applied to underwater coral reef images: (a) input image; (b) Schechner’s method [3]; (c) Bazeille’s method [18]; (d) Fattal’s method [7]; (e) Nicholas’s method [19]; (f) He’s method [9]; (g) Xiao’s method [20]; (h) Ancuti’s method [6]; (i) Chiang’s method [14]; and (j) the proposed method. We used the refined transmission map to clearly compare all the methods. Figure 9 shows the probability of detection map using HDR-VDP2-IQA measurements [21]. This metric is based on the human visual system and is therefore sensitive to three types of structural changes: loss of visible contrast (green), amplification of invisible contrast (blue), and reversal of visible contrast (red). The proposed method mainly amplified the contrast (blue). Some locations exhibited reverse contrast (red), and only a few locations exhibited a loss (green) of contrast. We can see that there was a large difference between the other methods and the proposed method (Fig. 9(j)).
16
(a)
(c)
(e)
(g)
(b)
(d)
(f)
(h)
Figure 9. Probability of detection map for coral reefs: (a) Bazeille’s method [18]; (b) Fattal’s method [7]; (c) Nicholas’s method [19]; (d) He’s method [9]; (e) Xiao’s method [20]; (f) Ancuti’s method [6]; (g) Chiang’s method [14]; and (h) the proposed method.
17 In addition to the visual analysis mentioned above, we conducted a quantitative analysis using some statistical values of the images (see Table 1). We considered the peak signal to noise ratio (PSNR) [22], the quality mean-opinion-score (Q-MOS) [21], the structural similarity (SSIM) [23], and the CPU time. Let Ai and Bi be the i-th pixels of the original image (A) and the distorted image (B), respectively. The MSE and PSNR values for the two images are given by
1 MSE N
N
( A B ) , 2
i 1
i
(31)
i
and
L2 PSNR 10log10 MSE ,
(32)
where PSNR means the peak signal to noise ratio (values are over 0 and a larger value is best). An multi-scale SSIM method was proposed in [23] for assessing image quality. For input images A and B, let μA, σA, and σAB denote the mean of A, the variance of A, and the covariance of A and B. The parameters of relative importance (α, β, γ) are equal to 1. Then,
SSIM ( x, y)
(2 A B C1 )(2 AB C 2 ) ( A2 B2 C1 )( A2 B2 C 2 )
,
(33)
where C1 and C2 are small constants. SSIM is called the structural similarity (values are between 0 (worst) and 1 (best)). The objective quality predictions do not map directly to the subjective mean opinion scores (MOS). There is a non-linear mapping function between subjective and objective predictions. A novel logistic function was proposed in [23, 27] to account for this mapping, that is,
Q MOS
100 , 1 exp( q1 (Q q2 ))
(34)
where Q is the pooling function that produced the strongest correction for the quality databases. The Q-MOS value is between 0 (worst) and 100 (best). Table 1 displays the numerical results of the QMOS, PSNR, and SSIM for several images. These results indicate that our approach effectively
18 removed scatter.
Table 1. Comparative Analysis of Different Descattering Methods (see Figure 8). Methods
PSNR
Q-MOS
SSIM
CPU Time [s]
Schechner [3]
15.7184
40.8985
0.3362
―
Bazeille [18]
18.4609
49.8972
0.6157
4.55
Fattal [7]
28.1155
91.9044
0.8328
20.05
Nicholas [19]
24.8454
78.0455
0.6184
90.09
He [9]
21.4759
92.5893
0.8191
30.85
Xiao [20]
24.6261
92.3767
0.8030
14.64
Ancuti [4]
21.7877
82.1602
0.7937
20.29
Chiang [14]
25.3353
90.3737
0.8258
11.76
Proposed
26.2918
91.5225
0.8293
10.83
Figure 10 shows other experimental results using different methods. The images contained 600×424 pixels. The proposed method performed the best. Table 2 shows the PSNR and SSIM values for Fig. 10. The proposed method performed better than the other methods. Table 2. Comparative Analysis of Different Descattering Methods (see Figure 10). Methods
PSNR
Q-MOS
SSIM
CPU Time [s]
Bazeille [18]
13.0844
19.8556
0.2061
1.98
Fattal [7]
11.7316
19.2204
0.4581
11.48
Nicholas [19]
18.9630
43.0762
0.7763
167.86
He [9]
23.2120
77.8366
0.9642
116.22
19 Xiao [20]
24.5089
66.0857
0.9493
104.98
Ancuti [4]
27.0414
39.8664
0.9074
62.76
Chiang [14]
27.3216
71.7899
0.9597
38.73
Proposed
27.7829
77.6780
0.9698
28.68
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
20
(i) Figure 10. Different methods for underwater coral image reconstruction: (a) input image; (b) Bazeille’s method [18]; (c) Fattal’s method [7]; (d) Nicholas’s method [19]; (e) He’s method [9]; (f) Xiao’s method [20]; (g) Ancuti’s method [6]; (h) Chiang’s method [14]; and (i) the proposed method. 4.2 Performance Analysis using Real-world Images We used a deep-sea image from KAIKO ROV [25], which was taken when surveying Muroto at the Nankai Trough, Japan. The images were provided by the Japan Agency for Marine-Earth Science and Technology (JAMSTEC), and were recorded on August 11, 1997. This image was captured at latitude 32.3515°N and longitude 134.5368°E. The water depth was 3627 meters (see Fig. 11 for details).
Figure 11. Location of captured deep-sea images.
21
(a)
(b)
(c)
(d)
(e)
(f)
22 (g)
(h)
(i) Figure 12. Different methods for deep-sea image reconstruction: (a) input image; (b) Bazeille’s method [18]; (c) Fattal’s method [7]; (d) Retinex [24]; (e) He’s method [9]; (f) Xiao’s method [20]; (g) Ancuti’s method [6]; (h) Chiang’s method [14]; and (i) the proposed method.
Figure 12 shows that Bazeille’s (Fig. 12(b)) and Fattal’s (Fig. 12(c)) methods give very distorted results. There were also some turbidity artefacts in the results of Retinex’s (Fig. 12(d)) [24], Xiao’s (Fig. 12(f)) and Ancuti’s (Fig. 12(g)) methods. The color was distorted in the result of Chiang’s method (Fig. 12(h)). He’s method (Fig. 12(e)) performed much better than the other techniques, but was computationally time consuming (it took approximately 30 seconds to process the 600×400 pixel input image). Nicolas’s method could not process the input image, because of the low contrast. The objects derived using the proposed method were much clearer than the other methods.
4.3 Performance Analysis for Color Correction Figure 13 illustrates the results obtained using the different methods for a color chart in a water tank. In this experiment, we added 200 mg/L deep-sea soil to the water. Bazeille’s, Fattal’s, Chiang’s, and Ancuti’s methods distorted the color. Some scatter remained in the results from He’s, Nicholas’, and Xiao’s methods. As shown in Fig. 13, the proposed method effectively removed haze and correctly
23 recovered the color. We can conclude that our method can correctly recover the colors of underwater scenes in suitable illumination.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
24
(i)
(j)
Figure 13. Simulation results of the color chart in turbid water: (a) turbid image; (b) Bazeille’s result [18]; (c) Fattal’s result [7]; (d) Nicholas’s result [19]; (e) He’s result [9]; (f) Xiao’s result [20]; (g) Ancuti’s result [6]; (h) Chiang’s result [14]; (i) the proposed result; and (j) true color chart.
Figure 14. Color accuracy metric ∆E for measuring the color relations of color chart. We measured the color accuracy with the International Commission on Illumination (CIE) distance metric ∆E [26]. The metric ∆E represents the Euclidean distance between two color images A, B in a Lab color space, calculated from their L, a, and b values by the following formula: E ( LA LB )2 (a A aB )2 (bA bB )2
(35) where the smaller of ∆E value, the more similar of two images. In this experiment, we cropped 42 color blocks of each method in Figure 13. Figure 14 shows the average ∆E value of different methods. Obviously, the proposed method can recover the distorted color well.
25 Figure 15 illustrates the PSNR and SSIM values of the different methods using nine images with known ground truths. The underwater images are were taken at turbidities of 1 mg/L, 2 mg/L, 10 mg/L, 20 mg/L, 100 mg/L, 200 mg/L, 300 mg/L, 400 mg/L, and 500 mg/L. Figure 14(a) shows that the average PSNR value of the proposed method was approximately 1 dB better than the other methods. In low-turbidity water (<100 mg/L), the average SSIM value was larger than the conventional methods.
(a)
(b) Figure 15. (a) Comparison of PSNR values for different methods, and (b) comparison of SSIM values for different methods using nine images with known ground truths.
26
However, when the turbidity increased, the SSIM values of Bazeille’s method became higher than the proposed method. A visual analysis suggests that although the structure of the image was improved, Bazeille’s method distorted the color. In contrast, the proposed method effectively removed the scatter and recovered the color information. 5. Conclusions In this paper, we developed and implemented a novel image enhancement technique for underwater optical image reconstruction. We proposed a de-vignetting method for removing the footprint in an underwater image. We also proposed a simple prior based on the difference between the attenuation of the different color channels, which inspired us to estimate the transmission map using the red and blue dual channels. We applied a weighted guided trigonometric filter, which preserves edges, removes noise, and reduces the computation time. Furthermore, the proposed spectral-based underwater image color correction method successfully created colorful underwater images that were better than the results of state-of-the-art methods. Limitations: During these experiments, we found that an increase in the turbid sediment reduces the image contrast. This may result in an inability to accurately estimate ambient light in highly turbid water. Another issue is that the proposed method cannot remove the highlights of LED flashes. Other problems such as camera reflection and lens distortion should be considered in future work. Acknowledgements This work was partially supported by the Grant in Aid for Japan Society for the Promotion of Science (No.15F15077), Research Fund of State Key Laboratory of Ocean Engineering in Shanghai Jiaotong University (OEK1315), Research Fund of State Key Laboratory of Marine Geology in Tongji University (MGK1407), and Grant in Aid for Japan Society for the Promotion of Science
27 (No.13J10713). The first author also wishes to thank the Japan Agency for Marine-Earth Science and Technology (JAMSTEC) for offering the deep-sea images and videos. Author Contributions The work presented here was a collaboration between all the authors. Conflicts of Interest The authors declare no conflict of interest. References and Notes 1.
D.M. Kocak; F.R. Dalgleish; F.M. Caimi; Y.Y. Schechner. A focus on recent developments and trends in underwater imaging. Mar. Technol. Soc. J., 42(1), (2008), 52–67.
2.
R. Schettini; S. Corchs. Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP J. Adv. Sig. Pr., 746052, (2010), 1-15.
3.
Y.Y. Schechner; Y. Averbuch. Regularized image recovery in scattering media. IEEE T. Pattern Anal., 29(9), (2007), 1655–1660.
4.
T. Treibitz; Y.Y. Schechner. Turbid scene enhancement using multi-directional illumination fusion. IEEE T. Image Process., 21(11), (2012), 4662–4667.
5.
B. Ouyang; F.R. Dalgleish; F.M. Caimi; A.K. Vuorenkoski; T.E. Giddings; J.J. Shirron. Image enhancement for underwater pulsed laser line scan imaging system. In Proceedings of SPIE, 8372, (2012), 1-12.
6.
C. Ancuti; C.O. Ancuti; T. Haber; P. Bekaert. Enhancing underwater images and videos by fusion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (2012), 81–88.
7.
R. Fattal. Signal image dehazing. In SIGGRAPH, (2008), 1–9.
8.
H. Lu; Y. Li; L. Zhang; S. Serikawa. Enhancing underwater image by dehazing and colorization. International Review on Computers & Software, 7(7), (2012), 3470–3474.
28 9.
K. He; J. Sun; X. Tang. Single image haze removal using dark channel prior. IEEE T. Pattern Anal., 33(12), (2011), 2341–2353.
10. K. He; J. Sun; X. Tang. Guided image filtering. In Proceedings of the 11th European Conference on Computer Vision, 1, (2010), 1–14. 11. S. Narasimhan; S. Nayar. Contrast restoration of weather degraded images. IEEE T. Pattern Anal., 25(6), 2003, 713–724. 12. K. Sooknanan; A. Kokaram; D. Corrigan. Improving underwater visibility using vignetting correction. In Proceedings of SPIE, 8305, (2012), 1-8. 13. Y. Zheng; S. Lin; S.B. Kang; R. Xiao; J.C. Gee; C. Kambhamettu. Single-image vignetting correction from gradient distribution symmetries. IEEE T. Pattern Anal., 35(6), (2013), 1480– 1494. 14. J.Y. Chiang; Y.C. Chen. Underwater image enhancement by wavelength compensation and dehazing. IEEE T. Image Process., 21(4), (2012), 1756–1769. 15. S. Serikawa; H. Lu. Underwater image dehazing using joint trilateral filter. Comput. Elec. En., 41(1), (2014), 41–50. 16. H. Lu; Y. Li; S. Serikawa. Underwater image enhancement using guided trigonometric bilateral filter and fast automatic color correction. In Proceedings of IEEE International Conference on Image Processing, (2013), 3412–3416. 17. D.L. Bongiorno; M. Bryson; D.G. Dansereau; S.B. Williams. Spectral characterization of COTS RGB cameras using a linear variable edge filter. In Proceedings of SPIE-IS&T, 8660, (2013), 1-8. 18. S. Bazeille; I. Quidu; L. Jaulin; J.P. Malkasse. Automatic underwater image pre-processing. In Proceedings of Caracterisation Du Milieu Marin, (2006), 1–8. 19. C-B Nicholas; M. Anush; R.M. Eustice. Initial results in underwater single image dehazing. In Proceedings of IEEE OCEANS, (2010), 1–8. 20. C. Xiao; J. Gan. Fast image dehazing using guided joint bilateral filter. Visual Comput., 28(6–8), (2012), 713–721.
29 21. R. Mantiuk; K.J. Kim; A.G. Rempel; W. Heidrich. HDR-VDP2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM T. Graphic., 30(4), (2011), 40– 52. 22. M. Chambah; D. Semani; A. Renouf; P. Courtellemont; A. Rizzi. Underwater color constancy: enhancement of automatic live fish recognition. In Proceedings of SPIE, 5293, (2004), 157–168. 23. Z. Wang; A. C. Bovik; H. R. Sheikh; E. P. Simoncelli. Image quality assessment: from error measurement to structural similarity. IEEE T. Image Process., 13(1), (2004), 600–612. 24. M. Bertalmio; V. Caselles; E. Provenzi; A. Rizzi. Perceptual color correction through variational techniques. IEEE T. Image Process., 16(4), (2007), 1058–1072. 25. Japan Agency for Marine-Earth Science and Technology, http://www.jamstec.go.jp/e/ 26. O. Pele; M. Werman. Improving perceptual color difference using basic color terms. (2012), http://arXiv:1211:5556, 1-15.
30
Highlights: We propose the guided trigonometric filter to refine the coarse depth transmission map. The single de-vignetting method is introduced to remove the footprint. The spectral properties-based color correction method is proposed to recover the scene color.