- Email: [email protected]
Wrapped phase denoising using convolutional neural networks
Optics and Lasers in Engineering 128 (2020) 105999
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Wrapped phase denoising using convolutional neural networks Ketao Yan a, Yingjie Yu a,∗, Tao Sun a, Anand Asundi b, Qian Kemao c a
Department of Precision Mechanical Engineering, Shanghai University, Shanghai 200072, China Managing Director, d’Optron Pte Ltd, Singapore 638075, Singapore c School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798, Singapore b
a r t i c l e
i n f o
Keywords: Wrapped phase Convolutional neural networks Denoising
a b s t r a c t We propose a wrapped phase denoising method based on convolutional neural networks (CNN), which can effectively denoise a noisy wrapped phase. The noisy numerator and denominator of the arctangent function are firstly denoised by CNN, and then the filtered numerator and denominator use the arctangent function to obtain the clean wrapped phase. We experimentally verify the denoising performance using various wrapped phase that contains different noise conditions, where the denoised wrapped phase can achieve a satisfactory unwrapping performance using the existing simple unwrapping method. In addition, the proposed method is further demonstrated through the comparison of the existing methods, and shows an accurate denoising result without adjusting any parameters.
1. Introduction Interferometry provides high measurement accuracy and is widely used for shape, deformation, reflective index, etc. [1]. Most interferogram analysis methods give a wrapped phase distributing between -𝜋 and 𝜋. Phase unwrapping is thus needed to reconstruct the true phase from the wrapped phase. However, noise in a wrapped phase is an obstacle for successful phase unwrapping, especially when a simple phase unwrapping method is used [2,3], and thus denoising is necessary. The effective denoising methods include the sine/cosine average filter (SCAF) [2,3], the modulus 2𝜋 filtering method [4], the windowed Fourier ridges (WFR) algorithm [5,6], the windowed Fourier filtering (WFF) algorithm [5,6], and the extended WFF [7]. However, most of these methods often require a high computational cost and experiences for parameter adjustment. Therefore, it is of interest to develop a new method without these problems. Deep learning [8] as a recent powerful tool has been applied in numerous fields, such as image segmentation [9], image style transfer [10], image generation [11], and classification [12]. In the optics field, deep learning has also been shown promising for fringe pattern denoising [13], wavefront sensing [14], speckle noise reduction [15] and phase aberration compensation [16]. Recently, phase unwrapping using neural networks [17–19] was proposed, among which, in Ref. [19], the wrapped phase was first denoised and then integral multiple was estimated.
∗
In this paper, we propose to train a convolutional neural network to denoise wrapped phase maps. The trained network can significantly reduce the noise in a wrapped phase. Consequently, we can more easily unwrap phase using the existing simple unwrapping algorithm. Our method outperforms the conventional average filter and median filter, and performs similarly to WFF, verifying its excellent performance. The comparison with the existing deep learning phase unwrapping method further demonstrates the performance of our denoising method. The advantages of our work are as follows, (1) High effectiveness: The proposed method can eliminate severe noise in a wrapped phase. In our current network, noise level up to a signal to noise ratio (SNR) value of −4 dB can be successfully removed; (2) Parameter free: The trained network can provide a denoising result without adjusting any parameters. As mentioned earlier, Ref. [19] also uses neural networks to estimate the clean wrapped phase from the noisy wrapped phase. However, the severe noise in sampled data results in many false phase jumps. If these false phase jumps are not handled carefully in the denoising process, the denoised result cannot be correctly unwrapped. Different from Ref. [19], our method denoises the noisy numerator and denominator of the arctangent function to obtain the clean numerator and denominator, and subsequently obtains the clean wrapped phase using the arctangent function. Our method can effectively eliminate the influence of false jumps.
Corresponding author. E-mail address: [email protected] (Y. Yu).
https://doi.org/10.1016/j.optlaseng.2019.105999 Received 12 July 2019; Received in revised form 19 December 2019; Accepted 30 December 2019 0143-8166/© 2019 Elsevier Ltd. All rights reserved.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 1. Simulated wrapped phase analysis: (a) the clean wrapped phase; (b) the true phase; (c) the noisy wrapped phase; (d) the unwrapping phase of (c) using line scanning.
Fig. 2. Schematic diagram of the propose method for denoising: Note that A(x, y) and B(x, y) are separately input to CNN to obtain the corresponding nA (x, y) and nB (x, y).
Table 1 The configuration of CNN.
2. Proposed CNN for denoising wrapped phase 2.1. The problem statement An interference fringe pattern can be mathematically expressed as: 𝐼(𝑥, 𝑦) = 𝐼0 (𝑥, 𝑦) + 𝑟(𝑥, 𝑦) cos [𝜑(𝑥, 𝑦)]
(1)
where (x, y) is the spatial coordinate, I0 (x, y), r(x, y) and 𝜑(x, y) are the background intensity, the amplitude and the desired phase distribution, respectively. The majority of the fringe demodulation techniques calculate the phase as: [ ] 𝜑𝑤 (𝑥, 𝑦) = arctan 𝐴(𝑥, 𝑦)∕𝐵 (𝑥, 𝑦) (2) where A(x, y) and B(x, y) are functions of the recorded intensity. Note that, due to the arctangent function, the obtained phase is wrapped, i.e., 𝜑𝑤 (𝑥, 𝑦) ∈ (−𝜋, 𝜋], with its relationship to 𝜑(x, y) as follows: 𝜑(𝑥, 𝑦) = 𝜑𝑤 (𝑥, 𝑦) + 2𝑘𝜋
(𝑘 ∈ Z)
(3)
As an example, Fig. 1(a) and 1(b) show the wrapped phase and the true phase, respectively. Recovering 𝜑(x, y) from 𝜑w (x, y) is called phase unwrapping. In practical measurement, noise is unavoidable, which significantly affects phase unwrapping. Fig. 1(c) is a simulated noisy wrapped phase, which is unwrapped through line scanning phase unwrapping [20]. Line scanning is vulnerable to noise and often gives unsuccessful unwrapping results, as seen from Fig. 1(d). However, the simplicity of such an unwrapping technique is both theoretically and practically attractive. Thus denoising for a wrapped phase is necessary to make simple unwrapping techniques applicable. 2.2. The proposed denoising method To suppress the noise without distorting the phase distribution, in Fig. 2, the proposed CNN denoising method takes the numerator A(x, y) and the denominator B(x, y) of Eq. (2) as input, one at a time, and outputs the corresponding noise distribution nA (x, y) and nB (x, y), respectively. Then the clean numerator𝐴, (𝑥, 𝑦)and the denominator 𝐵 , (𝑥, 𝑦) can be obtained by subtracting the predicted noise from the input noisy wrapped phase. Finally, the clean wrapped phase can be easily obtained by an arctangent function. As can be seen from Fig. 1(c), the noise leads
Layer
Filter number
Filter size
The feature map
2-D convolution ResNet (1) ResNet (2) ResNet (3) ResNet (4) ResNet (5) ResNet (6) ResNet (7) 2-D convolution
16 16 32 64 128 64 32 16 1
3 3 3 3 3 3 3 3 1
40 40 40 40 40 40 40 40 40
× × × × × × × × ×
3 3 3 3 3 3 3 3 1
× × × × × × × × ×
40 40 40 40 40 40 40 40 40
× × × × × × × × ×
16 16 32 64 128 64 32 16 1
to many false phase jumps, which can be successfully removed after the wrapped phase is successfully denoised. 2.3. The proposed CNN architecture The denoising architecture of the proposed CNN is illustrated in Fig. 3, which is designed based on the residual network (ResNet) [21] architecture. The network architecture consists of three parts: input part, nonlinear mapping part, and output part. Input part is the noisy image of size 40 × 40 pixels, and output part is estimated noise. Fig. 3(a) shows the nonlinear mapping part, consisting of a convolutional layer of 16 filters of size 3 × 3, 7 ResNets, and a convolutional layer of 1 filter of size 1 × 1. Fig. 3(b) shows the 4 residual blocks of each ResNet with 3 × 3 kernels. Meanwhile, rectified linear unit (ReLU) [22] activation function and batch normalization (BN) [23] are used for nonlinear mapping and to speed up training, respectively. At the last layer of Fig. 3(a), the predicted noise is extracted by a 1 × 1 convolution layer. The number of kernels is first increased to extract more features and then is reduced to get a single channel of output result. The details of the configuration are illustrated in Table 1. The loss function for training the network is defined as: ℜ(𝑥, 𝑦) = 𝜛 −1 ×
𝜛 ∑ 𝑖=1
‖𝜍𝑖 − 𝜏𝑖 ‖2 ‖ ‖
(4)
where 𝜍 i is the output of CNN, 𝜏 i is ground truth, ϖ is the minibatch number of input data.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 3. The architecture of the proposed CNN: (a) the architecture of CNN; (b) the architecture of ResNet.
Fig. 4. Zernike polynomials.
The Adam [24] (Adaptive Moment Estimation) algorithm is adopted to optimize the parameters and minimize the loss function. The initial learning rate is set to a large value of 0.001 and automatically multiplied by 0.7 times after every 5 epochs. The training is completed after 50 epochs, which takes approximately 8 h on a PC with Intel(R) Core(TM) i7-7820X [email protected] GHz × 13. The entire training process is implemented using the framework of Pytorch and accelerated computation with the GeForce GTX 1080 (NVIDIA).
] 4 × 𝜋 × 𝐻(𝑥, 𝑦) + 𝑛1 (𝑥, 𝑦) 𝜆
[ 𝐵(𝑥, 𝑦) = 𝑏 × cos
] 4 × 𝜋 × 𝐻(𝑥, 𝑦) + 𝑛2 (𝑥, 𝑦) 𝜆
𝑁 ∑ 𝑛=1
𝓁𝑛 𝜂𝑛 (𝑥𝑘 , 𝑦𝑘 )
(7)
(5)
(6)
3. Experiments and analysis
A training dataset is required to train the network. Following Eq. (2), the numerator and denominator are specified as: [
𝐻(𝑥, 𝑦) =
where N is the number of polynomial terms, 𝜂 n (xk , yk ) and 𝓁 n represent the Zernike polynomials and coefficients, respectively. As shown in Fig. 4, we randomly take the values of the aberration coefficients which are multiplied by piston, tilt, focus, and astigmatism to represent the actual wavefronts. In our work, 44,000 groups of input data with the size of 40 × 40 pixels are simulated using MATLAB according to Eqs. (5) and (6). n1 (x, y) and n2 (x, y) are also simulated as the corresponding ground truth.
2.4. Training datasets preparation
𝐴(𝑥, 𝑦) = 𝑏 × sin
Furthermore, the wavefront H(x, y) is represented by a Zernike polynomial which is often used to describe the aberrations in optical tests [25]:
where b is the amplitude taking a constant value in the range of (50– 120); H(x, y) is the actual wavefront; 𝜆is the wavelength (632.8 nm); n1 (x, y) and n2 (x, y) are the Gaussian additive noise with a zero mean (𝜇 = 0) and different standard deviations (𝜎 = 0–60).
Various experiments and comparisons are carried out to verify the performance of the proposed method. The denoising result of CNN is unwrapped using line scanning to demonstrate that we can use a simple method to unwrap phase after CNN denoising.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 5. Experiment results: rows (a)-(d): simulated data with different noise levels (SNR = 4, 2, 0, and −2); columns I–V: the noisy wrapped phase, the unwrapping result of I using line scanning, the denoising result using CNN, the unwrapping result of III using line scanning (CNN-line scanning), the true phase.
Fig. 6. The detail comparison of 100th row from columns IV and V in Fig. 5:(a), (b), (c), and (d) is the corresponding rows (a), (b), (c), and (d) in Fig. 5, respectively.
3.1. Denoising performance on simulated phase 3.1.1. Performance on different phase distributions Wrapped phase data with the size of 200 × 200 pixels and different noise are simulated and denoised, as shown in column I of Fig. 5. Line scanning phase unwrapping is not successful due to the noise (column II). However, if the noisy wrapped phase is denoised by CNN (column III), line scanning becomes successful (column IV). The true phase is shown in column V for visual comparison. Fig. 6 shows the quantitative comparison of the 100th row from columns IV and V in Fig. 5, where
perfect agreement is observed. The root mean square errors (RMSEs) of CNN-line scanning in 100th row are shown in Table 2. The calculation time approximately takes 4.2 s for denoising a 200 × 200 numerator or denominator. Therefore, the whole calculation time for denoising a 200 × 200 wrapped phase approximately is 8.4 s. 3.1.2. Performance on different phase frequencies Peaks functions with different frequencies are simulated and analyzed, and Fig. 7(a) shows the noisy peaks functions of size 256 × 256 at a fixed SNR value of 5 dB. The proposed method yields effective results
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 7. Different phase ranges analysis using the proposed method: Row (a) is noisy wrapped phase; Row (b) is denoising result; Row (c) is the unwrapping result of (b) (CNN-line scanning).
Fig. 8. Error curve under different phase frequencies (from peaks × 1 to peaks × 11) using CNN-line scanning.
Table 2 RMSEs of CNN-line scanning in 100th row, CNN-line scanning, and WFF-line scanning. Method
a
b
c
d
CNN-line scanning (the 100th row) CNN-line scanning WFF-line scanning
0.0429 0.0603 0.0599
0.0845 0.0707 0.0717
0.0714 0.0760 0.0927
0.0913 0.0911 0.1048
as shown in Fig. 7(b), which can be easily wrapped as shown in Fig. 7(c). Fig. 8 gives the RMSEs of the peaks function under different multiples (from peaks × 1 to peaks × 11), where the PV values of peaks function range from 14.66 to 161.21 rad. Although the RMSE increases with the phase frequency, it is less than 0.055 rad, quantitatively demonstrating the effectiveness of the CNN-line scanning.
reproduced in Fig. 9. Denoising using AF appears to blur images, as expected (column I). MF also does not perform satisfactorily (column II). As a result, the unwrapping by line scanning is unfortunately not successful. The data in column I of Fig. 5 is further processed by WFF to verify the accuracy of the proposed method, and the result is given in Fig. 10. WFF is effective in denoising a noisy wrapped phase (column I), and obtains a smooth unwrapping phase using line scanning (column II). The error maps of WFF-line scanning and CNN-line scanning are shown in columns III and IV, and both methods clearly provide lower error distribution. To objectively show the performance of CNN, RMSEs of WFF-line scanning and CNN-line scanning are further analyzed and provided in Table 2, where both methods have similar RMSE values. The advantage of CNN is that no parameters should be tuned after training 3.3. Comparison with deep learning phase unwrapping
3.2. Comparison with classical filters The conventional average filter (AF) and median filter (MF) are attempted for the wrapped phase maps in column I of Fig. 5, which are
The one-step phase unwrapping method using deep learning is realized according to the information provided in Ref. [18] and then compared with CNN-line scanning. Ten wrapped phase maps are simulated
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 9. Experiment results: Column I is the denoising result using AF under window size 3 × 3 and 5 × 5; Column II is the denoising result using MF under window size 3 × 3 and 5 × 5.
Fig. 10. Experiment results: Column I is the denoising result using WFF; Column II is the unwrapping result of I using line scanning (WFFline scanning); Column III is error map of WFFline scanning; Column IV is error map of CNNline scanning.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 11. Simulated data at an SNR value of 0 dB. Fig. 12. RMSE comparison between CNN-line scanning and one-step phase unwrapping.
with the SNR value ranging from 10 dB to −4 dB, among which, those with an SNR value of 0 dB are shown in Fig. 11. Both CNN-line scanning and one-step phase unwrapping can effectively perform phase unwrapping. The average RMSEs of these two methods are shown in Fig. 12. The better performance of our method is probably because our method adopts a two-step strategy, i.e., denoising and unwrapping, and thus the denoising is specially attended, while one-step phase unwrapping performs the unwrapping in just one step. We emphasize that, although we have followed all the implementation details in Ref. [18], the above results from the one-step phase unwrapping method may not be optimal. In addition, with noise suppression before and/or after their process, we believe that their performance could also be further improved. Thus, further investigation will be carried on as future work studied in our research progress. Nevertheless, the comparison results clearly reveal the effectiveness of our proposed CNN-line scanning.
3.4. Denoising performance in real experiment A real wrapped phase with a size of 500 × 260 pixels is also processed. The original wrapped phase is shown in Fig. 13(a), which is imperfectly unwrapped by line scanning as shown in Fig. 13(b). Such problem is often encountered in real measurements. The denoising result of CNN is shown in Fig. 13(c), which is successful unwrapped by line scanning as shown in Fig. 13(d). For comparison, Fig. 13(e) and 13(f) show the WFF denoising and line scanning phase unwrapping result, respectively. The quantitative comparison of the unwrapping results (CNN-line scanning, WFF-line scanning) from the 200th column and 100th row are shown in Fig. 14(a) and 14(b), respectively, where two methods produce very similar results. It can be concluded the proposed method has an effective denoising performance in the real data.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 13. Experiment results: (a) the real wrapped phase; (b) the unwrapped phase of (a) using line scanning; (c) the denoising result using CNN; (d) the unwrapped result of (c) using line scanning (CNN-line scanning); (e) the denoising result using WFF; (f) the unwrapped result of (e) using line scanning (WFF-line scanning).
Fig. 14. The detailed comparison of CNN-line scanning and WFF-line scanning: (a) the 200th column pixels; (b) the 100th row pixels.
4. Conclusions In this paper, wrapped phase denoising using CNN is proposed, where the noisy numerator and denominator of the arctangent function are firstly denoised by CNN. From the filtered numerator and denominator, the clean wrapped phase is computed. The proposed method is shown to be effective through extensive comparisons with the average filter, median filter and windowed Fourier filtering. With the effective denoising, a wrapped phase can be easily unwrapped using an existing
simple unwrapping method. The feasibility of phase unwrapping strategy is demonstrated by comparing with an existing deep learning unwrapping method. In addition to the effectiveness, the proposed CNN denoising method is able to directly provide denoising result without adjusting any parameters. Declaration of Competing Interest The authors declare that they have no competing interests
K. Yan, Y. Yu and T. Sun et al.
CRediT authorship contribution statement Ketao Yan: Data curation, Writing - original draft, Conceptualization, Methodology. Yingjie Yu: Visualization, Investigation. Tao Sun: Software, Validation. Anand Asundi: Supervision. Qian Kemao: Writing - review & editing. Acknowledgments This work was supported by the National Natural Science Foundation of China (NSFC)(51775326). The authors would like to thank Dr. Jianglei Di from Northwestern Polytechnical University for the helping of this paper. The authors also would like to thank Dr. Paraskvi Tornari from Institute of Electronic Structure and Laser, Foundation for Research and Technology-Hellas for the helping of this paper. References [1] Wyant JC. Dynamic interferometry. Opt Photon News 2003;14(4):36–41. doi:10.1364/OPN.14.4.000036. [2] Huntley JM. Random phase measurement errors in digital speckle pattern interferometry. Opt Lasers Eng 1997;26(2–3):131–50. doi:10.1016/0143 -8166(95)00109-3. [3] Aebischer HA, Waldner S. A simple and effective method for filtering speckleinterferometric phase fringe patterns. Opt Commun 1999;162(4–6):205–10. doi:10.1016/S0030-4018(99)00116-9. [4] Medina OM, Estrada JC, Lópezc YY. Filtering optical wrapped phase images algorithm. Opt Eng 2017;56(11):111704. doi:10.1117/1.OE.56.11.111704. [5] Kemao Q. Two-dimensional windowed Fourier transform for fringe pattern analysis: principles, applications and implementations. Opt Lasers Eng 2007;45(2):304–17. doi:10.1016/j.optlaseng.2005.10.012. [6] Kemao Q, Wang H, Gao W. Windowed Fourier transform for fringe pattern analysis: theoretical analyses. Appl Opt 2008;47(29):5408–19 10.1364/AO.47.005408. [7] Yatabe K, Oikawa Y. Convex optimization-based windowed Fourier filtering with multiple windows for wrapped-phase denoising. Appl Opt 2016;55(17):4632–41. doi:10.1364/AO.55.004632. [8] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436–44. doi:10.1038/nature14539. [9] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention; 2015. p. 234–41.
Optics and Lasers in Engineering 128 (2020) 105999 [10] Gatys LA, Ecker AS, Bethge M. Image style transfer using convolutional neural networks. In: IEEE conference on computer vision and pattern recognition; 2016. p. 2414–23. [11] Dosovitskiy A, Springenberg JT, Tatarchenko M, Brox T. Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans Pattern Anal Mach Intell 2017;39(4):692–705. doi:10.1109/TPAMI.2016.2567384. [12] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Neural information processing systems; 2012. p. 1097–105. [13] Yan K, Yu Y, Huang C, Sui L, Qian K, Asundi A. Fringe pattern denoising based on deep learning. Opt Commun 2019;437:148–52. doi:10.1016/j.optcom. 2018.12.058. [14] Nishizaki Y, Valdivia M, Horisaki R, Kitaguchi K, Saito M, Tanida J, Vera E. Deep learning wavefront sensing. Opt Express 2019;27(1):240–51 10.1364/OE.27.000240. [15] Jeon W, Jeong W, Son K, Yang H. Speckle noise reduction for digital holographic images using multi-scale convolutional neural networks. Opt Lett 2018;43(17):4240–3. doi:10.1364/OL.43.004240. [16] Nguyen T, Bui V, Lam V, Raub CB, Chang LC, Nehmetallah G. Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection. Opt Express 2017;25(13):15043–57. doi:10.1364/OE.25.015043. [17] Spoorthi GE, Gorthi S, Gorthi RKSS. PhaseNet: a deep convolutional neural network for two-dimensional phase unwrapping. IEEE Signal Process Lett. 2019;26(1):54–8. doi:10.1109/LSP.2018.2879184. [18] Wang K, Li Y, Kemao Q, Di J, Zhao J. One-step robust deep learning phase unwrapping. Opt Express 2019;27(10):15100–15 10.1364/OE.27.015100. [19] Zhang J, Tian X, Shao J, Luo H, Liang R. Phase unwrapping in optical metrology via denoised and convolutional segmentation networks. Opt Express 2019;27(10):14903–12. doi:10.1364/OE.27.014903. [20] Itoh K. Analysis of the phase unwrapping algorithm. Appl Opt 1982;21(14) 2470– 2470. doi:10.1364/AO.21.002470. [21] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition; 2016. p. 770–8. [22] Nair V, Hinton G. Rectified linear units improve restricted boltzmann machines. In: 27th international conference on machine learning; 2010. p. 807–14. [23] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: The 32nd international conference on machine learning; 2015. p. 448–56. [24] Kingma D, Ba J. Adam: a method for stochastic optimization. 3rd international conference on learning representations; 2015. [25] Wyant JC, Creath K. Basic wavefront aberration theory for optical metrology. In: Chapter 1 in appl opt opt eng; 1992. p. 28–9.
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Wrapped phase denoising using convolutional neural networks Ketao Yan a, Yingjie Yu a,∗, Tao Sun a, Anand Asundi b, Qian Kemao c a
Department of Precision Mechanical Engineering, Shanghai University, Shanghai 200072, China Managing Director, d’Optron Pte Ltd, Singapore 638075, Singapore c School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798, Singapore b
a r t i c l e
i n f o
Keywords: Wrapped phase Convolutional neural networks Denoising
a b s t r a c t We propose a wrapped phase denoising method based on convolutional neural networks (CNN), which can effectively denoise a noisy wrapped phase. The noisy numerator and denominator of the arctangent function are firstly denoised by CNN, and then the filtered numerator and denominator use the arctangent function to obtain the clean wrapped phase. We experimentally verify the denoising performance using various wrapped phase that contains different noise conditions, where the denoised wrapped phase can achieve a satisfactory unwrapping performance using the existing simple unwrapping method. In addition, the proposed method is further demonstrated through the comparison of the existing methods, and shows an accurate denoising result without adjusting any parameters.
1. Introduction Interferometry provides high measurement accuracy and is widely used for shape, deformation, reflective index, etc. [1]. Most interferogram analysis methods give a wrapped phase distributing between -𝜋 and 𝜋. Phase unwrapping is thus needed to reconstruct the true phase from the wrapped phase. However, noise in a wrapped phase is an obstacle for successful phase unwrapping, especially when a simple phase unwrapping method is used [2,3], and thus denoising is necessary. The effective denoising methods include the sine/cosine average filter (SCAF) [2,3], the modulus 2𝜋 filtering method [4], the windowed Fourier ridges (WFR) algorithm [5,6], the windowed Fourier filtering (WFF) algorithm [5,6], and the extended WFF [7]. However, most of these methods often require a high computational cost and experiences for parameter adjustment. Therefore, it is of interest to develop a new method without these problems. Deep learning [8] as a recent powerful tool has been applied in numerous fields, such as image segmentation [9], image style transfer [10], image generation [11], and classification [12]. In the optics field, deep learning has also been shown promising for fringe pattern denoising [13], wavefront sensing [14], speckle noise reduction [15] and phase aberration compensation [16]. Recently, phase unwrapping using neural networks [17–19] was proposed, among which, in Ref. [19], the wrapped phase was first denoised and then integral multiple was estimated.
∗
In this paper, we propose to train a convolutional neural network to denoise wrapped phase maps. The trained network can significantly reduce the noise in a wrapped phase. Consequently, we can more easily unwrap phase using the existing simple unwrapping algorithm. Our method outperforms the conventional average filter and median filter, and performs similarly to WFF, verifying its excellent performance. The comparison with the existing deep learning phase unwrapping method further demonstrates the performance of our denoising method. The advantages of our work are as follows, (1) High effectiveness: The proposed method can eliminate severe noise in a wrapped phase. In our current network, noise level up to a signal to noise ratio (SNR) value of −4 dB can be successfully removed; (2) Parameter free: The trained network can provide a denoising result without adjusting any parameters. As mentioned earlier, Ref. [19] also uses neural networks to estimate the clean wrapped phase from the noisy wrapped phase. However, the severe noise in sampled data results in many false phase jumps. If these false phase jumps are not handled carefully in the denoising process, the denoised result cannot be correctly unwrapped. Different from Ref. [19], our method denoises the noisy numerator and denominator of the arctangent function to obtain the clean numerator and denominator, and subsequently obtains the clean wrapped phase using the arctangent function. Our method can effectively eliminate the influence of false jumps.
Corresponding author. E-mail address: [email protected] (Y. Yu).
https://doi.org/10.1016/j.optlaseng.2019.105999 Received 12 July 2019; Received in revised form 19 December 2019; Accepted 30 December 2019 0143-8166/© 2019 Elsevier Ltd. All rights reserved.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 1. Simulated wrapped phase analysis: (a) the clean wrapped phase; (b) the true phase; (c) the noisy wrapped phase; (d) the unwrapping phase of (c) using line scanning.
Fig. 2. Schematic diagram of the propose method for denoising: Note that A(x, y) and B(x, y) are separately input to CNN to obtain the corresponding nA (x, y) and nB (x, y).
Table 1 The configuration of CNN.
2. Proposed CNN for denoising wrapped phase 2.1. The problem statement An interference fringe pattern can be mathematically expressed as: 𝐼(𝑥, 𝑦) = 𝐼0 (𝑥, 𝑦) + 𝑟(𝑥, 𝑦) cos [𝜑(𝑥, 𝑦)]
(1)
where (x, y) is the spatial coordinate, I0 (x, y), r(x, y) and 𝜑(x, y) are the background intensity, the amplitude and the desired phase distribution, respectively. The majority of the fringe demodulation techniques calculate the phase as: [ ] 𝜑𝑤 (𝑥, 𝑦) = arctan 𝐴(𝑥, 𝑦)∕𝐵 (𝑥, 𝑦) (2) where A(x, y) and B(x, y) are functions of the recorded intensity. Note that, due to the arctangent function, the obtained phase is wrapped, i.e., 𝜑𝑤 (𝑥, 𝑦) ∈ (−𝜋, 𝜋], with its relationship to 𝜑(x, y) as follows: 𝜑(𝑥, 𝑦) = 𝜑𝑤 (𝑥, 𝑦) + 2𝑘𝜋
(𝑘 ∈ Z)
(3)
As an example, Fig. 1(a) and 1(b) show the wrapped phase and the true phase, respectively. Recovering 𝜑(x, y) from 𝜑w (x, y) is called phase unwrapping. In practical measurement, noise is unavoidable, which significantly affects phase unwrapping. Fig. 1(c) is a simulated noisy wrapped phase, which is unwrapped through line scanning phase unwrapping [20]. Line scanning is vulnerable to noise and often gives unsuccessful unwrapping results, as seen from Fig. 1(d). However, the simplicity of such an unwrapping technique is both theoretically and practically attractive. Thus denoising for a wrapped phase is necessary to make simple unwrapping techniques applicable. 2.2. The proposed denoising method To suppress the noise without distorting the phase distribution, in Fig. 2, the proposed CNN denoising method takes the numerator A(x, y) and the denominator B(x, y) of Eq. (2) as input, one at a time, and outputs the corresponding noise distribution nA (x, y) and nB (x, y), respectively. Then the clean numerator𝐴, (𝑥, 𝑦)and the denominator 𝐵 , (𝑥, 𝑦) can be obtained by subtracting the predicted noise from the input noisy wrapped phase. Finally, the clean wrapped phase can be easily obtained by an arctangent function. As can be seen from Fig. 1(c), the noise leads
Layer
Filter number
Filter size
The feature map
2-D convolution ResNet (1) ResNet (2) ResNet (3) ResNet (4) ResNet (5) ResNet (6) ResNet (7) 2-D convolution
16 16 32 64 128 64 32 16 1
3 3 3 3 3 3 3 3 1
40 40 40 40 40 40 40 40 40
× × × × × × × × ×
3 3 3 3 3 3 3 3 1
× × × × × × × × ×
40 40 40 40 40 40 40 40 40
× × × × × × × × ×
16 16 32 64 128 64 32 16 1
to many false phase jumps, which can be successfully removed after the wrapped phase is successfully denoised. 2.3. The proposed CNN architecture The denoising architecture of the proposed CNN is illustrated in Fig. 3, which is designed based on the residual network (ResNet) [21] architecture. The network architecture consists of three parts: input part, nonlinear mapping part, and output part. Input part is the noisy image of size 40 × 40 pixels, and output part is estimated noise. Fig. 3(a) shows the nonlinear mapping part, consisting of a convolutional layer of 16 filters of size 3 × 3, 7 ResNets, and a convolutional layer of 1 filter of size 1 × 1. Fig. 3(b) shows the 4 residual blocks of each ResNet with 3 × 3 kernels. Meanwhile, rectified linear unit (ReLU) [22] activation function and batch normalization (BN) [23] are used for nonlinear mapping and to speed up training, respectively. At the last layer of Fig. 3(a), the predicted noise is extracted by a 1 × 1 convolution layer. The number of kernels is first increased to extract more features and then is reduced to get a single channel of output result. The details of the configuration are illustrated in Table 1. The loss function for training the network is defined as: ℜ(𝑥, 𝑦) = 𝜛 −1 ×
𝜛 ∑ 𝑖=1
‖𝜍𝑖 − 𝜏𝑖 ‖2 ‖ ‖
(4)
where 𝜍 i is the output of CNN, 𝜏 i is ground truth, ϖ is the minibatch number of input data.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 3. The architecture of the proposed CNN: (a) the architecture of CNN; (b) the architecture of ResNet.
Fig. 4. Zernike polynomials.
The Adam [24] (Adaptive Moment Estimation) algorithm is adopted to optimize the parameters and minimize the loss function. The initial learning rate is set to a large value of 0.001 and automatically multiplied by 0.7 times after every 5 epochs. The training is completed after 50 epochs, which takes approximately 8 h on a PC with Intel(R) Core(TM) i7-7820X [email protected] GHz × 13. The entire training process is implemented using the framework of Pytorch and accelerated computation with the GeForce GTX 1080 (NVIDIA).
] 4 × 𝜋 × 𝐻(𝑥, 𝑦) + 𝑛1 (𝑥, 𝑦) 𝜆
[ 𝐵(𝑥, 𝑦) = 𝑏 × cos
] 4 × 𝜋 × 𝐻(𝑥, 𝑦) + 𝑛2 (𝑥, 𝑦) 𝜆
𝑁 ∑ 𝑛=1
𝓁𝑛 𝜂𝑛 (𝑥𝑘 , 𝑦𝑘 )
(7)
(5)
(6)
3. Experiments and analysis
A training dataset is required to train the network. Following Eq. (2), the numerator and denominator are specified as: [
𝐻(𝑥, 𝑦) =
where N is the number of polynomial terms, 𝜂 n (xk , yk ) and 𝓁 n represent the Zernike polynomials and coefficients, respectively. As shown in Fig. 4, we randomly take the values of the aberration coefficients which are multiplied by piston, tilt, focus, and astigmatism to represent the actual wavefronts. In our work, 44,000 groups of input data with the size of 40 × 40 pixels are simulated using MATLAB according to Eqs. (5) and (6). n1 (x, y) and n2 (x, y) are also simulated as the corresponding ground truth.
2.4. Training datasets preparation
𝐴(𝑥, 𝑦) = 𝑏 × sin
Furthermore, the wavefront H(x, y) is represented by a Zernike polynomial which is often used to describe the aberrations in optical tests [25]:
where b is the amplitude taking a constant value in the range of (50– 120); H(x, y) is the actual wavefront; 𝜆is the wavelength (632.8 nm); n1 (x, y) and n2 (x, y) are the Gaussian additive noise with a zero mean (𝜇 = 0) and different standard deviations (𝜎 = 0–60).
Various experiments and comparisons are carried out to verify the performance of the proposed method. The denoising result of CNN is unwrapped using line scanning to demonstrate that we can use a simple method to unwrap phase after CNN denoising.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 5. Experiment results: rows (a)-(d): simulated data with different noise levels (SNR = 4, 2, 0, and −2); columns I–V: the noisy wrapped phase, the unwrapping result of I using line scanning, the denoising result using CNN, the unwrapping result of III using line scanning (CNN-line scanning), the true phase.
Fig. 6. The detail comparison of 100th row from columns IV and V in Fig. 5:(a), (b), (c), and (d) is the corresponding rows (a), (b), (c), and (d) in Fig. 5, respectively.
3.1. Denoising performance on simulated phase 3.1.1. Performance on different phase distributions Wrapped phase data with the size of 200 × 200 pixels and different noise are simulated and denoised, as shown in column I of Fig. 5. Line scanning phase unwrapping is not successful due to the noise (column II). However, if the noisy wrapped phase is denoised by CNN (column III), line scanning becomes successful (column IV). The true phase is shown in column V for visual comparison. Fig. 6 shows the quantitative comparison of the 100th row from columns IV and V in Fig. 5, where
perfect agreement is observed. The root mean square errors (RMSEs) of CNN-line scanning in 100th row are shown in Table 2. The calculation time approximately takes 4.2 s for denoising a 200 × 200 numerator or denominator. Therefore, the whole calculation time for denoising a 200 × 200 wrapped phase approximately is 8.4 s. 3.1.2. Performance on different phase frequencies Peaks functions with different frequencies are simulated and analyzed, and Fig. 7(a) shows the noisy peaks functions of size 256 × 256 at a fixed SNR value of 5 dB. The proposed method yields effective results
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 7. Different phase ranges analysis using the proposed method: Row (a) is noisy wrapped phase; Row (b) is denoising result; Row (c) is the unwrapping result of (b) (CNN-line scanning).
Fig. 8. Error curve under different phase frequencies (from peaks × 1 to peaks × 11) using CNN-line scanning.
Table 2 RMSEs of CNN-line scanning in 100th row, CNN-line scanning, and WFF-line scanning. Method
a
b
c
d
CNN-line scanning (the 100th row) CNN-line scanning WFF-line scanning
0.0429 0.0603 0.0599
0.0845 0.0707 0.0717
0.0714 0.0760 0.0927
0.0913 0.0911 0.1048
as shown in Fig. 7(b), which can be easily wrapped as shown in Fig. 7(c). Fig. 8 gives the RMSEs of the peaks function under different multiples (from peaks × 1 to peaks × 11), where the PV values of peaks function range from 14.66 to 161.21 rad. Although the RMSE increases with the phase frequency, it is less than 0.055 rad, quantitatively demonstrating the effectiveness of the CNN-line scanning.
reproduced in Fig. 9. Denoising using AF appears to blur images, as expected (column I). MF also does not perform satisfactorily (column II). As a result, the unwrapping by line scanning is unfortunately not successful. The data in column I of Fig. 5 is further processed by WFF to verify the accuracy of the proposed method, and the result is given in Fig. 10. WFF is effective in denoising a noisy wrapped phase (column I), and obtains a smooth unwrapping phase using line scanning (column II). The error maps of WFF-line scanning and CNN-line scanning are shown in columns III and IV, and both methods clearly provide lower error distribution. To objectively show the performance of CNN, RMSEs of WFF-line scanning and CNN-line scanning are further analyzed and provided in Table 2, where both methods have similar RMSE values. The advantage of CNN is that no parameters should be tuned after training 3.3. Comparison with deep learning phase unwrapping
3.2. Comparison with classical filters The conventional average filter (AF) and median filter (MF) are attempted for the wrapped phase maps in column I of Fig. 5, which are
The one-step phase unwrapping method using deep learning is realized according to the information provided in Ref. [18] and then compared with CNN-line scanning. Ten wrapped phase maps are simulated
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 9. Experiment results: Column I is the denoising result using AF under window size 3 × 3 and 5 × 5; Column II is the denoising result using MF under window size 3 × 3 and 5 × 5.
Fig. 10. Experiment results: Column I is the denoising result using WFF; Column II is the unwrapping result of I using line scanning (WFFline scanning); Column III is error map of WFFline scanning; Column IV is error map of CNNline scanning.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 11. Simulated data at an SNR value of 0 dB. Fig. 12. RMSE comparison between CNN-line scanning and one-step phase unwrapping.
with the SNR value ranging from 10 dB to −4 dB, among which, those with an SNR value of 0 dB are shown in Fig. 11. Both CNN-line scanning and one-step phase unwrapping can effectively perform phase unwrapping. The average RMSEs of these two methods are shown in Fig. 12. The better performance of our method is probably because our method adopts a two-step strategy, i.e., denoising and unwrapping, and thus the denoising is specially attended, while one-step phase unwrapping performs the unwrapping in just one step. We emphasize that, although we have followed all the implementation details in Ref. [18], the above results from the one-step phase unwrapping method may not be optimal. In addition, with noise suppression before and/or after their process, we believe that their performance could also be further improved. Thus, further investigation will be carried on as future work studied in our research progress. Nevertheless, the comparison results clearly reveal the effectiveness of our proposed CNN-line scanning.
3.4. Denoising performance in real experiment A real wrapped phase with a size of 500 × 260 pixels is also processed. The original wrapped phase is shown in Fig. 13(a), which is imperfectly unwrapped by line scanning as shown in Fig. 13(b). Such problem is often encountered in real measurements. The denoising result of CNN is shown in Fig. 13(c), which is successful unwrapped by line scanning as shown in Fig. 13(d). For comparison, Fig. 13(e) and 13(f) show the WFF denoising and line scanning phase unwrapping result, respectively. The quantitative comparison of the unwrapping results (CNN-line scanning, WFF-line scanning) from the 200th column and 100th row are shown in Fig. 14(a) and 14(b), respectively, where two methods produce very similar results. It can be concluded the proposed method has an effective denoising performance in the real data.
K. Yan, Y. Yu and T. Sun et al.
Optics and Lasers in Engineering 128 (2020) 105999
Fig. 13. Experiment results: (a) the real wrapped phase; (b) the unwrapped phase of (a) using line scanning; (c) the denoising result using CNN; (d) the unwrapped result of (c) using line scanning (CNN-line scanning); (e) the denoising result using WFF; (f) the unwrapped result of (e) using line scanning (WFF-line scanning).
Fig. 14. The detailed comparison of CNN-line scanning and WFF-line scanning: (a) the 200th column pixels; (b) the 100th row pixels.
4. Conclusions In this paper, wrapped phase denoising using CNN is proposed, where the noisy numerator and denominator of the arctangent function are firstly denoised by CNN. From the filtered numerator and denominator, the clean wrapped phase is computed. The proposed method is shown to be effective through extensive comparisons with the average filter, median filter and windowed Fourier filtering. With the effective denoising, a wrapped phase can be easily unwrapped using an existing
simple unwrapping method. The feasibility of phase unwrapping strategy is demonstrated by comparing with an existing deep learning unwrapping method. In addition to the effectiveness, the proposed CNN denoising method is able to directly provide denoising result without adjusting any parameters. Declaration of Competing Interest The authors declare that they have no competing interests
K. Yan, Y. Yu and T. Sun et al.
CRediT authorship contribution statement Ketao Yan: Data curation, Writing - original draft, Conceptualization, Methodology. Yingjie Yu: Visualization, Investigation. Tao Sun: Software, Validation. Anand Asundi: Supervision. Qian Kemao: Writing - review & editing. Acknowledgments This work was supported by the National Natural Science Foundation of China (NSFC)(51775326). The authors would like to thank Dr. Jianglei Di from Northwestern Polytechnical University for the helping of this paper. The authors also would like to thank Dr. Paraskvi Tornari from Institute of Electronic Structure and Laser, Foundation for Research and Technology-Hellas for the helping of this paper. References [1] Wyant JC. Dynamic interferometry. Opt Photon News 2003;14(4):36–41. doi:10.1364/OPN.14.4.000036. [2] Huntley JM. Random phase measurement errors in digital speckle pattern interferometry. Opt Lasers Eng 1997;26(2–3):131–50. doi:10.1016/0143 -8166(95)00109-3. [3] Aebischer HA, Waldner S. A simple and effective method for filtering speckleinterferometric phase fringe patterns. Opt Commun 1999;162(4–6):205–10. doi:10.1016/S0030-4018(99)00116-9. [4] Medina OM, Estrada JC, Lópezc YY. Filtering optical wrapped phase images algorithm. Opt Eng 2017;56(11):111704. doi:10.1117/1.OE.56.11.111704. [5] Kemao Q. Two-dimensional windowed Fourier transform for fringe pattern analysis: principles, applications and implementations. Opt Lasers Eng 2007;45(2):304–17. doi:10.1016/j.optlaseng.2005.10.012. [6] Kemao Q, Wang H, Gao W. Windowed Fourier transform for fringe pattern analysis: theoretical analyses. Appl Opt 2008;47(29):5408–19 10.1364/AO.47.005408. [7] Yatabe K, Oikawa Y. Convex optimization-based windowed Fourier filtering with multiple windows for wrapped-phase denoising. Appl Opt 2016;55(17):4632–41. doi:10.1364/AO.55.004632. [8] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436–44. doi:10.1038/nature14539. [9] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention; 2015. p. 234–41.
Optics and Lasers in Engineering 128 (2020) 105999 [10] Gatys LA, Ecker AS, Bethge M. Image style transfer using convolutional neural networks. In: IEEE conference on computer vision and pattern recognition; 2016. p. 2414–23. [11] Dosovitskiy A, Springenberg JT, Tatarchenko M, Brox T. Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans Pattern Anal Mach Intell 2017;39(4):692–705. doi:10.1109/TPAMI.2016.2567384. [12] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Neural information processing systems; 2012. p. 1097–105. [13] Yan K, Yu Y, Huang C, Sui L, Qian K, Asundi A. Fringe pattern denoising based on deep learning. Opt Commun 2019;437:148–52. doi:10.1016/j.optcom. 2018.12.058. [14] Nishizaki Y, Valdivia M, Horisaki R, Kitaguchi K, Saito M, Tanida J, Vera E. Deep learning wavefront sensing. Opt Express 2019;27(1):240–51 10.1364/OE.27.000240. [15] Jeon W, Jeong W, Son K, Yang H. Speckle noise reduction for digital holographic images using multi-scale convolutional neural networks. Opt Lett 2018;43(17):4240–3. doi:10.1364/OL.43.004240. [16] Nguyen T, Bui V, Lam V, Raub CB, Chang LC, Nehmetallah G. Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection. Opt Express 2017;25(13):15043–57. doi:10.1364/OE.25.015043. [17] Spoorthi GE, Gorthi S, Gorthi RKSS. PhaseNet: a deep convolutional neural network for two-dimensional phase unwrapping. IEEE Signal Process Lett. 2019;26(1):54–8. doi:10.1109/LSP.2018.2879184. [18] Wang K, Li Y, Kemao Q, Di J, Zhao J. One-step robust deep learning phase unwrapping. Opt Express 2019;27(10):15100–15 10.1364/OE.27.015100. [19] Zhang J, Tian X, Shao J, Luo H, Liang R. Phase unwrapping in optical metrology via denoised and convolutional segmentation networks. Opt Express 2019;27(10):14903–12. doi:10.1364/OE.27.014903. [20] Itoh K. Analysis of the phase unwrapping algorithm. Appl Opt 1982;21(14) 2470– 2470. doi:10.1364/AO.21.002470. [21] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition; 2016. p. 770–8. [22] Nair V, Hinton G. Rectified linear units improve restricted boltzmann machines. In: 27th international conference on machine learning; 2010. p. 807–14. [23] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: The 32nd international conference on machine learning; 2015. p. 448–56. [24] Kingma D, Ba J. Adam: a method for stochastic optimization. 3rd international conference on learning representations; 2015. [25] Wyant JC, Creath K. Basic wavefront aberration theory for optical metrology. In: Chapter 1 in appl opt opt eng; 1992. p. 28–9.