- Email: [email protected]
Image registration model and algorithm for multi-focus images
Accepted Manuscript
Image Registration Model and Algorithm for Multi-focus Images Xiaohua Xia , Gang Dang , Yunshi Yao , Jia Liang PII: DOI: Reference:
S0167-8655(16)30352-X 10.1016/j.patrec.2016.12.005 PATREC 6695
To appear in:
Pattern Recognition Letters
Received date: Revised date: Accepted date:
21 March 2016 17 October 2016 10 December 2016
Please cite this article as: Xiaohua Xia , Gang Dang , Yunshi Yao , Jia Liang , Image Registration Model and Algorithm for Multi-focus Images, Pattern Recognition Letters (2016), doi: 10.1016/j.patrec.2016.12.005
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Image Registration Model and Algorithm for Multi-focus Images Xiaohua Xiae-mail: [email protected], Gang Dang, Yunshi Yao and Jia Liang Key Laboratory of Road Construction Technology and Equipment of MOE, Chang’an University, Xi’an 710064, China Corresponding author.
*
CR IP T
Highlights A model and an algorithm are proposed for multi-focus image registration.
The model contains only one parameter, which has explicit physical meaning. They improve the registration accuracy and are better than the widely used method.
AN US
Factors affecting image registration error are analyzed, and suggestions are given.
AC
CE
PT
ED
M
Graphical abstract
ABSTRACT
Accurate registration of multi-focus images is essential in the fields of image fusion and shape from focus. A new image registration model, which is based on the geometry of multi-focus imaging, is presented for multi-focus images. The model contains only one parameter to be calibrated, and the parameter has explicit physical meaning. Golden section
ACCEPTED MANUSCRIPT
search is utilized to calibrate the parameter in our global image registration algorithm. Experimental results show that the model and algorithm can improve registration accuracy of multi-focus images and the method is superior to the widely used method based on affine transformation. Finally, factors affecting the registration error of our method are analyzed by simulation, and suggestions are given to improve the registration accuracy in practical
CR IP T
applications.
Keywords:
Multi-focus image; Image registration model; Image registration algorithm; Error analysis
AN US
1. Introduction
Multi-focus images are a sequence of images with focus on different depths of the object. They are used in many applications such as image fusion (Benes et al., 2013; Huang and Jing, 2007; Ludusan and Lavialle, 2012; Wang et al., 2010), shape from focus (Karthik and
M
Rajagopalan, 2014; Mahmood and Choi, 2012; Minhas et al., 2011), etc. to extend the depth of focus or recover the 3D shape of object. In these applications, image registration is an
ED
indispensable step (Karthik and Rajagopalan, 2014; Shah et al., 2014). The purpose of image registration is to align the target image to the reference image according to the
PT
coordinate transformation model (Klein et al., 2009; Li et al., 2013). Accurate registration of
CE
multi-focus images is essential for image fusion and shape from focus because it has great effects on the performance of focus measures (De et al., 2006; Li and Yang, 2008).
AC
In the past few decades, plenty of image registration methods have been proposed for different applications such as medical imaging, remote sensing and computer vision (Ulaş et al., 2010; Zhang et al., 2014; Zitová and Flusser, 2003). However, few literatures refer to multi-focus image registration. Many literatures assume that the images have been registered(Li et al., 2012; Li and Yang, 2008; Li and Li, 2012; Zhang et al., 2016). The common method for multi-focus image registration is based on feature detection and establishes the mapping of features according to affine transformation model (De et al.,
ACCEPTED MANUSCRIPT
2006). Since the accuracy of feature detection in defocused regions is uncertain (Xia et al., 2014), the method does not ensure the registration accuracy of multi-focus images. To overcome the problem mentioned above, an image registration model and a global image registration algorithm are proposed for the registration of multi-focus images. The model is established according to the geometry of multi-focus imaging. It contains only one parameter to be calibrated, and the parameter has explicit physical meaning. The global
CR IP T
image registration algorithm utilizes golden section search to calibrate the parameter of the model.
The paper is organized as follows. The proposed image registration method, including the image registration model and the image registration algorithm, is presented in Section 2. In
AN US
Section 3, experimental results are given to verify the effectiveness of the method, and factors affecting image registration error are analyzed by simulation. Finally, conclusions are drawn and suggestions are given to improve image registration accuracy in practical applications in Section 4.
M
2. Image registration method
ED
2.1. Image registration model
The geometry of multi-focus imaging is shown in Fig. 1, where u is the initial object
PT
distance, v is the initial image distance, and δx is the lens moving distance for capturing another multi-focus image.
CE
Note that u is variable because of the fluctuation of the object surface while v is constant because it is the initial image distance. The magnification of the imaging system can be
AC
expressed by: m
v u
(1)
When the lens moves a distance δx to the object, the magnification becomes: m
v x u x
(2)
The relationship between the two magnifications can be established by a scale factor:
ACCEPTED MANUSCRIPT x 1 m v s m 1 x u
Fig. 1. Geometry of multi-focus imaging
CR IP T
(3)
Assume that δx is far smaller than u. The assumption coincides with the fact that δx is
s 1
x v
constant
(4)
AN US
millimeter level when u is meter level or more. The scale factor becomes:
Based on the assumption, we can use a scale factor to establish the relationship between
M
the target and reference images. Note that the scale factor in equation (4) is not the best
ED
scale factor to register images because it is always less than the real scale factor in equation (3). To reduce image registration error, we do not adopt the scale factor in equation (4) but
PT
use a scale factor to be calibrated for image registration. The relationship between the target and reference images can be expressed by: (5)
AC
CE
x s ( x xo ) xo y s ( y yo ) yo
where (x, y) and (x′, y′) are respectively the image coordinates of the target and reference images; (xo, yo) and (x′o, y′o) are the corresponding image center coordinates; s* is the scale factor to be calibrated. The image registration model in equation (5) contains only one parameter, which means the ratio between the magnifications of the two multi-focus images. 2.2. Image Registration algorithm
ACCEPTED MANUSCRIPT
Golden section search is utilized in our global image registration algorithm to calibrate the scale factor s*. The algorithm is as follows: a) Define the object function: F s
f x, y g x, y
(6)
M
where f(*) and g(*) are respectively the intensity of the target and reference images; M is
CR IP T
the pixel number of the overlapping region of the multi-focus images. For a given scale factor s, M can be computed by counting the number of pixels in the reference image that can correspond to the pixels in the target image according to equation (5).
Suppose the error tolerance of the scale factor is ε. The setting of ε depends on the
AN US
accuracy of image processing and the image size. For the common pixel-level image processing accuracy that the error is not permitted to exceed 0.5 pixels, the error tolerance ε can be set by: 1
(7)
W H2 2
M
ED
where W and H are the image width and height respectively. b) Set the search interval of the scale factor to be [s1, s2]. Make sure s1≤s*≤s2. Usually,
PT
the magnifications of multi-focus images are close to each other. It means the scale factor s*
CE
is close to 1. Customarily, we set s1=0.80 and s2=1/s1=1.25. The search interval [0.80, 1.25] is sufficient for most cases of multi-focus image registration. Moreover, the setting that s1<1
AC
and s2>1 makes us not pay attention to which image should be selected as the reference image.
c) Given a division ratio r=0.618, the search interval is divided by two internal points: x1 s2 r s2 s1 x2 s1 r s2 s1
(8)
d) Compute F(x1) and F(x2). If F(x1)≥F(x2), s1=x1; else s2=x2.
ACCEPTED MANUSCRIPT
e) Compute the width of the new interval w=s2 - s1. If w≤ε, s*= (s1+ s2)/2; else go to Step c). 3. Experimental verification and error analysis 3.1. Experimental verification To verify the proposed model and algorithm, eight groups of multi-focus images are used for image registration. They are shown in Fig. 2, and the focused regions are marked with
CR IP T
ellipses. The multi-focus images are captured by a digital camera with an APS-C sensor and a zoom lens with the focal length varying from 18mm to 135mm. The size of the APS-C sensor is 22.3×14.9mm, and the resolution is 5184×3456. MSE (Mean square error) is a common objective evaluation for estimating the accuracy of image registration (Xia et al.,
AN US
2014). Less MSE means more accurate image registration. MSEs of the eight groups of images before registration, registered by the widely used method based on affine transformation model (the traditional method) and registered by the proposed method are shown in Table 1.
M
From Table 1, we can see that the MSEs of each group of images registered by the
ED
traditional method and the proposed method are less than that before registration. It means both the traditional method and the proposed method can improve the accuracy of image
PT
registration. Moreover, the MSE of each group of images registered by the proposed method is less than that by the traditional method. It means the image registration by the proposed
CE
method is more accurate than that by the traditional method. The proposed method is superior to the widely used method based on affine transformation in multi-focus image
AC
registration.
Further, ANOVA analysis (analysis of variance) is used to interpret the experimental data. It can determine the influence of the quality factor on the measurable characteristics of the research object (Zębala and Kowalczyk, 2015). The analysis is based on the normalized registration error of each group of multi-focus images, and it is carried out for a level of significance of 5%, with a confidence level of 95%. The ANOVA analysis results are shown in Table 2. Apparently, the F-ratio of the proposed method is greater than F0.05, 1.23=4.28, and it has statistical significance on the registration error. However, the F-ratio of the
ACCEPTED MANUSCRIPT
traditional method is smaller than F0.05, 1.23, and it does not have statistical significance on the registration error. It results from the uncertainty of feature detection in defocused regions in the traditional image registration method. P-value of factors indicates that the
M
AN US
CR IP T
proposed method is significantly contributing towards image registration.
ED
Fig. 2. Eight groups of multi-focus images
Registered by the
Registered by the
registration
traditional method
proposed method
Group 1
593.32
120.94
114.25
Group 2
769.39
440.63
321.70
Group 3
892.56
836.36
642.23
Group 4
765.08
730.09
691.40
Group 5
290.96
262.06
250.35
Group 6
688.52
676.42
638.65
Group 7
180.51
179.74
174.82
Group 8
1706.87
1651.86
1633.33
CE
Images
PT
Before
AC
Table 1. MSEs of multi-focus images
ACCEPTED MANUSCRIPT
Table 2. Analysis of variance (ANOVA) of the traditional method and the proposed method on the normalized registration error of the eight groups of multi-foucs images Sum of
Degree of
squares
freedom
Traditional
0.1379
Proposed
Mean square
F-ratio
P-value
1
0.1379
2.55
0.1254
0.2634
1
0.2634
4.86
0.0387
Error
1.1373
21
0.0542
Total
1.4182
23
CR IP T
Factor
The performance of the proposed method is tested in shape from focus. The objects used in experiments are of stepped shape. The multi-focus images of them are shown in Fig. 3,
AN US
and the focused regions are marked with ellipses. They are captured by a digital camera with an APS-C sensor and a lens with the focal length of 135mm. The size of the sensor is 22.3×14.9mm, and the resolution is 5184×3456. The shape from focus algorithm used in experiments is from Said Pertuz’s software 1 . The focus measure used in experiments is
M
Modified Laplacian (Nayar and Nakagawa, 1994; Pertuz et al., 2013; Xia et al., 2016). To
ED
reduce the computation of shape from focus, the multi-focus images are subsampled to one tenth of the raw images in size. The comparison between the shape from focus results by the
PT
proposed image registration method and the traditional image registration method based on
AC
CE
affine transformation is shown in Fig. 4.
Fig. 3. The multi-focus images of the obejcts used for shape from focus in experiments 1
https://cn.mathworks.com/matlabcentral/fileexchange/55103-shape-from-focus
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Fig. 4. The comparison of different shape from focus results
ED
We compute the false pixel number in the results and use it as the criterion of the accuracy of shape from focus. In the experimental result of the objects of two books, the
PT
false pixel number of the result by the proposed image registration method is 1566, and the false pixel number of the result by the traditional image registration method is 1611. In the
CE
experimental result of the objects of texts, the false pixel number of the result by the proposed image registration method is 473, and the false pixel number of the result by the
AC
traditional image registration method is 813. The false pixel numbers of the results by the proposed image registration methods are smaller than these by the traditional image registration methods. It means the accuracy of shap from focus by the proposed image registration method is better than that by the traditional method. The proposed image registration method is superior to the widely used method based on affine transformation in multi-focus image registration. 3.2. Error analysis
ACCEPTED MANUSCRIPT
In the proposed method, a constant scale factor is used to establish the relationship of multi-focus images. The registration error arises from the difference between the constant scale factor s* and the real scale factor s. For the pixel at (x, y) in the target image and the corresponding pixel at (x’, y’) in the reference image, the registration error can be computed by: x xo y yo 2
2
s* s
(9)
CR IP T
E
The maximum of equation (9) is difficult to compute because the real scale factor s is related to the deep informantion of the object (see equation (3)). But it can be estimated by: Emax
W2 H2 max smax s , s smin 4 4
(10)
AN US
where smax and smin are respectively the maximum and minimum of the real scale factor in equation (3). Equality holds when max(smax-s*, s*-smin) occurs at the corner of the image. Note that s* is the calibration result of the scale factor and it is related to the shape of the
umax
umin
1 1
x v
(11)
x
ED
1 s umax umin
M
object. Here we replace it by the average scale factor for simulation:
u
PT
where umax and umin are respectively the maximum and minimum of the object distance. Thus, the simulation model for estimating the maximum of the registration error is: W2 H2 max smax s , s smin 4 4
CE
Emax
(12)
AC
From equation (3), we can see that lens moving distance and object distance range are two factors affecting the registration error. The effects of the two factors on registration error are simulated according to equation (12). The simulation condition is: the sensor size is 22.3×14.9mm and the resolution is 5184×3456; the focal length is 50mm; the initial object distance is 1m. The simulation results are shown in Fig. 5. Fig. 5 (a) shows the effect of lens moving distance on registration error. Fig. 5 (b) shows the effect of object distance range on registration error. The maximum registration error increases as lens moving distance increases, and the relationship of them is approximately linear (see Fig. 5 (a)). Therefore,
ACCEPTED MANUSCRIPT
small lens moving distance is beneficial to improving the accuracy of image registration. The maximum registration error increases as the object distance range increases, but the increase rate (slop of the curve) is falling (see Fig. 5 (b)). Moreover, the effect of object distance range on registration error weakens (the descent range becomes small) as lens moving distance decreases. Therefore, small object distance range is beneficial to improving the accuracy of image registration, and small lens moving distance can reduce the effect of
AN US
CR IP T
object distance range on registration error.
Fig. 5. The effects of lens moving distance δx and object distance range δu on registration
M
error
ED
The minimum object distance is another factor affecting the registration error. The effect of the minimum object distance is simulated under the following condition: the sensor size
PT
is 22.3×14.9mm and the resolution is 5184×3456; the focal length is 50mm; the initial
CE
object distance is the minimum object distance. The simulation results are shown in Fig. 6. Fig. 6 (a) shows the effect of the minimum object distance on registration error under
AC
different object distance ranges when lens moving distance is 1mm, and Fig. 6 (b) shows the effect of that under different lens moving distances when the object distance range is 10m. The maximum registration error decreases as the minimum object distance increases. The decrease rate is high when the minimum object distance is small (<5m). Therefore, moving the camera away from the object can improve the registration accuracy effectively when the camera is close to the object.
ACCEPTED MANUSCRIPT
CR IP T
Fig. 6 The effect of the minimum object distance umin on registration error
From equation (3), we can see that the initial image distance also affects the scale factor. For a given object distance, the image distance depends on the focus length according to the
AN US
Gaussian lens law. So the registration error is related to the focal length f. The effect of focal length on registration error is simulated. The simulated results are shown in Fig. 7. Fig. 7 (a) shows the effect of focal length on registration error under different ranges of object distance, with the simulation condition that the lens moving distance is 1mm, the
M
sensor size is 22.3×14.9mm (APS-C sensor), and the sensor resolution is 5184×3456. Fig. 7 (b) shows the effect of focal length on registration error under different lens moving
ED
distances, with the simulation condition that the range of object distance is 1~10m, the sensor size is 22.3×14.9mm (APS-C sensor), and the sensor resolution is 5184×3456. Fig. 7
PT
(c) shows the effect of focal length on registration error under different sensor sizes, with the simulation condition that the lens moving distance is 1mm, the range of object distance
CE
is 1~10m. The sensor sizes adopted in simulation are the medium-format sensor with the size of 43.8×32.8mm and the resolution of 8256×6192, the full-frame sensor with the size of
AC
35.9×24mm and the resolution of 7360×4912, the APS-C sensor with the size of 22.3×14.9mm and the resolution of 5184×3456, and the 1.5” sensor with the size of 14×18.7mm and the resolution of 4352×2904. The maximum registration error decreases as the focal length increases, but the decrease is tiny. All the curves are approximately flat. The maximum decrease of the maximum registration error in the simulation is 0.45 pixels, which occurs when the lens moving distance is 2.0mm, the range of object distance is 1m~10m, and the focal length changes from 18mm to 135mm. The effect of focal length on registration error is far smaller than that of lens moving distance, object distance range and
ACCEPTED MANUSCRIPT
the minimum object distance. Telephoto lens can be used to improve the accuracy of image
AN US
CR IP T
registration when high image registration accuracy is required.
ED
M
Fig. 7. The effect of focal length f on registration error
4. Conclusions
PT
We have proposed an image registration model and a global image registration algorithm for multi-focus images. The model contains only one parameter to be calibrated, and the
CE
parameter has explicit physical meaning. The global image registration algorithm utilizes golden section search to calibrate the parameter in the model. The effects of lens moving
AC
distance, object distance range, the minimum object distance and focal length on registration error are analyzed by simulation. The following suggestions are given to improve the image registration accuracy of multi-focus images in practical applications: a) Reduce the lens moving distance to capture multi-focus images; b) Reduce the depth of object if possible; c) Move the camera away from the object; d) Adopt telephoto lens if requiring high image registration accuracy.
ACCEPTED MANUSCRIPT
Acknowledgments The authors would like to thank the editor and reviewers for improving the quality of the paper. This work is supported by National Natural Science Foundation of China (NSFC) (51208044) and the Fundamental Research Funds for the Central Universities
CR IP T
(310825161007, 310825163408, 310825161005, 310825161009). References
1. Benes, R., Dvorak, P., Faundez-Zanuy, M., Espinosa-Duró, V., Mekyska, J., 2013. Multi-focus thermal image fusion. Pattern Recognition Letters 34, 536-544.
AN US
2. De, I., Chanda, B., Chattopadhyay, B., 2006. Enhancing effective depth-of-field by image fusion using mathematical morphology. Image and Vision Computing 24, 1278-1287.
3. Huang, W., Jing, Z., 2007. Evaluation of focus measures in multi-focus image fusion.
M
Pattern Recognition Letters 28, 493-500.
4. Karthik, S., Rajagopalan, A.N., 2014. Underwater microscopic shape from focus,
ED
2014 22nd International Conference on Pattern Recognition. IEEE, Stockholm, pp. 2107 - 2112.
PT
5. Klein, S., Pluim, J.P.W., Staring, M., Viergever, M.A., 2009. Adaptive Stochastic Gradient Descent Optimisation for Image Registration. International Journal of
CE
Computer Vision 81, 227-239. 6. Li, B., Wang, W., Ye, H., 2013. Multi-sensor image registration based on algebraic
AC
projective invariants. Opt. Express 21, 9824-9838.
7. Li, H., Chai, Y., Yin, H., Liu, G., 2012. Multifocus image fusion and denoising scheme based on homogeneity similarity. Optics Communications 285, 91–100.
8. Li, S., Yang, B., 2008. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing 26, 971–979.
ACCEPTED MANUSCRIPT
9. Li, W., Li, Y., 2012. Overall well focused catadioptric image acquisition with multifocal images: a model based method. IEEE Transactions on Image Processing 21, 3697-3706. 10.Ludusan, C., Lavialle, O., 2012. Multifocus image fusion and denoising: A variational approach. Pattern Recognition Letters 33, 1388-1396. 11.Mahmood, M.T., Choi, T.S., 2012. Nonlinear approach for enhancement of image
2873.
CR IP T
focus volume in shape from focus. IEEE Transactions on Image Processing 21, 2866-
12.Minhas, R., Mohammed, A.A., Wu, Q.M.J., 2011. Shape from focus using fast discrete curvelet transform. Pattern Recognition 44, 839-853.
13.Nayar, S.K., Nakagawa, Y., 1994. Shape from focus. IEEE Transactions on Pattern
AN US
Analysis and Machine Intelligence 16, 824-831.
14.Pertuz, S., Puig, D., Garcia, M.A., 2013. Analysis of focus measure operators for shape-from-focus. Pattern Recognition 46, 1415-1432.
15.Shah, P., Srikanth, T.V., Merchant, S.N., Desai, U.B., 2014. Multimodal image/video
M
fusion rule using generalized pixel significance based on statistical properties of the neighborhood. Signal, Image and Video Processing 8, 723-738.
ED
16.Ulaş, B., Udupa, J.K., Bai, L., 2010. The role of intensity standardization in medical image registration. Pattern Recognition Letters 31, 315-323.
PT
17.Wang, Z., Ma, Y., Gu, J., 2010. Multi-focus image fusion using PCNN. Pattern
CE
Recognition 43, 2003-2016. 18.Xia, X., Fang, S., Xiao, Y., 2014. High resolution image fusion algorithm based on
AC
multi-focused region extraction. Pattern Recognition Letters 45, 115-120. 19.Xia, X., Yao, Y., Liang, J., Fang, S., Yang, Z., Cui, D., 2016. Evaluation of focus measures for the autofocus of line scan cameras. Optik - International Journal for Light and Electron Optics 127, 7762-7775.
20.Zębala, W., Kowalczyk, R., 2015. Estimating the effect of cutting data on surface roughness and cutting force during WC-Co turning with PCD tool using Taguchi design and ANOVA analysis. International Journal of Advanced Manufacturing Technology 77, 2241-2256.
ACCEPTED MANUSCRIPT
21.Zhang, K., Li, X., Zhang, J., 2014. A Robust Point-Matching Algorithm for Remote Sensing Image Registration. IEEE Geoscience and Remote Sensing Letters 11, 469473. 22.Zhang, Y., Chen, L., Zhao, Z., Jia, J., 2016. Multi-focus image fusion based on cartoon-texture image decomposition. Optik - International Journal for Light and Electron Optics 127, 1291-1296. 23.Zitová, B., Flusser, J., 2003. Image registration methods: a survey. Image and Vision
AC
CE
PT
ED
M
AN US
CR IP T
Computing 21, 977-1000.
Image Registration Model and Algorithm for Multi-focus Images Xiaohua Xia , Gang Dang , Yunshi Yao , Jia Liang PII: DOI: Reference:
S0167-8655(16)30352-X 10.1016/j.patrec.2016.12.005 PATREC 6695
To appear in:
Pattern Recognition Letters
Received date: Revised date: Accepted date:
21 March 2016 17 October 2016 10 December 2016
Please cite this article as: Xiaohua Xia , Gang Dang , Yunshi Yao , Jia Liang , Image Registration Model and Algorithm for Multi-focus Images, Pattern Recognition Letters (2016), doi: 10.1016/j.patrec.2016.12.005
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Image Registration Model and Algorithm for Multi-focus Images Xiaohua Xiae-mail: [email protected], Gang Dang, Yunshi Yao and Jia Liang Key Laboratory of Road Construction Technology and Equipment of MOE, Chang’an University, Xi’an 710064, China Corresponding author.
*
CR IP T
Highlights A model and an algorithm are proposed for multi-focus image registration.
The model contains only one parameter, which has explicit physical meaning. They improve the registration accuracy and are better than the widely used method.
AN US
Factors affecting image registration error are analyzed, and suggestions are given.
AC
CE
PT
ED
M
Graphical abstract
ABSTRACT
Accurate registration of multi-focus images is essential in the fields of image fusion and shape from focus. A new image registration model, which is based on the geometry of multi-focus imaging, is presented for multi-focus images. The model contains only one parameter to be calibrated, and the parameter has explicit physical meaning. Golden section
ACCEPTED MANUSCRIPT
search is utilized to calibrate the parameter in our global image registration algorithm. Experimental results show that the model and algorithm can improve registration accuracy of multi-focus images and the method is superior to the widely used method based on affine transformation. Finally, factors affecting the registration error of our method are analyzed by simulation, and suggestions are given to improve the registration accuracy in practical
CR IP T
applications.
Keywords:
Multi-focus image; Image registration model; Image registration algorithm; Error analysis
AN US
1. Introduction
Multi-focus images are a sequence of images with focus on different depths of the object. They are used in many applications such as image fusion (Benes et al., 2013; Huang and Jing, 2007; Ludusan and Lavialle, 2012; Wang et al., 2010), shape from focus (Karthik and
M
Rajagopalan, 2014; Mahmood and Choi, 2012; Minhas et al., 2011), etc. to extend the depth of focus or recover the 3D shape of object. In these applications, image registration is an
ED
indispensable step (Karthik and Rajagopalan, 2014; Shah et al., 2014). The purpose of image registration is to align the target image to the reference image according to the
PT
coordinate transformation model (Klein et al., 2009; Li et al., 2013). Accurate registration of
CE
multi-focus images is essential for image fusion and shape from focus because it has great effects on the performance of focus measures (De et al., 2006; Li and Yang, 2008).
AC
In the past few decades, plenty of image registration methods have been proposed for different applications such as medical imaging, remote sensing and computer vision (Ulaş et al., 2010; Zhang et al., 2014; Zitová and Flusser, 2003). However, few literatures refer to multi-focus image registration. Many literatures assume that the images have been registered(Li et al., 2012; Li and Yang, 2008; Li and Li, 2012; Zhang et al., 2016). The common method for multi-focus image registration is based on feature detection and establishes the mapping of features according to affine transformation model (De et al.,
ACCEPTED MANUSCRIPT
2006). Since the accuracy of feature detection in defocused regions is uncertain (Xia et al., 2014), the method does not ensure the registration accuracy of multi-focus images. To overcome the problem mentioned above, an image registration model and a global image registration algorithm are proposed for the registration of multi-focus images. The model is established according to the geometry of multi-focus imaging. It contains only one parameter to be calibrated, and the parameter has explicit physical meaning. The global
CR IP T
image registration algorithm utilizes golden section search to calibrate the parameter of the model.
The paper is organized as follows. The proposed image registration method, including the image registration model and the image registration algorithm, is presented in Section 2. In
AN US
Section 3, experimental results are given to verify the effectiveness of the method, and factors affecting image registration error are analyzed by simulation. Finally, conclusions are drawn and suggestions are given to improve image registration accuracy in practical applications in Section 4.
M
2. Image registration method
ED
2.1. Image registration model
The geometry of multi-focus imaging is shown in Fig. 1, where u is the initial object
PT
distance, v is the initial image distance, and δx is the lens moving distance for capturing another multi-focus image.
CE
Note that u is variable because of the fluctuation of the object surface while v is constant because it is the initial image distance. The magnification of the imaging system can be
AC
expressed by: m
v u
(1)
When the lens moves a distance δx to the object, the magnification becomes: m
v x u x
(2)
The relationship between the two magnifications can be established by a scale factor:
ACCEPTED MANUSCRIPT x 1 m v s m 1 x u
Fig. 1. Geometry of multi-focus imaging
CR IP T
(3)
Assume that δx is far smaller than u. The assumption coincides with the fact that δx is
s 1
x v
constant
(4)
AN US
millimeter level when u is meter level or more. The scale factor becomes:
Based on the assumption, we can use a scale factor to establish the relationship between
M
the target and reference images. Note that the scale factor in equation (4) is not the best
ED
scale factor to register images because it is always less than the real scale factor in equation (3). To reduce image registration error, we do not adopt the scale factor in equation (4) but
PT
use a scale factor to be calibrated for image registration. The relationship between the target and reference images can be expressed by: (5)
AC
CE
x s ( x xo ) xo y s ( y yo ) yo
where (x, y) and (x′, y′) are respectively the image coordinates of the target and reference images; (xo, yo) and (x′o, y′o) are the corresponding image center coordinates; s* is the scale factor to be calibrated. The image registration model in equation (5) contains only one parameter, which means the ratio between the magnifications of the two multi-focus images. 2.2. Image Registration algorithm
ACCEPTED MANUSCRIPT
Golden section search is utilized in our global image registration algorithm to calibrate the scale factor s*. The algorithm is as follows: a) Define the object function: F s
f x, y g x, y
(6)
M
where f(*) and g(*) are respectively the intensity of the target and reference images; M is
CR IP T
the pixel number of the overlapping region of the multi-focus images. For a given scale factor s, M can be computed by counting the number of pixels in the reference image that can correspond to the pixels in the target image according to equation (5).
Suppose the error tolerance of the scale factor is ε. The setting of ε depends on the
AN US
accuracy of image processing and the image size. For the common pixel-level image processing accuracy that the error is not permitted to exceed 0.5 pixels, the error tolerance ε can be set by: 1
(7)
W H2 2
M
ED
where W and H are the image width and height respectively. b) Set the search interval of the scale factor to be [s1, s2]. Make sure s1≤s*≤s2. Usually,
PT
the magnifications of multi-focus images are close to each other. It means the scale factor s*
CE
is close to 1. Customarily, we set s1=0.80 and s2=1/s1=1.25. The search interval [0.80, 1.25] is sufficient for most cases of multi-focus image registration. Moreover, the setting that s1<1
AC
and s2>1 makes us not pay attention to which image should be selected as the reference image.
c) Given a division ratio r=0.618, the search interval is divided by two internal points: x1 s2 r s2 s1 x2 s1 r s2 s1
(8)
d) Compute F(x1) and F(x2). If F(x1)≥F(x2), s1=x1; else s2=x2.
ACCEPTED MANUSCRIPT
e) Compute the width of the new interval w=s2 - s1. If w≤ε, s*= (s1+ s2)/2; else go to Step c). 3. Experimental verification and error analysis 3.1. Experimental verification To verify the proposed model and algorithm, eight groups of multi-focus images are used for image registration. They are shown in Fig. 2, and the focused regions are marked with
CR IP T
ellipses. The multi-focus images are captured by a digital camera with an APS-C sensor and a zoom lens with the focal length varying from 18mm to 135mm. The size of the APS-C sensor is 22.3×14.9mm, and the resolution is 5184×3456. MSE (Mean square error) is a common objective evaluation for estimating the accuracy of image registration (Xia et al.,
AN US
2014). Less MSE means more accurate image registration. MSEs of the eight groups of images before registration, registered by the widely used method based on affine transformation model (the traditional method) and registered by the proposed method are shown in Table 1.
M
From Table 1, we can see that the MSEs of each group of images registered by the
ED
traditional method and the proposed method are less than that before registration. It means both the traditional method and the proposed method can improve the accuracy of image
PT
registration. Moreover, the MSE of each group of images registered by the proposed method is less than that by the traditional method. It means the image registration by the proposed
CE
method is more accurate than that by the traditional method. The proposed method is superior to the widely used method based on affine transformation in multi-focus image
AC
registration.
Further, ANOVA analysis (analysis of variance) is used to interpret the experimental data. It can determine the influence of the quality factor on the measurable characteristics of the research object (Zębala and Kowalczyk, 2015). The analysis is based on the normalized registration error of each group of multi-focus images, and it is carried out for a level of significance of 5%, with a confidence level of 95%. The ANOVA analysis results are shown in Table 2. Apparently, the F-ratio of the proposed method is greater than F0.05, 1.23=4.28, and it has statistical significance on the registration error. However, the F-ratio of the
ACCEPTED MANUSCRIPT
traditional method is smaller than F0.05, 1.23, and it does not have statistical significance on the registration error. It results from the uncertainty of feature detection in defocused regions in the traditional image registration method. P-value of factors indicates that the
M
AN US
CR IP T
proposed method is significantly contributing towards image registration.
ED
Fig. 2. Eight groups of multi-focus images
Registered by the
Registered by the
registration
traditional method
proposed method
Group 1
593.32
120.94
114.25
Group 2
769.39
440.63
321.70
Group 3
892.56
836.36
642.23
Group 4
765.08
730.09
691.40
Group 5
290.96
262.06
250.35
Group 6
688.52
676.42
638.65
Group 7
180.51
179.74
174.82
Group 8
1706.87
1651.86
1633.33
CE
Images
PT
Before
AC
Table 1. MSEs of multi-focus images
ACCEPTED MANUSCRIPT
Table 2. Analysis of variance (ANOVA) of the traditional method and the proposed method on the normalized registration error of the eight groups of multi-foucs images Sum of
Degree of
squares
freedom
Traditional
0.1379
Proposed
Mean square
F-ratio
P-value
1
0.1379
2.55
0.1254
0.2634
1
0.2634
4.86
0.0387
Error
1.1373
21
0.0542
Total
1.4182
23
CR IP T
Factor
The performance of the proposed method is tested in shape from focus. The objects used in experiments are of stepped shape. The multi-focus images of them are shown in Fig. 3,
AN US
and the focused regions are marked with ellipses. They are captured by a digital camera with an APS-C sensor and a lens with the focal length of 135mm. The size of the sensor is 22.3×14.9mm, and the resolution is 5184×3456. The shape from focus algorithm used in experiments is from Said Pertuz’s software 1 . The focus measure used in experiments is
M
Modified Laplacian (Nayar and Nakagawa, 1994; Pertuz et al., 2013; Xia et al., 2016). To
ED
reduce the computation of shape from focus, the multi-focus images are subsampled to one tenth of the raw images in size. The comparison between the shape from focus results by the
PT
proposed image registration method and the traditional image registration method based on
AC
CE
affine transformation is shown in Fig. 4.
Fig. 3. The multi-focus images of the obejcts used for shape from focus in experiments 1
https://cn.mathworks.com/matlabcentral/fileexchange/55103-shape-from-focus
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Fig. 4. The comparison of different shape from focus results
ED
We compute the false pixel number in the results and use it as the criterion of the accuracy of shape from focus. In the experimental result of the objects of two books, the
PT
false pixel number of the result by the proposed image registration method is 1566, and the false pixel number of the result by the traditional image registration method is 1611. In the
CE
experimental result of the objects of texts, the false pixel number of the result by the proposed image registration method is 473, and the false pixel number of the result by the
AC
traditional image registration method is 813. The false pixel numbers of the results by the proposed image registration methods are smaller than these by the traditional image registration methods. It means the accuracy of shap from focus by the proposed image registration method is better than that by the traditional method. The proposed image registration method is superior to the widely used method based on affine transformation in multi-focus image registration. 3.2. Error analysis
ACCEPTED MANUSCRIPT
In the proposed method, a constant scale factor is used to establish the relationship of multi-focus images. The registration error arises from the difference between the constant scale factor s* and the real scale factor s. For the pixel at (x, y) in the target image and the corresponding pixel at (x’, y’) in the reference image, the registration error can be computed by: x xo y yo 2
2
s* s
(9)
CR IP T
E
The maximum of equation (9) is difficult to compute because the real scale factor s is related to the deep informantion of the object (see equation (3)). But it can be estimated by: Emax
W2 H2 max smax s , s smin 4 4
(10)
AN US
where smax and smin are respectively the maximum and minimum of the real scale factor in equation (3). Equality holds when max(smax-s*, s*-smin) occurs at the corner of the image. Note that s* is the calibration result of the scale factor and it is related to the shape of the
umax
umin
1 1
x v
(11)
x
ED
1 s umax umin
M
object. Here we replace it by the average scale factor for simulation:
u
PT
where umax and umin are respectively the maximum and minimum of the object distance. Thus, the simulation model for estimating the maximum of the registration error is: W2 H2 max smax s , s smin 4 4
CE
Emax
(12)
AC
From equation (3), we can see that lens moving distance and object distance range are two factors affecting the registration error. The effects of the two factors on registration error are simulated according to equation (12). The simulation condition is: the sensor size is 22.3×14.9mm and the resolution is 5184×3456; the focal length is 50mm; the initial object distance is 1m. The simulation results are shown in Fig. 5. Fig. 5 (a) shows the effect of lens moving distance on registration error. Fig. 5 (b) shows the effect of object distance range on registration error. The maximum registration error increases as lens moving distance increases, and the relationship of them is approximately linear (see Fig. 5 (a)). Therefore,
ACCEPTED MANUSCRIPT
small lens moving distance is beneficial to improving the accuracy of image registration. The maximum registration error increases as the object distance range increases, but the increase rate (slop of the curve) is falling (see Fig. 5 (b)). Moreover, the effect of object distance range on registration error weakens (the descent range becomes small) as lens moving distance decreases. Therefore, small object distance range is beneficial to improving the accuracy of image registration, and small lens moving distance can reduce the effect of
AN US
CR IP T
object distance range on registration error.
Fig. 5. The effects of lens moving distance δx and object distance range δu on registration
M
error
ED
The minimum object distance is another factor affecting the registration error. The effect of the minimum object distance is simulated under the following condition: the sensor size
PT
is 22.3×14.9mm and the resolution is 5184×3456; the focal length is 50mm; the initial
CE
object distance is the minimum object distance. The simulation results are shown in Fig. 6. Fig. 6 (a) shows the effect of the minimum object distance on registration error under
AC
different object distance ranges when lens moving distance is 1mm, and Fig. 6 (b) shows the effect of that under different lens moving distances when the object distance range is 10m. The maximum registration error decreases as the minimum object distance increases. The decrease rate is high when the minimum object distance is small (<5m). Therefore, moving the camera away from the object can improve the registration accuracy effectively when the camera is close to the object.
ACCEPTED MANUSCRIPT
CR IP T
Fig. 6 The effect of the minimum object distance umin on registration error
From equation (3), we can see that the initial image distance also affects the scale factor. For a given object distance, the image distance depends on the focus length according to the
AN US
Gaussian lens law. So the registration error is related to the focal length f. The effect of focal length on registration error is simulated. The simulated results are shown in Fig. 7. Fig. 7 (a) shows the effect of focal length on registration error under different ranges of object distance, with the simulation condition that the lens moving distance is 1mm, the
M
sensor size is 22.3×14.9mm (APS-C sensor), and the sensor resolution is 5184×3456. Fig. 7 (b) shows the effect of focal length on registration error under different lens moving
ED
distances, with the simulation condition that the range of object distance is 1~10m, the sensor size is 22.3×14.9mm (APS-C sensor), and the sensor resolution is 5184×3456. Fig. 7
PT
(c) shows the effect of focal length on registration error under different sensor sizes, with the simulation condition that the lens moving distance is 1mm, the range of object distance
CE
is 1~10m. The sensor sizes adopted in simulation are the medium-format sensor with the size of 43.8×32.8mm and the resolution of 8256×6192, the full-frame sensor with the size of
AC
35.9×24mm and the resolution of 7360×4912, the APS-C sensor with the size of 22.3×14.9mm and the resolution of 5184×3456, and the 1.5” sensor with the size of 14×18.7mm and the resolution of 4352×2904. The maximum registration error decreases as the focal length increases, but the decrease is tiny. All the curves are approximately flat. The maximum decrease of the maximum registration error in the simulation is 0.45 pixels, which occurs when the lens moving distance is 2.0mm, the range of object distance is 1m~10m, and the focal length changes from 18mm to 135mm. The effect of focal length on registration error is far smaller than that of lens moving distance, object distance range and
ACCEPTED MANUSCRIPT
the minimum object distance. Telephoto lens can be used to improve the accuracy of image
AN US
CR IP T
registration when high image registration accuracy is required.
ED
M
Fig. 7. The effect of focal length f on registration error
4. Conclusions
PT
We have proposed an image registration model and a global image registration algorithm for multi-focus images. The model contains only one parameter to be calibrated, and the
CE
parameter has explicit physical meaning. The global image registration algorithm utilizes golden section search to calibrate the parameter in the model. The effects of lens moving
AC
distance, object distance range, the minimum object distance and focal length on registration error are analyzed by simulation. The following suggestions are given to improve the image registration accuracy of multi-focus images in practical applications: a) Reduce the lens moving distance to capture multi-focus images; b) Reduce the depth of object if possible; c) Move the camera away from the object; d) Adopt telephoto lens if requiring high image registration accuracy.
ACCEPTED MANUSCRIPT
Acknowledgments The authors would like to thank the editor and reviewers for improving the quality of the paper. This work is supported by National Natural Science Foundation of China (NSFC) (51208044) and the Fundamental Research Funds for the Central Universities
CR IP T
(310825161007, 310825163408, 310825161005, 310825161009). References
1. Benes, R., Dvorak, P., Faundez-Zanuy, M., Espinosa-Duró, V., Mekyska, J., 2013. Multi-focus thermal image fusion. Pattern Recognition Letters 34, 536-544.
AN US
2. De, I., Chanda, B., Chattopadhyay, B., 2006. Enhancing effective depth-of-field by image fusion using mathematical morphology. Image and Vision Computing 24, 1278-1287.
3. Huang, W., Jing, Z., 2007. Evaluation of focus measures in multi-focus image fusion.
M
Pattern Recognition Letters 28, 493-500.
4. Karthik, S., Rajagopalan, A.N., 2014. Underwater microscopic shape from focus,
ED
2014 22nd International Conference on Pattern Recognition. IEEE, Stockholm, pp. 2107 - 2112.
PT
5. Klein, S., Pluim, J.P.W., Staring, M., Viergever, M.A., 2009. Adaptive Stochastic Gradient Descent Optimisation for Image Registration. International Journal of
CE
Computer Vision 81, 227-239. 6. Li, B., Wang, W., Ye, H., 2013. Multi-sensor image registration based on algebraic
AC
projective invariants. Opt. Express 21, 9824-9838.
7. Li, H., Chai, Y., Yin, H., Liu, G., 2012. Multifocus image fusion and denoising scheme based on homogeneity similarity. Optics Communications 285, 91–100.
8. Li, S., Yang, B., 2008. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing 26, 971–979.
ACCEPTED MANUSCRIPT
9. Li, W., Li, Y., 2012. Overall well focused catadioptric image acquisition with multifocal images: a model based method. IEEE Transactions on Image Processing 21, 3697-3706. 10.Ludusan, C., Lavialle, O., 2012. Multifocus image fusion and denoising: A variational approach. Pattern Recognition Letters 33, 1388-1396. 11.Mahmood, M.T., Choi, T.S., 2012. Nonlinear approach for enhancement of image
2873.
CR IP T
focus volume in shape from focus. IEEE Transactions on Image Processing 21, 2866-
12.Minhas, R., Mohammed, A.A., Wu, Q.M.J., 2011. Shape from focus using fast discrete curvelet transform. Pattern Recognition 44, 839-853.
13.Nayar, S.K., Nakagawa, Y., 1994. Shape from focus. IEEE Transactions on Pattern
AN US
Analysis and Machine Intelligence 16, 824-831.
14.Pertuz, S., Puig, D., Garcia, M.A., 2013. Analysis of focus measure operators for shape-from-focus. Pattern Recognition 46, 1415-1432.
15.Shah, P., Srikanth, T.V., Merchant, S.N., Desai, U.B., 2014. Multimodal image/video
M
fusion rule using generalized pixel significance based on statistical properties of the neighborhood. Signal, Image and Video Processing 8, 723-738.
ED
16.Ulaş, B., Udupa, J.K., Bai, L., 2010. The role of intensity standardization in medical image registration. Pattern Recognition Letters 31, 315-323.
PT
17.Wang, Z., Ma, Y., Gu, J., 2010. Multi-focus image fusion using PCNN. Pattern
CE
Recognition 43, 2003-2016. 18.Xia, X., Fang, S., Xiao, Y., 2014. High resolution image fusion algorithm based on
AC
multi-focused region extraction. Pattern Recognition Letters 45, 115-120. 19.Xia, X., Yao, Y., Liang, J., Fang, S., Yang, Z., Cui, D., 2016. Evaluation of focus measures for the autofocus of line scan cameras. Optik - International Journal for Light and Electron Optics 127, 7762-7775.
20.Zębala, W., Kowalczyk, R., 2015. Estimating the effect of cutting data on surface roughness and cutting force during WC-Co turning with PCD tool using Taguchi design and ANOVA analysis. International Journal of Advanced Manufacturing Technology 77, 2241-2256.
ACCEPTED MANUSCRIPT
21.Zhang, K., Li, X., Zhang, J., 2014. A Robust Point-Matching Algorithm for Remote Sensing Image Registration. IEEE Geoscience and Remote Sensing Letters 11, 469473. 22.Zhang, Y., Chen, L., Zhao, Z., Jia, J., 2016. Multi-focus image fusion based on cartoon-texture image decomposition. Optik - International Journal for Light and Electron Optics 127, 1291-1296. 23.Zitová, B., Flusser, J., 2003. Image registration methods: a survey. Image and Vision
AC
CE
PT
ED
M
AN US
CR IP T
Computing 21, 977-1000.