Journal Pre-proof Local-Adaptive and Outlier-Tolerant Image Alignment Using RBF Approximation
Jing Li, Baosong Deng, Maojun Zhang, Ye Yan, Zhengming Wang PII:
S0262-8856(20)30022-6
DOI:
https://doi.org/10.1016/j.imavis.2020.103890
Reference:
IMAVIS 103890
To appear in:
Image and Vision Computing
Received date:
7 March 2019
Revised date:
12 January 2020
Accepted date:
28 January 2020
Please cite this article as: J. Li, B. Deng, M. Zhang, et al., Local-Adaptive and OutlierTolerant Image Alignment Using RBF Approximation, Image and Vision Computing(2020), https://doi.org/10.1016/j.imavis.2020.103890
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
© 2020 Published by Elsevier.
Journal Pre-proof
Local-Adaptive and Outlier-Tolerant Image Alignment Using RBF Approximation Jing Li
1,2,∗
, Maojun Zhang 3 , Ye Yan Wang 3
1
and Zhengming
National Innovation Institute of Defense Technology (NIIDT), Academy of Military Science;
[email protected] 2 Tianjin Artificial Intelligence Innovation Center (TAIIC) College of System Engineering, National University of Defense Technology (NUDT)
-p
ro
3
, Baosong Deng
of
1
1,2,∗
Abstract
Jo
ur
na
lP
re
Image alignment is a crucial step to generate a high quality panorama. The state-of-the-art approaches use local-adaptive transformations to deal with multi-view parallax, but still suffer from unreliable feature correspondences and high computational cost. In this paper, we propose a local-adaptive and outlier-tolerant image alignment method using RBF (radial basis function) approximation. To eliminate the visible artifacts, the input images are warped according to a constructed projection error function, whose parameters are estimated by solving a linear system. The outliers are efficiently removed by screening out the abnormal weights of RBFs, such that better alignment quality can be achieved compared to the existing approaches. Moreover, a weight assignment strategy is introduced to further address the overfitting issues caused by extrapolation, and hence the global projectivity can be well preserved. The proposed method is computationally efficient, whose performance is verified by comparative experiments on several challenging cases. Keywords: image alignment; radial basis function; scattered data approximation; outlier removal; computer vision 1. Introduction Image alignment is one of the most active research areas of computer vision in the past few years. Given a group of overlapped images, image Preprint submitted to Image and Vision Computing
January 12, 2020
Journal Pre-proof
Jo
ur
na
lP
re
-p
ro
of
alignment aims at establishing the accurate mapping relationship between images. Since the optical centers of cameras are usually unfixed while capturing different images, the shape or location of an object may vary in different views. This phenomenon is called multi-view parallax which leads to structure misalignment in the resulted panorama [1]. Parallax handling is still one of the most challenging tasks in research on image alignment, and numerous approaches have been proposed in the past decade to address such issues. Applying an optimal global transformation [2, 3] over the whole image plane cannot accurately align the natural images with parallax. For input images with poor and repetitive textures, the results of global alignment is shown in Fig. 1a, which contains significant ghosting. To achieve better alignment accuracy, the local-adaptive alignment approaches are introduced which divide the image plane into grid meshes and estimate space-varying transformations over the mesh cells. Among the state-of-the-art approaches, CPW [4] (content-preserving warps) and APAP [5] (as-projective-as-possible) stitching are representative. Utilizing the geometric distribution of the feature correspondences, CPW smoothly adjusts the locations of mesh vertexes derived from the global transformation. APAP directly estimates local-adaptive transformations by weighting the matching data according to their distances from current cell center. However, their performance highly relies on the quality of feature correspondences. Visible artifacts may arise if there are incorrect or insufficient matches. Fig. 1b and Fig. 1c respectively show the results of CPW and APAP, where visible artifacts appear. Instead of constructing a geometric transformation model, we interpret image alignment as a scattered data approximation problem, which can be efficiently solved with the help of radial basis functions (RBFs). Given a few feature correspondences, a rough global transformation is estimated first. By applying RBF approximation, a projection error function is then constructed, whose parameters are obtained by solving a specific linear system. Based on this error function, the reprojected images are warped to eliminate the misalignment. Observing that the correctness of a match is highly related to the weight of the RBF attached to it, the outliers can be efficiently removed by screening out the unnaturally distributed weights such that the localadaptive and outlier-tolerant image alignment is achieved. To further address the overfitting issues caused by extrapolation, a weight assignment strategy is introduced which suppresses the deformation near image borders. Then the global projectivity is well preserved. Fig. 1d shows the alignment results using 2
Journal Pre-proof
(b) CPW
-p
ro
of
(a) Global alignment
(c) APAP
(d) Ours
lP
re
Figure 1: Image alignments results of a building scene. For the highlighted areas, visible artifacts are circled in red. (a) presents the result of global alignment [2], which contains significant ghosting. (b) and (c) are the results of CPW [4] and APAP [5] respective, in which visible artifacts appear. (d) is the result of our method.
Jo
ur
na
the proposed approach, in which visible artifacts are successfully eliminated and no abnormal distortion exists. The rest of the paper is organized as follows: Section 2 reviews the literature related to image alignment and RBF approximation. Section 3 presents the proposed alignment method in detail. Its complexity and efficiency are analyzed in Section 4. Experimental results and comparisons of different methods are presented in Section 5. Section 6 concludes the paper. 2. Related Works
2.1. Content-Preserving Warps The content-preserving warp (CPW) approaches attempt to eliminate the projection errors of global transformation by warping the misaligned image content. The idea of CPW is originated from animation process in computer graphics [6]. The image plane is divided into a grid mesh and a cost function containing three terms (specifically, the local alignment term, the global alignment term and the smoothness term) is defined with respect to the shifted locations of grid nodes [4]. The optimal warp is then obtained by 3
Journal Pre-proof
na
lP
re
-p
ro
of
minimizing the cost function. CPW performs well in preserving the shapes and textures of image content while warping the original images, hence it is widely used in image resizing [7, 8, 9, 10], video stabilization [4, 11] and image stitching [12, 13, 14]. Jia and Tang [15] detect and match one dimensional features along the seam, and warp the image according to the feature correspondences. Seamless panoramic image can then be obtained by employing gradient domain fusion. Zhang and Liu [12] accurately align image with parallax by warping the reprojected images. Cheng and Chuang [13] first build the correspondences between grid nodes in the overlapping region, and then minimize the overall cost function to achieve both accurate alignment and global projectivity preserving. Lin et al. [14] propose the SEAGULL (seam-guided local alignment) approach, in which the optimal warp is reached by iterate the location of seam and the grid nodes. Lin et al. [16] propose the MPA (mesh-based photometric alignment) approach that minimizes pixel intensity difference instead of geometric distance between feature correspondences. It achieves satisfactory alignment quality in both textured and monochromic image areas.Cell size of the grid mesh is a crucial parameter of the CPW approaches. To achieve more accurate alignment, smaller size is preferred. However, the results might be unstable if there exist matching errors within such small grid cells. Thus the alignment quality of the CPW approaches is limited.
Jo
ur
2.2. Local-Adaptive Transformations The local-adaptive transformation approaches focus on constructing accurate projection models between images. Gao et al. [17] introduce the idea of local planar transformation. They propose the DHW (dual-homography warping) approach to deal with the scenes consisting of two dominating planes. However, DWH does not perform well for more complicated scenes. Lin et al. [18] propose the SVA (smoothly varying affine) transformations to more accurately align the local image areas. Zaragoza et al. [5] proposed the APAP (as-projective-as-possible) stitching approach, which computes the local transformations by weighting the matching data. For input images of wide field-of-views, APAP may cause perspective distortions in the non-overlapping regions. To address this issue, Chang et al. [19] propose the SPHP (shape-preserving half-projective) warp, which is a halfprojective approach. Lin et al. [20] propose a similar approach named as ANAP (as-natural-as-possible) stitching, by which the local homographies can be smoothly changed to a global similarity transformation while the 4
Journal Pre-proof
of
current location is moving away from the overlapping region. To remove incorrect matches in local areas, Zhang et al. [21] propose a local DLT (direct linear transformation) scheme in their multi-viewpoint panorama stitching method. Herrmann et al. [22] propose a multiple registration approach to stitch images with significant depth variation or object motion, and the MRF (Markov Random Field) inference technique is extended to search for the seam over multiple registrations. The local-adaptive transformation approaches usually perform well with small cell size, but still suffer from limited alignment accuracy caused by matching errors.
Jo
ur
na
lP
re
-p
ro
2.3. RBF Approximation The Radial Basis Function (RBF) is a powerful tool for large scattered data approximation [23]. Several approaches using different RBF types are briefly introduced and compared in Majdisova and Skala’s work [24]. Since the mesh-free approximation problem is converted to the solution of a linear system [25], RBF approximation is efficient for high dimensional spaces. Therefore, it is widely used in facial expression [26], surface detection [27] and image registration [28, 29, 30, 31]. Arad and Resisfeld [32] introduce RBF as a tool of image warping using few anchor points. As a special type of RBF approximation approaches, TPS (thin plate spline) [33] reflects the shape of a physically bent thin metal sheet, and is used for elastic image registration [29, 28, 34] in computer vision. A comparative study of different transformation functions including the TPS and multiquadric RBFs for elastic image registration is given by Zagorchev and Goshtasby [35]. The performance of RBFs in three dimensional spaces is evaluated by Beatson et al. [36]. By adjusting the smoothness parameters of RBFs, Bozzini et al. [37] achieve stable approximation of noisy scattered data. Concentrated on medical image registration, Zhang et al. [38] propose a sparse constrained transformation model based on RBF expansion, which is robust to noisy correspondences and with high registration accuracy while preserving the topology of deformation field. Li et al. [34] propose a efficient and flexible image alignment approach, which employs TPS to eliminate the parallax issues. A Bayesian model of feature refinement is also proposed to adaptively remove the incorrect local matches. The reliability of matching data is also a key issue for the RBF approximation approaches. However, it still have not been paid enough research attention up to now.
5
Journal Pre-proof
3. Image Alignment Using RBF Approximation
-p
x ˜0 ∼ H˜ x
ro
of
3.1. Global Alignment Given a group of matched feature points {xi , x0 i }, i = 1, · · · , n in the input images I and I 0 , image alignment estimate a point-wise mapping relationship between I and I 0 by interpolating or fitting these discrete correspondences based on a certain transformation model. In the multi-view imaging system, since the depth of scene is unknown, approaches based on global alignment usually approximate the three-dimensional transformations as two-dimensional projections. By employing the DLT [39] (Direct Linear Transformation) solver, the global transformation between an arbitrary pair of corresponding points {x, x0 } can be estimated as (1)
na
lP
re
where H is the homography indicating the 2D projective transformation between images, x ˜ = (x, y, 1)T and x ˜0 = (x0 , y 0 , 1)T are respectively the homogeneous coordinates of x = (x, y)T and x0 = (x0 , y 0 )T , and ∼ indicates equality up to scale. (1) is equivalent to ( x0 = (h11 x + h12 y + h13 )/(h31 x + h32 y + h33 ) (2) y 0 = (h21 x + h22 y + h23 )/(h31 x + h32 y + h33 )
Jo
ur
where hkj , k, j = 1, 2, 3 is the entry of H at location (i, j). Denoting the projections of x and x0 on the panoramic image plane as u and u0 , then u ˜ ∼ Hu x ˜
(3)
u ˜ 0 ∼ H0 u x ˜0
(4)
where u ˜ and u ˜ 0 are u and u0 in homogeneous coordinates, the homography Hu and H0 u respectively indicate the global transformations from I and I 0 to the panoramic image plane, and they meet the relation H0 u = Hu H−1 . Without loss of generality, we select the image plane of I as the panoramic image plane, then Hu = I and H0 u = Hu H−1 = H−1 . 3.2. Local Alignment using RBF Approximation For ideal cases without parallax, the global transformation is sufficient to achieve accurate image alignment, which means u0 = u. However, for general 6
Journal Pre-proof
cases, the projection errors will arise, then u0 6= u. The error function over the whole panoramic image plane can be defined as g (u) = u0 − u
(5)
wi φ (ku − ui k2 ) + AT u ˜
(6)
-p
g (u) =
n X
ro
of
Local alignment is the process of computing g (u) = (g (u) , h (u))T based on the discrete matching errors gi = g (ui ) = (gi , hi )T of the given correspondences, where g (u) and h (u) are respectively the horizontal and vertical components of g (u), and similarly, gi and hi are the two coordinate components of gi . According to the basic theories of RBF approximation [30, 40, 24], g (u) can be expressed as
i=1
Jo
ur
na
lP
re
where φ (ku − ui k2 ) is the RBF centered at ui , and its value only depends on the Euclidean distance ku − ui k2 , wi = (ωi , υi )T is the weight of φ (ku − ui k2 ), and A = (a, b) is the weight of the linear component u ˜ , the T T vectors a = (α1 , α2 , α3 ) and b = (β1 , β2 , β3 ) are corresponding to the two coordinate components of g (u) respectively. Geometrically, the term AT u ˜ in (6) represents the linear approximation result of the error function g (u), and then g (u) is further accumulated by RBFs centered at all matched feature points. The weights W = (w1 , · · · , wn )T and A meet the following linear constraint K + λI P W G = (7) PT 0 A 0 where K = (φ (ku − ui k2 )) ∈ Rn×n , P = (˜ u1 , · · · , u ˜ n )T , G = (g1 , · · · , gn )T , the coefficient λ is used to balance the influence of the accuracy and the smoothness of approximation. The constraints g (ui ) = (gi , hi )T are strictly met if λ = 0. While λ is becoming larger, the approximation error increases but g (u) tends to be smoother. The value of W and A can be computed by solving (7), and then the analytical expression of the projections at an arbitrary location u = (u, v)T can be obtained by plugging them into (6). The numerical computation of g (u) need to accumulate the values of n RBFs. It is still a time-consuming task to repeat this procedure for all pixels. To achieve better alignment efficiency, a uniform grid mesh of Cx × Cy cells is set over the panoramic image plane, and only g (u) on the grid nodes are 7
Journal Pre-proof
of
computed using (6). With negligible loss of accuracy, the projection errors of all the other pixels can be directly obtained by linear interpolation. The projection error can be effectively eliminated by warping the reprojected images according to the error function g (u), more specifically, by pulling u0 = u + g (u) to be coincide with u. In the input image I, the corresponding point of u is x = u, and it is obtained as x ˜0 ∼ H˜ u0 in I 0 . The reprojection and warping can be enclosed in the transformation of look-up-tables (LUTs), and accomplished by inverse mapping with bilinear interpolation [41].
lP
re
-p
ro
3.3. Selection of RBFs Selecting proper RBF is the key step to form the accurate error function and achieve satisfactory alignment quality. Denoting the radius as r = |u − ui | and the critical radius as σ if it exists, commonly used types of RBFs are listed in Table 1. The curves of the listed RBFs are depicted in Fig. 2, where all the four RBFs containing the parameter σ are set with σ = 1.0 to keep consistency. Table 1: Commonly used RBFs Expression
na
RBF
φ (r) = r r 2 /σ 2 φ (r) = ep φ (r) = 1 + r2 /σ 2 φ (r) = 1/p1 + r2 /σ 2 φ (r) = 1/ 1 + r2 /σ 2 φ (r) = r2 ln r φ (r) = r3
Jo
ur
Linear Gaussion Multiquadric Inverse quadratic Inverse multiquadric TPS Cubic
Parameters
Convergence
σ σ σ σ -
Divergent Convergent Divergent Convergent Convergent Divergent Divergent
According to their convergence, the listed RBFs can be divided into two categories: The convergent ones and the divergent ones. For a convergent RBF, the critical radius σ reflects its declining speed and effective coverage. A small σ improves flexibility of approximation that contributes to alignment accuracy, but also raises the risk of structural distortion. Conversely, a large σ improves the smoothness of warping that contributes to structure preserving, but also leads to alignment errors. The proper value of σ balancing the flexibility and smoothness depends on the geometric distances between feature points. Since the features of natural images are usually non-uniformly distributed, it brings about lots of additional tasks to search for a proper σ attached to every feature point. Meanwhile, since the value of a divergent 8
Journal Pre-proof
3.5 Gaussian Inverse quadratic Inverse multiquadratic Linear Multiquadric TPS Cubic
3 2.5
(r)
2 1.5 1
of
0.5
-0.5 0
0.5
ro
0
1
r
1.5
re
-p
Figure 2: Curves of the commonly used RBFs. The critical radius σ is set to 1.0 for all the RBFs.
ur
na
lP
RBF does not converge to 0 over the whole image plane, its flexibility will not be influenced by σ. The critical radius σ of a divergent RBF (e.g. the multiquadric RBF) mainly serve as a supplementary of the coefficient λ in (7) to adjust the smoothness of warping. The stable alignment can be much easier obtained using the divergent RBFs than the convergent ones. Therefore, from the perspective of robustness, the four divergent RBFs (namely, the linear, multiquadric, TPS and cubic RBFs) are selected for approximating the error function while aligning images. The computed projection error functions and the corresponding local alignment results are presented in Fig. 3.
Jo
3.4. Local Outlier Removal The existence of outliers is inevitable due to the complexity of natural images, especially for image areas with poor or repetitive textures. Most of the existing approaches remove outliers using RANSAC [42] (Random Sample Consensus) which minimizes the global projection errors even if the local transformations are estimated. In such circumstances, some inliers might be mistakenly removed while a few outliers are still reserved, and consequentially, the resulted panoramic image will contain undesirable distortions. Zhanget al. [21] proposed a local DLT approach to remove outliers for every local areas centered at a matched feature point. However, it does not apply to poor-textured areas due to the lack of sufficient features.
9
Journal Pre-proof
20
10 5
h(u,v)
-20
-40
0 -5 -10
-60 1500
-15 1500 1000
1000 500 0
u
1500
500
1000
-500
0
500 0
u
v
h(u,v)
g(u,v)
ro
5
-20
-40
0 -5 -10
-60 1500
-15 1500 1000
1000 500 1500
500
1000
-500
0
500 0
u
v
1500
500
1000
0
-500
v
re
0
-500
(c)
10
0
u
0
v
(b)
20
(e)
(f)
lP
(d)
20
10
0
0
h(u,v)
-20 -40
-10
na
g(u,v)
500
1000
of
(a)
1500
-p
g(u,v)
0
-20
-60 -80 1500
-30 1500
1000
1000
500 0
1500
500
1000
0
-500
500 0
u
v
ur
u
Jo
(g)
g(u,v)
0
500
1000
0
-500
v
(h)
(i)
20 10
h(u,v)
50
1500
0 -10
-50 -20 -100 1500
-30 1500 1000
1000 500
u
0
1500
500
1000
(j)
v
0
-500
500
u
0
1500
500
1000
(k)
0
-500
v
(l)
Figure 3: Projection error functions computed using different RBFs. (a) and (b) are the components of the projection error function computed using the linear RBF. (c) is the corresponding alignment result obtained by warping the reprojected images. Similarly, (d) (e) and (f ) are with the multiquadric RBF, (g) (h) and (i) are with the TPS RBF, (j) (k) and (l) are with the cubic RBF.
10
Journal Pre-proof
of
As analyzed by Li et al. [34], the weight wi = (ωi , υi )T reflects the difference between the projection error gi = (gi , hi )T at ui and projection errors of other feature points around ui . Based on a few idealized assumptions, it canPbe deduced Pnthat ωi and υi obey the normal distribution. Moren over, ω = i=1 i i=1 υi = 0 according to (7). Then ωi ∼ N (0, σω ) and υi ∼ N (0, συ ), where σω and συ are the standard deviations. An intuitive conclusion can be further inferred that a match is likely to be incorrect if its projection error significantly differs from the others nearby. Then a Bayesian model can be built by which the local outliers can be removed iteratively. The specific algorithm flow is as follows:
lP
re
-p
ro
1. Fomulate (7) based on the matches (ui , u0 i ), i = 1, · · · , n, and set the match number n` = n; 2. Solve (7) to get the weights (ωi , υi ), i = 1, · · · , n` ; 3. Compute the standard deviation σω and συ , and mark all matches with |ωi | > 3σω or |υi | > 3συ ; 4. If the number of marked matches is less than εn` , then goto 6; 5. Remove all rows and columns regarding the marked matches from (7), and update the match number n` , then goto 2; 6. The reserved unmarked matches are regarded as correct.
Jo
ur
na
The threshold ε is set to 0.0027 as calculated in the framework of Bayesian inference [34]. This approach is first proposed for the TPS, but its theoretical derivation is independent of the type of RBFs. Therefore, it can be directly extended to any RBF types. In practice, a part of outliers are removed by the DLT solver combined with global RANSAC, and a loose threshold is set to keep the maximum number of inliers. The intermediate result is shown in Fig.4a, which still contains several local outliers. Then the local Bayesian approach is applied to the rest matches. As an example, the result using the linear RBF is shown in Fig.4b, where the local outliers are effectively removed. 3.5. Overfitting Suppression The accurate alignment of the overlapping region can be achieved by RBF Approximation. However, directly extrapolating the estimated deformation to the non-overlapping region will lead to overfitting issues. The linear component AT u ˜ introduced in (6) could somewhat suppress the undesirable trend, but the objects approaching image borders still might be 11
Journal Pre-proof
(b)
of
(a)
-p
ro
Figure 4: Local outlier removal. The removed matches are marked as red, and the reserved ones are blue. (a) is the result of a loose global RANSAC, which still contains several local outliers. (b) is the outlier removal result of the local Bayesian approach, in which only the correct matches are reserved.
na
lP
re
highly distorted. This phenomenon can be observed in the third column of Fig.3, where the image content near the border is excessively stretched and some straight lines are bent. The better solution should reduce the influence of error function g (u) to image warping while the image point u is moving away from the overlapping region. Hence, we set the panoramic image coordinate up as up = u + µg (u)
(8)
up = u0 − (1 − µ) g (u)
(9)
Jo
ur
and according to (5),
where the weighting parameter µ ∈ [0, 1], and µ linearly varies from 0 to −−→ 1 along the direction OO0 on the panoramic image plane [20], as illustrated in Fig. 5. Specifically, µ is computed as −−→ −−−→ D−−→ −−−→E −−−→ 2 µ = M Pd / M M 0 = M P , M M 0 / M M 0 (10) ¯ µ is close to 0 near the image border away For the reprojected image I, from the overlapping region according to (8). Similarly, for the reprojected image I¯0 , 1 − µ is close to 0 near the image border on the opposite side according to (9). The distortions caused by extrapolation is eliminated without sacrificing the accuracy of alignment. 12
Journal Pre-proof
I' P
I O'
O
M'
Pd
of
M
ro
Figure 5: Weight assignment for overfitting suppression. I¯ and I¯0 are the reprojected images on the panoramic image plane, and O and O0 are the two reprojected −−−→ image centers. −−−→0 The weighting parameter µ of a image point P is computed as µ = M Pd / M M , where
-p
M and M 0 is the intersections of OO0 and the image borders, and Pd is the projection of P on M M 0 .
ur
na
lP
re
Given the panoramic image coordinate up , the coordinate u in the original reprojected image I¯ can be computed by employing the fixed-point iteration on (8). Then the image coordinate u0 can be directly obtained by plugging u into (5). The input image coordinates corresponding to up are further acquired as x = u and x ˜0 ∼ H˜ u0 . With the help of overfitting suppression, the image alignment results using the four selected RBFs are shown in Fig. 6. Compared with the results in Fig. 3, The global projectivity is better preserved. 4. Complexity and Efficiency Analysis
Jo
To more concretely expound the proposed method, its methodology is summarized as follows: The basic framework of RBF approximation is built in Section 3.1 and Section 3.2. For a pair of input images with n matches, a global transformation is estimated first using the DLT solver, whose computational complexity is O (n2 ). To eliminate the misalignment caused by global transformation, a projection error function is then constructed to guide the warping of images, and the computational complexity of solving its weighting parameters is O (n3 ). The parameter estimation is repeated a few times for local outlier removal in Section 3.4. For the overwhelming majority cases, it will terminate within 3 − 10 iterations that is negligible compared to the matching number n. Therefore, the overall computational complexity to construct the analytical projection error is O (n3 ). 13
(b) Multiquadric RBF
Jo
ur
na
lP
(a) Linear RBF
re
-p
ro
of
Journal Pre-proof
(c) TPS RBF
(d) Cubic RBF
Figure 6: Image alignments results with overfitting suppression. (a) (b) (c) and (d) are respectively the results using the linear, multiquadric, TPS and cubic RBFs.
14
Journal Pre-proof
re
-p
ro
of
The value of projection errors should be obtained by summing the n weighted RBFs. For input images of size X × Y , the computational complexity of such numerical computation is O (nXY ). A uniform grid mesh of Cx × Cy cells is set over the image plane to accelerate the computation. Then the computational complexity is reduced to O (nCx Cy + XY ). In Section 3.5, the overfitting issues is suppressed by weighting the deformation of images. The main computational cost lies on the fixed-point iteration for seeking the original coordinates, which actually converges very fast in 10 iterations. Benefiting from the meshing strategy, the additional computational complexity introduced by the above iteration can be reduced to O (Cx Cy ). The following bilinear interpolation and image projection operation are all with a O (XY ) complexity. In short, the overall computational complexity of our method is O (n3 + nCx Cy + XY ), which is much lower than the other local-adaptive alignment approaches.
lP
5. Results
Jo
ur
na
A group of experiments are carried out to evaluate the performance of our method. As reviewed in Section 2, in image alignment stage, the state-of-theart approaches can be roughly divided into two groups: the ones employing content-preserving warps [4, 12, 13, 14] and the ones with local-adaptive transformations [17, 18, 5, 19, 20]. Similar alignment techniques are applied by different approach of each group. Therefore, in addition to global alignment [2], two representative approach CPW [4] and APAP [5] are selected for the comparative experiments. The feature points are detected and matched using a SIFT [43] implementation included in the mexopencv [44] kit. Given the same matching data, all the compared methods are evaluated in terms of alignment quality and computational efficiency. The parameters of CPW and APAP are set as recommended by the corresponding papers. While estimating the global transformation, the threshold of RANSAC is set between 0.02 − 0.08 for all testing image databases. Since the movement of a grid node in CPW is highly dependent on the feature correspondences around it, the alignment quality is very sensitive to the reliability of these matches. To ensure that a stable alignment result can be obtained using CPW, the cell size of its uniform grid mesh is set to 100 × 100 pixels that is a relatively large value. For APAP and the proposed approach using different RBF types, the cell size is set to 10 × 10 pixels for the sake of 15
Journal Pre-proof
alignment accuracy. Given the input images of size X × Y , the parameters with best alignment quality is applied for all the four tested RBF types. Denoting C = (X + Y ) /2, the specific parameter settings is listed in Table. 2. Table 2: Parameter settings for RBF approximation
Linear Multiquadric TPS Cubic
φ (r) = p r φ (r) = 1 + r2 /σ 2 φ (r) = r2 ln r φ (r) = r3
λ
σ
0 −0.01C 0.01C 2 0.1C 3
0.01C -
of
Expression
ro
RBF
Jo
ur
na
lP
re
-p
5.1. Qualitative Comparison on Alignment Quality Except for the alignment results of the building database presented in the previous sections, Fig. 7 shows another challenging case, in which the results of different approaches are presented by rows. For each resulted image, three representative areas in the overlapping region is highlighted by colored boxes. Row 1 shows the results of global alignment, in which significant projection errors can be observed in all the three highlighted areas. The results of CPW are shown in Row 2. Accurate alignment is achieved in the second highlighted area (the green boxed area). However, since some incorrect local matches still exist, visible artifacts appear in the first and third highlighted areas (the blue and red boxed areas). Due to the same reason, the results of APAP also contain slight ghosting effects, as shown in Row 3. Row 4-7 respectively show the results of our approach using the linear, multiquadric, TPS and cubic RBFs. By adopting the local outlier removal technique provided in Section 3.4, the reserved feature correspondences are much more robust than the ones filtered using global RANSAC. All the three highlighted areas of the four resulted images are accurately alignment. Additionally, benefited from the overfitting suppression strategy introduced in Section 3.5, abnormal distortions that may arise from extrapolating the error functions are successfully avoided, and the global projectivity is preserved. 5.2. Quantitative Comparison on Alignment Quality For more objective evaluation of alignment quality, the SSIM (structural similarity) [45] index is selected as the quantitative metric to compare different approaches. Fig. 8 presents the image databases used in the comparative
16
Journal Pre-proof
Global
ro
of
CPW
re
-p
APAP
TPS
Jo
ur
Multiquadric
na
lP
Linear
Cubic
Figure 7: Comparison of alignment quality17on a stairs scene. Row 1: Result of global alignment. Row 2: Result of CPW [4]. Row 3: Result of APAP [5]. Row 4-7: Results of the linear, multiquadric, TPS, cubic RBFs respectively.
(b) chess girl
(c) fence
re
-p
(a) temple
ro
of
Journal Pre-proof
(f) railtracks
(h) intersection
(i) tower
(k) stairs
(l) work table
na
Jo
ur
(g) building
(e) garden
lP
(d) cabin
(j) wall
Figure 8: Databases for quantitative comparison.
18
Journal Pre-proof
re
-p
ro
of
experiments, in which the testing images are collected from the public available databases [17, 18, 5, 19, 20, 34] or captured by ourselves. Only the overlapping regions are extracted to compute SSIM of each aligned image pair. The scores of all the seven tested approaches are listed in Table 3. For the upper six cases with either small or large parallax, all the six localadaptive approaches could achieve high alignment accuracy and get better scores than global alignment. Input images of the following six cases contain weakly-textured or repetitively-textured areas, which is very likely to produce incorrect matches. The alignment accuracy of CPW and APAP are influenced by these matching errors, such that their scores are higher then global alignment but still not good enough. Compared to the existing approaches, better outlier removal scheme is provided by the proposed approach. Therefore, alignment results using all the four RBFs get significantly higher scores. They all achieve satisfactory alignment accuracy, and hence resulted similar scores.
lP
Table 3: Comparison of alignment quality using SSIM. Global
CPW
APAP
Linear
Multiquadric
TPS
Cubic
temple [17] chess girl [18] fence [20] cabin garden [19] railtracks [5]
0.6877 0.7336 0.8062 0.5072 0.7685 0.5359
0.8610 0.7833 0.8790 0.9130 0.8067 0.9158
0.8920 0.7895 0.8863 0.8899 0.8068 0.8809
0.9283 0.8011 0.9029 0.9657 0.8444 0.9334
0.9301 0.7971 0.9040 0.9660 0.8352 0.9294
0.9261 0.7993 0.8966 0.9604 0.8352 0.9330
0.9284 0.7943 0.8964 0.9616 0.8165 0.9220
building intersection [20] tower wall stairs work table [34]
0.7361 0.4715 0.8755 0.7145 0.5427 0.6149
0.9154 0.5763 0.9131 0.9039 0.8789 0.8271
0.9042 0.5418 0.9257 0.8780 0.9178 0.8906
0.9705 0.7799 0.9855 0.9649 0.9461 0.9586
0.9709 0.7874 0.9855 0.9662 0.9458 0.9579
0.9691 0.7557 0.9846 0.9646 0.9458 0.9582
0.9719 0.7872 0.9842 0.9609 0.9474 0.9411
Jo
ur
na
Database
5.3. Alignment Quality Evaluation by Stages The main contribution of the proposed method lies on three aspects: 1) the RBF-based error function, 2) the local outlier removal technique, and 3) the overfitting suppression strategy. For more detailed alignment quality evaluation, we quantitatively compare the SSIM index of the alignment results at different processing stages. As shown in Table 4, the results of global alignment are presented as baselines. For all the four RBF types, the local alignment results only employing the RBF-based error function are listed 19
Journal Pre-proof
-p
ro
of
in the first row of each case. Significantly higher scores are achieved compared with those of global alignment. By additionally introducing the local outlier removal, the reliability of the computed error functions is enhanced. Therefore, better scores (presented in the second row of each case) could be obtained for most databases. Without sacrificing the alignment accuracy, the overfitting suppression strategy is adopted to better preserving the global projectivity of the resulted panoramas. This is verified by the very closing scores of the posterior two stages (the second and third row of each case). It is worth noting that the scores obtained after all the three stages can slightly varies due to the randomness brought in by the RANSAC processing within the second stage. Therefore, the corresponding scores listed in Table 3 and Table 4 are approximately equal but still with very small differences.
lP
re
5.4. Efficiency Evaluation by Stages The computational complexity of the proposed method has been analyzed in Section 4. For convenience of efficiency evaluation, it can be divided into three stages:
Jo
ur
na
1. Parameter estimation (PE) of the projection error function, together with local outlier removal. The feature correspondences and the previously estimated global transformation are taken as input, and the analytical error function is generated as output. The computational complexity of this stage is O (n3 ). 2. Numerical computation (NC) of the error function values. The estimated weighting parameters of the error function are taken as input, and the pixel-wise projection errors are generated as output. The computational complexity of this stage is O (nCx Cy + XY ). 3. Overfitting suppression (OS) to reduce the abnormal distortion caused by extrapolation issue, together with the construction of LUTs. The pixel-wise projection errors and the global transformation are taken as input, and the accurate mapping relationship between images is generated as output. The computational complexity of this stage is O (XY ). We evaluate the efficiency of our method in form of elapsed time used to align different image. The experimental environment is with a 3.7 GHz Intel i7 CPU and a 32 GB memory. The timing starts after feature matching and ends before image reprojection. For all the four RBF types, the elapsed time 20
Journal Pre-proof
Database
Global
Stage
Linear
Multiquadric
TPS
Cubic
temple
0.6891
RBF-based error function + Local outlier removal + Overfitting suppression
0.9086 0.9178 0.9267
0.9085 0.9195 0.9284
0.9053 0.9141 0.9242
0.9067 0.9169 0.9268
chess girl
0.7325
RBF-based error function + Local outlier removal + Overfitting suppression
0.7973 0.7964 0.7990
of
0.7943 0.7974 0.7992
0.7958 0.7918 0.7948
fence
0.8059
RBF-based error function + Local outlier removal + Overfitting suppression
0.9009 0.9037 0.9038
0.8998 0.9031 0.9040
0.8987 0.9022 0.9019
0.8924 0.8953 0.8978
cabin
0.5023
-p
Table 4: Comparison of alignment quality by stages using SSIM
RBF-based error function + Local outlier removal + Overfitting suppression
0.9544 0.9691 0.9662
0.9519 0.9621 0.9603
0.9537 0.9651 0.9615
garden
0.7685
RBF-based error function + Local outlier removal + Overfitting suppression
0.8316 0.8442 0.8466
0.8294 0.8319 0.8346
0.8277 0.8423 0.8455
0.8205 0.8194 0.8210
railtracks
0.5327
RBF-based error function + Local outlier removal + Overfitting suppression
0.9149 0.9326 0.9339
0.9102 0.9266 0.9290
0.9156 0.9304 0.9328
0.9077 0.9192 0.9218
building
0.8172
RBF-based error function + Local outlier removal + Overfitting suppression
0.9519 0.9735 0.9729
0.9513 0.9740 0.9733
0.9436 0.9712 0.9707
0.9610 0.9730 0.9721
intersection
0.4825
RBF-based error function + Local outlier removal + Overfitting suppression
0.7234 0.8081 0.7965
0.7181 0.8143 0.8003
0.7137 0.7884 0.7756
0.7223 0.8230 0.8019
0.8767
RBF-based error function + Local outlier removal + Overfitting suppression
0.9752 0.9817 0.9799
0.9757 0.9818 0.9801
0.9772 0.9852 0.9821
0.9785 0.9860 0.9839
0.7003
RBF-based error function + Local outlier removal + Overfitting suppression
0.9592 0.9631 0.9658
0.9609 0.9638 0.9667
0.9586 0.9638 0.9659
0.9600 0.9569 0.9614
stairs
0.5832
RBF-based error function + Local outlier removal + Overfitting suppression
0.9247 0.9508 0.9518
0.9232 0.9502 0.9515
0.9163 0.9534 0.9546
0.9285 0.9456 0.9476
work table
0.6135
RBF-based error function + Local outlier removal + Overfitting suppression
0.9259 0.9614 0.9600
0.9223 0.9577 0.9566
0.9050 0.9611 0.9597
0.9158 0.9450 0.9451
wall
ro
0.9541 0.9695 0.9660
re
lP
na
ur
Jo
tower
21
0.7978 0.7956 0.7978
Journal Pre-proof
separated by stages are recorded and listed in Table 5. For each database, the time consumption of the three stages are usually within one order of magnitude. Consistent with the analysis of computational complexity in Section 4, the time consumption of the PE stage mainly depend on the matching number, and that of the OS stage mainly depends on the image size, while the NC stage is related to the both variables.
re
-p
ro
of
5.5. Comparison of Computational Efficiency The computational efficiency of the proposed approach is further evaluated by comparison with the other local-adaptive approaches, namely, CPW [4] and APAP [5]. The time consumption of different approaches under the same experimental setting is presented in Table 6. The elapsed time of our method using four different RBF types are constantly much lower than the other two methods for all the listed cases. Although the four RBFs are with similar performance, their computational efficiency still can be sorted in descending order as: 1) linear, 2) multiquadric, 3) TPS, 4) cubic.
lP
6. Conclusions
Jo
ur
na
A local-adaptive and outlier-tolerant image alignment method is proposed in this paper. First, by applying RBF approximation, a projection error function is constructed to compensate the misalignment caused by global transformation. Second, an outlier removal technique based on Bayesian inference is introduced to enhance the reliability of feature correspondences. Third, a weight assignment strategy is presented to further address the overfitting issues caused by extrapolation. The proposed approach achieves satisfactory alignment quality with high outlier tolerance and low computational cost. Its performance is comprehensively evaluated by a group of comparative experiments on several challenging case. We will concentrate on extending our work to multi-image and video alignment in the future. References [1] R. Szeliski, Image alignment and stitching: A tutorial, Foundations and R in Computer Graphics and Vision 2 (1) (2006) 1–104 (2006). Trends [2] M. Brown, D. G. Lowe, Recognising panoramas., in: ICCV, Vol. 3, 2003, p. 1218 (2003). 22
Journal Pre-proof
Image resolution
Matching number
Stage
Linear
Multiquadric
TPS
Cubic
temple [17]
730 × 487
841
PE NC OS
0.0476 0.0289 0.0465
0.0560 0.0343 0.0440
0.0585 0.0584 0.0446
0.0999 0.1450 0.0410
chess girl [18]
1008 × 755
913
PE NC OS
0.0411 0.0442 0.0905
0.0427 0.0391 0.0518
0.0593 0.0661 0.0504
0.0901 0.1496 0.0533
fence [20]
816 × 1088
682
PE NC OS
0.0275 0.0452 0.1156
0.0366 0.0519 0.1091
0.0557 0.1097 0.1112
0.0666 0.3086 0.1218
cabin
1280 × 960
3646
PE NC OS
2.1159 0.1987 0.1200
2.1444 0.2393 0.1190
2.2305 0.5440 0.1361
2.5006 1.6796 0.1174
PE NC OS
0.5319 0.2951 0.4317
0.5784 0.3091 0.3499
0.6346 0.8354 0.3721
0.7356 2.8312 0.3683
PE NC OS
2.6712 0.3546 0.2683
2.4663 0.4055 0.2425
2.3238 1.1024 0.2385
2.7024 3.9339 0.2705
963
PE NC OS
0.0728 0.0450 0.1105
0.0508 0.0486 0.0881
0.0674 0.0929 0.0835
0.0507 0.2765 0.1009
933
PE NC OS
0.0563 0.0655 0.1759
0.0597 0.0692 0.1689
0.0812 0.1412 0.1581
0.0647 0.4627 0.1568
1863
PE NC OS
0.2977 0.1130 0.1536
0.2697 0.1330 0.1137
0.2937 0.2798 0.1271
0.3871 0.8093 0.1187
0.1719 0.1256 0.2245
0.1953 0.1520 0.1920
0.1728 0.3387 0.2001
0.2182 1.0779 0.1889
2000 × 1500
tower
1155 × 876
Jo
intersection [20]
ur
720 × 960
building
960 × 1280
-p
re
railtracks [5]
2821
lP
1440 × 1920
4147
na
garden [19]
of
Database
ro
Table 5: Elapsed time using different RBF types (s).
wall
960 × 1280
1311
PE NC OS
stairs
1080 × 1440
4676
PE NC OS
2.5741 0.2143 0.1390
2.4803 0.2716 0.1300
2.5326 0.5951 0.1423
3.0713 1.8992 0.1233
work table [34]
2160 × 1440
1801
PE NC OS
0.2456 0.1436 0.2542
0.2345 0.1763 0.2182
0.2973 0.3999 0.2034
0.3073 1.2468 0.2009
23
Journal Pre-proof
Table 6: Comparison of overall elapsed time (s). CPW
APAP
Linear
Multiquadric
TPS
Cubic
temple [17] chess girl [18] fence [20] cabin garden [19] railtracks [5] building intersection [20] tower wall stairs work table [34]
0.9900 1.4105 3.1626 3.5643 13.4106 8.8983 2.3925 5.5854 3.4254 5.9673 3.9964 8.5521
2.5164 3.7178 7.9867 18.2352 66.4439 54.4424 5.6675 12.5874 16.7388 25.8513 25.5696 31.1640
0.1230 0.1759 0.1884 2.4345 1.2587 3.2941 0.2283 0.2977 0.5643 0.5220 2.9274 0.6434
0.1343 0.1336 0.1977 2.5027 1.2373 3.1144 0.1876 0.2979 0.5163 0.5392 2.8819 0.6290
0.1616 0.1758 0.2765 2.9106 1.8421 3.6646 0.2438 0.3805 0.7005 0.7116 3.2700 0.9007
0.2859 0.2930 0.4970 4.2975 3.9351 6.9069 0.4281 0.6843 1.3151 1.4851 5.0938 1.7549
ro
of
Database
re
-p
[3] M. Brown, D. G. Lowe, Automatic panoramic image stitching using invariant features, International Journal of Computer Vision 74 (1) (2007) 59–73 (2007).
lP
[4] F. Liu, M. Gleicher, H. Jin, A. Agarwala, Content-preserving warps for 3d video stabilization, in: ACM Transactions on Graphics (TOG), Vol. 28, ACM, 2009, p. 44 (2009).
ur
na
[5] J. Zaragoza, T.-J. Chin, Q.-H. Tran, M. S. Brown, D. Suter, Asprojective-as-possible image stitching with moving dlt, Pattern Analysis and Machine Intelligence, IEEE Transactions on 36 (7) (2014) 1285–1298 (2014).
Jo
[6] T. Igarashi, T. Moscovich, J. F. Hughes, As-rigid-as-possible shape manipulation, in: ACM transactions on Graphics (TOG), Vol. 24, ACM, 2005, pp. 1134–1141 (2005). [7] L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content-driven video-retargeting, in: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, IEEE, 2007, pp. 1–6 (2007). [8] Y.-S. Wang, C.-L. Tai, O. Sorkine, T.-Y. Lee, Optimized scale-andstretch for image resizing, in: ACM Transactions on Graphics (TOG), Vol. 27, ACM, 2008, p. 118 (2008). [9] R. Carroll, M. Agrawala, A. Agarwala, Optimizing content-preserving
24
Journal Pre-proof
projections for wide-angle images, ACM Trans. Graph. 28 (3) (2009) 43–1 (2009). [10] K. He, H. Chang, J. Sun, Rectangling panoramic images via warping, ACM Transactions on Graphics (TOG) 32 (4) (2013) 79 (2013). [11] Y. Nie, T. Su, Z. Zhang, H. Sun, G. Li, Dynamic video stitching via shakiness removing, IEEE Transactions on Image Processing (2017).
ro
of
[12] F. Zhang, F. Liu, Parallax-tolerant image stitching, in: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 2014, pp. 3262–3269 (2014).
re
-p
[13] Y.-S. Chen, Y.-Y. Chuang, Natural image stitching with the global similarity prior, in: European Conference on Computer Vision, Springer, 2016, pp. 186–201 (2016).
lP
[14] K. Lin, N. Jiang, L.-F. Cheong, M. Do, J. Lu, Seagull: Seam-guided local alignment for parallax-tolerant image stitching, in: European Conference on Computer Vision, Springer, 2016, pp. 370–385 (2016).
na
[15] J. Jia, C.-K. Tang, Image stitching using structure deformation, Pattern Analysis and Machine Intelligence, IEEE Transactions on 30 (4) (2008) 617–631 (2008).
Jo
ur
[16] K. Lin, N. Jiang, S. Liu, L.-F. Cheong, M. D. J. Lu, Direct photometric alignment by mesh deformation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2405–2413 (2017). [17] J. Gao, S. J. Kim, M. S. Brown, Constructing image panoramas using dual-homography warping, in: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 2011, pp. 49–56 (2011). [18] W.-Y. Lin, S. Liu, Y. Matsushita, T.-T. Ng, L.-F. Cheong, Smoothly varying affine stitching, in: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 2011, pp. 345–352 (2011). [19] C.-H. Chang, Y. Sato, Y.-Y. Chuang, Shape-preserving half-projective warps for image stitching, in: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, 2014, pp. 3254–3261 (2014). 25
Journal Pre-proof
[20] C.-C. Lin, S. U. Pankanti, K. N. Ramamurthy, A. Y. Aravkin, Adaptive as-natural-as-possible image stitching, in: Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, IEEE, 2015, pp. 1155– 1163 (2015).
of
[21] G. Zhang, Y. He, W. Chen, J. Jia, H. Bao, Multi-viewpoint panorama construction with wide-baseline images, IEEE Transactions on Image Processing 25 (7) (2016) 3099–3111 (2016).
-p
ro
[22] C. Herrmann, C. Wang, R. Strong Bowen, E. Keyder, M. Krainin, C. Liu, R. Zabih, Robust image stitching with multiple registrations, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 53–67 (2018).
re
[23] R. L. Hardy, Theory and applications of the multiquadric-biharmonic method 20 years of discovery 1968–1988, Computers & Mathematics with Applications 19 (8-9) (1990) 163–208 (1990).
lP
[24] Z. Majdisova, V. Skala, Radial basis function approximations: comparison and applications, Applied Mathematical Modelling 51 (2017) 728–743 (2017).
na
[25] G. E. Fasshauer, Meshfree approximation methods with MATLAB, Vol. 6, World Scientific, 2007 (2007).
Jo
ur
[26] N. Arad, N. Dyn, D. Reisfeld, Y. Yeshurun, Image warping by radial basis functions: Application to facial expressions, CVGIP: Graphical models and image processing 56 (2) (1994) 161–172 (1994). [27] J. Pilet, V. Lepetit, P. Fua, Fast non-rigid surface detection, registration and realistic augmentation, International Journal of Computer Vision 76 (2) (2008) 109–122 (2008). doi:10.1007/s11263-006-0017-9. URL http://dx.doi.org/10.1007/s11263-006-0017-9 [28] K. Rohr, H. S. Stiehl, R. Sprengel, W. Beil, T. M. Buzug, J. Weese, M. Kuhn, Point-based elastic registration of medical image data using approximating thin-plate splines, in: Visualization in Biomedical Computing, Springer, 1996, pp. 297–306 (1996).
26
Journal Pre-proof
[29] R. Sprengel, K. Rohr, H. S. Stiehl, Thin-plate spline approximation for image registration, in: Engineering in Medicine and Biology Society, 1996. Bridging Disciplines for Biomedicine. Proceedings of the 18th Annual International Conference of the IEEE, Vol. 3, IEEE, 1996, pp. 1190–1191 (1996).
of
[30] A. Bartoli, A. Zisserman, Direct Estimation of Non-Rigid Registrations, BMVA, 2004, pp. 899–908 (2004).
ro
[31] J. Ashburner, A fast diffeomorphic image registration algorithm, Neuroimage 38 (1) (2007) 95–113 (2007).
-p
[32] N. Arad, D. Reisfeld, Image warping using few anchor points and radial functions, in: Computer Graphics Forum, Vol. 14, Wiley Online Library, 1995, pp. 35–46 (1995).
lP
re
[33] F. L. Bookstein, Principal warps: Thin-plate splines and the decomposition of deformations, IEEE Transactions on Pattern Analysis & Machine Intelligence (6) (1989) 567–585 (1989).
na
[34] J. Li, Z. Wang, S. Lai, Y. Zhai, M. Zhang, Parallax-tolerant image stitching based on robust elastic warping, IEEE Transactions on Multimedia 20 (7) (2018) 1672–1687 (2018).
ur
[35] L. Zagorchev, A. Goshtasby, A comparative study of transformation functions for nonrigid image registration, IEEE transactions on image processing 15 (3) (2006) 529–538 (2006).
Jo
[36] R. Beatson, M. Powell, A. Tan, Fast evaluation of polyharmonic splines in three dimensions, IMA Journal of Numerical Analysis 27 (3) (2007) 427–450 (2007). [37] M. Bozzini, L. Lenarduzzi, M. Rossini, Polyharmonic splines: An approximation method for noisy scattered data of extra-large size, Applied Mathematics and Computation 216 (1) (2010) 317–331 (2010). [38] Z. Zhang, X. Yang, Y.-R. Li, G. Chen, Sparse constrained transformation model based on radial basis function expansion: Application to cardiac and brain image registration, IEEE Access 6 (2018) 42631–42646 (2018).
27
Journal Pre-proof
[39] Z. Zhang, Parameter estimation techniques: A tutorial with application to conic fitting, Image and vision Computing 15 (1) (1997) 59–76 (1997). [40] A. De Boer, M. Van der Schoot, H. Bijl, Mesh deformation based on radial basis function interpolation, Computers & structures 85 (11-14) (2007) 784–795 (2007).
of
[41] P. S. Heckbert, Fundamentals of texture mapping and image warping (1989).
ro
[42] M. A. Fischler, R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM 24 (6) (1981) 381–395 (1981).
re
-p
[43] D. G. Lowe, Distinctive image features from scale-invariant keypoints, International journal of computer vision 60 (2) (2004) 91–110 (2004).
lP
[44] kyamagu, Collection and a development kit of matlab mex functions for opencv library, http://kyamagu.github.io/mexopencv/, online; accessed 28-Jan-2019 (Jan. 2019).
Jo
ur
na
[45] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, Image Processing, IEEE Transactions on 13 (4) (2004) 600–612 (2004).
28
Journal Pre-proof
Highlights: Natural images usually cannot be accurately aligned using traditional approaches. Up-to-date approaches suffer from unreliable matches and high computational cost. Radial basis function approximation is used to efficiently align
of
natural images.
ro
Incorrect matches are removed by analyzing parameters of the
-p
approximation.
lP
Jo ur
na
deformations.
re
Overfitting issues are addressed by smoothly weighting the image
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8