- Email: [email protected]
Super-resolution AFM imaging based on compressive sensing
Journal Pre-proofs Full Length Article Super-resolution AFM imaging based on compressive sensing Guoqiang Han, Luyao Lv, Gaopeng Yang, Yixiang Niu PII: DOI: Reference:
S0169-4332(19)34048-6 https://doi.org/10.1016/j.apsusc.2019.145231 APSUSC 145231
To appear in:
Applied Surface Science
Received Date: Revised Date: Accepted Date:
11 August 2019 28 November 2019 29 December 2019
Please cite this article as: G. Han, L. Lv, G. Yang, Y. Niu, Super-resolution AFM imaging based on compressive sensing, Applied Surface Science (2019), doi: https://doi.org/10.1016/j.apsusc.2019.145231
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
© 2019 Published by Elsevier B.V.
Super-resolution AFM imaging based on compressive sensing Guoqiang Hana, b, *, Luyao Lva, Gaopeng Yanga and Yixiang Niua a School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou 350 108, People’s Republic of China b Key Laboratory of Fluid Power and Intelligent ElectroHydraulic Control, Fuzhou U niversity, Fuzhou 350108, People’s Republic of China Email: [email protected]
Abstract: Atomic force microscopy (AFM) is a powerful and ultra-precision instrument in nano-scale, which is widely used in many fields. It is a complex and timeconsuming process for AFM imaging. Most of the original AFM images are with low resolution. For nano-scale measurement and imaging, it is very important to obtain super-resolution images. In most cases, super-resolution imaging takes a long time and the quality of imaging is unsatisfactory. In this regard, a novel super-resolution imaging method based on compressed sensing (CS) technology is proposed in AFM. In the experiment, six samples with different morphology were used to test the effect of superresolution image reconstruction with different upscaling factors (2, 3 and 4). The quality of reconstructed image is analyzed and evaluated by image evaluation metrics (PSNR and SSIM). The relationship between the reconstruction quality of different images and the actual TV or TV/R of sample images is analyzed, which can provide a preliminary basis for predicting the imaging quality. Comparing with other superresolution imaging methods, the proposed method has achieved better imaging effect both visually and quantitatively. In summary, super-resolution imaging method based on CS not only has high imaging quality but also has high speed. Keywords: Atomic force microscopy (AFM); Compressed sensing (CS); Super resolution (SR); Measurement matrix;
1. Introduction The super-resolution imaging is a technique that aims to obtain a high-resolution image from a low -resolution image or a low-resolution data [1]. It is widely used in video surveillance, image printing, criminal investigation analysis, medical image processing, satellite imaging and other fields. The super-resolution imaging methods are mainly based on hardware or software. Super-resolution imaging through hardware is achieved by reducing the size of pixels and increasing the size of sensors. Because of the complexity of manufacturing process and cost of sensors and optical components, it is difficult to obtain high-resolution images in many situations [2]. Therefore, software-based methods are usually preferred to the hardware-based solutions. And the software-based methods are mainly divided into three kinds. First of all, the interpolation-based approach is a valid way to realize super-resolution imaging. The interpolation-based methods, such as bilinear and bicubic interpolation, require less computation [3, 4]. But it will result in highly blurred edges and many image details are always lost. In addition, another method is to use the reconstruction-based approach to realize super-resolution imaging, such as iterative back projection (IBP), projection onto convex sets (POCS) and maximum a posteriori (MAP) [5-7]. The reconstructionbased method is a course in which prior knowledge is added to the imaging process as a constraint to realize super-resolution imaging [8]. Unfortunately, the convergence speed of this method is slow. Last but not least, super-resolution imaging can also be achieved through learning-based methods. Learning-based methods are roughly divided into neighborhood embedding method (NE), sparse representation method (SR) and the convolutional neural network method(CNN) [9-11]. The relationship between
low-resolution image and high-resolution image is identified by the external database. However, a large amount of calculation and storage space is also required for this method. Atomic Force Microscope (AFM) is usually used to measure the topography of conductive, nonconductive samples and semiconductor [12, 13]. On the premise of following Nyquist-Shannon sampling theorem, traditional AFM imaging often takes a lot of time. Irreversible damage is often caused by long-time contact and interaction between probe and sample. In particular, probes are prone to wear or damage. The quality of AFM imaging will be greatly reduced. Accordingly, low-resolution images are often obtained in traditional AFM. Even though sometimes high-resolution images can be obtained through optimization, their enhancement is extremely limited. Therefore, fast super-resolution technology should be applied to AFM. Applying CS technique with specially designed measurement matrices can effectively speed up AFM imaging [14, 15]. The super-resolution technique base on CS can be regarded as a reconstruction-based approach or a learning-based approach in super-resolution imaging. Learning-based CS method is to identify dictionaries that represent highresolution patches in a sparse way, which introduce additional information of similar structures in images into the dictionary of CS framework [16]. But it takes a long time to train the dictionary. Our proposed method belongs to a reconstruction-based approach and can be used in AFM. Although other CS-based reconstruction algorithms can achieve super-resolution, they may not be suitable for AFM because the measurement principles of different instruments are different. Super-resolution imaging based on the proposed method is achieved by optimized measurement matrix. This method does not require additional dictionary training and other preprocessing. Compressive sensing theory (CS) theory shows that if the signal is sparse in a certain area, it can be reconstructed from fewer samples than the original signal [17]. Redundancy in signals is exploited. The low-dimensional projection of the highdimensional signal is obtained from the measurement matrix, and then the original signal can be restored almost perfectly from a small number of measured values by using appropriate reconstruction algorithm. For the super-resolution algorithm based on CS, better results can be obtained by choosing the appropriate measurement matrix and reconstruction algorithm. In our work, a large number of experiments have been carried out to prove the advantages of CS in super-resolution AFM imaging by comparing with different superresolution imaging methods, especially when the upscaling factor is large. In addition, imaging effect of six samples with different morphology was analyzed. TVAL3 algorithm was used to reconstruct the AFM images in super-resolution imaging based on CS. In a word, the CS technology was a promising method to realize super-resolution AFM imaging.
2. Super-resolution imaging methods in AFM 2.1 Imaging principle of traditional AFM The principle of AFM is to measure the longitudinal displacement of the tip by
controlling the constant interaction force between the tip and the atoms on the sample surface [18]. Van der Waals force is the general force between atoms. The working modes of the AFM can be divided into three main categories: contact mode, non-contact mode and tapping mode [19]. And the measurement mode of the AFM microscope is point-to-point scanning. The choice of tip scanning patterns in AFM depends on different scenarios. Grating scanning pattern is commonly used in traditional AFM. In general, the probe starts at the upper left corner of the sample and sweeps a horizontal line to the right. Next, the probe is quickly swept back to the next point on the left, and then the second horizontal line is swept. On the basis of this fixed path and sequence, the whole sample can be scanned until the last horizontal line. According to the different resolution, the corresponding measurement points are set by step size to meet the requirements. A diagram of raster scanning is shown in Fig. 1. In traditional AFM scanning, both scanning and imaging are performed simultaneously. In other words, the scanning mode is line-by-line real-time imaging. Scanning and imaging end at the same time. Therefore, the scanning time is the imaging time. On account of traditional AFM scanning is extremely time-consuming, it is urgent to find a faster imaging method.
Fig. 1 Raster Scanning Routine of AFM.
2.2 Super-resolution imaging in CS-AFM Using CS theory, the signal can be reconstructed from fewer samples than the original signal. In other words, Fast AFM imaging can be achieved through compressed sensing (CS) with low sampling rate. Consider an unknown signal X of length N containing only K nonzero elements. It is called K-sparse. If it takes M times liner measurements to sample the signal X, it means it takes fewer measurements than signal dimension. (1) Y X where Y is a measurement of length M with M>N. is an M N measurement matrix. Since the process is non-adaptive, the measurement matrix is selected beforehand. CS can recover the signal X from significantly fewer measurements, only M cK log
N , K
suggesting the potential of significant cost reduction in digital data acquisition. And c is a small constant and K denotes the number of non-zero elements in the signal X. Many signals are not sparse. They can be not directly used in compressive sensing. Fortunately, most natural signals are compressible, that is the signal X could be transformed into sparse form through sparse basis .
X (2) where is the sparse representation of signal X. is an N N basis transform matrix. Then, the Eq. (1) changes into (3) Y X A where A is M N sensing matrix which should satisfy the restricted isometry property (RIP) [20]. Compressed sensing theory mainly includes three aspects: sparse representation, measurement matrix and reconstruction algorithm. In CS theory, the measured signal is required to be sparse or compressible. In practice, natural signals, most of which are not strictly sparse, are mostly compressible. Therefore, the signal can be sparse, such as discrete cosine change, discrete wavelet transformation and so on. Common compressed sensing measurement matrices can be divided into two categories: deterministic measurement matrices and random measurement matrices. Most of the energy of the original signal can be collected by the measurement matrix. Suitable algorithms are needed to solve the above undetermined equation. The sparse reconstruction algorithm in CS can be used. Sparse reconstruction algorithm can be classified into convex optimization algorithm and greedy iteration algorithm. TV Minimization by Augmented Lagrangian and Alternating Direction Algorithms (TVAL3) belongs to the convex optimization algorithms. TVAL3 effectively combines an alternating direction technique with a nonmonotone line search to minimize the augmented Lagrangian function at each iteration which could significantly accelerate the convergence [21]. In CS problems, the precise preservation of edges or boundaries is achieved by TV, which makes the reconstructed image clearer. The advantage of TV is that the sparse images can be reconstructed by TV. The noiseless discrete TV model for CS reconstruction can be written as
min TV(X) subject to Y=AX
(4)
where TV X i , j
X
X i , j X i , j 1 X i , j 2
i 1, j
2
(5)
where Xi, j is the value of the matrix at pixel (i, j) and X represents the signal that needs to be reconstructed. Since most of the signals in nature are sparse in the gradient domain, there is no need to sparse initial image signals when TVAL3 algorithm is used as the reconstruction algorithm, which greatly simplifies the imaging process.
Fig. 2 Super-resolution AFM Imaging by using Compressed Sensing.
As shown in Fig. 2, the low-resolution image or data is obtained by raster scanning with large step in AFM, and the final high-resolution image is obtained in CS-AFM through reconstruction algorithm. In reality, super-resolution imaging is an inverse process of down-sampling. The CS technology includes compression process and reconstruction process. The compression process in CS is assumed to be the downsampling process, which is a mathematical process of getting low-resolution image or data from high-resolution image or data. In the actual AFM measurement, the highresolution image is unknown and need to be reconstructed. In addition, the purpose of reconstruction process is to reconstruct unknown data points and finally obtain a highresolution image. As illustrated in Fig. 3, CS technology is used to realize the superresolution imaging of AFM image. Given an initial low-resolution image or undersampling data, the corresponding high-resolution image can be correctly restored under certain conditions through CS method. Y R M represents the low-resolution image,
X R N represents the high-resolution image, XK represents the blank template for * N high-resolution images and X R represents the reconstructed high-resolution
image. The pixels in the low-resolution image Y are projected into the blank template of the required high-resolution image according to a certain rule. The rest of the pixel values are set to zero. Besides, the filled blank template XK is equivalent to the undersampling of the required high-resolution image X. The under-sampled sampling point is selected by a certain rule. The low-resolution image Y is used as the input image of the compressed sensing algorithm. Low-resolution image Y and the blank template X K is known. Therefore, the measurement matrix can be obtained by equation (1). Constructing a special measurement matrix is the focus of CS method, which extracts certain rows from the unit matrix according to the position of the pixels in the blank template of the highresolution image. In other words, the measurement matrix is a variant of the unit matrix.
Its subordination is shown as:
mn
0 1 0 0 0 1 0 0 0
0 0 0 0 1 0
(6)
Each measurement of CS typically relies on a linear combination of many elements of the signal. However, they are difficult to be applied in AFM due to the point-like nature of the AFM probe tip. The AFM probe tip only measures a single pixel at a time. Therefore, a specially designed measurement matrix is needed to measure the AFM sample. An identity matrix with some of its row removed is a valid choice in AFM application as the measurement matrix [22]. In each row of , there is only a single one and zeros elsewhere. Such a measurement matrix ensures that only a single pixel of the image is required for each measurement. This unique measurement matrix could seem as AFM tip trajectory. By substituting low-resolution image Y and measurement matrix into equation (1), the reconstructed high-resolution image X * can be obtained by the TVAL3 algorithm. As shown in Fig. 3, the specific implementation of super-resolution imaging with the upscaling factor of 2 is achieved by using a special measurement matrix. Low-
34 158 resolution image Y is known. A practical representation of measurement 2 213 matrix can be obtained from Fig. 3. Specific manifestation is shown as 1 0 0 0
0 0 0 0
0 1 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 1 0
0 0 0 0
0 0 0 1
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
(7)
37 99 156 96 60 103 141 96 can be obtained by introducing High-resolution image X * 6 110 209 88 64 101 132 95
these two known conditions into equation (1).
Fig. 3 Super-resolution imaging schematic based on compressed sensing with the upscaling factor of 2.
2.3 Imaging time for CS-AFM In order to evaluate the performance of compressed sensing algorithm, the AFM imaging time needs to be reasonably estimated. The imaging time in CS-AFM includes scanning time and reconstruction time. The imaging time of traditional AFM is only scanning time. In the scanning process of AFM, it should be noted that the tip of the AFM can only measure one point at a time. In each measurement, the tip is pulled down until the sample is touched, height data is measured, and then retrieved and moved to the next point in the sequence. The scan time can be computed approximately as follows: (8) t Sr T where Sr=M/N is the sampling rate, and T is the conventional AFM acquisition time. And M, N denotes the length of measured value, the length of the original signal. After introducing CS technology, a low-resolution image scanned by AFM can be reconstructed to a high-resolution image. If the sampling rate is low, the scanning time can be saved to a large extent. The reconstruction time usually takes less 20 seconds by using a computer with general hardware configuration. The time estimation of the image obtained by three different methods is shown in Fig. 4. The scanning rate is set to 1 Hz for the low-resolution (256×256) and the scanning rate is set to 0.25 Hz for the high-resolution (1024×1024). In other words, 256 measurement points are scanned per second in AFM. In this way, the scanning speed of low-resolution image is the same as that of high-resolution image. Scanning a low-resolution image in AFM is faster, but the quality of the image is poor and the details are blurred. Even though the quality of the high-resolution image scanned in traditional AFM is satisfactory, it takes a lot of time. After introducing CS technology, satisfactory image quality can be obtained in a short time. The imaging time can be saved while the imaging quality is guaranteed by using CS.
(a)
(b)
(c)
Fig. 4 The R-6G images obtained by three different methods. (a) Low-resolution image (256×256) obtained by traditional AFM. Imaging time=256s. (b) High-resolution (1024×1024) image obtained by traditional AFM. Imaging time=4096s. (c) High-resolution image (1024×1024) obtained by using CSAFM. Imaging time=276s.
3. Traditional super-resolution methods 3.1 Bicubic interpolation In the early image interpolation algorithms, nearest neighbor interpolation is widely used because of uncomplicated calculation. But jagged edges and mosaics are produced by using Bicubic. The subsequent bilinear interpolation method can make up for the defect of the nearest neighbor interpolation method and solve the above problems, but the high-frequency part and details of the image will be weakened. In super-resolution imaging, the bicubic interpolation method is superior to the low-order interpolation method when the magnification is large. The calculation of the pixel value not only considers the influence of the surrounding pixel value, but also the change rate of the pixel value between adjacent points into the pixel value of the pixel to be calculated. Assuming that the A size of the source image is m×n, the size of the target image B is M×N after scaling the K times. Each pixel of A is known, and B is unknown. To get the value of each pixel (X, Y) in the target image B, it is necessary to find the corresponding pixel (x, y) in the source image A. According to the proportional relation x/X = m/M = 1/K, the corresponding coordinates of B (X, Y) on A are A (x, y) = A (X/K, Y/K). Then, the pixel values (X, Y) of the target image B are calculated according to the weight of the last 16 pixels in the source image A, which is calculated based on the bicubic basis function [4]. The value of B pixels (x, y) is equal to the weighted superposition of 16 pixels. And data are fitted by bicubic interpolation using a specific basis function [23]. The bicubic interpolation basis function is as follows: 3 2 w 2 w 1 3 2 S ( w) w 5 w 8 w 4 0
w 1 1 w <2 w 2
(9)
where S is the weight corresponding to the above 16 pixels. And w denotes the distance from pixel (x, y) in image A to pixel (X, Y) in image B. 3.2 Iterative back-projection Iterative back projection(IBP) is a kind of spatial super-resolution imaging method [5]. In this approach, the high-resolution image is finally obtained by iterative back projection of the errors of the simulated low-resolution image and the observed lowresolution image. And degradation model w is set for initial estimated image, and k simulated low-resolution images Li (i=1,2,…,k) are generated according to degraded parameters: the deviation between the simulated low-resolution image and the original n low-resolution image is projected back to the initial estimation Yˆ of the high-
resolution image by using back-projection operator HBP. The high-resolution estimation n image Yˆ obtained by the nth iteration is Lˆn by using degradation model w. The
estimated image is updated by iteration formula until the error meets the requirement
[24]. The iterative process is as follows:
Yˆ n 1 Yˆ n H BP L wLˆn
(10)
where Yˆ n , Yˆ n 1 are high resolution images obtained by the n, n+1 iterations, Lˆn is the approximation of Y after n iterations to simulate low resolution images and HBP is a back projection kernel function, which can correctly determine the contribution of errors. Low computational complexity is the characteristic of using IBP. However, illconditioned and no unique solution are still the problems to be solved in superresolution. The proper back projection operator HBP is difficult to be obtained. And Effectively utilizing the prior knowledge of images is not suitable for this method. 3.3 Projection onto convex sets Projection onto convex sets (POCS) algorithm is a super-resolution imaging method which starts from the initial guessed image and corrects the image successively according to the error [6]. By intersecting the solution space of high-resolution image with a series of constraints (non-negative, energy boundedness, consistency of observation data, local smoothness, etc.) representing the properties of high-resolution image, a smaller solution space can be obtained. Starting from a point in high resolution image space, POCS method is used to find the next point satisfying all constrained convex sets, and finally the estimation of high-resolution image is obtained. For the initial value X 0 , let each constraint set define a convex set projection operator AK . The projection operator AK of convex set is used to iteratively project the initial estimate X0 of high-resolution image to obtain ideal high-resolution image. And the calculation process is as follows: X n AN
A3 A2 A1 X 0
(11)
where X0 denotes the high-resolution image of the initial estimation, Xn represents the final high-resolution image obtained after n iterations and AK is a projection operator that projects arbitrary signal X onto a closed convex set. Making full use of prior knowledge is achieved by projection method of convex sets. However, the convergence process depends on the selection of initial values, and the solution is unstable. 3.4 The Papoulis-Gerchberg (P-G) algorithm The P-G algorithm is used in super-resolution imaging in modified form when lowresolution images are available [25]. After motion estimation, low-resolution image data is projected onto the high-dimensional mesh. The unknown pixel value is initially set to zero. A high-resolution image with initial estimation is obtained. Then the image is passed through a low-pass filter whose cut-off frequency is assumed to be the bandwidth of the input image, which results in blurred images [26]. However, due to filtering, the size of known pixel values also decreases. In order to recover the high
frequency components lost due to filtering, the original values are forcibly restored to the known pixels. The whole process is repeated until convergence. The result is a reconstructed image. Because of the steep cut-off in frequency domain, the highresolution image generated has jagged phenomenon at the edge. In addition, it relies heavily on the fact that the measured (known) pixel value is the value obtained in the reconstructed high-resolution image. In addition, they should be completely free from noise. Therefore, it can’t effectively compensate the measurement data of ambiguity and noise. 3.5 The super-resolution convolutional neural network method As a main learning-based method in SR, deep convolution neural networks (CNNs) have attracted more and more attention in recent years. The deep convolution neural network methods include SRCNN, FSRCNN, ESCPN, VDSR and DRCN [11, 27-30]. SRCNN is a basic convolutional neural network algorithm. In SRCNN algorithm, three convolution layers are used to represent the steps of feature extraction, non-linear transformation and image reconstruction. Firstly, the patches are extracted from lowresolution images, and each patch is represented as a high-dimensional vector. These vectors form a set of feature maps whose number is equal to the dimension of the vector. Then each high-dimensional vector is mapped nonlinearly to another high-dimensional vector. Each mapping vector conceptually represents a high-resolution patch. These vectors form another set of feature maps. Finally, the stitching representation of the above high-resolution patch is aggregated to generate the final high-resolution image. This image is expected to be similar to the original high-resolution image. For the convolution neural network algorithm, the structure of each layer and the specific algorithm are very important, and the image training library should meet the requirements of super-resolution imaging. Four or more layers and larger filter sizes can be used to further improve performance. But the SRCNN method is trained for a single upscaling factor. If a new scaling factor is needed, the model must be retrained.
4. Experiments A train of experiments were conducted using six samples include R-6G, prions peptide, graphene oxide, nematode C. elegans, hexagonal domains in lithium niobite and CS-20NG [31-34]. The first five samples are provided by NT-MDT from Russia. The last sample is a calibrated grating (CS-20NG) scanned in our laboratory. Fig. 5 shows two-dimensional and three-dimensional AFM images obtained in contact mode. The resolution of all images is 1024×1024 pixels. The surface morphologies of the six samples are different. The low- resolution images are down-sampled from these highresolution images. Low-resolution images are the initial images of the experiment. In this way, the difference between the original image and reconstructed image can be evaluated more conveniently. Moreover, the size of the low-resolution image is determined by the upscaling factor. The super-resolution reconstruction effect in CSAFM is analyzed with different upscaling factors of 2, 3 and 4. The low-resolution image was imported into MATLAB to perform experiments. The simulation was carried
out on a typical WIN7 operating system, 3.4 GHz AMD A4 and 8 GB memory personal computer.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 5 The 2D and 3D AFM images of testing samples. (a) R-6G. (b) prions peptide. (c) graphene oxide. (d) nematode C. elegans. (e) hexagonal domains in lithium niobite. (f) CS-20NG.
Another important issue is to evaluate the image quality. When the original image and the contrast image are known, the traditional way is to measure the quality of the contrast image by Peak signal-to-noise ratio (PSNR). PSNR is still an important index of image quality evaluation. PSNR is defined as. h w max I min I 2 1 2 PSNR 10log10 h w 2 I1 i, j I 2 i, j i 1 j 1
(12)
where, h and w are respectively expressed as the number of rows and columns of the image. I1 i, j and I 2 i, j are respectively expressed as the gray value of the original image and the new image at i, j . The structural similarity (SSIM) index is a full-reference quality metric that compares local patterns of pixel intensities that have been normalized for luminance and contrast [35]. SSIM is defined as, SSIM ( x, y )
2
x
2 x
y
C1 2 xy C2
C1 x2 y2 C2 2 y
(13)
1
1 where x N
N
x
i
1
1 N 2 2 is the mean intensity of x . x xi x is the N 1 i 1
standard deviation (the square root of variance) of y . xy is the covariance of x and y . We choose C1 K1R
1 N xi x yi y N 1 i 1
and C2 K2 R . R is the
2
2
dynamic range of the pixel values. K 1 = 0.01 and K 2 =0.03 are small constants. Table 1. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor 2. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
2
2
2
2
2
2
PSNR
50.493
41.840
40.412
48.987
45.143
47.371
SSIM
0.996
0.934
0.957
0.997
0.993
0.992
TIME
7.305
7.690
7.301
7.344
7.326
7.243
PSNR
40.484
37.024
35.662
47.695
37.687
40.902
SSIM
0.972
0.875
0.907
0.988
0.944
0.966
TIME
11.211
22.783
18.932
16.329
22.032
10.258
PSNR
34.919
35.212
33.717
33.246
34.569
38.053
SSIM
0.961
0.826
0.860
0.983
0.907
0.937
TIME
174.685
172.827
189.864
198.572
186.212
161.116
PSNR
37.381
34.586
33.349
41.266
34.901
37.993
SSIM
0.965
0.794
0.851
0.982
0.912
0.932
TIME
157.253
132.650
155.821
164.139
194.570
165.183
PSNR
37.698
35.531
34.096
44.042
35.066
38.471
SSIM
0.968
0.834
0.871
0.995
0.916
0.940
TIME
21.020
19.952
20.597
15.508
18.321
22.343
PSNR
47.497
31.189
27.962
44.964
21.635
29.762
SSIM
0.994
0.729
0.673
0.996
0.744
0.817
TIME
137.374
152.871
149.715
132.297
150.698
140.338
bicubic
IBP
POCS
PG
CS
SRCNN
5. Results and discussion It takes a long time to get a high-resolution image by using AFM scanning. Therefore, introducing CS technology is a good solution. A low-resolution image or data is scanned and then a high-resolution image can be obtained using CS technology. In a word, the high-resolution imaging time in AFM can be shortened by using CS technology. In order to evaluate the performance of CS-based super-resolution imaging in AFM, a series of simulation experiments were carried out using six test samples. The proposed high-resolution imaging method based on CS is compared with the bicubic interpolation, the IBP (Iterative back-projection), the POCS (Projection onto Convex Sets), the P-G (Papoulis-Gerchberg), and the SRCNN (super-resolution convolutional
neural network) method. Tables 1,2 and 3 summarize the results, showing PSNR, SSIM and reconstruction time for reconstructed images. As confirmed in table 1, PSNR and SSIM of all reconstructed images have reached the maximum by using bicubic method, slightly higher than ones by using the proposed CS method when the upscaling factor is 2. As shown in table 1, satisfactory reconstruction results are achieved by employing CS method. For example, the PSNR value of reconstructed nematode image by CS method is 44.042 dB, which indicates that the imaging quality is commendable. Only the PSNR of the reconstructed nematode image is not the best by CS method in table 2. And the PSNR of the reconstructed nematode image achieved by CS method is 45.657 dB, only 0.994 dB lower than by using IBP. The highest PSNR and SSIM of other reconstructed images can be obtained by CS method. Specifically, as shown in table 3, the optimal PSNR and SSIM of all reconstructed images have been obtained by using the proposed method with the upscaling factor of 4. The experimental results are indicative that the proposed imaging method in AFM can realize fast SR imaging with different upscaling factors well, especially with larger upscaling factor. Table 2. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor of 3. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
3
3
3
3
3
3
PSNR
37.871
35.002
33.790
44.271
35.164
38.381
SSIM
0.967
0.767
0.859
0.995
0.915
0.935
TIME
7.441
7.573
7.475
7.675
7.582
7.380
PSNR
39.395
35.991
34.673
46.651
36.479
39.392
SSIM
0.953
0.806
0.889
0.982
0.933
0.963
TIME
17.537
15.737
16.745
20.738
20.707
13.749
PSNR
32.769
34.585
32.999
30.196
33.345
36.861
SSIM
0.949
0.787
0.838
0.977
0.889
0.926
TIME
233.394
245.608
246.622
248.598
243.029
242.747
PSNR
37.059
34.273
33.165
39.049
34.618
37.320
SSIM
0.963
0.724
0.833
0.979
0.911
0.925
TIME
325.855
308.632
280.793
253.383
302.950
254.940
PSNR
40.835
37.028
35.773
45.657
39.461
40.619
SSIM
0.978
0.836
0.904
0.997
0.966
0.964
TIME
20.539
20.437
20.700
20.504
20.413
20.684
PSNR
23.585
23.543
18.984
17.924
11.188
19.879
SSIM
0.881
0.422
0.304
0.962
0.549
0.520
TIME
140.219
138.736
138.193
125.669
141.937
147.369
bicubic
IBP
POCS
PG
CS
SRCNN
As illustrated in Fig. 6, it indicates the reconstruction results of graphene oxide images with different upscaling factors and SR imaging method. The imaging quality by using bicubic, IBP, POCS and PG algorithms decreases with the increase of upscaling factor. With the increase of upscaling factor, the imaging effect of CS method
does not deteriorate significantly. The larger the upscaling factor, the higher the accuracy of the super resolution algorithm is required. The SR imaging method based on CS achieves the best imaging effect with the upscaling factor of 4. The effect of imaging by SRCNN method is worst, which has a great relationship with the suitability of external training library. Due to the diversity of AFM images and the imaging characteristics of AFM, the appropriate training library for AFM images is difficult to obtain. The high-resolution blocks obtained by SRCNN method are from the external image database, which are not real details. This kind of algorithm is also called "image fantasy". The effect of imaging is closely related to the appropriateness of the external training library. If the external library is not suitable, the reconstructed image may have erroneous high-frequency details. Moreover, a large number of computations are also necessary for this method and the imaging effect depends on the accuracy of training algorithm. The super-resolution method based on bicubic is the fastest, and it only takes less than 10 seconds to reconstruct an image. But the image quality obtained by using bicubic is unsatisfactory with the larger upscaling factor. Compared with other algorithms, higher imaging quality can be obtained by using CS algorithm on the premise of guaranteeing imaging time. It only takes about 20 seconds to reconstruct an AFM image. The imaging time includes scanning time and reconstruction time in CSAFM. Super-resolution imaging based on CS method can not only improve the scanning speed of AFM, but also save the reconstruction time of AFM image. Table 3. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor of 4. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
4
4
4
4
4
4
PSNR
32.782
32.120
30.348
41.284
29.863
33.622
SSIM
0.920
0.659
0.765
0.993
0.814
0.867
TIME
7.612
7.645
7.632
7.654
7.674
7.594
PSNR
36.288
33.964
32.496
43.631
33.437
36.638
SSIM
0.919
0.714
0.838
0.969
0.870
0.932
TIME
11.476
11.358
11.397
11.616
11.394
11.782
PSNR
30.597
33.941
32.124
27.660
31.309
35.769
SSIM
0.931
0.752
0.809
0.964
0.850
0.906
TIME
187.355
172.056
172.358
172.239
168.659
198.61
PSNR
36.331
33.776
32.474
36.662
34.207
37.074
SSIM
0.957
0.655
0.798
0.973
0.907
0.916
TIME
479.198
454.612
465.924
418.387
394.657
429.23
PSNR
37.723
35.402
33.863
44.146
36.322
38.189
SSIM
0.961
0.778
0.861
0.996
0.929
0.940
TIME
21.215
21.042
20.878
20.999
20.875
21.158
PSNR
31.547
30.686
28.608
38.784
27.930
32.327
SSIM
0.881
0.604
0.677
0.988
0.728
0.805
TIME
130.737
141.793
141.671
129.387
142.701
144.933
bicubic
IBP
POCS
PG
CS
SRCNN
Figures 7, 8, 9, 10, 11, 12, 13 and 14 show the visual results by using different imaging approaches with different upscaling factors (2, 3 and 4). From Fig. 7, even if the PSNR and SSIM of reconstructed graphene oxide image by using CS method is not the best with the upscaling factor of 2, a better visual imaging quality has been obtained. Although the PSNR is the highest, the graphene oxide image reconstructed by bicubic method is extremely dark and the details are distorted in Fig.7. As can be seen from Fig.8, even if the PSNR is not ideal, the nematode images reconstructed by POCS or SRCNN method have good visual effect. This is because the structure of nematode is plain, and the details of the image are less. As illustrated in Fig. 9, although the PSNR values of the R-6G images reconstructed by using Bicubic, POCS, PG and SRCNN methods exceed 30 dB, the visual effect of the reconstructed images is unacceptable. This phenomenon may be caused by the nature of the image itself and such quantitative metrics (PSNR or SSIM) may not correlate well with visual perception. The image reconstructed by IBP is acceptable, but the part of the image is blurred. In this case, the reconstructed image in CS-AFM is clear and the detail features present well. As shown in Fig. 10, the reconstruction of prions peptide images by using PG and SRCNN failed with the upscaling factor of 4. It is because the morphological structure of prions peptide is complex. The local fluctuation of sample structure is large. At this time, for other algorithms, the reconstruction results are acceptable, but the local details of images are lost. From Fig.11, visually, all the algorithms can complete the imaging, but the details of the imaging are not distinct. As shown in Fig.12-14, because of the simple surface structure of these three samples (nematode, hexagonal and CS-20NG), their imaging effect using different algorithms is visually acceptable. In most cases, poor visual imaging quality and low PSNR are given by using the SRCNN method. It is because the AFM image library is inappropriate and not well trained. That is to say, the SRCCN algorithm and training library are mismatched. The ideal PSNR of reconstructed images should be not less than 35 (dB). For samples with different morphologies, the super-resolution method based on CS can make the reconstructed image obtain desired PNSR and visual effect. That is to say, the super-resolution imaging method based on CS can obtain ideal imaging effect, especially when the upscaling factor is large. From the above data, it can be seen that the PSNR and SSIM of constructed images of different samples are different. In CS-based SR imaging method, TVAL3 is adopted as the reconstruction algorithm. Therefore, the TV value of original image should be related to the quality of imaging. Just as expect, the connection between image evaluation index PSNR and the value of TV/R of original AFM images is shown in Fig. 15. As can be seen from the graph, the fitting curve based on characteristics of six samples shows that the smaller the TV/R value of the image, the better the reconstruction quality of image. The sample image with a large TV and a small range is more difficult to reconstruct. It is because the sample surface is too complex and has many nanostructures. There are too many local details in the image to obtain a good PSNR of the reconstructed image. From the aspect of SSIM, it is harder to maintain the structure of a sample image that has a very small TV in reconstruction. As shown in Fig. 16, SSIM is positively correlated with TV. This may indicate that when the TV
value is small, the local image will have over-smoothing phenomenon, which leads to poor imaging quality. To sum up, the sample morphology leads to the difference of TV/R or TV of sample image, which is connected to the quality of imaging. The imaging quality can be predicted by estimate the TV or TV/R of the sample images. Therefore, for different imaging requirements, the appropriate upscaling factor, reconstruction algorithm and parameters can be selected according to the samples with the different morphology. As shown in Fig. 17, super-resolution imaging is realized by using CS technology with the different upscaling factors. First, in our laboratory, a low-resolution image was obtained by scanning the Biaxially Oriented Polypropylene (BOPP) sample in AFM. This low-resolution image is the initial image in the experiment. Then, the required high-resolution image is obtained from a low-resolution image by using the CS technology with the different upscaling factors. With the increase of upscaling factor and image resolution, the image becomes more and more clear, and the image shows more details. Super-resolution imaging based on CS is an effective method to improve the image resolution in AFM.
(a)
(b)
(c)
Fig. 6 The imaging evaluation for the graphene oxide by using different algorithms. (a) upscaling factor and PSNR. (b) upscaling factor and SSIM. (c) upscaling factor and reconstruction time.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 7 The images of graphene oxide with the upscaling factor of 2.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 8 The images of nematode C. elegans with the upscaling factor of 3.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 9 The images of R-6G with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 10 The images of prions peptide with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 11 The images of graphene oxide with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 12 The images of nematode C. elegans with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 13 The images of hexagonal domains in lithium niobite with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 14 The images of CS-20NG with the upscaling factor of 4.
(a)
(b)
(c)
Fig. 15 The relationship between reconstruction quality metric PSNR and the value of TV/R. The range
of the whole image is expressed as R. (a) upscaling factor=2. (b) upscaling factor=3. (c) upscaling factor=4.
(a)
(b)
(c)
Fig. 16 The relationship between reconstruction quality metric SSIM and the value of TV. (a) upscaling factor=2. (b) upscaling factor=3. (c) upscaling factor=4.
(a)
(b)
(c)
(d)
Fig. 17 The images of BOPP by using CS method with the different upscaling factors. (a) original lowresolution image (256×256). (b) upscaling factor=2, resolution=512×512. (c) upscaling factor=3, resolution=768×768. (c) upscaling factor=4, resolution=1024×1024.
6. Conclusions A novel super-resolution method based on compressed sensing is proposed, which is applied in AFM imaging. A special measurement matrix and TVAL3 reconstruction algorithm are used in the SR-AFM imaging based on CS. Compared with other superresolution methods, the proposed method can improve the imaging quality on the premise of ensuring certain imaging time. Through analyzing the relationship between the imaging effect and the TV/R or TV value of the original image, the quality of imaging is related to the sample morphology. For different samples, the upscaling factors, reconstruction algorithm and parameters should be adopted according to different imaging requirement. In most cases, the ideal visual and quantitative imaging effect can be obtained by using the SR imaging method based on CS when the upscaling factor is large. In a word, the proposed CS method, which uses a special measurement matrix and TVAL3 algorithm, is an effective fast super-resolution imaging method in
AFM. The proposed SR imaging method in CS-AFM can not only save imaging time greatly, reduce the interaction between probe and sample, reduce the wear of sample and probe, but also realize high-speed and large-scale imaging in AFM.
Acknowledgement The authors were supported financially by General program of the Natural Science Foundation of Fujian Province, China (Grant No.2019J01632). References [1] T.S.H, R. Y. Tsai, Multiple frame image restoration and registration,Advances in Computer Vision and Image Processing, (1984) 317-339. [2] K. Nasrollahi, T.B. Moeslund, Super-resolution: a comprehensive survey, Machine Vision and Applications, 25 (2014) 1423-1468. [3] E. Seeram, Digital image processing, Radiologic technology, 75 (2004) 435-452. [4] J. Parker, R.V. Kenyon, D.E. Troxel, Comparison of interpolating methods for image resampling, IEEE transactions on medical imaging, 2 (1983) 31-39. [5] M. Irani, S. Peleg, Improving resolution by image registration, Cvgip-Graphical Models and Image Processing, 53 (1991) 231-239. [6] H. Stark, P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, 6 (1989) 1715-1726. [7] R.R. Schultz, R.L. Stevenson, A bayesian-approach to image expansion for improved definition, IEEE Transactions on Image Processing, 3 (1994) 233-242. [8] S. Jian, Z. Xu, H.Y. Shum, Image super-resolution using gradient profile prior, IEEE Conference on Computer Vision & Pattern Recognition, (2008)24-26. [9] H. Chang, D.Y. Yeung, Y. Xiong, Super-resolution through neighbor embedding, IEEE Computer Society Conference on Computer Vision & Pattern Recognition, 2004. [10] Y. Jianchao, W. John, H. Thomas, Image Super-Resolution via Sparse Representation, IEEE Transactions on Image Processing, 19 (2010) 2861-2873. [11] D. Chao, C.L. Chen, K. He, X. Tang, Learning a Deep Convolutional Network for Image Super-Resolution, 2014. [12] G. Binnig, C.F. Quate, C. Gerber, Atomic Force Microscope, Physical Review Letters, 56 (1986) 930-933. [13] J. Zhong, and J. Yan , Seeing is Believing: Atomic Force Microscopy Imaging For Nanomaterials Research, RSC Advances, 6 (2016)1103-1121. [14] C.S. Oxvig, T. Arildsen, T. Larsen, Structure assisted compressed sensing reconstruction of undersampled AFM images, Ultramicroscopy, 172 (2017) 1-9. [15] G.Q. Han, B. Lin, Y.L. Lin, Reconstruction of atomic force microscopy image using compressed sensing, Micron, 105 (2018) 1-10. [16] S.J. Sreeja, M. Wilscy, Single image super-resolution based on compressive sensing and TV minimization sparse recovery for remote sensing images, Intelligent Computational Systems, 2014. [17] D.L. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006) 1289-1306. [18] F.J. Giessibl, Advances in atomic force microscopy, Rev. Mod. Phys., 75 (2003) 949983.
[19] S.K. Das, F.R. Badal, M.A. Rahman, M.A. Islam, S.K. Sarker, N. Paul, Improvement of Alternative Non-Raster Scanning Methods for High Speed Atomic Force Microscopy: A Review, IEEE Access, 7 (2019) 115603-115624. [20] E.J. Candes, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information, 2006. [21] C.B. Li, W.T. Yin, H. Jiang, Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013) 507-530. [22] S.B. Andersson, L.Y. Pao, Non-raster sampling in atomic force microscopy: A compressed sensing approach, American Control Conference, 2012. [23] R. G. Keys, Cubic convolution interpolation for digital image processing, IEEE Trans. on Acoust. Speech. & Signal Processing, 37(1981). [24] Park, Sung Cheol , M. K. Park , and M. G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Processing Magazine, 20(2003)2136. [25] R. Optics, Super-resolution through Error Energy Reduction, J. Modern Optics, 21 (1974) 709-720. [26] P. Chatterjee, S. Mukherjee, S. Chaudhuri, G. Seetharaman, Application Of PapoulisGerchberg Method In Image Super-Resolution and Inpainting, Computer Journal, 52 (2009) 80-89. [27] C. Dong, C.C. Loy, X. Tang, Accelerating the Super-Resolution Convolutional Neural Network, (2016). [28] W. Shi, J. Caballero, F. Huszár, J. Totz, A.P. Aitken, R. Bishop, D. Rueckert, Z. Wang, Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, Computer Vision & Pattern Recognition, 2016. [29] J. Kim, J.K. Lee, K.M. Lee, Accurate Image Super-Resolution Using Very Deep Convolutional Networks, IEEE Conference on Computer Vision & Pattern Recognition, 2016. [30] J. Kim, J.K. Lee, K.M. Lee, Deeply-Recursive Convolutional Network for Image Super-Resolution, (2015). [31] Image courtesy of Valery V. Prokhorov, Inst. of Bioorganic Chemistry RAS, Moscow, Russia. [32] Image courtesy Dr. Hannes C. Schniepp, t. C. o. W. M., Williamsburg, United States. [33] Image courtesy of Prof. M.Muller and N.Matsko, ETH Center, Zurich, Switzerland. [34] Image courtesy of T. Jung, A. Hoffmann, E. Soergel, University of Bonn. [35] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, 13 (2004) 600-612.
author statement Guoqiang Han: Conceptualization, Methodology, Writing - Review & Editing, Data Curation, Resources, Project administration, Funding acquisition, Supervision. Luyao Lv: Software, Validation, Formal analysis, Data curation, Writing- Original draft preparation. Gaopeng Yang: Data Curation, Investigation, Supervision. Yixiang Niu: Visualization, Investigation.
Declaration of Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Highlights: 1. The CS theory is introduced to solve the problems of time-consuming and low resolution of AFM imaging. 2. Six different samples and six different algorithms are used to estimate the results of superresolution imaging. 3. A large number of experiments show that CS algorithm has a good effect on super-resolution imaging of AFM images, especially when the upscaling factor is large. 4. By analyzing the TV or TV/R and surface morphology of the original image of the AFM sample, it is found that the image quality is largely determined by the morphology of the sample.
S0169-4332(19)34048-6 https://doi.org/10.1016/j.apsusc.2019.145231 APSUSC 145231
To appear in:
Applied Surface Science
Received Date: Revised Date: Accepted Date:
11 August 2019 28 November 2019 29 December 2019
Please cite this article as: G. Han, L. Lv, G. Yang, Y. Niu, Super-resolution AFM imaging based on compressive sensing, Applied Surface Science (2019), doi: https://doi.org/10.1016/j.apsusc.2019.145231
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
© 2019 Published by Elsevier B.V.
Super-resolution AFM imaging based on compressive sensing Guoqiang Hana, b, *, Luyao Lva, Gaopeng Yanga and Yixiang Niua a School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou 350 108, People’s Republic of China b Key Laboratory of Fluid Power and Intelligent ElectroHydraulic Control, Fuzhou U niversity, Fuzhou 350108, People’s Republic of China Email: [email protected]
Abstract: Atomic force microscopy (AFM) is a powerful and ultra-precision instrument in nano-scale, which is widely used in many fields. It is a complex and timeconsuming process for AFM imaging. Most of the original AFM images are with low resolution. For nano-scale measurement and imaging, it is very important to obtain super-resolution images. In most cases, super-resolution imaging takes a long time and the quality of imaging is unsatisfactory. In this regard, a novel super-resolution imaging method based on compressed sensing (CS) technology is proposed in AFM. In the experiment, six samples with different morphology were used to test the effect of superresolution image reconstruction with different upscaling factors (2, 3 and 4). The quality of reconstructed image is analyzed and evaluated by image evaluation metrics (PSNR and SSIM). The relationship between the reconstruction quality of different images and the actual TV or TV/R of sample images is analyzed, which can provide a preliminary basis for predicting the imaging quality. Comparing with other superresolution imaging methods, the proposed method has achieved better imaging effect both visually and quantitatively. In summary, super-resolution imaging method based on CS not only has high imaging quality but also has high speed. Keywords: Atomic force microscopy (AFM); Compressed sensing (CS); Super resolution (SR); Measurement matrix;
1. Introduction The super-resolution imaging is a technique that aims to obtain a high-resolution image from a low -resolution image or a low-resolution data [1]. It is widely used in video surveillance, image printing, criminal investigation analysis, medical image processing, satellite imaging and other fields. The super-resolution imaging methods are mainly based on hardware or software. Super-resolution imaging through hardware is achieved by reducing the size of pixels and increasing the size of sensors. Because of the complexity of manufacturing process and cost of sensors and optical components, it is difficult to obtain high-resolution images in many situations [2]. Therefore, software-based methods are usually preferred to the hardware-based solutions. And the software-based methods are mainly divided into three kinds. First of all, the interpolation-based approach is a valid way to realize super-resolution imaging. The interpolation-based methods, such as bilinear and bicubic interpolation, require less computation [3, 4]. But it will result in highly blurred edges and many image details are always lost. In addition, another method is to use the reconstruction-based approach to realize super-resolution imaging, such as iterative back projection (IBP), projection onto convex sets (POCS) and maximum a posteriori (MAP) [5-7]. The reconstructionbased method is a course in which prior knowledge is added to the imaging process as a constraint to realize super-resolution imaging [8]. Unfortunately, the convergence speed of this method is slow. Last but not least, super-resolution imaging can also be achieved through learning-based methods. Learning-based methods are roughly divided into neighborhood embedding method (NE), sparse representation method (SR) and the convolutional neural network method(CNN) [9-11]. The relationship between
low-resolution image and high-resolution image is identified by the external database. However, a large amount of calculation and storage space is also required for this method. Atomic Force Microscope (AFM) is usually used to measure the topography of conductive, nonconductive samples and semiconductor [12, 13]. On the premise of following Nyquist-Shannon sampling theorem, traditional AFM imaging often takes a lot of time. Irreversible damage is often caused by long-time contact and interaction between probe and sample. In particular, probes are prone to wear or damage. The quality of AFM imaging will be greatly reduced. Accordingly, low-resolution images are often obtained in traditional AFM. Even though sometimes high-resolution images can be obtained through optimization, their enhancement is extremely limited. Therefore, fast super-resolution technology should be applied to AFM. Applying CS technique with specially designed measurement matrices can effectively speed up AFM imaging [14, 15]. The super-resolution technique base on CS can be regarded as a reconstruction-based approach or a learning-based approach in super-resolution imaging. Learning-based CS method is to identify dictionaries that represent highresolution patches in a sparse way, which introduce additional information of similar structures in images into the dictionary of CS framework [16]. But it takes a long time to train the dictionary. Our proposed method belongs to a reconstruction-based approach and can be used in AFM. Although other CS-based reconstruction algorithms can achieve super-resolution, they may not be suitable for AFM because the measurement principles of different instruments are different. Super-resolution imaging based on the proposed method is achieved by optimized measurement matrix. This method does not require additional dictionary training and other preprocessing. Compressive sensing theory (CS) theory shows that if the signal is sparse in a certain area, it can be reconstructed from fewer samples than the original signal [17]. Redundancy in signals is exploited. The low-dimensional projection of the highdimensional signal is obtained from the measurement matrix, and then the original signal can be restored almost perfectly from a small number of measured values by using appropriate reconstruction algorithm. For the super-resolution algorithm based on CS, better results can be obtained by choosing the appropriate measurement matrix and reconstruction algorithm. In our work, a large number of experiments have been carried out to prove the advantages of CS in super-resolution AFM imaging by comparing with different superresolution imaging methods, especially when the upscaling factor is large. In addition, imaging effect of six samples with different morphology was analyzed. TVAL3 algorithm was used to reconstruct the AFM images in super-resolution imaging based on CS. In a word, the CS technology was a promising method to realize super-resolution AFM imaging.
2. Super-resolution imaging methods in AFM 2.1 Imaging principle of traditional AFM The principle of AFM is to measure the longitudinal displacement of the tip by
controlling the constant interaction force between the tip and the atoms on the sample surface [18]. Van der Waals force is the general force between atoms. The working modes of the AFM can be divided into three main categories: contact mode, non-contact mode and tapping mode [19]. And the measurement mode of the AFM microscope is point-to-point scanning. The choice of tip scanning patterns in AFM depends on different scenarios. Grating scanning pattern is commonly used in traditional AFM. In general, the probe starts at the upper left corner of the sample and sweeps a horizontal line to the right. Next, the probe is quickly swept back to the next point on the left, and then the second horizontal line is swept. On the basis of this fixed path and sequence, the whole sample can be scanned until the last horizontal line. According to the different resolution, the corresponding measurement points are set by step size to meet the requirements. A diagram of raster scanning is shown in Fig. 1. In traditional AFM scanning, both scanning and imaging are performed simultaneously. In other words, the scanning mode is line-by-line real-time imaging. Scanning and imaging end at the same time. Therefore, the scanning time is the imaging time. On account of traditional AFM scanning is extremely time-consuming, it is urgent to find a faster imaging method.
Fig. 1 Raster Scanning Routine of AFM.
2.2 Super-resolution imaging in CS-AFM Using CS theory, the signal can be reconstructed from fewer samples than the original signal. In other words, Fast AFM imaging can be achieved through compressed sensing (CS) with low sampling rate. Consider an unknown signal X of length N containing only K nonzero elements. It is called K-sparse. If it takes M times liner measurements to sample the signal X, it means it takes fewer measurements than signal dimension. (1) Y X where Y is a measurement of length M with M>N. is an M N measurement matrix. Since the process is non-adaptive, the measurement matrix is selected beforehand. CS can recover the signal X from significantly fewer measurements, only M cK log
N , K
suggesting the potential of significant cost reduction in digital data acquisition. And c is a small constant and K denotes the number of non-zero elements in the signal X. Many signals are not sparse. They can be not directly used in compressive sensing. Fortunately, most natural signals are compressible, that is the signal X could be transformed into sparse form through sparse basis .
X (2) where is the sparse representation of signal X. is an N N basis transform matrix. Then, the Eq. (1) changes into (3) Y X A where A is M N sensing matrix which should satisfy the restricted isometry property (RIP) [20]. Compressed sensing theory mainly includes three aspects: sparse representation, measurement matrix and reconstruction algorithm. In CS theory, the measured signal is required to be sparse or compressible. In practice, natural signals, most of which are not strictly sparse, are mostly compressible. Therefore, the signal can be sparse, such as discrete cosine change, discrete wavelet transformation and so on. Common compressed sensing measurement matrices can be divided into two categories: deterministic measurement matrices and random measurement matrices. Most of the energy of the original signal can be collected by the measurement matrix. Suitable algorithms are needed to solve the above undetermined equation. The sparse reconstruction algorithm in CS can be used. Sparse reconstruction algorithm can be classified into convex optimization algorithm and greedy iteration algorithm. TV Minimization by Augmented Lagrangian and Alternating Direction Algorithms (TVAL3) belongs to the convex optimization algorithms. TVAL3 effectively combines an alternating direction technique with a nonmonotone line search to minimize the augmented Lagrangian function at each iteration which could significantly accelerate the convergence [21]. In CS problems, the precise preservation of edges or boundaries is achieved by TV, which makes the reconstructed image clearer. The advantage of TV is that the sparse images can be reconstructed by TV. The noiseless discrete TV model for CS reconstruction can be written as
min TV(X) subject to Y=AX
(4)
where TV X i , j
X
X i , j X i , j 1 X i , j 2
i 1, j
2
(5)
where Xi, j is the value of the matrix at pixel (i, j) and X represents the signal that needs to be reconstructed. Since most of the signals in nature are sparse in the gradient domain, there is no need to sparse initial image signals when TVAL3 algorithm is used as the reconstruction algorithm, which greatly simplifies the imaging process.
Fig. 2 Super-resolution AFM Imaging by using Compressed Sensing.
As shown in Fig. 2, the low-resolution image or data is obtained by raster scanning with large step in AFM, and the final high-resolution image is obtained in CS-AFM through reconstruction algorithm. In reality, super-resolution imaging is an inverse process of down-sampling. The CS technology includes compression process and reconstruction process. The compression process in CS is assumed to be the downsampling process, which is a mathematical process of getting low-resolution image or data from high-resolution image or data. In the actual AFM measurement, the highresolution image is unknown and need to be reconstructed. In addition, the purpose of reconstruction process is to reconstruct unknown data points and finally obtain a highresolution image. As illustrated in Fig. 3, CS technology is used to realize the superresolution imaging of AFM image. Given an initial low-resolution image or undersampling data, the corresponding high-resolution image can be correctly restored under certain conditions through CS method. Y R M represents the low-resolution image,
X R N represents the high-resolution image, XK represents the blank template for * N high-resolution images and X R represents the reconstructed high-resolution
image. The pixels in the low-resolution image Y are projected into the blank template of the required high-resolution image according to a certain rule. The rest of the pixel values are set to zero. Besides, the filled blank template XK is equivalent to the undersampling of the required high-resolution image X. The under-sampled sampling point is selected by a certain rule. The low-resolution image Y is used as the input image of the compressed sensing algorithm. Low-resolution image Y and the blank template X K is known. Therefore, the measurement matrix can be obtained by equation (1). Constructing a special measurement matrix is the focus of CS method, which extracts certain rows from the unit matrix according to the position of the pixels in the blank template of the highresolution image. In other words, the measurement matrix is a variant of the unit matrix.
Its subordination is shown as:
mn
0 1 0 0 0 1 0 0 0
0 0 0 0 1 0
(6)
Each measurement of CS typically relies on a linear combination of many elements of the signal. However, they are difficult to be applied in AFM due to the point-like nature of the AFM probe tip. The AFM probe tip only measures a single pixel at a time. Therefore, a specially designed measurement matrix is needed to measure the AFM sample. An identity matrix with some of its row removed is a valid choice in AFM application as the measurement matrix [22]. In each row of , there is only a single one and zeros elsewhere. Such a measurement matrix ensures that only a single pixel of the image is required for each measurement. This unique measurement matrix could seem as AFM tip trajectory. By substituting low-resolution image Y and measurement matrix into equation (1), the reconstructed high-resolution image X * can be obtained by the TVAL3 algorithm. As shown in Fig. 3, the specific implementation of super-resolution imaging with the upscaling factor of 2 is achieved by using a special measurement matrix. Low-
34 158 resolution image Y is known. A practical representation of measurement 2 213 matrix can be obtained from Fig. 3. Specific manifestation is shown as 1 0 0 0
0 0 0 0
0 1 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 1 0
0 0 0 0
0 0 0 1
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
(7)
37 99 156 96 60 103 141 96 can be obtained by introducing High-resolution image X * 6 110 209 88 64 101 132 95
these two known conditions into equation (1).
Fig. 3 Super-resolution imaging schematic based on compressed sensing with the upscaling factor of 2.
2.3 Imaging time for CS-AFM In order to evaluate the performance of compressed sensing algorithm, the AFM imaging time needs to be reasonably estimated. The imaging time in CS-AFM includes scanning time and reconstruction time. The imaging time of traditional AFM is only scanning time. In the scanning process of AFM, it should be noted that the tip of the AFM can only measure one point at a time. In each measurement, the tip is pulled down until the sample is touched, height data is measured, and then retrieved and moved to the next point in the sequence. The scan time can be computed approximately as follows: (8) t Sr T where Sr=M/N is the sampling rate, and T is the conventional AFM acquisition time. And M, N denotes the length of measured value, the length of the original signal. After introducing CS technology, a low-resolution image scanned by AFM can be reconstructed to a high-resolution image. If the sampling rate is low, the scanning time can be saved to a large extent. The reconstruction time usually takes less 20 seconds by using a computer with general hardware configuration. The time estimation of the image obtained by three different methods is shown in Fig. 4. The scanning rate is set to 1 Hz for the low-resolution (256×256) and the scanning rate is set to 0.25 Hz for the high-resolution (1024×1024). In other words, 256 measurement points are scanned per second in AFM. In this way, the scanning speed of low-resolution image is the same as that of high-resolution image. Scanning a low-resolution image in AFM is faster, but the quality of the image is poor and the details are blurred. Even though the quality of the high-resolution image scanned in traditional AFM is satisfactory, it takes a lot of time. After introducing CS technology, satisfactory image quality can be obtained in a short time. The imaging time can be saved while the imaging quality is guaranteed by using CS.
(a)
(b)
(c)
Fig. 4 The R-6G images obtained by three different methods. (a) Low-resolution image (256×256) obtained by traditional AFM. Imaging time=256s. (b) High-resolution (1024×1024) image obtained by traditional AFM. Imaging time=4096s. (c) High-resolution image (1024×1024) obtained by using CSAFM. Imaging time=276s.
3. Traditional super-resolution methods 3.1 Bicubic interpolation In the early image interpolation algorithms, nearest neighbor interpolation is widely used because of uncomplicated calculation. But jagged edges and mosaics are produced by using Bicubic. The subsequent bilinear interpolation method can make up for the defect of the nearest neighbor interpolation method and solve the above problems, but the high-frequency part and details of the image will be weakened. In super-resolution imaging, the bicubic interpolation method is superior to the low-order interpolation method when the magnification is large. The calculation of the pixel value not only considers the influence of the surrounding pixel value, but also the change rate of the pixel value between adjacent points into the pixel value of the pixel to be calculated. Assuming that the A size of the source image is m×n, the size of the target image B is M×N after scaling the K times. Each pixel of A is known, and B is unknown. To get the value of each pixel (X, Y) in the target image B, it is necessary to find the corresponding pixel (x, y) in the source image A. According to the proportional relation x/X = m/M = 1/K, the corresponding coordinates of B (X, Y) on A are A (x, y) = A (X/K, Y/K). Then, the pixel values (X, Y) of the target image B are calculated according to the weight of the last 16 pixels in the source image A, which is calculated based on the bicubic basis function [4]. The value of B pixels (x, y) is equal to the weighted superposition of 16 pixels. And data are fitted by bicubic interpolation using a specific basis function [23]. The bicubic interpolation basis function is as follows: 3 2 w 2 w 1 3 2 S ( w) w 5 w 8 w 4 0
w 1 1 w <2 w 2
(9)
where S is the weight corresponding to the above 16 pixels. And w denotes the distance from pixel (x, y) in image A to pixel (X, Y) in image B. 3.2 Iterative back-projection Iterative back projection(IBP) is a kind of spatial super-resolution imaging method [5]. In this approach, the high-resolution image is finally obtained by iterative back projection of the errors of the simulated low-resolution image and the observed lowresolution image. And degradation model w is set for initial estimated image, and k simulated low-resolution images Li (i=1,2,…,k) are generated according to degraded parameters: the deviation between the simulated low-resolution image and the original n low-resolution image is projected back to the initial estimation Yˆ of the high-
resolution image by using back-projection operator HBP. The high-resolution estimation n image Yˆ obtained by the nth iteration is Lˆn by using degradation model w. The
estimated image is updated by iteration formula until the error meets the requirement
[24]. The iterative process is as follows:
Yˆ n 1 Yˆ n H BP L wLˆn
(10)
where Yˆ n , Yˆ n 1 are high resolution images obtained by the n, n+1 iterations, Lˆn is the approximation of Y after n iterations to simulate low resolution images and HBP is a back projection kernel function, which can correctly determine the contribution of errors. Low computational complexity is the characteristic of using IBP. However, illconditioned and no unique solution are still the problems to be solved in superresolution. The proper back projection operator HBP is difficult to be obtained. And Effectively utilizing the prior knowledge of images is not suitable for this method. 3.3 Projection onto convex sets Projection onto convex sets (POCS) algorithm is a super-resolution imaging method which starts from the initial guessed image and corrects the image successively according to the error [6]. By intersecting the solution space of high-resolution image with a series of constraints (non-negative, energy boundedness, consistency of observation data, local smoothness, etc.) representing the properties of high-resolution image, a smaller solution space can be obtained. Starting from a point in high resolution image space, POCS method is used to find the next point satisfying all constrained convex sets, and finally the estimation of high-resolution image is obtained. For the initial value X 0 , let each constraint set define a convex set projection operator AK . The projection operator AK of convex set is used to iteratively project the initial estimate X0 of high-resolution image to obtain ideal high-resolution image. And the calculation process is as follows: X n AN
A3 A2 A1 X 0
(11)
where X0 denotes the high-resolution image of the initial estimation, Xn represents the final high-resolution image obtained after n iterations and AK is a projection operator that projects arbitrary signal X onto a closed convex set. Making full use of prior knowledge is achieved by projection method of convex sets. However, the convergence process depends on the selection of initial values, and the solution is unstable. 3.4 The Papoulis-Gerchberg (P-G) algorithm The P-G algorithm is used in super-resolution imaging in modified form when lowresolution images are available [25]. After motion estimation, low-resolution image data is projected onto the high-dimensional mesh. The unknown pixel value is initially set to zero. A high-resolution image with initial estimation is obtained. Then the image is passed through a low-pass filter whose cut-off frequency is assumed to be the bandwidth of the input image, which results in blurred images [26]. However, due to filtering, the size of known pixel values also decreases. In order to recover the high
frequency components lost due to filtering, the original values are forcibly restored to the known pixels. The whole process is repeated until convergence. The result is a reconstructed image. Because of the steep cut-off in frequency domain, the highresolution image generated has jagged phenomenon at the edge. In addition, it relies heavily on the fact that the measured (known) pixel value is the value obtained in the reconstructed high-resolution image. In addition, they should be completely free from noise. Therefore, it can’t effectively compensate the measurement data of ambiguity and noise. 3.5 The super-resolution convolutional neural network method As a main learning-based method in SR, deep convolution neural networks (CNNs) have attracted more and more attention in recent years. The deep convolution neural network methods include SRCNN, FSRCNN, ESCPN, VDSR and DRCN [11, 27-30]. SRCNN is a basic convolutional neural network algorithm. In SRCNN algorithm, three convolution layers are used to represent the steps of feature extraction, non-linear transformation and image reconstruction. Firstly, the patches are extracted from lowresolution images, and each patch is represented as a high-dimensional vector. These vectors form a set of feature maps whose number is equal to the dimension of the vector. Then each high-dimensional vector is mapped nonlinearly to another high-dimensional vector. Each mapping vector conceptually represents a high-resolution patch. These vectors form another set of feature maps. Finally, the stitching representation of the above high-resolution patch is aggregated to generate the final high-resolution image. This image is expected to be similar to the original high-resolution image. For the convolution neural network algorithm, the structure of each layer and the specific algorithm are very important, and the image training library should meet the requirements of super-resolution imaging. Four or more layers and larger filter sizes can be used to further improve performance. But the SRCNN method is trained for a single upscaling factor. If a new scaling factor is needed, the model must be retrained.
4. Experiments A train of experiments were conducted using six samples include R-6G, prions peptide, graphene oxide, nematode C. elegans, hexagonal domains in lithium niobite and CS-20NG [31-34]. The first five samples are provided by NT-MDT from Russia. The last sample is a calibrated grating (CS-20NG) scanned in our laboratory. Fig. 5 shows two-dimensional and three-dimensional AFM images obtained in contact mode. The resolution of all images is 1024×1024 pixels. The surface morphologies of the six samples are different. The low- resolution images are down-sampled from these highresolution images. Low-resolution images are the initial images of the experiment. In this way, the difference between the original image and reconstructed image can be evaluated more conveniently. Moreover, the size of the low-resolution image is determined by the upscaling factor. The super-resolution reconstruction effect in CSAFM is analyzed with different upscaling factors of 2, 3 and 4. The low-resolution image was imported into MATLAB to perform experiments. The simulation was carried
out on a typical WIN7 operating system, 3.4 GHz AMD A4 and 8 GB memory personal computer.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 5 The 2D and 3D AFM images of testing samples. (a) R-6G. (b) prions peptide. (c) graphene oxide. (d) nematode C. elegans. (e) hexagonal domains in lithium niobite. (f) CS-20NG.
Another important issue is to evaluate the image quality. When the original image and the contrast image are known, the traditional way is to measure the quality of the contrast image by Peak signal-to-noise ratio (PSNR). PSNR is still an important index of image quality evaluation. PSNR is defined as. h w max I min I 2 1 2 PSNR 10log10 h w 2 I1 i, j I 2 i, j i 1 j 1
(12)
where, h and w are respectively expressed as the number of rows and columns of the image. I1 i, j and I 2 i, j are respectively expressed as the gray value of the original image and the new image at i, j . The structural similarity (SSIM) index is a full-reference quality metric that compares local patterns of pixel intensities that have been normalized for luminance and contrast [35]. SSIM is defined as, SSIM ( x, y )
2
x
2 x
y
C1 2 xy C2
C1 x2 y2 C2 2 y
(13)
1
1 where x N
N
x
i
1
1 N 2 2 is the mean intensity of x . x xi x is the N 1 i 1
standard deviation (the square root of variance) of y . xy is the covariance of x and y . We choose C1 K1R
1 N xi x yi y N 1 i 1
and C2 K2 R . R is the
2
2
dynamic range of the pixel values. K 1 = 0.01 and K 2 =0.03 are small constants. Table 1. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor 2. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
2
2
2
2
2
2
PSNR
50.493
41.840
40.412
48.987
45.143
47.371
SSIM
0.996
0.934
0.957
0.997
0.993
0.992
TIME
7.305
7.690
7.301
7.344
7.326
7.243
PSNR
40.484
37.024
35.662
47.695
37.687
40.902
SSIM
0.972
0.875
0.907
0.988
0.944
0.966
TIME
11.211
22.783
18.932
16.329
22.032
10.258
PSNR
34.919
35.212
33.717
33.246
34.569
38.053
SSIM
0.961
0.826
0.860
0.983
0.907
0.937
TIME
174.685
172.827
189.864
198.572
186.212
161.116
PSNR
37.381
34.586
33.349
41.266
34.901
37.993
SSIM
0.965
0.794
0.851
0.982
0.912
0.932
TIME
157.253
132.650
155.821
164.139
194.570
165.183
PSNR
37.698
35.531
34.096
44.042
35.066
38.471
SSIM
0.968
0.834
0.871
0.995
0.916
0.940
TIME
21.020
19.952
20.597
15.508
18.321
22.343
PSNR
47.497
31.189
27.962
44.964
21.635
29.762
SSIM
0.994
0.729
0.673
0.996
0.744
0.817
TIME
137.374
152.871
149.715
132.297
150.698
140.338
bicubic
IBP
POCS
PG
CS
SRCNN
5. Results and discussion It takes a long time to get a high-resolution image by using AFM scanning. Therefore, introducing CS technology is a good solution. A low-resolution image or data is scanned and then a high-resolution image can be obtained using CS technology. In a word, the high-resolution imaging time in AFM can be shortened by using CS technology. In order to evaluate the performance of CS-based super-resolution imaging in AFM, a series of simulation experiments were carried out using six test samples. The proposed high-resolution imaging method based on CS is compared with the bicubic interpolation, the IBP (Iterative back-projection), the POCS (Projection onto Convex Sets), the P-G (Papoulis-Gerchberg), and the SRCNN (super-resolution convolutional
neural network) method. Tables 1,2 and 3 summarize the results, showing PSNR, SSIM and reconstruction time for reconstructed images. As confirmed in table 1, PSNR and SSIM of all reconstructed images have reached the maximum by using bicubic method, slightly higher than ones by using the proposed CS method when the upscaling factor is 2. As shown in table 1, satisfactory reconstruction results are achieved by employing CS method. For example, the PSNR value of reconstructed nematode image by CS method is 44.042 dB, which indicates that the imaging quality is commendable. Only the PSNR of the reconstructed nematode image is not the best by CS method in table 2. And the PSNR of the reconstructed nematode image achieved by CS method is 45.657 dB, only 0.994 dB lower than by using IBP. The highest PSNR and SSIM of other reconstructed images can be obtained by CS method. Specifically, as shown in table 3, the optimal PSNR and SSIM of all reconstructed images have been obtained by using the proposed method with the upscaling factor of 4. The experimental results are indicative that the proposed imaging method in AFM can realize fast SR imaging with different upscaling factors well, especially with larger upscaling factor. Table 2. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor of 3. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
3
3
3
3
3
3
PSNR
37.871
35.002
33.790
44.271
35.164
38.381
SSIM
0.967
0.767
0.859
0.995
0.915
0.935
TIME
7.441
7.573
7.475
7.675
7.582
7.380
PSNR
39.395
35.991
34.673
46.651
36.479
39.392
SSIM
0.953
0.806
0.889
0.982
0.933
0.963
TIME
17.537
15.737
16.745
20.738
20.707
13.749
PSNR
32.769
34.585
32.999
30.196
33.345
36.861
SSIM
0.949
0.787
0.838
0.977
0.889
0.926
TIME
233.394
245.608
246.622
248.598
243.029
242.747
PSNR
37.059
34.273
33.165
39.049
34.618
37.320
SSIM
0.963
0.724
0.833
0.979
0.911
0.925
TIME
325.855
308.632
280.793
253.383
302.950
254.940
PSNR
40.835
37.028
35.773
45.657
39.461
40.619
SSIM
0.978
0.836
0.904
0.997
0.966
0.964
TIME
20.539
20.437
20.700
20.504
20.413
20.684
PSNR
23.585
23.543
18.984
17.924
11.188
19.879
SSIM
0.881
0.422
0.304
0.962
0.549
0.520
TIME
140.219
138.736
138.193
125.669
141.937
147.369
bicubic
IBP
POCS
PG
CS
SRCNN
As illustrated in Fig. 6, it indicates the reconstruction results of graphene oxide images with different upscaling factors and SR imaging method. The imaging quality by using bicubic, IBP, POCS and PG algorithms decreases with the increase of upscaling factor. With the increase of upscaling factor, the imaging effect of CS method
does not deteriorate significantly. The larger the upscaling factor, the higher the accuracy of the super resolution algorithm is required. The SR imaging method based on CS achieves the best imaging effect with the upscaling factor of 4. The effect of imaging by SRCNN method is worst, which has a great relationship with the suitability of external training library. Due to the diversity of AFM images and the imaging characteristics of AFM, the appropriate training library for AFM images is difficult to obtain. The high-resolution blocks obtained by SRCNN method are from the external image database, which are not real details. This kind of algorithm is also called "image fantasy". The effect of imaging is closely related to the appropriateness of the external training library. If the external library is not suitable, the reconstructed image may have erroneous high-frequency details. Moreover, a large number of computations are also necessary for this method and the imaging effect depends on the accuracy of training algorithm. The super-resolution method based on bicubic is the fastest, and it only takes less than 10 seconds to reconstruct an image. But the image quality obtained by using bicubic is unsatisfactory with the larger upscaling factor. Compared with other algorithms, higher imaging quality can be obtained by using CS algorithm on the premise of guaranteeing imaging time. It only takes about 20 seconds to reconstruct an AFM image. The imaging time includes scanning time and reconstruction time in CSAFM. Super-resolution imaging based on CS method can not only improve the scanning speed of AFM, but also save the reconstruction time of AFM image. Table 3. The results of PSNR (dB), SSIM and image reconstruction time (sec) for images of six samples with the upscaling factor of 4. sample
R-6G
prions
graphene
nematode
hexagonal
CS-20NG
scale
4
4
4
4
4
4
PSNR
32.782
32.120
30.348
41.284
29.863
33.622
SSIM
0.920
0.659
0.765
0.993
0.814
0.867
TIME
7.612
7.645
7.632
7.654
7.674
7.594
PSNR
36.288
33.964
32.496
43.631
33.437
36.638
SSIM
0.919
0.714
0.838
0.969
0.870
0.932
TIME
11.476
11.358
11.397
11.616
11.394
11.782
PSNR
30.597
33.941
32.124
27.660
31.309
35.769
SSIM
0.931
0.752
0.809
0.964
0.850
0.906
TIME
187.355
172.056
172.358
172.239
168.659
198.61
PSNR
36.331
33.776
32.474
36.662
34.207
37.074
SSIM
0.957
0.655
0.798
0.973
0.907
0.916
TIME
479.198
454.612
465.924
418.387
394.657
429.23
PSNR
37.723
35.402
33.863
44.146
36.322
38.189
SSIM
0.961
0.778
0.861
0.996
0.929
0.940
TIME
21.215
21.042
20.878
20.999
20.875
21.158
PSNR
31.547
30.686
28.608
38.784
27.930
32.327
SSIM
0.881
0.604
0.677
0.988
0.728
0.805
TIME
130.737
141.793
141.671
129.387
142.701
144.933
bicubic
IBP
POCS
PG
CS
SRCNN
Figures 7, 8, 9, 10, 11, 12, 13 and 14 show the visual results by using different imaging approaches with different upscaling factors (2, 3 and 4). From Fig. 7, even if the PSNR and SSIM of reconstructed graphene oxide image by using CS method is not the best with the upscaling factor of 2, a better visual imaging quality has been obtained. Although the PSNR is the highest, the graphene oxide image reconstructed by bicubic method is extremely dark and the details are distorted in Fig.7. As can be seen from Fig.8, even if the PSNR is not ideal, the nematode images reconstructed by POCS or SRCNN method have good visual effect. This is because the structure of nematode is plain, and the details of the image are less. As illustrated in Fig. 9, although the PSNR values of the R-6G images reconstructed by using Bicubic, POCS, PG and SRCNN methods exceed 30 dB, the visual effect of the reconstructed images is unacceptable. This phenomenon may be caused by the nature of the image itself and such quantitative metrics (PSNR or SSIM) may not correlate well with visual perception. The image reconstructed by IBP is acceptable, but the part of the image is blurred. In this case, the reconstructed image in CS-AFM is clear and the detail features present well. As shown in Fig. 10, the reconstruction of prions peptide images by using PG and SRCNN failed with the upscaling factor of 4. It is because the morphological structure of prions peptide is complex. The local fluctuation of sample structure is large. At this time, for other algorithms, the reconstruction results are acceptable, but the local details of images are lost. From Fig.11, visually, all the algorithms can complete the imaging, but the details of the imaging are not distinct. As shown in Fig.12-14, because of the simple surface structure of these three samples (nematode, hexagonal and CS-20NG), their imaging effect using different algorithms is visually acceptable. In most cases, poor visual imaging quality and low PSNR are given by using the SRCNN method. It is because the AFM image library is inappropriate and not well trained. That is to say, the SRCCN algorithm and training library are mismatched. The ideal PSNR of reconstructed images should be not less than 35 (dB). For samples with different morphologies, the super-resolution method based on CS can make the reconstructed image obtain desired PNSR and visual effect. That is to say, the super-resolution imaging method based on CS can obtain ideal imaging effect, especially when the upscaling factor is large. From the above data, it can be seen that the PSNR and SSIM of constructed images of different samples are different. In CS-based SR imaging method, TVAL3 is adopted as the reconstruction algorithm. Therefore, the TV value of original image should be related to the quality of imaging. Just as expect, the connection between image evaluation index PSNR and the value of TV/R of original AFM images is shown in Fig. 15. As can be seen from the graph, the fitting curve based on characteristics of six samples shows that the smaller the TV/R value of the image, the better the reconstruction quality of image. The sample image with a large TV and a small range is more difficult to reconstruct. It is because the sample surface is too complex and has many nanostructures. There are too many local details in the image to obtain a good PSNR of the reconstructed image. From the aspect of SSIM, it is harder to maintain the structure of a sample image that has a very small TV in reconstruction. As shown in Fig. 16, SSIM is positively correlated with TV. This may indicate that when the TV
value is small, the local image will have over-smoothing phenomenon, which leads to poor imaging quality. To sum up, the sample morphology leads to the difference of TV/R or TV of sample image, which is connected to the quality of imaging. The imaging quality can be predicted by estimate the TV or TV/R of the sample images. Therefore, for different imaging requirements, the appropriate upscaling factor, reconstruction algorithm and parameters can be selected according to the samples with the different morphology. As shown in Fig. 17, super-resolution imaging is realized by using CS technology with the different upscaling factors. First, in our laboratory, a low-resolution image was obtained by scanning the Biaxially Oriented Polypropylene (BOPP) sample in AFM. This low-resolution image is the initial image in the experiment. Then, the required high-resolution image is obtained from a low-resolution image by using the CS technology with the different upscaling factors. With the increase of upscaling factor and image resolution, the image becomes more and more clear, and the image shows more details. Super-resolution imaging based on CS is an effective method to improve the image resolution in AFM.
(a)
(b)
(c)
Fig. 6 The imaging evaluation for the graphene oxide by using different algorithms. (a) upscaling factor and PSNR. (b) upscaling factor and SSIM. (c) upscaling factor and reconstruction time.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 7 The images of graphene oxide with the upscaling factor of 2.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 8 The images of nematode C. elegans with the upscaling factor of 3.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 9 The images of R-6G with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 10 The images of prions peptide with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 11 The images of graphene oxide with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 12 The images of nematode C. elegans with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(d) POCS
(e) PG
(f) CS
(g) SRCNN
Fig. 13 The images of hexagonal domains in lithium niobite with the upscaling factor of 4.
(a) original
(b) Bicubic
(c) IBP
(e) PG
(f) CS
(g) SRCNN
(d) POCS
Fig. 14 The images of CS-20NG with the upscaling factor of 4.
(a)
(b)
(c)
Fig. 15 The relationship between reconstruction quality metric PSNR and the value of TV/R. The range
of the whole image is expressed as R. (a) upscaling factor=2. (b) upscaling factor=3. (c) upscaling factor=4.
(a)
(b)
(c)
Fig. 16 The relationship between reconstruction quality metric SSIM and the value of TV. (a) upscaling factor=2. (b) upscaling factor=3. (c) upscaling factor=4.
(a)
(b)
(c)
(d)
Fig. 17 The images of BOPP by using CS method with the different upscaling factors. (a) original lowresolution image (256×256). (b) upscaling factor=2, resolution=512×512. (c) upscaling factor=3, resolution=768×768. (c) upscaling factor=4, resolution=1024×1024.
6. Conclusions A novel super-resolution method based on compressed sensing is proposed, which is applied in AFM imaging. A special measurement matrix and TVAL3 reconstruction algorithm are used in the SR-AFM imaging based on CS. Compared with other superresolution methods, the proposed method can improve the imaging quality on the premise of ensuring certain imaging time. Through analyzing the relationship between the imaging effect and the TV/R or TV value of the original image, the quality of imaging is related to the sample morphology. For different samples, the upscaling factors, reconstruction algorithm and parameters should be adopted according to different imaging requirement. In most cases, the ideal visual and quantitative imaging effect can be obtained by using the SR imaging method based on CS when the upscaling factor is large. In a word, the proposed CS method, which uses a special measurement matrix and TVAL3 algorithm, is an effective fast super-resolution imaging method in
AFM. The proposed SR imaging method in CS-AFM can not only save imaging time greatly, reduce the interaction between probe and sample, reduce the wear of sample and probe, but also realize high-speed and large-scale imaging in AFM.
Acknowledgement The authors were supported financially by General program of the Natural Science Foundation of Fujian Province, China (Grant No.2019J01632). References [1] T.S.H, R. Y. Tsai, Multiple frame image restoration and registration,Advances in Computer Vision and Image Processing, (1984) 317-339. [2] K. Nasrollahi, T.B. Moeslund, Super-resolution: a comprehensive survey, Machine Vision and Applications, 25 (2014) 1423-1468. [3] E. Seeram, Digital image processing, Radiologic technology, 75 (2004) 435-452. [4] J. Parker, R.V. Kenyon, D.E. Troxel, Comparison of interpolating methods for image resampling, IEEE transactions on medical imaging, 2 (1983) 31-39. [5] M. Irani, S. Peleg, Improving resolution by image registration, Cvgip-Graphical Models and Image Processing, 53 (1991) 231-239. [6] H. Stark, P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, 6 (1989) 1715-1726. [7] R.R. Schultz, R.L. Stevenson, A bayesian-approach to image expansion for improved definition, IEEE Transactions on Image Processing, 3 (1994) 233-242. [8] S. Jian, Z. Xu, H.Y. Shum, Image super-resolution using gradient profile prior, IEEE Conference on Computer Vision & Pattern Recognition, (2008)24-26. [9] H. Chang, D.Y. Yeung, Y. Xiong, Super-resolution through neighbor embedding, IEEE Computer Society Conference on Computer Vision & Pattern Recognition, 2004. [10] Y. Jianchao, W. John, H. Thomas, Image Super-Resolution via Sparse Representation, IEEE Transactions on Image Processing, 19 (2010) 2861-2873. [11] D. Chao, C.L. Chen, K. He, X. Tang, Learning a Deep Convolutional Network for Image Super-Resolution, 2014. [12] G. Binnig, C.F. Quate, C. Gerber, Atomic Force Microscope, Physical Review Letters, 56 (1986) 930-933. [13] J. Zhong, and J. Yan , Seeing is Believing: Atomic Force Microscopy Imaging For Nanomaterials Research, RSC Advances, 6 (2016)1103-1121. [14] C.S. Oxvig, T. Arildsen, T. Larsen, Structure assisted compressed sensing reconstruction of undersampled AFM images, Ultramicroscopy, 172 (2017) 1-9. [15] G.Q. Han, B. Lin, Y.L. Lin, Reconstruction of atomic force microscopy image using compressed sensing, Micron, 105 (2018) 1-10. [16] S.J. Sreeja, M. Wilscy, Single image super-resolution based on compressive sensing and TV minimization sparse recovery for remote sensing images, Intelligent Computational Systems, 2014. [17] D.L. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006) 1289-1306. [18] F.J. Giessibl, Advances in atomic force microscopy, Rev. Mod. Phys., 75 (2003) 949983.
[19] S.K. Das, F.R. Badal, M.A. Rahman, M.A. Islam, S.K. Sarker, N. Paul, Improvement of Alternative Non-Raster Scanning Methods for High Speed Atomic Force Microscopy: A Review, IEEE Access, 7 (2019) 115603-115624. [20] E.J. Candes, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information, 2006. [21] C.B. Li, W.T. Yin, H. Jiang, Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013) 507-530. [22] S.B. Andersson, L.Y. Pao, Non-raster sampling in atomic force microscopy: A compressed sensing approach, American Control Conference, 2012. [23] R. G. Keys, Cubic convolution interpolation for digital image processing, IEEE Trans. on Acoust. Speech. & Signal Processing, 37(1981). [24] Park, Sung Cheol , M. K. Park , and M. G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Processing Magazine, 20(2003)2136. [25] R. Optics, Super-resolution through Error Energy Reduction, J. Modern Optics, 21 (1974) 709-720. [26] P. Chatterjee, S. Mukherjee, S. Chaudhuri, G. Seetharaman, Application Of PapoulisGerchberg Method In Image Super-Resolution and Inpainting, Computer Journal, 52 (2009) 80-89. [27] C. Dong, C.C. Loy, X. Tang, Accelerating the Super-Resolution Convolutional Neural Network, (2016). [28] W. Shi, J. Caballero, F. Huszár, J. Totz, A.P. Aitken, R. Bishop, D. Rueckert, Z. Wang, Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, Computer Vision & Pattern Recognition, 2016. [29] J. Kim, J.K. Lee, K.M. Lee, Accurate Image Super-Resolution Using Very Deep Convolutional Networks, IEEE Conference on Computer Vision & Pattern Recognition, 2016. [30] J. Kim, J.K. Lee, K.M. Lee, Deeply-Recursive Convolutional Network for Image Super-Resolution, (2015). [31] Image courtesy of Valery V. Prokhorov, Inst. of Bioorganic Chemistry RAS, Moscow, Russia. [32] Image courtesy Dr. Hannes C. Schniepp, t. C. o. W. M., Williamsburg, United States. [33] Image courtesy of Prof. M.Muller and N.Matsko, ETH Center, Zurich, Switzerland. [34] Image courtesy of T. Jung, A. Hoffmann, E. Soergel, University of Bonn. [35] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, 13 (2004) 600-612.
author statement Guoqiang Han: Conceptualization, Methodology, Writing - Review & Editing, Data Curation, Resources, Project administration, Funding acquisition, Supervision. Luyao Lv: Software, Validation, Formal analysis, Data curation, Writing- Original draft preparation. Gaopeng Yang: Data Curation, Investigation, Supervision. Yixiang Niu: Visualization, Investigation.
Declaration of Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Highlights: 1. The CS theory is introduced to solve the problems of time-consuming and low resolution of AFM imaging. 2. Six different samples and six different algorithms are used to estimate the results of superresolution imaging. 3. A large number of experiments show that CS algorithm has a good effect on super-resolution imaging of AFM images, especially when the upscaling factor is large. 4. By analyzing the TV or TV/R and surface morphology of the original image of the AFM sample, it is found that the image quality is largely determined by the morphology of the sample.