Variational JPEG artifacts suppression based on high-order MRFs

Author’s Accepted Manuscript Variational JPEG artifacts suppression based on high-order MRFs Yunjin Chen

www.elsevier.com/locate/image

PII: DOI: Reference:

S0923-5965(16)30189-8 http://dx.doi.org/10.1016/j.image.2016.12.006 IMAGE15155

To appear in: Signal Processing : Image Communication Received date: 2 February 2016 Revised date: 21 December 2016 Accepted date: 21 December 2016 Cite this article as: Yunjin Chen, Variational JPEG artifacts suppression based on high-order MRFs, Signal Processing : Image Communication, http://dx.doi.org/10.1016/j.image.2016.12.006 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting galley proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Variational JPEG artifacts suppression based on high-order MRFsI Yunjin Chen Institute for Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16, A-8010 Graz, Austria

Abstract The Block-wise Discrete Cosine Transform (B-DCT) based compression technique has been widely used in image and video coding standards. However, at high compression ratios, the coded images inevitably contain annoying blocking artifacts. In this paper, the author proposes a novel variational model for blocking artifacts suppression, which combines a discriminatively trained Fields of Experts (FoE) image prior model and the indicator function of the quantization constraint set (QCS). The FoE prior model is a filterbased higher-order Markov Random Fields (MRF) model, and it has proven to be effective for many image restoration problems. The resulting variational model leads to a generally difficult non-convex optimization problem, which can be efficiently solved by a recently proposed non-convex optimization algorithm. Numerical experiments show that the proposed deblocking approach leads to visually strongly comparable performance to state-of-the-art deblocking methods across a range of compression levels. Furthermore, our method can achieve higher PSNR-B results, which is a block-sensitive index, specialized for deblocked image evaluation and correlates well with subjective quality. Besides, the proposed model comes along with the additional advantage of high efficiency. Keywords: JPEG deblocking, Fields of Experts, non-convex optimization, MRFs

1. Introduction B-DCT has been successfully exploited in lossy image and video compression, such as JPEG, MPEG and H.263. Although it is well-known that the B-DCT coded images suffer from the so-called blocking effect, especially severe at low bit rate (i.e., high compression ratio), the classic JPEG standard is still the most popular lossy compression technique, and the dominant majority of the pictures circulated on the Internet is compressed by this standard. As a result of this fact, the development of advanced and efficient post-processing techniques to reduce the blocking artifacts is still a very active research topic. Recovery of B-DCT coded images has attracted a lot of research attention since early 1980s [17, 14], and therefore there are hundreds of publications to deal with this problem. Reviewing all the works of image deblocking is surely beyond the scope of this paper. Instead we briefly pick out some representative works. As a matter of fact, one can view the blocking artifacts from different aspects, which will lead to different solutions. Viewing the blocking artifacts as noise with certain structure, the deblocking problem corresponds to image denoising. There are different types of approaches proposed starting from this point of view: (1) image filtering technique [17, 14, 8, 24]; (2) wavelet thresholding technique which originates from image denoising for Gaussian white noise [22, 12]; (3) deblocking via sparse representation using a general dictionary trained by I This work was supported by the Austrian Science Fund (FWF) under the China Scholarship Council (CSC) Scholarship Program and the START project BIVISION, No. Y729. Email address: [email protected] (Yunjin Chen)

Preprint submitted to Signal Processing: Image Communication

the K-singular value decomposition (K-SVD) algorithm [11], whose original purpose is also for Gaussian noise removal; and (4) a recently introduced non-parametric image restoration model based on Regression Tree Fields (RTF) [10], which defines a framework leveraging the advantages of existing deblocking methods by incorporating their predictions into the filed model, and therefore generates a state of the art for image deblocking. On the other hand, the compression operation can be viewed as a degradation process, and many image restoration approaches are proposed to recover the original image, for example, algorithms based on projection onto convex sets (POCS) [9, 13]. In these methods, the prior information of the original image is defined as several convex sets, and an iterative projection algorithm is exploited to search the recovered image. Two commonly used convex sets are the QCS and the smoothness constraint set (SCS). The POCS-based methods are effective for reducing blocking artifacts. However, the basic difficulty is to construct appropriate convex sets to represent the image prior information of particular types. In general, image restoration is a typical inverse problem. One of the most successful approaches to solve inverse problems is to minimize a suitable energy functional whose minimizer provides a trade-off between a regularization term and a data term. Up to now, researchers have investigated the regularization technique for image deblocking, see for example [2, 4, 1, 19, 7]. These methods differ from each other in the regularization term or data term. Concerning the regularizer, the widely used Total Variation (TV), the Total Generalized Variation (TGV) [3] of second order and a higher-order MRF model December 23, 2016

based on the FoE image prior [18] have been investigated in [2, 4], [1] and [19] respectively. There are two methods to define the data term. One is the noise model [19], and the other is the indicator function of the QCS [2, 4, 1]. Many previous works have demonstrated that the FoE image prior model is more expressive for natural image modeling than hand-crafted models, such as TV and TGV models, since it explicitly captures the statistics properties of natural images. For the data term, although the Gaussian noise model exploited in [19] can produce better recovery results than its predecessor, it is only a coarse approximate model for the quantization noise (blocking artifacts). However, the indicator function of the QCS exactly depicts the image compression process, as the QCS defines an accurate convex set of original image, and therefore, it is a more accurate data fidelity term. As a consequence, it is interesting to combine the FoE-based prior term with the QCSbased data term. As far as we are aware, there is no previous work to investigate this model.

2.1. JPEG compression and the QCS In our work, we only consider gray-value images1 . A detailed illustration of the JPEG compression and decompression procedure can be found in [20]. In the step of quantization, the transformed DCT coefficients of each 8 × 8 block are pointwise divided by the quantization matrix, and then the values are rounded to integer, which is where the loss of data takes place, as the rounding operation is a mapping of “∞ → 1”. Given an integer number d, any number in the interval [d − 21 , d + 12 ] is a possible candidate for the original number which is rounded to d. With the compressed image data, we only know the integer coefficient data (di,q j )1≤i, j≤8 , where q indicates a 8 × 8 block indexed by q, and the quantization matrix (Qi, j )1≤i, j≤8 . Therefore, the possible original DCT coefficients, which yield (di,q j ) in the quantization and rounding step are given by the interval 1 1 S i,q j = [Qi, j (di,q j − ), Qi, j (di,q j + )] . 2 2

1.1. Our contributions In this paper, we introduce a novel variational model for image deblocking based on the FoE image prior model [18] and the QCS. This model incorporates better modes for both the regularization term and data term, and therefore defines a more accurate variational model for this task. Especially, the exploited QCS is usually lacking in traditional JPEG deblocking approaches. As mentioned before, image deblocking has a long history, and there exist numerous image deblocking methods. In order to demonstrate the effectiveness of our proposed method, we compare our approach with four representative methods: (1) a state-of-the-art method based on RTF [10]; (2) a method based on dictionary learning and sparse representation [4]; and two similar variational approaches (3) FoE prior based method (different in data term) [19] and (4) TGV regularized model (different in regularization term) [1]. Numerical experiments show that in terms of a blocksensitive index, PSNR-B, our variational model with a discriminatively trained FoE image regularization can obtain strongly competitive results to the state-of-the-art image deblocking results [10], and significantly surpass the results of (1) a previous deblocking model also based on the FoE prior [19]; (2) TGV regularized deblocking model [1]; (3) deblocking method based on TV regularization and sparse representation [4] and (4) the image filtering based algorithm - SADCT [8]. In additional, our deblocking algorithm is relative easy to understand and implement, and also well-suited for GPU parallel computation, which will make the inference procedure extremely efficient.

This result is for the block q. For the full size image, we just need to repeat this result for each distinct block. All the intervals S i,q j associated with each 8 × 8 block form the so-called QCS, which is simply a box constraint determining all possible source images. In order to simplify the notation, the interval S is represented by two column vectors a ∈ RN and b ∈ RN , which correspond to the lower and upper bounds of the intervals S i,p j , respectively. 2.2. The FoE prior utilized in our deblocking model The FoE model is defined by a set of linear filters. According to [18, 6, 5], the student-t distribution based FoE image prior model for an image u is formulated as XNk E FoE (u) = αi φ(ki ∗ u) , (2.1) i=1

where φ(ki ∗ u) = p=1 φ((ki ∗ u) p ), N is the number of pixels in image u, Nk is the number of filters, ki is a set of learned linear filters with the corresponding weights αi > 0, ki ∗ u denotes the convolution of the filter ki with a two-dimensional image u, and φ(·) denotes the Lorentzian potential function, PN

φ(x) = log(1 + x2 ) , which is derived from the student-t distribution. Note that the FoE energy is non-convex w.r.t. u. In this work, we directly make use of the learned filters of a previous work [6], as shown in Figure 1. 2.3. Variational model for image deblocking Recall that, in our formulations, an image u of size m × n is written as a column vector u ∈ RN with N = m × n. We further define a highly sparse matrix D ∈ RN×N , which makes

2. A novel variational model for image deblocking We first give a brief overview of the JPEG compression process, and introduce the QCS, which will be employed in our model. Then we propose a variational model based on the FoE prior and the QCS, and introduce an efficient algorithm to solve the corresponding optimization problem.

1 Note that extending our model to color images is straightforward. For color images, we can separately deal with the three channels of the compressed image in the YCbCr color space.

2

(0.44,1.89)

(0.48,1.56)

(0.35,2.12)

(1.25,0.59)

(1.14,0.55)

(1.13,0.54)

(0.68,0.85)

(0.72,0.79)

(0.68,0.84)

(0.71,0.79)

(0.45,1.22)

(0.42,1.26)

(0.55,0.95)

(0.79,0.64)

(0.63,0.78)

(0.50,0.97)

(0.81,0.56)

(0.34,1.31)

(0.42,1.04)

(0.31,1.39)

(0.27,1.54)

(0.37,1.13)

(0.34,1.19)

(0.29,1.36)

(0.27,1.31)

(0.71,0.49)

(0.25,1.32)

(0.20,1.32)

(0.13,1.58)

(0.12,1.59)

(0.11,1.58)

(0.09,1.56)

(0.08,1.57)

(0.08,1.56)

(0.03,1.57)

(0.03,1.60)

(0.02,1.60)

(0.02,1.60)

(0.02,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.01,1.60)

(0.00,1.60)

In (2.5), un − τ∇F (un ) is the forward gradient descent step, γ un − un−1 is the inertial term, and (I + τ∂G)−1 is the backward step. Casting (2.3) in the form of (2.4), we have X Nk F(u) = αi φ(ki ∗ u) and G(u) = IS (Du) . (2.6) i=1

It is clear that F(u) is smooth and G(u) is convex, and hence the iPiano algorithm can be applied. In order to use this algorithm, we need to calculate the gradient of F and the proximal map with respect to G. It is easy to check that XNk ∇u F(u) = αi Ki> φ0 (Ki u) , (2.7)

Figure 1: 48 learned filters of size 7 × 7 exploited in our despeckling model. The first number in the bracket is the norm of the filter and the second one is the weight αi .

Du equivalent to the B-DCT transform applied to the twodimensional image u. Given the compressed data, the QCS is given as the box constraint S = [a, b], and the set of possible source image to generate this compressed data is defined as

i=1

where Ki is an N × N highly sparse matrix, which is implemented as 2D convolution of the image u with filter kernel ki , i.e., Ki u ⇔ ki ∗u, φ0 (Ki u) = (φ0 ((Ki u)1 ), · · · , φ0 ((Ki u)N ))> ∈ RN , with φ0 (x) = 2x/(1 + x2 ). The proximal map with respect to G is given as the following minimization problem

U = {u ∈ RN |(Du) p ∈ [a p , b p ]} . Now we can define our variational model, which reads as X Nk αi φ(ki ∗ u) . (2.2) min E(u) = i=1

u∈U

(I + τ∂G)−1 (ˆu) = arg min u

This is a constrained optimization problem, and it can be rewritten as XNk min E(u) = αi φ(ki ∗ u) + IS (Du) , (2.3) where

kDu − Dˆuk22 = (u − uˆ )> D> D(u − uˆ ) = (u − uˆ )> (u − uˆ ) = ku − uˆ k22 .

   0 if Du ∈ S , IS (Du) =   ∞ else .

For problem 2.8, let

In this formulation, we directly exploit the convex set S instead of set U, as S is a box constraint, which is easier to handle than U.

c = Du, cˆ = Dˆu . Note that the connection between c and u (also cˆ and uˆ ) is a mapping of one-to-one. It turns out that

2.4. Solving the variational deblocking model An immediate question about the proposed image deblocking model is how to solve it. Due to the non-convexity of the prior term and the non-smoothness of the data term, it turns out that (2.3) is a very hard optimization problem. In this work, we resort to a recently published non-convex optimization algorithm - iPiano [16] to solve them. The iPiano algorithm is designed for a structured non-smooth non-convex optimization problem, which is composed of a smooth (possibly non-convex) function F and a convex (possibly non-smooth) function G: arg min H(u) = F(u) + G(u) . u

(2.8)

As DCT is a orthogonal transform, i.e., D> D = DD> = I, then we have

i=1

u

ku − uˆ k22 + τIS (Du) . 2

arg min u

ku − uˆ k22 kc − cˆ k22 + τIS (Du) ⇐⇒ arg min + τIS (c) . c 2 2

Obviously, the solution for the minimization problem of right side is given as the following point-wise projection onto the interval    cˆ p if cˆ p ∈ S p = [a p , b p ]     c˜ p =  b p if cˆ p > b p     a p if cˆ p < a p . Finally, the the solution of u is given as

(2.4)

u˜ = D> c˜

The iPiano algorithm is based on a forward-backward splitting scheme with an inertial force term. Its basic update rule is simple and given as    un+1 = (I + τ∂G)−1 un − τ∇F (un ) + γ un − un−1 , (2.5)

(2.9)

In summary, the overall variational deblocking process using the iPiano algorithm is outlined in Algorithm 2.1. 3. Experimental results

where τ and γ are step sizes. The term (I + τ∂G)−1 signifies the standard proximal mapping [15], given as

In order to evaluate the performance of our proposed variational image deblocking model based on a set of learned filters, we applied the proposed model to suppress the blocking

ku − uˆ k22 + τG(u) . ProxτG (ˆu) = (I + τ∂G)−1 (ˆu) = min u 2 3

Algorithm 2.1 The overall JPEG deblocking process using the iPiano algorithm Input: The JPEG compressed image y and its associated box constraint S Initialization: set Iter > 0, Choose γ = 0.8, l−1 = 1, η = 1.2, and initialize u0 = y and set u−1 = u0 For n = 0 : (Iter − 1)

exploited: two conventional objective measurements for general image restoration tasks, PSNR and SSIM index [21], as well as a specialized index for the JPEG deblocked image quality assessment, named PSNR-B [23], which is sensitive to the block artifacts. The PSNR-B index considers the blocking effect factor (BEF), which is also shown in our numerical results. It is pointed out in [23] that the standard PSNR index is not in accord with subjective judgment for the assessment of the deblocked images. In our experiments, we also observed that the quality index of PSNR, as well as SSIM, is indeed perceptually questionable, see the deblocked result in Figure 2 and 3 for an illustration. These two examples also verify that the block-sensitive image quality index PSNR-B correlates better with subjective quality. However, in this paper, we still keep the results of perceptually questionable PSNR/SSIM for the sake of completeness.

1. Conduct a line search to find the smallest nonnegative integer i such that with ln = ηi ln−1 , F(un+1 ) ≤ F (un ) + h∇F (un ) , un+1 − un i +

ln n+1 ku − un k22 . 2

is satisfied, where F and ∇F is defined as (2.6) and (2.7), respectively; 2. Set ln = ηi ln−1 , τn = 1.99 (1 − γ) /ln ;

3.1.1. Evaluation on BSDS500 As we know, image deblocking performance of a specific method varies greatly for different image contents. In order to make a fair comparison with other competing methods, we conducted deblocking experiments over a standard test dataset - BSDS500, consisting of 200 natural images, which was firstly used in [10] for deblocking performance evaluation. We followed the test procedure in [10]. The test images were converted to gray-value, and scaled by a factor of 0.5, resulting images of size 240 × 160. We distorted the images by JPEG blocking artifacts. We considered three compression quality settings q = 10, 20 and 30 for the JPEG encoder. The average PSNR/SSIM/PSNR-B/BEF results of the considered methods over the test dataset are summarized in Table 1. Remember that the block-sensitive image quality index PSNR-B is a better metric for quality assessment of deblocked images, as PSNR/SSIM correlate poorly with subjective judgment. In terms of the PSNR-B measurements, one can see that our variational model has surpassed all the competing methods and has achieved slightly higher performance than a state-ofthe-art work [10]. We present some deblocking results of our method and other competing methods in Figure 4 and 5 for the case of compression quality q = 10, in Figure 6 and 7 for the compression quality q = 20, and Figure 8 for the compression quality q = 30. Concerning the visual inspection, generally speaking, in the case of relatively high compression rate, e.g., q = 10, the results given by TGV based method are over-smoothed as the minimizer of the TGV regularized model is piece-wise affine. The dictionary based method frequently fails to remove the blocking artifacts in the homogeneous regions, such as in the sky. The RTF based method, FoE prior regularized model and our approach usually generate visually plausible deblocking results, but there are still some residual blocking artifacts in deblocked images produced by the FoE prior regularized model in previous work [19],thus inferior PSNR-B results. See Figure 4 and 5 for example.

3. Compute ∇u F (un ) according to (2.7); 4. Compute un+1 according to (2.5), (2.8) and (2.9); end For uˆ = uIter ; Solution: Estimated underlying HR image uˆ . artifacts in JPEG compressed images with different compression quality q. The compressed images are quantized using the quantization matrix Qq = round(50Q50 /q), where Q50 is the standard quantization matrix [20] given as   11 10 16 24 40 51 61   16   12 12 14 19 26 58 60 55    13 16 24 40 57 69 56   14  14 17 22 29 51 87 80 62  Q50 =   22 37 56 68 109 103 77   18   24 35 55 64 81 104 113 92    64 78 87 103 121 120 101   49 72 92 95 98 112 100 103 99 3.1. Quantitative and qualitative evaluation We compared our variational models with four representative methods: (1) a state-of-the art deblocking method based on RTF [10]; (2) method based on dictionary learning and sparse representation (SR) [4]; as well as two variational approaches (3) FoE prior based method [19] and (4) TGV regularized model[1]. For the sparse representation based method, we used the dictionary model without TV regularization term. For the RTF based model, we exploited the PSNRRTFSADCT system, which includes SA-DCT [8] as a base method and is optimized for the PSNR performance measure. Note that the implementation of the RTF algorithm for JPEG deblocking is not available and we only have the deblocked images on the BSDS500 database for a few quality q. As a consequence, we are not able to present the deblocking results of RTF for test images other than the BSDS500 database. In order to have a comprehensive quantitative evaluation of the considered approaches, three objective quality metrics are

3 http://r0k.us/graphics/kodak/

4

(b) JPEG image q = 10 (27.14/24.84/90.88/87.96)

(a) Clean image (PSNR/PSNR-B/SSIM×100/BEF)

(c) SA-DCT deblocking (27.87/27.30//92.32/14.81)

(d) Ours (27.80/27.80/92.10/0.00)

Figure 2: Deblocking results for an image from Kodak Lossless True Color Image Suite the case of quality q = 10. It is clear that our method leads to better visually pleasant deblocking result than the SA-DCT algorithm, as the latter fails to remove the annoying blocking artifacts in the sky region. However, in contrast, our result has inferior PSNR and SSIM value. For this example, the PSNR and SSIM metrics are perceptually questionable. In terms of the block-sensitive index, PSNR-B, our method results in better performance, achieving a significant gain of 0.5dB over SA-DCT. Furthermore, the deblocking results only achieve a gain less than 1dB in terms of PSNR over the original JPEG decoder; while there is a significant improvement about 3dB in terms of PSNR-B. 3 in

Quality q

Evaluation

JPEG decoder

TGV [1] deblocking

Dictionary SR [4]

FoE [19] deblocking

RTF [10]

SA-DCT [8]

Ours

10

PSNR PSNR-B SSIM BEF

26.59 23.63 76.10 152.5

26.96 26.41 77.80 20.94

27.15 26.22 77.87 38.49

27.40 26.33 79.10 41.44

27.68 27.12 79.47 19.26

27.43 26.53 78.64 35.41

27.44 27.33 78.94 3.8

20

PSNR PSNR-B SSIM BEF

28.77 25.76 84.47 96.01

29.01 28.28 85.03 18.71

29.03 28.39 83.76 17.54

29.54 28.10 86.47 36.3

29.83 28.99 86.68 18.56

29.46 28.06 85.98 37.37

29.57 29.34 86.09 4.73

30

PSNR PSNR-B SSIM BEF

30.05 27.10 88.04 69.97

30.25 29.44 88.33 16

30.13 29.58 86.35 11.16

30.82 29.16 89.63 31.49

31.14 30.25 89.85 14.59

30.67 29.05 89.24 33.17

30.90 30.60 89.35 4.45

Table 1: JPEG deblocking results for natural images in terms of PSNR/PSNR-B/SSIM(×100)/BEF index on the BSDS500 database. We compared our method to four representative image deblocking methods, including a state-of-the-art deblocking method based on RTF [10]. Note that PSNR and SSIM do not correlate well with subjective quality. The PSNR-B index, which is designed specifically to assess blocky and deblocked images is a better quality metric for the task of JPEG deblocking. Note that for the BEF index, smaller value is better.

to solve our proposed models, and typically it takes 40 iterations to converge to a stationary point. For the investigated algorithms, we make use of the codes provided by the authors

3.2. Run time We conducted a direct run time comparison for different deblocking algorithms based on CPU implementation. In Table 2, we show the average run time of the considered deblocking methods on 240 × 160 images. We use the iPiano algorithm 5

(a) Clean image (PSNR/PSNR-B/SSIM×100/BEF)

(b) JPEG image q = 10 (29.73/26.87/92.05/64.40)

(c) SA-DCT deblocking (31.12/30.77/94.59/4.21)

(d) Ours (31.11/31.11/94.56/0.00)

Figure 3: Deblocking results for the flower image from Kodak Lossless True Color Image Suite in the case of quality q = 10. Our method leads better visual performance as the deblocked image with SA-DCT still contains some visible blocking artifacts in the highlighted regions. However, both methods have equivalent PSNR and SSIM value, while our method achieves higher PSNR-B. Therefore, the PSNR-B index correlates better with subjective quality.

T(s) PSNR PSNR-B SSIM BEF

TGV [1] deblocking 7.42 26.96 26.41 77.80 20.94

Dictionary SR [4] 7.23 27.15 26.22 77.87 38.49

FoE [19] deblocking 50 27.40 26.33 79.10 41.44

SADCT [8]

Ours

3.53 27.43 26.53 78.64 35.41

5.29 (0.058) 27.44 27.33 78.94 3.8

Table 2: Typical run time (CPU computation) of the deblocking methods for a 240 × 160 image (q = 10) on a server (Intel X5675, 3.07GHz). The highlighted number is the run time of the GPU implementation.

as is4 . Our proposed deblocking model comes along with the additional advantage of simplicity, as it solely involves convolution of filters with an image. Therefore, our model is well-suited to GPU parallel computation. Table 3 presents the GPU computation time of our deblocking model for different image dimensions. Note that the TGV based method [1] and the previous FoE prior based method [19] are also suited for GPU parallel com-

putation, but we didn’t implement it.5 Also note that the RTF based approach, e.g., the PSNRRTFSADCT system relies on the output of the SADCT algorithm [8]. However, our proposed model is an independent algorithm, which dose not rely on the output of any existing methods. 4. Conclusion In this paper, we have proposed a novel variational model for image deblocking based on (1) an expressive image prior

4

However, we are not able to present the runtime of RTF method [10], as its implementation is not available. But we know it relies on the output of the SADCT method, thus its computation time is the runtime of SADCT plus the execution time of RTF (i.e., RTF based method is slower than the SADCT algorithm).

5 In the work of [1], for the TGV based deblocking algorithm (1000 iterations), the authors quoted a GPU computation time of 1.2s for the image size 512 × 512 based on NVIDIA Geforce GTX 580.

6

(b) JPEG q = 10 (24.99/22.24/81.16/182.31) (c) TGV[1] (25.44/25.07/83.77/16.47)

(a) Clean image

(e) FoE[19] (25.71/24.50/84.37/56.49)

(f) SA-DCT[8] (25.80/24.56/84.46/56.38)

(g) RTF[10] (26.17/25.40/85.31/30.33)

(d) Dic. SR[4] (25.58/24.50/84.06/52.28)

(h) Ours (25.83/25.83/84.59/0.00)

Figure 4: Image deblocking results for an image compressed by JPEG encoder with quality q = 10 on the BSDS500 database. PSNR/PSNR-B/SSIM ×100/BEF results are shown in the bracket. Note that some methods fail to remove the blocking artefacts in the sky region indicated by red rectangle.

(b) JPEG q = 10 (24.80/21.82/81.23/212.58) (c) TGV[1] (25.10/24.36/84.68/36.94)

(a) Clean image

(e) FoE[19] (25.48/23.82/85.08/85.61)

(f) SA-DCT[8] (25.52/23.88/84.97/83.65)

(g) RTF[10] (26.01/25.16/86.25/35.44)

(d) Dic. SR[4] (25.23/23.70/83.79/82.40)

(h) Ours (25.76/25.75/85.73/0.42)

Figure 5: Image deblocking results for an image compressed by JPEG encoder with quality q = 10 on the BSDS500 database. PSNR/PSNR-B/SSIM ×100/BEF results are shown in the bracket. Again, some methods fail to remove the blocking artefacts in the sky region indicated by red rectangle.

Ours(ms) SADCT(ms)

240 × 160 58.2 3530

480 × 320 153 11790

5122 230 17650

10242 822 70978

a generally demanding non-convex minimization problem and we introduced recently proposed algorithm - iPiano to solve it. We directly applied our discriminatively trained FoE image prior model in the case of Gaussian denoising to the deblocking model. Experimental results on a set of natural images demonstrate that our deblocking model with the learned FoE prior model leads to strongly competitive performance to the stateof-the art method. Our proposed model comes along with the additional advantage that the inference is extremely efficient. The GPU based implementation can conduct image deblocking in less than 1s for image sizes up to 1024 × 1024.

Table 3: The run time of our deblocking approach for different image size by using GPU computation (based on NVIDIA Geforce GTX 780Ti). We also present the CPU computation time for the SADCT algorithm (based on Intel X5675, 3.07GHz), which is the strongest competitor in terms of run time.

model - FoE model and (2) the indicator function of the QCS, which exactly defines a convex set of possible source images given the compressed image. This new variational model poses 7

(a) Clean image

(b) JPEG q = 20 (32.12/29.22/86.72/37.99)

(c) TGV[1] (32.29/31.58/87.44/6.82)

(d) Dic. SR[4] (31.85/31.45/85.17/4.11)

(e) FoE[19] (32.71/31.74/88.15/8.71)

(f) SA-DCT[8] (32.61/31.99/87.58/5.46)

(g) RTF[10] (32.88/32.13/88.32/6.31)

(h) Ours (32.57/32.24)/87.90/2.83

Figure 6: Image deblocking results for an image compressed by JPEG encoder with quality q = 20 on the BSDS500 database. PSNR/PSNR-B/SSIM ×100/BEF results are shown in the bracket.

(a) Clean image

(e) FoE[19] (27.45/24.90/87.35/93.51)

(b) JPEG q = 20 (26.58/23.26/85.05/164.14) (c) TGV[1] (27.33/26.30/86.33/32.16)

(f) SA-DCT[8] (27.58/24.88/87.21/97.84)

(g) RTF[10] (28.77/26.96/88.61/44.77)

(d) Dic. SR[4] (27.75/25.97/85.71/55.24)

(h) Ours (28.20/27.39/87.86/19.97)

Figure 7: Image deblocking results for an image compressed by JPEG encoder with quality q = 20 on the BSDS500 database. PSNR/PSNR-B/SSIM ×100/BEF results are shown in the bracket.

References [7] [1] Bredies, K., Holler, M., 2012. Artifact-free JPEG decompression with total generalized variation. In: VISAPP (1). pp. 12–21. [2] Bredies, K., Holler, M., 2012. A total variation-based JPEG decompression model. SIAM J. Imaging Sciences 5 (1), 366–393. [3] Bredies, K., Kunisch, K., Pock, T., 2010. Total generalized variation. SIAM Journal on Imaging Sciences 3 (3), 492–526. [4] Chang, H., Ng, M. K., Zeng, T., 2014. Reducing artifact in JPEG decompression via a learned dictionary. IEEE Transactions on Signal Processing 62 (3), 718–728. [5] Chen, Y., Pock, T., Ranftl, R., Bischof, H., 2013. Revisiting loss-specific training of filter-based MRFs for image restoration. In: GCPR. pp. 271– 281. [6] Chen, Y., Ranftl, R., Pock, T., 2014. Insights into analysis operator learn-

[8]

[9]

[10]

[11]

8

ing: From patch-based sparse models to higher order MRFs. IEEE Transactions on Image Processing 23 (3), 1060–1072. Do, Q. B., Beghdadi, A., Luong, M., 2010. A new adaptive image posttreatment for deblocking and deringing based on total variation method. In: 10th International Conference on Information Sciences, Signal Processing and their Applications, 2010. pp. 464–467. Foi, A., Katkovnik, V., Egiazarian, K., 2007. Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. Image Processing, IEEE Transactions on 16 (5), 1395–1411. Gan, X., Liew, A. W.-C., Yan, H., 2005. A smoothness constraint set based on local statistics of BDCT coefficients for image postprocessing. Image and Vision Computing 23 (8), 731–737. Jancsary, J., Nowozin, S., Rother, C., 2012. Loss-specific training of nonparametric image restoration models: A new state of the art. In: ECCV. pp. 112–125. Jung, C., Jiao, L., Qi, H., Sun, T., 2012. Image deblocking via sparse

(a) Clean image

(e) FoE[19] (29.51/27.68/91.27/38.18)

(b) JPEG q = 30 (28.68/25.98/89.26/75.92) (c) TGV[1] (28.91/28.37/90.46/11.03)

(f) SA-DCT[8] (29.22/27.26/90.76/44.47)

(g) RTF[10] (29.80/29.14/91.37/11.19)

(d) Dic. SR[4] (29.21/28.60/89.45/11.90)

(h) Ours (29.57/29.57/91.14/0.00)

Figure 8: Image deblocking results for an image compressed by JPEG encoder with quality q = 30 on the BSDS500 database. PSNR/PSNR-B/SSIM ×100/BEF results are shown in the bracket.

[12]

[13]

[14]

[15] [16]

[17] [18] [19]

[20] [21]

[22]

[23] [24]

representation. Signal Processing: Image Communication 27 (6), 663– 677. Liew, A.-C., Yan, H., 2004. Blocking artifacts suppression in block-coded images using overcomplete wavelet representation. IEEE Trans. Circuits Syst. Video Techn. 14 (4), 450–461. Liew, A.-C., Yan, H., Law, N.-F., 2005. POCS-based blocking artifacts suppression using a smoothness constraint set with explicit region modeling. IEEE Trans. Circuits Syst. Video Techn. 15 (6), 795–800. List, P., Joch, A., Lainema, J., Bjontegaard, G., Karczewicz, M., 2003. Adaptive deblocking filter. IEEE transactions on circuits and systems for video technology 13 (7), 614–619. Nesterov, Y., 2013. Introductory lectures on convex optimization: A basic course. Vol. 87. Springer Science & Business Media. Ochs, P., Chen, Y., Brox, T., Pock, T., 2014. iPiano: Inertial Proximal Algorithm for Nonconvex Optimization. SIAM J. Imaging Sciences 7 (2), 1388–1419. Reeve, H. C., Lim, J. S., 1983. Reduction of blocking effect in image coding. In: ICASSP. pp. 1212–1215. Roth, S., Black, M. J., 2009. Fields of experts. International Journal of Computer Vision 82 (2), 205–229. Sun, D., kuen Cham, W., 2007. Postprocessing of low bit-rate block DCT coded images based on a fields of experts prior. IEEE Transactions on Image Processing 16 (11), 2743–2751. Wallace, G. K., 1991. The JPEG still picture compression standard. Communications of the ACM 34 (4), 30–44. Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P., 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), 600–612. Xiong, Z., Orchard, M. T., Zhang, Y.-Q., 1997. A deblocking algorithm for JPEG compressed images using overcomplete wavelet representations. IEEE Trans. Circuits Syst. Video Techn. 7 (2), 433–437. Yim, C., Bovik, A. C., 2011. Quality assessment of deblocked images. Image Processing, IEEE Transactions on 20 (1), 88–98. Zhai, G., Zhang, W., Yang, X., Lin, W., Xu, Y., 2008. Efficient image deblocking based on postfiltering in shifted windows. IEEE Transactions on Circuits and Systems for Video Technology 18 (1), 122–126.

9