- Email: [email protected]
A requantization algorithm for the transcoding of JPEG images
ARTICLE IN PRESS
Signal Processing: Image Communication 21 (2006) 13–21 www.elsevier.com/locate/image
A requantization algorithm for the transcoding of JPEG images Jae Won Moona, Jong Seok Leeb, Nam Ik Choc,d, a
S/W Lab. Telecommunication R&D Center, Samsung Electronics, DongSuwon PO Box 105, Suwon, Korea b Voiceware Inc., Daechi-dong, Kangnam-gu, Seoul 135-500, Korea c School of Electrical Engineering, Seoul National University, Shillim-Dong, Kwanak-Ku, Seoul 151-744, Korea d INMC (Institute of New Media and Communication), Seoul National University, Seoul, Korea Received 2 December 2004; accepted 29 April 2005
Abstract Photos from digital camera are mostly lossy compressed in JPEG format, and almost all users do not save or cannot access the original image. Sometimes we need to reduce the size of this JPEG file for saving the disk space or for sending the image to the device with limited bandwidth and/or display size. Since the original image is not available, we need to recompress the already compressed photos. In the existing literature, it is revealed that the direct transcoding (decompressing the JPEG image and then compressing it again with a larger step size) generally results in lower quality images when compared to the one compressed from the original image with the same parameters. This discrepancy is due to the requantization error, and the existing algorithms for more efficient transcoding were focused on finding the requantization step sizes for reducing this error. In this paper, we propose an algorithm that manipulates the requantized DCT coefficients for enhancing the quality of transcoded image, regardless of the choice of requantization step sizes. For this purpose, we first locate the DCT coefficients which have the possibility of causing large requantization errors. But, since the exact requantization error cannot be computed without the original image, we define the error as the discontinuity of pixels around the block boundaries, between the given JPEG image and the targeting one. Then, for the possible variants of these coefficients, the errors are computed and the best combination of coefficients is found. Experimental results show that more coefficients are correctly quantized than the conventional direct requantization, and thus the PSNR as well as the subjective quality is much improved. r 2005 Elsevier B.V. All rights reserved. Keywords: JPEG; Image compression; Requantization; Transcoding; DCT coefficients
Corresponding author. School of Electrical Engineering,
Seoul National University, Shillim-Dong, Kwanak-Ku, Seoul 151-744, Korea. E-mail address: [email protected] (N.I. Cho).
1. Introduction The use of JPEG compressed images on the Internet and in our daily life is ever growing with
0923-5965/$ - see front matter r 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.image.2005.04.003
ARTICLE IN PRESS 14
J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
the advancement of compression technology and digital cameras. The output from the digital camera is usually a JPEG image, and most images on the image servers and home pages are provided in JPEG format. Sometimes, there is a need to convert the resolution and/or file size of the JPEG image that is received from a server or a digital camera. In most cases, the original uncompressed image is not available, and hence we have to requantize the compressed image for reducing the bit rates. It is well explained in [2,3] that the quality of this requantized image is worse than the one quantized from the original uncompressed image, although the targeting bitrate is the same. The requantization process is again illustrated in Fig. 1 for further discussion and explanation of the proposed algorithm, which includes some notations used in this paper. We also employ the ‘‘quality scale’’ of Independent JPEG Group (IJG) as in [5] for representing the degree of compression (size of quantization steps). In this paper, the quality scale is denoted by Qi, which ranges from 1 (poor) to 100 (excellent) according to [8]. Fig. 1(a) is the JPEG with the quality scale Q1, where the input is X ðu; vÞ and the output is Y Q1 ðu; vÞ. Fig. 1(b) is the requantization of Y Q1 ðu; vÞ by the quality scale of Q2 ðQ2oQ1Þ, which results in Y Q1_Q2 ðu; vÞ. More specifically, Y a ðu; vÞ denotes the JPEG image obtained by quantizing the original image by the quality scale a. And Y a_b ðu; vÞ represents the image resulting from the requantization of Y a ðu; vÞ by the quality scale b. The quality of Y Q1_Q2 ðu; vÞ is often much worse than Y Q2 ðu; vÞ, the reason for which is illustrated in Fig. 2. In the case of the DCT coefficient in the ‘‘case 1’’ position, it becomes the
(a)
(b) Fig. 1. Block diagram and notations for the requantization problem: (a) Quantization of original image by quality scale Q1, (b) requantization of the quantized image by quality scale Q2.
5
Fig. 2. Illustration of requantization error.
2mþ1 center of ½2m1 2 D2 ; 2 D2 when it is quantized by the quality scale Q2. However, when it is first quantized by step size D1 , and again by D2 (by requantization), then it becomes the center of 2mþ3 ½2mþ1 2 D2 ; 2 D2 , which results in larger quantization error. This additional error is called the requantization error. In the case of the coefficient 2nþ5 in the ‘‘case 2,’’ since ½2nþ3 2 D1 ; 2 D1 lies inside of 2mþ1 2mþ3 ½ 2 D2 ; 2 D2 , there is no requantization error. When there are many ‘‘case 1’’ situations for the Q1–Q2 pair (i.e., many quantization steps of Q1 overlap with those of Q2), the requantization error becomes large. The analyses and methods for reducing the requantization error have been studied in many literatures [1,3,4,6–9], and most of the conventional requantization schemes have been focused on finding better Q2 for the given Q1. More precisely, let us consider the requantization of Lena image compressed by JPEG provided by the Independent JPEG Group [5]. The original Lena image is compressed by the quality 75 as in [3], and the result is requantized with the various quality parameters. Fig. 3(a) shows the PSNRs of Y Q2 ðu; vÞ and Y 75_Q2 ðu; vÞ, which shows that Y 75_Q2 ðu; vÞ is much worse than Y Q2 ðu; vÞ. But, since Y 75_Q2 ðu; vÞ does not have the same bit rates as Y Q2 ðu; vÞ, the PSNR with respect to bpp (bits per pixel) is also shown in Fig. 3(b) for fair comparison. As can be observed from the figures, Y 75_Q2 ðu; vÞ with lower Q2 or bpp often shows better quality than the ones with higher Q2 (and also bpp) [3]. As stated previously, the conventional algorithms have been focused on finding these better
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
quantization parameters, and there have been no algorithm for correcting the coefficients which are wrongly quantized by requantization. That is, there have been no efforts for fixing the ‘‘case 1’’ in Fig. 2. In this paper, we propose an algorithm that corrects the wrongly quantized coefficients by Lena
37.5
Y(Q2)
37
Y(75_Q2)
36.5
PSNR
36 35.5 35 34.5
15
using smoothness constrain as defined in the deblocking algorithms [10,11]. More specifically, when requantizing Y Q1 ðu; vÞ, the coefficients of Y Q1_Q2 ðu; vÞ are adjusted so that the pixels of Y Q1_Q2 ðu; vÞ around the block boundaries keep the continuity with those of Y Q1 ðu; vÞ. From the simulation, it is found that many of wrong coefficients are fixed by the proposed algorithm, and thus the PSNR is improved. By the proposed algorithm, we can have much choice for the targeting bit rates than the existing algorithms. This paper is organized as follows. In Section 2, we define the smoothness constraint and introduce the algorithm for the correction of requantized coefficients. In Section 3, experimental results are provided for various images and cases. In Section 4, conclusions are given.
34 33.5 60
55
(a)
50 quality parameter
45
40
We first introduce the basic idea of proposed algorithm, and then the detailed procedure. Some techniques for reducing the computational load are also discussed.
Lena
38.5
Y(Q2)(u,v)
38
2. Proposed requantization algorithm
Y(75_Q2)(u,v)
37.5 37
2.1. Object function for the correction of requantized coefficients
PSNR
36.5 36 35.5 35 34.5 34 33.5
0.8
0.75
(b)
0.7
0.65 bpp
0.6
0.55
0.5
Fig. 3. Requantization results of Lena image: (a) PSNR vs. quality scale, (b) PSNR vs. bpp.
As stated previously, some of the coefficients may fall into the wrong position as shown in Fig. 2 when the quantization steps of Q1 overlap with those of Q2. For example, Table 1 shows the number of wrongly quantized coefficients in the case of Y 75_Q2 ðu; vÞ. It should be noted that the bpp of Y Q2 ðu; vÞ is much different from that of Y 75_Q2 ðu; vÞ. Since it is fair to compare the images
Table 1 Number of wrongly quantized coefficients in the case of Y 75_Q2 ðu; vÞ Q2
60
55
50
45
40
No. of wrong coeff. PSNR of Y Q2 ðu; vÞ PSNR of Y 75_Q2 ðu; vÞ PSNR of Y q2 ðu; vÞ ðq2Þ Y q2 ðu; vÞ Y 75_Q2 ðu; vÞ
13162 37.2 35.9 38.1 (70) 2.2
14873 36.9 35.0 38.1 (70) 3.1
18452 36.6 33.9 38.0 (69) 4.1
7021 36.2 35.8 36.0 (41) 0.2
4009 35.0 34.9 35.2 (32) 0:2
Total number of coefficients are 262,144. Y q2 ðu; vÞ is the image with the closest bpp (bits per pixel) compared with Y 75_Q2 ðu; vÞ.
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
16
with the (almost) same bpp, we find the images which are directly compressed from the original image and have almost the same bpp as Y 75_Q2 ðu; vÞ. The quality scale in this case is denoted as q2. As expected, it can be observed from the last row of Table 1 that the requantized image has lower PSNR than the directly quantized one (Y q2 ) when there are more wrong coefficients. In order to improve the quality of requantized image, we attempt to correct these wrong coefficients without the original image. The correction is of course to reduce the requantization error. But since we do not have the original image, we need to define a new error measure which would reflect the difference from the original image. There would be many kinds of such error measure, but the continuity along the block boundary is considered here. More precisely, the basic notion is that the reduction of discontinuity along the block boundary would correct some of wrong coefficients. Specifically, the pixels of Y Q1_Q2 ðu; vÞ around the block boundaries should keep the continuity with those of neighboring blocks as in the case of postprocessing (deblocking) algorithms [10,11]. But unlike the deblocking algorithms, we have the image of higher quality, i.e., Y Q1 ðu; vÞ. Hence, we define the error measure as the difference of pixels of Y Q1_Q2 ðu; vÞ around the boundary from those of neighboring blocks of Y Q1 ðu; vÞ. In this paper, the smoothness constraint is defined the same as the one in [11], except that the neighboring blocks are from the image of higher quality. Fig. 4 shows a block (Bi ) of Y Q1_Q2 ðu; vÞ surrounded by neighboring blocks (Ak ’s) of Y Q1 ðu; vÞ and pixels in each block. The smoothness constraint is defined as Fi ¼ 2t þ 2b þ 2r þ 2l ,
(1)
where 2t
¼
7 X 1 k¼0
2
ð3ti;1 ðkÞ ti;2 ðkÞÞ
2 1 ð3bip;1 ðkÞ bip;2 ðkÞÞ 2
ð2Þ
is the cost for the smoothness of upper boundary of the block, and 2b ; 2r ; 2l are defined similarly as 2t . Now, the DCT coefficients are adjusted to minimize this Fi . But not all coefficients need to be
B
Fig. 4. A block Bi of Y Q1_Q2 ðu; vÞ, the coefficients of which will be corrected. Also shown are the neighboring blocks Ak ’s from Y Q1 ðu; vÞ. Small blocks are pixels in each block.
adjusted. A coefficient needs to be adjusted only when the quantization steps of Q1 overlap with those of Q2. In other words, we need not control the coefficient when the quantization step of Q1 lies inside that of Q2. For example, we need not adjust the coefficient that is inside 2nþ5 ½2nþ3 2 D1 ; 2 D1 in Fig. 2, because it is inside 2mþ1 ½ 2 D2 ; 2mþ3 2 D2 . Also, only two cases are considered for each coefficient, namely to keep the coefficient or move it to the center of either left or right step. For example, in the case of the 2mþ3 coefficient in ½2mþ1 2 D2 ; 2 D2 in Fig. 2, we check two cases: to keep it in the current position or 2mþ1 move it to the center of ½2m1 2 D2 ; 2 D2 (to the left step of current position), because the center of 2nþ3 ½2nþ1 2 D1 ; 2 D1 (the step that overlaps with 2mþ1 2mþ1 2mþ3 ½ 2 D2 ; 2mþ3 2 D2 ) is inside of ½ 2 D2 ; 2 D2 . If one of the steps in the case of Q1 overlaps with 2mþ3 ½2mþ1 2 D2 ; 2 D2 to the right, and if the center of it 2mþ3 is inside ½2mþ1 2 D2 ; 2 D2 , then we need to consider moving it to the right. In summary, not all coefficients are controlled. We need to consider keeping or moving the coefficients only when the quantization steps of Q1 overlap with those of Q2 such that the center of
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
17
2.2. Correction of requantized coefficient
2 2M=2 . It is found from the experiment that this method gives almost the same performance as the one that corrects all the M coefficients at once. Finally, it is noted that the coefficients in the higher frequency bands have larger effects than the lower ones, because the quantization step size in the higher band is larger. Hence, ‘‘more significant’’ coefficients are the ones in the higher band in the zig-zag order.
The procedure for the correction of requantized coefficient can be summarized as follows.
3. Experimental results
the former step is inside of the steps of the latter. Hence, the number of coefficients to be considered depends on Q1–Q2 pair. When the Q1–Q2 pair is the optimal one as in the conventional algorithms [1,3,4,6–9], there would be less coefficients to be considered, and if it not the case, then there are many coefficients to be corrected.
1. Determine the coefficients to be corrected. If the quantization step containing a coefficient of Y Q1 ðu; vÞ is not inside that of the quality scale Q2, then the coefficient is a candidate to be corrected. 2. If there are M (among 64) coefficients to be corrected, since each coefficient is ‘‘kept’’ or ‘‘moved’’ to either left or right, there are 2M different cases to be considered. The object function in (1) is computed for these 2M different cases. 3. Find the combination of ‘‘keep’’ and ‘‘move’’ of the coefficients that minimizes (1). Note that only one coefficient is changed when changing a combination of coefficients to another if we arrange ‘‘keep’’ and ‘‘move’’ in a Gray code style. For example, when two coefficients are considered and if we denote ‘‘keep’’ as 0 and ‘‘move’’ as 1, we can arrange the order of pattern as 00, 01, 11, 10. Then only a single coefficient is changed at each case and thus only a single multiplication per pixel is needed for IDCT. Since 48 pixels in Bi are involved for the computation of (1) as shown in Fig. 4, 48 multiplications and some additional computations for (1) are needed for each case. Hence, at least 48 2M multiplications are needed for the computation of error. This is fairly small when M is small, but it grows exponentially with respect to M. Hence, in order to avoid excessive computations, we first correct only half of the coefficients, and then the rest. That is, we first correct more significant M=2 coefficients, and then correct the rest later. Then the number of combinations are reduced from 2M to
The experiment is performed for several 512 512 images: Lena, Pepper, Barbara and Baboon. First, Table 2 shows that many coefficients are corrected by applying the proposed algorithm. Some figures in Table 1 (number of wrong coefficients before applying the proposed algorithm) are also repeated here for comparison. It can be observed that the number of wrong coefficients become smaller in most cases down to Q2 ¼ 50. As a result, the PSNR is improved as shown in Fig. 5(a), which shows the PSNR vs. Q2 in the case of Lena image. For fair comparison (as Fig. 3(b)), the PSNR with respect to bpp is also shown in Fig. 5(b). It can be observed that the PSNR is improved almost up to 3 dB by applying the proposed algorithm. For the subjective comparison, the directly requantized image at 0.82 bpp (33.88 dB) and a result of the proposed algorithm (36.83 dB at 0.78 bpp) are shown in Fig. 6. It can be seen that the proposed method improves the overall quality, especially in the low frequency region. In the case of high frequency region, the Table 2 Number of wrong coefficients before and after applying the proposed algorithm
Lena Pepper Baboon Barbara
Q2
60
55
50
45
40
Before After Before After Before After Before After
13162 11978 15202 13903 28943 29094 17492 18265
14873 10898 17443 12751 32214 27211 19545 17138
18452 11415 21443 13093 41423 29475 25179 18170
7021 8982 8086 9870 15993 20252 10171 13134
4009 7119 4405 7643 10348 16193 6222 10767
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
18
Lena
37.5 37
Lena
38
Y(75_Q2)(u,v) Y(Q2)(u,v) Proposed
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
37.5
36.5
37 PSNR
PSNR
36 35.5 35
36
34.5
35.5
34
35
33.5 60 (a)
36.5
55
50 Q2
45
34.5 0.8
40 (b)
0.75
0.7
0.65
0.6
bpp
Fig. 5. Comparison of transcoded image and the result of applying the proposed algorithm: (a) Comparison of PSNR vs. Q2 (Lena), (b) comparison of PSNR vs. bpp.
Fig. 6. Comparison of subjective quality: (a) Direct requantization (33.88 dB at 0.82 bpp), (b) proposed algorithm (36.83 dB at 0.78 bpp), (c) magnification of (a), (d) magnification of (b).
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
texture (e.g. the texture in the hat) is a little blurred because the object function is the smoothness constraint. That is, the edges around the block boundaries are sometimes smoothed when the proposed algorithm is applied. This is also evidenced in the results of Baboon and Barbara images, which have wide texture regions. More specifically, the number of wrong coefficients sometimes increases even at high Q2, as shown in the case of Q2 ¼ 60 in Table 2. Hence, if we wish to further improve the subjective quality, it might be better to adaptively apply the algorithm with respect to the frequency contents of the region. Fig. 7 shows the PSNR vs. bpp for other images: Pepper, Baboon and Barbara. In all the figures, the
28
PSNR
34.5 34
27
33.5
26
33
25
0.85
0.8
0.75
0.7
0.65
24 1.9 1.85 1.8 1.75 1.7 1.65 1.6 1.55 1.5 1.45 1.4
0.6
(b)
bpp
bpp
barbara
34.5
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
34 33.5 33 PSNR
PSNR
29
0.9
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
30
35
32.5 0.95
baboon
31 Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
35.5
(a)
proposed algorithm shows consistent degradation from the Y Q2 ðu; vÞ whereas the direct requantization shows some inconsistency. It is also observed that the performance of the proposed algorithm is better at the range of moderate bpp, but becomes a little poorer than the direct requantization in the low bpps. This can also be explained from Table 2, which shows that the number of wrong coefficients decreases for very low Q2. The main reason for this is that the number of step size overlapping decreases for the low Q2. That is, a quantization step of the targeting bitrate is so large at the very low Q2, thus many quantization steps of the original quality scale lies in it. Also, when the targeting step size is very large, wrong
Pepper
36
32.5 32 31.5 31 30.5 30 29.5 1.2
(c)
1.15
1.1
19
1.05
1
0.95
0.9
bpp
Fig. 7. Comparison for other images: (a) Pepper, (b) Baboon, (c) Barbara.
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
20
Fig. 8. Comparison of subjective quality at very low bitrate: (a) Direct requantization (35.4 dB at 0.64 bpp), (b) proposed algorithm (35.1 dB at 0.63 bpp).
Table 3 Average number of coefficients to be corrected per block
Lena
36.2
60
55
50
45
40
Lena Pepper Baboon Barbara
2.4 2.6 5.8 3.6
3.2 3.7 7.1 4.3
4.5 5.2 10.1 6.2
0.9 1.1 2.5 1.5
0.5 0.5 1.4 0.8
35.8 PSNR
Q2
method1 method2
36
35.6 35.4 35.2
manipulation of coefficients results in larger objective error. However, since the image quality at these low quantization scale is usually very low, with many block artifacts, the subjective quality of the proposed algorithm is still better as shown in Fig. 8. It can be seen that the result of proposed method has less block artifacts as if a deblocking algorithm is applied. For the comments on the computational loads, the number of coefficients to be corrected is shown in Table 3. The number is not so high and thus the computational load is not very serious though the proposed algorithm needs the computation of Oð2M Þ. But for the reduction of computational loads, we first correct first M=2 coefficients and then the rest as stated in the previous section. By doing so, the computation time is greatly reduced, while yielding almost the same performance as the case of correcting all the coefficients at once. Fig. 9 shows the comparison of two methods (Method 1: correction of all the coefficients at once. Method 2:
35 34.8 0.54
0.56
0.58
0.6
0.62 bpp
0.64
0.66
0.68
0.7
Fig. 9. Comparison of the performance for the case of correcting M=2 coefficients first and the rest later.
correction of M=2 coefficients first and then the rest), which shows that there is almost no difference in the performance.
4. Conclusions In this paper, we have proposed an algorithm for the correction of DCT coefficients in the transcoding of JPEG images. We first find the DCT coefficients which may cause large requantization error. Each of the coefficients is either kept in the current position or moved to the right or left
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
depending on the relationships of quantization step sizes. Then for each combination of these coefficients, the error measure that reflect the smoothness around the block boundary is computed and the combination with the least error is found. This results in the reduced number of wrong coefficients, and thus yields improved images. A modified method for the reduction of computational load is also proposed. The experimental results show that the proposed algorithm corrects many of the wrongly requantized coefficients, and improves the PSNR up to 3 dB in some cases. Also, we can have a wider choice of targeting bit rates than the conventional algorithms, which select the optimal quantization parameters. It is expected that the proposed algorithm can be applied to the transcoding of moving pictures, especially to the correction of coefficients in intraframes. References [1] P.A.A. Assunc- a˜o, M. Ghanbari, Optimal transcoding of compressed video, IEEE Int. Conf. Image Process. 1 (October 1997) 739–742. [2] P.A.A. Assunc- a˜o, M. Ghanbari, A frequency-domain video transcoder for dynamic bit-rate reduction of
[3]
[4]
[5] [6]
[7]
[8]
[9]
[10]
[11]
21
MPEG-2 bit streams, IEEE Trans. Circuits and Syst. Video Technol. 8 (8) (December 1998) 953–967. H.H. Bauschke, C.H. Hamilton, M.S. Macklem, J.S. McMichael, N.R. Swart, A requantization-based method for recompressing JPEG images, IEEE Int. Conf. on Acoustics, Speech, Signal Process. 3 (May 2002) 2489–2492. H.H. Bauschke, C.H. Hamilton, M.S. Macklem, J.S. McMichael, N.R. Swart, Recompression of JPEG images by requantization, IEEE Trans. Image Process. 12 (7) (July 2003) 843–849. http://www.ijg.org Y. Nakajima, H. Hori, T. Kanoh, Rate conversion of MPEG coded video by re-quantization process, IEEE Int. Conf. Image Process. 3 (October 1995) 408–411. H. Sorial, W.E. Lynch, A. Vincent, Selective requantization for transcoding of MPEG compressed video, IEEE Int. Conf. Multimedia and Expo. 1 (August 2000) 200–217. H. Sorial, W.E. Lynch, A. Vincent, Estimating Laplacian parameters of DCT coefficients for requantization in the transcoding of MPEG-2 video, IEEE Int. Conf. Image Process. 1 (September 2000) 956–959. O. Werner, Requantization for transcoding of MPEG-2 intraframes, IEEE Trans. Image Process. 8 (2) (February 1999) 179–191. Y. Yang, N.P. Galatsanos, Projection-based spatially adaptive reconstruction of block transform compressed images, IEEE Trans. Image Process. 4 (7) (July 1995) 896–908. A. Zakhor, Iterative procedures for reduction of block effects in transform image coding, IEEE Trans. on Circuits and Syst. Video Technol. 2 (1) (March 1992) 91–95.
Signal Processing: Image Communication 21 (2006) 13–21 www.elsevier.com/locate/image
A requantization algorithm for the transcoding of JPEG images Jae Won Moona, Jong Seok Leeb, Nam Ik Choc,d, a
S/W Lab. Telecommunication R&D Center, Samsung Electronics, DongSuwon PO Box 105, Suwon, Korea b Voiceware Inc., Daechi-dong, Kangnam-gu, Seoul 135-500, Korea c School of Electrical Engineering, Seoul National University, Shillim-Dong, Kwanak-Ku, Seoul 151-744, Korea d INMC (Institute of New Media and Communication), Seoul National University, Seoul, Korea Received 2 December 2004; accepted 29 April 2005
Abstract Photos from digital camera are mostly lossy compressed in JPEG format, and almost all users do not save or cannot access the original image. Sometimes we need to reduce the size of this JPEG file for saving the disk space or for sending the image to the device with limited bandwidth and/or display size. Since the original image is not available, we need to recompress the already compressed photos. In the existing literature, it is revealed that the direct transcoding (decompressing the JPEG image and then compressing it again with a larger step size) generally results in lower quality images when compared to the one compressed from the original image with the same parameters. This discrepancy is due to the requantization error, and the existing algorithms for more efficient transcoding were focused on finding the requantization step sizes for reducing this error. In this paper, we propose an algorithm that manipulates the requantized DCT coefficients for enhancing the quality of transcoded image, regardless of the choice of requantization step sizes. For this purpose, we first locate the DCT coefficients which have the possibility of causing large requantization errors. But, since the exact requantization error cannot be computed without the original image, we define the error as the discontinuity of pixels around the block boundaries, between the given JPEG image and the targeting one. Then, for the possible variants of these coefficients, the errors are computed and the best combination of coefficients is found. Experimental results show that more coefficients are correctly quantized than the conventional direct requantization, and thus the PSNR as well as the subjective quality is much improved. r 2005 Elsevier B.V. All rights reserved. Keywords: JPEG; Image compression; Requantization; Transcoding; DCT coefficients
Corresponding author. School of Electrical Engineering,
Seoul National University, Shillim-Dong, Kwanak-Ku, Seoul 151-744, Korea. E-mail address: [email protected] (N.I. Cho).
1. Introduction The use of JPEG compressed images on the Internet and in our daily life is ever growing with
0923-5965/$ - see front matter r 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.image.2005.04.003
ARTICLE IN PRESS 14
J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
the advancement of compression technology and digital cameras. The output from the digital camera is usually a JPEG image, and most images on the image servers and home pages are provided in JPEG format. Sometimes, there is a need to convert the resolution and/or file size of the JPEG image that is received from a server or a digital camera. In most cases, the original uncompressed image is not available, and hence we have to requantize the compressed image for reducing the bit rates. It is well explained in [2,3] that the quality of this requantized image is worse than the one quantized from the original uncompressed image, although the targeting bitrate is the same. The requantization process is again illustrated in Fig. 1 for further discussion and explanation of the proposed algorithm, which includes some notations used in this paper. We also employ the ‘‘quality scale’’ of Independent JPEG Group (IJG) as in [5] for representing the degree of compression (size of quantization steps). In this paper, the quality scale is denoted by Qi, which ranges from 1 (poor) to 100 (excellent) according to [8]. Fig. 1(a) is the JPEG with the quality scale Q1, where the input is X ðu; vÞ and the output is Y Q1 ðu; vÞ. Fig. 1(b) is the requantization of Y Q1 ðu; vÞ by the quality scale of Q2 ðQ2oQ1Þ, which results in Y Q1_Q2 ðu; vÞ. More specifically, Y a ðu; vÞ denotes the JPEG image obtained by quantizing the original image by the quality scale a. And Y a_b ðu; vÞ represents the image resulting from the requantization of Y a ðu; vÞ by the quality scale b. The quality of Y Q1_Q2 ðu; vÞ is often much worse than Y Q2 ðu; vÞ, the reason for which is illustrated in Fig. 2. In the case of the DCT coefficient in the ‘‘case 1’’ position, it becomes the
(a)
(b) Fig. 1. Block diagram and notations for the requantization problem: (a) Quantization of original image by quality scale Q1, (b) requantization of the quantized image by quality scale Q2.
5
Fig. 2. Illustration of requantization error.
2mþ1 center of ½2m1 2 D2 ; 2 D2 when it is quantized by the quality scale Q2. However, when it is first quantized by step size D1 , and again by D2 (by requantization), then it becomes the center of 2mþ3 ½2mþ1 2 D2 ; 2 D2 , which results in larger quantization error. This additional error is called the requantization error. In the case of the coefficient 2nþ5 in the ‘‘case 2,’’ since ½2nþ3 2 D1 ; 2 D1 lies inside of 2mþ1 2mþ3 ½ 2 D2 ; 2 D2 , there is no requantization error. When there are many ‘‘case 1’’ situations for the Q1–Q2 pair (i.e., many quantization steps of Q1 overlap with those of Q2), the requantization error becomes large. The analyses and methods for reducing the requantization error have been studied in many literatures [1,3,4,6–9], and most of the conventional requantization schemes have been focused on finding better Q2 for the given Q1. More precisely, let us consider the requantization of Lena image compressed by JPEG provided by the Independent JPEG Group [5]. The original Lena image is compressed by the quality 75 as in [3], and the result is requantized with the various quality parameters. Fig. 3(a) shows the PSNRs of Y Q2 ðu; vÞ and Y 75_Q2 ðu; vÞ, which shows that Y 75_Q2 ðu; vÞ is much worse than Y Q2 ðu; vÞ. But, since Y 75_Q2 ðu; vÞ does not have the same bit rates as Y Q2 ðu; vÞ, the PSNR with respect to bpp (bits per pixel) is also shown in Fig. 3(b) for fair comparison. As can be observed from the figures, Y 75_Q2 ðu; vÞ with lower Q2 or bpp often shows better quality than the ones with higher Q2 (and also bpp) [3]. As stated previously, the conventional algorithms have been focused on finding these better
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
quantization parameters, and there have been no algorithm for correcting the coefficients which are wrongly quantized by requantization. That is, there have been no efforts for fixing the ‘‘case 1’’ in Fig. 2. In this paper, we propose an algorithm that corrects the wrongly quantized coefficients by Lena
37.5
Y(Q2)
37
Y(75_Q2)
36.5
PSNR
36 35.5 35 34.5
15
using smoothness constrain as defined in the deblocking algorithms [10,11]. More specifically, when requantizing Y Q1 ðu; vÞ, the coefficients of Y Q1_Q2 ðu; vÞ are adjusted so that the pixels of Y Q1_Q2 ðu; vÞ around the block boundaries keep the continuity with those of Y Q1 ðu; vÞ. From the simulation, it is found that many of wrong coefficients are fixed by the proposed algorithm, and thus the PSNR is improved. By the proposed algorithm, we can have much choice for the targeting bit rates than the existing algorithms. This paper is organized as follows. In Section 2, we define the smoothness constraint and introduce the algorithm for the correction of requantized coefficients. In Section 3, experimental results are provided for various images and cases. In Section 4, conclusions are given.
34 33.5 60
55
(a)
50 quality parameter
45
40
We first introduce the basic idea of proposed algorithm, and then the detailed procedure. Some techniques for reducing the computational load are also discussed.
Lena
38.5
Y(Q2)(u,v)
38
2. Proposed requantization algorithm
Y(75_Q2)(u,v)
37.5 37
2.1. Object function for the correction of requantized coefficients
PSNR
36.5 36 35.5 35 34.5 34 33.5
0.8
0.75
(b)
0.7
0.65 bpp
0.6
0.55
0.5
Fig. 3. Requantization results of Lena image: (a) PSNR vs. quality scale, (b) PSNR vs. bpp.
As stated previously, some of the coefficients may fall into the wrong position as shown in Fig. 2 when the quantization steps of Q1 overlap with those of Q2. For example, Table 1 shows the number of wrongly quantized coefficients in the case of Y 75_Q2 ðu; vÞ. It should be noted that the bpp of Y Q2 ðu; vÞ is much different from that of Y 75_Q2 ðu; vÞ. Since it is fair to compare the images
Table 1 Number of wrongly quantized coefficients in the case of Y 75_Q2 ðu; vÞ Q2
60
55
50
45
40
No. of wrong coeff. PSNR of Y Q2 ðu; vÞ PSNR of Y 75_Q2 ðu; vÞ PSNR of Y q2 ðu; vÞ ðq2Þ Y q2 ðu; vÞ Y 75_Q2 ðu; vÞ
13162 37.2 35.9 38.1 (70) 2.2
14873 36.9 35.0 38.1 (70) 3.1
18452 36.6 33.9 38.0 (69) 4.1
7021 36.2 35.8 36.0 (41) 0.2
4009 35.0 34.9 35.2 (32) 0:2
Total number of coefficients are 262,144. Y q2 ðu; vÞ is the image with the closest bpp (bits per pixel) compared with Y 75_Q2 ðu; vÞ.
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
16
with the (almost) same bpp, we find the images which are directly compressed from the original image and have almost the same bpp as Y 75_Q2 ðu; vÞ. The quality scale in this case is denoted as q2. As expected, it can be observed from the last row of Table 1 that the requantized image has lower PSNR than the directly quantized one (Y q2 ) when there are more wrong coefficients. In order to improve the quality of requantized image, we attempt to correct these wrong coefficients without the original image. The correction is of course to reduce the requantization error. But since we do not have the original image, we need to define a new error measure which would reflect the difference from the original image. There would be many kinds of such error measure, but the continuity along the block boundary is considered here. More precisely, the basic notion is that the reduction of discontinuity along the block boundary would correct some of wrong coefficients. Specifically, the pixels of Y Q1_Q2 ðu; vÞ around the block boundaries should keep the continuity with those of neighboring blocks as in the case of postprocessing (deblocking) algorithms [10,11]. But unlike the deblocking algorithms, we have the image of higher quality, i.e., Y Q1 ðu; vÞ. Hence, we define the error measure as the difference of pixels of Y Q1_Q2 ðu; vÞ around the boundary from those of neighboring blocks of Y Q1 ðu; vÞ. In this paper, the smoothness constraint is defined the same as the one in [11], except that the neighboring blocks are from the image of higher quality. Fig. 4 shows a block (Bi ) of Y Q1_Q2 ðu; vÞ surrounded by neighboring blocks (Ak ’s) of Y Q1 ðu; vÞ and pixels in each block. The smoothness constraint is defined as Fi ¼ 2t þ 2b þ 2r þ 2l ,
(1)
where 2t
¼
7 X 1 k¼0
2
ð3ti;1 ðkÞ ti;2 ðkÞÞ
2 1 ð3bip;1 ðkÞ bip;2 ðkÞÞ 2
ð2Þ
is the cost for the smoothness of upper boundary of the block, and 2b ; 2r ; 2l are defined similarly as 2t . Now, the DCT coefficients are adjusted to minimize this Fi . But not all coefficients need to be
B
Fig. 4. A block Bi of Y Q1_Q2 ðu; vÞ, the coefficients of which will be corrected. Also shown are the neighboring blocks Ak ’s from Y Q1 ðu; vÞ. Small blocks are pixels in each block.
adjusted. A coefficient needs to be adjusted only when the quantization steps of Q1 overlap with those of Q2. In other words, we need not control the coefficient when the quantization step of Q1 lies inside that of Q2. For example, we need not adjust the coefficient that is inside 2nþ5 ½2nþ3 2 D1 ; 2 D1 in Fig. 2, because it is inside 2mþ1 ½ 2 D2 ; 2mþ3 2 D2 . Also, only two cases are considered for each coefficient, namely to keep the coefficient or move it to the center of either left or right step. For example, in the case of the 2mþ3 coefficient in ½2mþ1 2 D2 ; 2 D2 in Fig. 2, we check two cases: to keep it in the current position or 2mþ1 move it to the center of ½2m1 2 D2 ; 2 D2 (to the left step of current position), because the center of 2nþ3 ½2nþ1 2 D1 ; 2 D1 (the step that overlaps with 2mþ1 2mþ1 2mþ3 ½ 2 D2 ; 2mþ3 2 D2 ) is inside of ½ 2 D2 ; 2 D2 . If one of the steps in the case of Q1 overlaps with 2mþ3 ½2mþ1 2 D2 ; 2 D2 to the right, and if the center of it 2mþ3 is inside ½2mþ1 2 D2 ; 2 D2 , then we need to consider moving it to the right. In summary, not all coefficients are controlled. We need to consider keeping or moving the coefficients only when the quantization steps of Q1 overlap with those of Q2 such that the center of
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
17
2.2. Correction of requantized coefficient
2 2M=2 . It is found from the experiment that this method gives almost the same performance as the one that corrects all the M coefficients at once. Finally, it is noted that the coefficients in the higher frequency bands have larger effects than the lower ones, because the quantization step size in the higher band is larger. Hence, ‘‘more significant’’ coefficients are the ones in the higher band in the zig-zag order.
The procedure for the correction of requantized coefficient can be summarized as follows.
3. Experimental results
the former step is inside of the steps of the latter. Hence, the number of coefficients to be considered depends on Q1–Q2 pair. When the Q1–Q2 pair is the optimal one as in the conventional algorithms [1,3,4,6–9], there would be less coefficients to be considered, and if it not the case, then there are many coefficients to be corrected.
1. Determine the coefficients to be corrected. If the quantization step containing a coefficient of Y Q1 ðu; vÞ is not inside that of the quality scale Q2, then the coefficient is a candidate to be corrected. 2. If there are M (among 64) coefficients to be corrected, since each coefficient is ‘‘kept’’ or ‘‘moved’’ to either left or right, there are 2M different cases to be considered. The object function in (1) is computed for these 2M different cases. 3. Find the combination of ‘‘keep’’ and ‘‘move’’ of the coefficients that minimizes (1). Note that only one coefficient is changed when changing a combination of coefficients to another if we arrange ‘‘keep’’ and ‘‘move’’ in a Gray code style. For example, when two coefficients are considered and if we denote ‘‘keep’’ as 0 and ‘‘move’’ as 1, we can arrange the order of pattern as 00, 01, 11, 10. Then only a single coefficient is changed at each case and thus only a single multiplication per pixel is needed for IDCT. Since 48 pixels in Bi are involved for the computation of (1) as shown in Fig. 4, 48 multiplications and some additional computations for (1) are needed for each case. Hence, at least 48 2M multiplications are needed for the computation of error. This is fairly small when M is small, but it grows exponentially with respect to M. Hence, in order to avoid excessive computations, we first correct only half of the coefficients, and then the rest. That is, we first correct more significant M=2 coefficients, and then correct the rest later. Then the number of combinations are reduced from 2M to
The experiment is performed for several 512 512 images: Lena, Pepper, Barbara and Baboon. First, Table 2 shows that many coefficients are corrected by applying the proposed algorithm. Some figures in Table 1 (number of wrong coefficients before applying the proposed algorithm) are also repeated here for comparison. It can be observed that the number of wrong coefficients become smaller in most cases down to Q2 ¼ 50. As a result, the PSNR is improved as shown in Fig. 5(a), which shows the PSNR vs. Q2 in the case of Lena image. For fair comparison (as Fig. 3(b)), the PSNR with respect to bpp is also shown in Fig. 5(b). It can be observed that the PSNR is improved almost up to 3 dB by applying the proposed algorithm. For the subjective comparison, the directly requantized image at 0.82 bpp (33.88 dB) and a result of the proposed algorithm (36.83 dB at 0.78 bpp) are shown in Fig. 6. It can be seen that the proposed method improves the overall quality, especially in the low frequency region. In the case of high frequency region, the Table 2 Number of wrong coefficients before and after applying the proposed algorithm
Lena Pepper Baboon Barbara
Q2
60
55
50
45
40
Before After Before After Before After Before After
13162 11978 15202 13903 28943 29094 17492 18265
14873 10898 17443 12751 32214 27211 19545 17138
18452 11415 21443 13093 41423 29475 25179 18170
7021 8982 8086 9870 15993 20252 10171 13134
4009 7119 4405 7643 10348 16193 6222 10767
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
18
Lena
37.5 37
Lena
38
Y(75_Q2)(u,v) Y(Q2)(u,v) Proposed
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
37.5
36.5
37 PSNR
PSNR
36 35.5 35
36
34.5
35.5
34
35
33.5 60 (a)
36.5
55
50 Q2
45
34.5 0.8
40 (b)
0.75
0.7
0.65
0.6
bpp
Fig. 5. Comparison of transcoded image and the result of applying the proposed algorithm: (a) Comparison of PSNR vs. Q2 (Lena), (b) comparison of PSNR vs. bpp.
Fig. 6. Comparison of subjective quality: (a) Direct requantization (33.88 dB at 0.82 bpp), (b) proposed algorithm (36.83 dB at 0.78 bpp), (c) magnification of (a), (d) magnification of (b).
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
texture (e.g. the texture in the hat) is a little blurred because the object function is the smoothness constraint. That is, the edges around the block boundaries are sometimes smoothed when the proposed algorithm is applied. This is also evidenced in the results of Baboon and Barbara images, which have wide texture regions. More specifically, the number of wrong coefficients sometimes increases even at high Q2, as shown in the case of Q2 ¼ 60 in Table 2. Hence, if we wish to further improve the subjective quality, it might be better to adaptively apply the algorithm with respect to the frequency contents of the region. Fig. 7 shows the PSNR vs. bpp for other images: Pepper, Baboon and Barbara. In all the figures, the
28
PSNR
34.5 34
27
33.5
26
33
25
0.85
0.8
0.75
0.7
0.65
24 1.9 1.85 1.8 1.75 1.7 1.65 1.6 1.55 1.5 1.45 1.4
0.6
(b)
bpp
bpp
barbara
34.5
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
34 33.5 33 PSNR
PSNR
29
0.9
Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
30
35
32.5 0.95
baboon
31 Y(Q2)(u,v) Y(75_Q2)(u,v) Proposed
35.5
(a)
proposed algorithm shows consistent degradation from the Y Q2 ðu; vÞ whereas the direct requantization shows some inconsistency. It is also observed that the performance of the proposed algorithm is better at the range of moderate bpp, but becomes a little poorer than the direct requantization in the low bpps. This can also be explained from Table 2, which shows that the number of wrong coefficients decreases for very low Q2. The main reason for this is that the number of step size overlapping decreases for the low Q2. That is, a quantization step of the targeting bitrate is so large at the very low Q2, thus many quantization steps of the original quality scale lies in it. Also, when the targeting step size is very large, wrong
Pepper
36
32.5 32 31.5 31 30.5 30 29.5 1.2
(c)
1.15
1.1
19
1.05
1
0.95
0.9
bpp
Fig. 7. Comparison for other images: (a) Pepper, (b) Baboon, (c) Barbara.
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
20
Fig. 8. Comparison of subjective quality at very low bitrate: (a) Direct requantization (35.4 dB at 0.64 bpp), (b) proposed algorithm (35.1 dB at 0.63 bpp).
Table 3 Average number of coefficients to be corrected per block
Lena
36.2
60
55
50
45
40
Lena Pepper Baboon Barbara
2.4 2.6 5.8 3.6
3.2 3.7 7.1 4.3
4.5 5.2 10.1 6.2
0.9 1.1 2.5 1.5
0.5 0.5 1.4 0.8
35.8 PSNR
Q2
method1 method2
36
35.6 35.4 35.2
manipulation of coefficients results in larger objective error. However, since the image quality at these low quantization scale is usually very low, with many block artifacts, the subjective quality of the proposed algorithm is still better as shown in Fig. 8. It can be seen that the result of proposed method has less block artifacts as if a deblocking algorithm is applied. For the comments on the computational loads, the number of coefficients to be corrected is shown in Table 3. The number is not so high and thus the computational load is not very serious though the proposed algorithm needs the computation of Oð2M Þ. But for the reduction of computational loads, we first correct first M=2 coefficients and then the rest as stated in the previous section. By doing so, the computation time is greatly reduced, while yielding almost the same performance as the case of correcting all the coefficients at once. Fig. 9 shows the comparison of two methods (Method 1: correction of all the coefficients at once. Method 2:
35 34.8 0.54
0.56
0.58
0.6
0.62 bpp
0.64
0.66
0.68
0.7
Fig. 9. Comparison of the performance for the case of correcting M=2 coefficients first and the rest later.
correction of M=2 coefficients first and then the rest), which shows that there is almost no difference in the performance.
4. Conclusions In this paper, we have proposed an algorithm for the correction of DCT coefficients in the transcoding of JPEG images. We first find the DCT coefficients which may cause large requantization error. Each of the coefficients is either kept in the current position or moved to the right or left
ARTICLE IN PRESS J. Won Moon et al. / Signal Processing: Image Communication 21 (2006) 13–21
depending on the relationships of quantization step sizes. Then for each combination of these coefficients, the error measure that reflect the smoothness around the block boundary is computed and the combination with the least error is found. This results in the reduced number of wrong coefficients, and thus yields improved images. A modified method for the reduction of computational load is also proposed. The experimental results show that the proposed algorithm corrects many of the wrongly requantized coefficients, and improves the PSNR up to 3 dB in some cases. Also, we can have a wider choice of targeting bit rates than the conventional algorithms, which select the optimal quantization parameters. It is expected that the proposed algorithm can be applied to the transcoding of moving pictures, especially to the correction of coefficients in intraframes. References [1] P.A.A. Assunc- a˜o, M. Ghanbari, Optimal transcoding of compressed video, IEEE Int. Conf. Image Process. 1 (October 1997) 739–742. [2] P.A.A. Assunc- a˜o, M. Ghanbari, A frequency-domain video transcoder for dynamic bit-rate reduction of
[3]
[4]
[5] [6]
[7]
[8]
[9]
[10]
[11]
21
MPEG-2 bit streams, IEEE Trans. Circuits and Syst. Video Technol. 8 (8) (December 1998) 953–967. H.H. Bauschke, C.H. Hamilton, M.S. Macklem, J.S. McMichael, N.R. Swart, A requantization-based method for recompressing JPEG images, IEEE Int. Conf. on Acoustics, Speech, Signal Process. 3 (May 2002) 2489–2492. H.H. Bauschke, C.H. Hamilton, M.S. Macklem, J.S. McMichael, N.R. Swart, Recompression of JPEG images by requantization, IEEE Trans. Image Process. 12 (7) (July 2003) 843–849. http://www.ijg.org Y. Nakajima, H. Hori, T. Kanoh, Rate conversion of MPEG coded video by re-quantization process, IEEE Int. Conf. Image Process. 3 (October 1995) 408–411. H. Sorial, W.E. Lynch, A. Vincent, Selective requantization for transcoding of MPEG compressed video, IEEE Int. Conf. Multimedia and Expo. 1 (August 2000) 200–217. H. Sorial, W.E. Lynch, A. Vincent, Estimating Laplacian parameters of DCT coefficients for requantization in the transcoding of MPEG-2 video, IEEE Int. Conf. Image Process. 1 (September 2000) 956–959. O. Werner, Requantization for transcoding of MPEG-2 intraframes, IEEE Trans. Image Process. 8 (2) (February 1999) 179–191. Y. Yang, N.P. Galatsanos, Projection-based spatially adaptive reconstruction of block transform compressed images, IEEE Trans. Image Process. 4 (7) (July 1995) 896–908. A. Zakhor, Iterative procedures for reduction of block effects in transform image coding, IEEE Trans. on Circuits and Syst. Video Technol. 2 (1) (March 1992) 91–95.