- Email: [email protected]
Deblocking filtering method using a perceptual map
Signal Processing: Image Communication 25 (2010) 527–534
Contents lists available at ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
Deblocking filtering method using a perceptual map Aladine Chetouani , Ghiles Mostafaoui, Azeddine Beghdadi Laboratory L2TI, Institute of Galilee, University Paris XIII, France
a r t i c l e in fo
abstract
Article history: Received 1 May 2009 Accepted 17 September 2009
A new method of deblocking is proposed. It aims to reduce the blocking artifacts in the compressed image by analyzing their visibility. A perceptual map is obtained using some Human Visual System (HVS) characteristics. This perceptual map is used as input to a recursive filter to reduce the blocking effect. The obtained results have been compared with a very recent efficient method. & 2009 Elsevier B.V. All rights reserved.
Keywords: Blocking effect Deblocking HVS Masking effect
1. Introduction To account for the non-stationarity of the image signal and the computational constraints, the image is often analyzed and processed on a block-by-block basis. There are other motivations behind the use of block-based image treatments such as real-time applications, parallel architectures and the limited channel bandwidth. However, block-based image-processing techniques may result in some annoying artifacts, such as blocking effects that may affect the image quality. For low bit rate, block-based coding methods produce a noticeable blocking effect. This is mainly due to the fact that the blocks are transformed and quantized independently. This annoying artefact appears as horizontal and vertical artificial contours and its visibility depends highly on the spatial intensity distribution in the image. Moreover, the Human Visual System (HVS) increases the perceived contrast between two adjacent regions. This is mainly due to inhibitory neural connections in the retina known as Mach effect [1]. Therefore, the visibility of the
Corresponding author.
E-mail addresses: [email protected] (A. Chetouani), [email protected] (G. Mostafaoui). [email protected] (A. Beghdadi). 0923-5965/$ - see front matter & 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.image.2009.09.006
blocking effect is more amplified due to the pronounced horizontal and vertical sensitivity of the HVS. Blocking effect has been widely studied and many ad hoc methods for measuring and reducing it have been proposed in the literature. In [2], a blocking effect measure based on boundary blocks analysis is proposed. To compute the blocking effect visibility vertically, the first step consists of computing the difference between each adjacent row (the same process is applied to obtain the visibility of the blocking effect horizontally). Then, a summation of the information of the boundaries and the information contained in the adjacent blocks of the boundaries is performed. Finally, after applying a zero crossing detection, the global measure is obtained by weighting the values achieved at the previous steps. In [3], a similar approach is proposed with some improvements. The masking effect is considered by taking into account only the maximum between the boundaries blocking effect measure and the zero crossing detection only if the computed visibility is superior to the Just Noticeable Distortion (JND). In [4], an iterative algorithm is applied for reducing the blocking effect artefact in the block transform-coded images by using a minimum mean square error filter. In [5], the blocking artefacts are modelled as 2D signals in the DCT-coded images. By taking into account some HVS properties, a visual parameter in the vicinity of the
528
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
inter-block boundary is used to remove the blocking effect. In [6], after forming a new block from two adjacent blocks 8 8, a zero masking is applied in the DCT domain to modify some particular DCT coefficients. In [7], Nosratinia proposed a method for blocking reduction where the processed image is obtained by averaging the shifted and JPEG-recompressed versions of the original compressed image. The bit rate recompression is obtained by reading the header file. In [8], a new technique based on a frequency analysis is proposed for reducing the blocking effects. The artefacts are modelled as 2-D step function between the neighbouring blocks. The blocks are classified into smooth and textured regions. Depending on the region type, an appropriate filter is applied. In [9], an artificial neural network is used in the DCT domain to reduce the blocking effect by removing some DCT coefficients of each intermediary block. Several other interesting methods for reducing the blocking effect have been also developed in [10,11]. In this work, we propose a different approach for reducing the visibility of the blocking effects without a prior information on the compression method. Indeed, the previous methods are based either on the size of blocks or on the compression type, which can be a disadvantage. Here, we compute a visibility map obtained by analyzing the visibility of the frontiers of adjacent regions by using some HVS characteristics namely Contrast Sensitivity Function (CSF), Cortex transform and masking effect. This visibility map is then used to control the blocking effect filtering by adapting, for each pixel, the filtering strength according to the associated visibility value. This paper is
organized as follows: Section 2 presents the method to obtain the visibility map and how to exploit it in the filtering process. Section 3 is dedicated to the results and the performance evaluation of our method. In this section we present some results obtained using different images. Finally, the last section contains some conclusions and perspectives. 2. The proposed method The main idea developed here is to reduce the blocking effect using perceptual information about the visibility of blocking effect. The first step is to extract from the distorted image a visibility map of the blocking effect. This visibility map is obtained using some well known HVS models, to analyze the visibility region by region in the degraded image. Once the visibility map computed, the following step consists in adapting for each pixel the filter parameter according to this visibility map. Therefore, the importance of each pixel becomes different according to its visibility. This perceptually adaptive filtering scheme is summarized in Fig. 1. 2.1. The blocking visibility map As explained above, the most annoying artifact induced by blocking effects is the increase of the boundary visibility between adjacent regions. To quantify this artefact, we propose using some HVS models to analyze the visibilities between all adjacent regions in a segmented image. The compressed image is first segmented into different regions. This segmentation produces two classes of
Distorted image
Segmentation
Iteration
Inter region visibility CSF, Cortex,Visual masking model
Global visibility map
Filtering process
Iterative criterion
Restored image
Fig. 1. Method flowchart.
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
regions C1 and C2. The first class corresponds to regions more affected by the distortions and the second one to those not or less affected by the distortions. This could be achieved by analyzing the spatial distribution of the pixels using the local variance as a homogeneity measure. Indeed, depending on the bit rate, blocking effect tends to create large uniform zones with null local variance. After classifying the pixels in the compressed image using the local variance criterion (C1 and C2 classes), a labelling process is applied to obtain delimited regions. More precisely:
A region of C1 is defined as a set of connected pixels
having the same gray-level and null local variance (Fig. 2b). A region of C2 is defined as a set of connected pixels with non-null local variance (Fig. 2c).
It is worth noticing that the visibility map is mainly used to point out the visible transitions between distorted regions (C1 class). Another point to mention is that the gray level of the transitions not affected by the compressed effect (C2 class) should not be taken into account in the filtering process. A result of segmentation is illustrated in Fig. 2d. Let us now define the following notations:
Ri is the ith region in the segmented image. Nij is the jth adjacent region of the region Ri. To compute the visibility between adjacent regions, the image is analyzed region by region. Let us consider a given region Ri and one of its adjacent region Nij. To analyze the visibility of the transition between these two regions, we define an image containing only this region on a black background and an image containing this region and one of its adjacent regions. Fig. 3c and d illustrates this representation. The underlying image model behind this description is that we assume that region Ri is the original image and the adjacent region Nij is considered as the added degradation. The visibility between this couple of images is then computed using an HVS-based distortion model. The most relevant characteristics of the HVS used here are its
529
directional and frequency selectivity, contrast sensitivity and masking. This approach has two main advantages. It allows:
A local analysis of the boundary visibilities between adjacent regions.
A perceptual guided extraction of the blocking visibility map without using a reference image (noncompressed image). Nevertheless, the boundary visibility between two regions depend not only on their structural information but also on the local contrasts and the global appearance of the image. This means that adding a black background (Fig. 3c and d) introduces small variations on the visibility coefficients. However, we will demonstrate in Section 2.2 that these errors can be easily handled. For the sake of completeness of the article, we will first present briefly some of the HVS characteristics used here before describing the process for extracting the visibility maps. The first HVS characteristic is the Contrast Sensitivity Function (CSF). It reflects the limitation of the HVS to discern fine details from a certain distance and under certain illumination conditions [12]. This HVS characteristic is modelled as a band-pass filter whose bandwidth depends on the viewing distance and also the type of image (luminance or chrominance). Here, we use the CSF luminance model of Daly [13]. Fig. 4 presents the CSF filter. The cortex transform is then used to model the frequency and directional selectivity of the HVS. It is a multi-channel decomposition using both Dom filter for the spatial frequency selectivity and Fan filter for the orientation selectivity [14]. Fig. 5 presents the frequency decomposition. Finally, visual masking refers to the reduction of the visibility of a signal (the target) in the presence of another (masked). Masking model attempts to explain how the presence of one signal affects the detectability of another signal in an image [15]. Different models exist. In this work, we use the masking model proposed by Daly [13]. The visibility threshold elevation is defined as follows: Elevationr;y ðx; yÞ ¼ ð1 þ ðk1 ðk2 jIsbr;y ðx; yÞjÞs Þb Þ1=b
Fig. 2. (a) Compressed image with a quality factor equal to 10, (b) regions of C1 class, (c) regions of C2 class and (d) segmented image.
ð1Þ
530
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Fig. 3. (a) Compressed image, (b) segmented image, (c) region Ri of the segmented image, (d) region Ri of the segmented image and one of its adjacent regions Nij, (e) visibility map obtained for the couple Ri and Nij, (f–j) examples of visibility maps obtained considering the region Ri and other adjacent regions, (k) visibility map obtained for Ri (considering all its adjacent regions) and (l) the global visibility map obtained for the image.
The masking function curve is shown in Fig. 6. To compute the visibility maps the images undergo different processes. First, the CSF filter is applied to each couple of images. Then, for each Cortex transformed image cortex transform, a differential visibility threshold elevation is calculated by applying Daly’s masking model. The result is used as input to a symmetric sigmoid function in the decision step in order to weight the masking effect as shown in Fig. 7. The global visibility map is then obtained by applying this process to all regions Ri in the image. The obtained visibility weights w are given by the following equation:
Fig. 4. Contrast Sensitivity Function (CSF).
w ðElevation ðr; y; x; yÞ; DVTðr; y; x; yÞÞ ¼
where k1=0.0153, k2=392.5, Isbr,y (x,y) is the signal value at the position (x,y) in the subband (r,y). s and b are the parameters depending on the frequency subband.
1 1 þ eðDVTðr;y;x;yÞElevationðr;y;x;yÞÞ
ð2Þ
Elevation(r,y,x,y) and DVT(r,y,x,y) are, respectively, the elevation obtained for the masked signal and the
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
531
Fig. 5. Cortex transform (multi-channel decomposition).
Fig. 9b and c represents the visibility blocking map and the associated 1-D profile, respectively. As can be seen from this example, the blocking visibility map clearly highlights the block frontiers. However, it decreases rapidly in the homogeneous regions. This result clearly shows that the quantification of the blocking effect is well expressed in the visibility map Fig. 10.
Differential visibility threshold elevation (DVT)
2.2. Filtering process
Masked signal contrast
0
Fig. 6. Masking function.
differential visibility threshold elevation obtained at the position (x,y) in the subband (r,y). The obtained results are presented in Fig. 4. Figs. 4(e–j) illustrate the visibility maps (Wji) obtained for a given region Ri by considering its different adjacent regions. At this stage, m visibility maps (Wji) are obtained for a given region Ri of the segmented image (where m describes the number of adjacent regions). The final visibility map obtained for a given region Ri is then computed as follows: W i ðx; yÞ ¼ maxfWji ðx; yÞg;
j ¼ 1...m
ð3Þ
As shown in Fig. 8, the pixel weight visibility differ depending on the considered adjacent region. The underlying idea behind Eq. (3) is to consider, for each pixel, only its higher visibility according to its local neighbourhood (adjacent regions). Fig. 4k and Fig. 8 represent the final visibility map Wi obtained for a given region Ri. One can notice that high visibilities are observed near the boundaries. Finally, the global visibility (V) map is obtained (Fig. 4l) by summing the elementary maps as follows: Xn W i ðx; yÞ ð4Þ Vðx; yÞ ¼ i¼1 where n represents the number of regions in the segmented image. To better illustrate the blocking effect visibility estimation, a synthetic image is considered (Fig. 9a).
One way to reduce the blocking effect is to smoothen the signal across the block boundaries. But this may result in loss of details and blurring. To overcome this side effect, a perceptual recursive filter is used. The basic idea is to exploit the visibility map in order to control the filtering process. Here, we chose the smoothing recursive Deriche filter [16], thanks to its separability and simplicity. Indeed, this filter depends only on one parameter a, which describes the filtering strength. Originally, this smoothing filter combined with a derivative filter has been used for edge detection. The smoothing is applied in both vertical and horizontal directions. For each direction two filters are defined: causal and anti-causal filters. Here, we use only the smoothing part of the Deriche filter defined as follows: y1ðnÞ ¼ k½xðnÞ þ gða 1Þxðn 1Þ þ gy1ðn 1Þ g2 y1ðn 2Þ
ð5Þ
y2ðnÞ ¼ k½gða þ 1Þxðn þ 1Þ g2 xðn þ 2Þ þ 2gy2ðn þ 1Þ g2 y2ðn þ 2Þ k¼
ð1 gÞ2 1 þ 2a g g2
and
g ¼ eðaÞ
ð6Þ ð7Þ
where y1(n) and y2(n) represent the output of the causal and anti-causal filters, respectively. k and g are the two parameters depending on a. To take into account the blocking visibility map, the parameter a is not fixed as in Deriche’s but varies as a function of the strength of blocking effect. The standard deviation of each pixel in the visibility map is then used as a local measure of the blocking effect strength.
532
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Visibility Weights (w)
Differential visibility threshold elevation (DVT)
w
Decision step
Masked signal contrast
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0 DVT (x, y)
Fig. 7. (a) Modified masking model, (b) the visibility weight function.
i
Wj (x, y)
W i (x, y)
Max
Fig. 8. Computation of the visibility map for a given region Ri.
Fig. 9. Simulated results: (a) synthetic blocking effect, (b) blocking visibility image and (c) 1-D profile of (b).
The a function used here is as follows:
b
2
aðx; yÞ ¼ pffiffiffiffiffiffi eð1=2Þðs½Vðx;yÞm=sÞ s 2p
ð8Þ
where s[V(x,y)] is the local standard deviation of the visibility map V5 at the location (x,y). m and s are the Gaussian parameters fixed to 0 and 2.5, respectively. b defines the range of a and its value is fixed to 2.5.
It is worth noticing that only the local standard deviation of the visibility map is used to filter the distorted image. This is mainly due to the fact that the added black backgrounds (used to compute inter-region visibilities) tend to uniformly amplify the visibility coefficient with the increase of the gray levels of the considered regions. This phenomenon is clearly illustrated by the 1-D profile of the visibility map presented in Fig. 9c.
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Using the standard deviations allows one to only consider the distribution of the coefficients around the local means of the visibilities and annihilate the errors induced by using black backgrounds. Another possibility is to use local or global contrast means instead of a black background. However, experimental tests show that this solution yields poor results. 3. Experimental results To test the efficiency of the proposed measure, experimental tests are performed using a set of natural
2.5
2
1.5
1
0.5
0 0
5
10 [V (x, y)] Fig. 10. Curve of (x,y).
15
20
533
images compressed at different bit rates. The experimental procedure is quite simple and does not require the original images or compression information. After computing the visibility map from the labelled image, the restored image is obtained by applying the smoothing part of the Deriche recursive filter using the visibility map. This procedure is applied iteratively and it is stopped when the difference between the average of the visibility map values obtained at the iteration t and t1 is smaller than a threshold T. Better results are obtained for T=2. The proposed method is compared to a recent efficient method proposed by Zhai et al. [17], which has been proven to outperform other similar methods. The first step of this method consists of filtering blocks that have higher visibility than a certain threshold based on HVS experience. Then, a quantization is applied in the DCT domain using the quantization matrix given in the header of the JPEG file. Fig. 11 illustrates the obtained results on a natural test image. Fig. 11a is the compressed test image. Fig. 11b is the filtered image by the Zhai method. Fig. 11c and d are the results of the proposed method obtained for the 1st and 3rd iterations, respectively. For a better visibility, a region is zoomed and shown in Fig. 12. The results obtained with the proposed method for one iteration are quite similar to those with Zhai et al.’s method. But for an increase of iterations better results are achieved with our method. Moreover, we notice that the homogeneous regions are more filtered than the texture regions, which permits to keep the details of image and decrease the visibility of the blocking effect. Indeed, the obtained visibility map confirms a decrease of visibility of blocking effect when the process is iterated.
Fig. 11. (a) Original JPEG compressed (Fq=10), (b) filtered image with Zhai method and (c–d) filtered images with the proposed method for the 1st and 3rd iterations, respectively.
534
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Fig. 12. (a) Zoomed compressed image with JPEG (Fq=10), (b) zoomed zone corresponding to Fig. 11.a, b, c and d, respectively.
4. Conclusion and perspectives An efficient deblocking method based on a recursive filtering exploiting the visibility map of blocking effect is proposed. The obtained results confirm the relevance of the visibility map in the design of the deblocking filter. The proposed method is compared to a recent efficient method. It is shown that our method provides better results than this method considered as the state-of-art. As perspective work, it would be interesting to adapt this method to video post-processing. In this case the visibility map should incorporate the temporal parameters. This method could also be extended to other block-based image compression techniques. References [1] W.T. Welford, The visual mach effect, Physics Education 3 (2) (1968) 83–88. [2] F. Pan, X. Lin, S. Rahardja, W. Lin, E. Ong, S. Yao, Z. Lu, X. Yang, A locally-adaptive algorithm for measuring blocking artifacts in images and videos, International Symposium on Circuits and Systems 3 (2004) 925–928. [3] Z. Wang, H. Sheikh, A. Bovik, No-reference perceptual quality assessment of jpeg compressed images, In Proceedings International Conference on Image Processing 1 (2002) 477–480. [4] I.H. Jang, N.C. Kim, H.J. So, Iterative blocking artefact reduction using a minimum mean square error filters in wavelet domain, Signal Processing 83 (2003) 2607–2619.
[5] F.X. Coudoux, M.G. Gazalet, P. Corlay, Reduction of blocking effect in DCT-coded images based on a visual perception criterion, Signal Processing: Image Communication 11 (3) (1998) 179–186. [6] B. Zeng, Reduction of blocking effect in DCT-coded images using zero-masking techniques, Signal Processing 79 (2) (1999) 205–211. [7] A. Nosratinia, Enhancement of JPEG-compressed images by re-application of JPEG, Journal of VLSI Signal Processing System (2001) 69–79. [8] S. Singh, V. Kumar, H.K. Verma, Reduction of blocking artefacts in JPEG compressed images, Digital Signal Processing 17 (1) (2007) 225–243. [9] C. Chang, C. Chan, C. Tseng, Removing blocking effects using an artificial neural network, Signal Processing 86 (2006) 2381–2387. [10] A. Zakhor, Iterative procedure for reduction of blocking effects in transform image coding, IEEE Transactions Circuits Systems Video Technology (1992) 91–94. [11] Y. Kim, C.S. Park, S.J. Ko, Fast POCS based post-processing technique for HDTV, IEEE Transactions on Consumer Electronics 49 (4) (2003) 1438–1447. [12] A.B. Watson, H.B. Barlow, J.G. Robson, What does the eye see best?, Nature 3 (1983) 419–422. [13] S. Daly, The visible differences predictor: an algorithm for the assessment of image fidelity, in: A.B. Watson (Ed.), Digital Images and Human Vision, MIT Press, 1993, pp. 179–206 ch 14. [14] A.B. Watson, The cortex transform: rapid computation of simulated neural images, Computer Vision Graphics and Image Processing 3 (1987) 311–327. [15] G. Legge, J. Foley, Contrast masking in human vision 70 (1980) 1458–4471. [16] R. Deriche, Using Canny’s criteria to derive a recursively implemented optimal edge detector, International Journal of Computer Vision (1987) 167–187. [17] G. Zhai, W. Zhang, X. Yang, W. Lin, Efficient image deblocking based on post filtering in shifted windows, IEEE Transactions on Circuits and Systems for Video Technology 18 (1) (2008) 122–126.
Contents lists available at ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
Deblocking filtering method using a perceptual map Aladine Chetouani , Ghiles Mostafaoui, Azeddine Beghdadi Laboratory L2TI, Institute of Galilee, University Paris XIII, France
a r t i c l e in fo
abstract
Article history: Received 1 May 2009 Accepted 17 September 2009
A new method of deblocking is proposed. It aims to reduce the blocking artifacts in the compressed image by analyzing their visibility. A perceptual map is obtained using some Human Visual System (HVS) characteristics. This perceptual map is used as input to a recursive filter to reduce the blocking effect. The obtained results have been compared with a very recent efficient method. & 2009 Elsevier B.V. All rights reserved.
Keywords: Blocking effect Deblocking HVS Masking effect
1. Introduction To account for the non-stationarity of the image signal and the computational constraints, the image is often analyzed and processed on a block-by-block basis. There are other motivations behind the use of block-based image treatments such as real-time applications, parallel architectures and the limited channel bandwidth. However, block-based image-processing techniques may result in some annoying artifacts, such as blocking effects that may affect the image quality. For low bit rate, block-based coding methods produce a noticeable blocking effect. This is mainly due to the fact that the blocks are transformed and quantized independently. This annoying artefact appears as horizontal and vertical artificial contours and its visibility depends highly on the spatial intensity distribution in the image. Moreover, the Human Visual System (HVS) increases the perceived contrast between two adjacent regions. This is mainly due to inhibitory neural connections in the retina known as Mach effect [1]. Therefore, the visibility of the
Corresponding author.
E-mail addresses: [email protected] (A. Chetouani), [email protected] (G. Mostafaoui). [email protected] (A. Beghdadi). 0923-5965/$ - see front matter & 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.image.2009.09.006
blocking effect is more amplified due to the pronounced horizontal and vertical sensitivity of the HVS. Blocking effect has been widely studied and many ad hoc methods for measuring and reducing it have been proposed in the literature. In [2], a blocking effect measure based on boundary blocks analysis is proposed. To compute the blocking effect visibility vertically, the first step consists of computing the difference between each adjacent row (the same process is applied to obtain the visibility of the blocking effect horizontally). Then, a summation of the information of the boundaries and the information contained in the adjacent blocks of the boundaries is performed. Finally, after applying a zero crossing detection, the global measure is obtained by weighting the values achieved at the previous steps. In [3], a similar approach is proposed with some improvements. The masking effect is considered by taking into account only the maximum between the boundaries blocking effect measure and the zero crossing detection only if the computed visibility is superior to the Just Noticeable Distortion (JND). In [4], an iterative algorithm is applied for reducing the blocking effect artefact in the block transform-coded images by using a minimum mean square error filter. In [5], the blocking artefacts are modelled as 2D signals in the DCT-coded images. By taking into account some HVS properties, a visual parameter in the vicinity of the
528
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
inter-block boundary is used to remove the blocking effect. In [6], after forming a new block from two adjacent blocks 8 8, a zero masking is applied in the DCT domain to modify some particular DCT coefficients. In [7], Nosratinia proposed a method for blocking reduction where the processed image is obtained by averaging the shifted and JPEG-recompressed versions of the original compressed image. The bit rate recompression is obtained by reading the header file. In [8], a new technique based on a frequency analysis is proposed for reducing the blocking effects. The artefacts are modelled as 2-D step function between the neighbouring blocks. The blocks are classified into smooth and textured regions. Depending on the region type, an appropriate filter is applied. In [9], an artificial neural network is used in the DCT domain to reduce the blocking effect by removing some DCT coefficients of each intermediary block. Several other interesting methods for reducing the blocking effect have been also developed in [10,11]. In this work, we propose a different approach for reducing the visibility of the blocking effects without a prior information on the compression method. Indeed, the previous methods are based either on the size of blocks or on the compression type, which can be a disadvantage. Here, we compute a visibility map obtained by analyzing the visibility of the frontiers of adjacent regions by using some HVS characteristics namely Contrast Sensitivity Function (CSF), Cortex transform and masking effect. This visibility map is then used to control the blocking effect filtering by adapting, for each pixel, the filtering strength according to the associated visibility value. This paper is
organized as follows: Section 2 presents the method to obtain the visibility map and how to exploit it in the filtering process. Section 3 is dedicated to the results and the performance evaluation of our method. In this section we present some results obtained using different images. Finally, the last section contains some conclusions and perspectives. 2. The proposed method The main idea developed here is to reduce the blocking effect using perceptual information about the visibility of blocking effect. The first step is to extract from the distorted image a visibility map of the blocking effect. This visibility map is obtained using some well known HVS models, to analyze the visibility region by region in the degraded image. Once the visibility map computed, the following step consists in adapting for each pixel the filter parameter according to this visibility map. Therefore, the importance of each pixel becomes different according to its visibility. This perceptually adaptive filtering scheme is summarized in Fig. 1. 2.1. The blocking visibility map As explained above, the most annoying artifact induced by blocking effects is the increase of the boundary visibility between adjacent regions. To quantify this artefact, we propose using some HVS models to analyze the visibilities between all adjacent regions in a segmented image. The compressed image is first segmented into different regions. This segmentation produces two classes of
Distorted image
Segmentation
Iteration
Inter region visibility CSF, Cortex,Visual masking model
Global visibility map
Filtering process
Iterative criterion
Restored image
Fig. 1. Method flowchart.
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
regions C1 and C2. The first class corresponds to regions more affected by the distortions and the second one to those not or less affected by the distortions. This could be achieved by analyzing the spatial distribution of the pixels using the local variance as a homogeneity measure. Indeed, depending on the bit rate, blocking effect tends to create large uniform zones with null local variance. After classifying the pixels in the compressed image using the local variance criterion (C1 and C2 classes), a labelling process is applied to obtain delimited regions. More precisely:
A region of C1 is defined as a set of connected pixels
having the same gray-level and null local variance (Fig. 2b). A region of C2 is defined as a set of connected pixels with non-null local variance (Fig. 2c).
It is worth noticing that the visibility map is mainly used to point out the visible transitions between distorted regions (C1 class). Another point to mention is that the gray level of the transitions not affected by the compressed effect (C2 class) should not be taken into account in the filtering process. A result of segmentation is illustrated in Fig. 2d. Let us now define the following notations:
Ri is the ith region in the segmented image. Nij is the jth adjacent region of the region Ri. To compute the visibility between adjacent regions, the image is analyzed region by region. Let us consider a given region Ri and one of its adjacent region Nij. To analyze the visibility of the transition between these two regions, we define an image containing only this region on a black background and an image containing this region and one of its adjacent regions. Fig. 3c and d illustrates this representation. The underlying image model behind this description is that we assume that region Ri is the original image and the adjacent region Nij is considered as the added degradation. The visibility between this couple of images is then computed using an HVS-based distortion model. The most relevant characteristics of the HVS used here are its
529
directional and frequency selectivity, contrast sensitivity and masking. This approach has two main advantages. It allows:
A local analysis of the boundary visibilities between adjacent regions.
A perceptual guided extraction of the blocking visibility map without using a reference image (noncompressed image). Nevertheless, the boundary visibility between two regions depend not only on their structural information but also on the local contrasts and the global appearance of the image. This means that adding a black background (Fig. 3c and d) introduces small variations on the visibility coefficients. However, we will demonstrate in Section 2.2 that these errors can be easily handled. For the sake of completeness of the article, we will first present briefly some of the HVS characteristics used here before describing the process for extracting the visibility maps. The first HVS characteristic is the Contrast Sensitivity Function (CSF). It reflects the limitation of the HVS to discern fine details from a certain distance and under certain illumination conditions [12]. This HVS characteristic is modelled as a band-pass filter whose bandwidth depends on the viewing distance and also the type of image (luminance or chrominance). Here, we use the CSF luminance model of Daly [13]. Fig. 4 presents the CSF filter. The cortex transform is then used to model the frequency and directional selectivity of the HVS. It is a multi-channel decomposition using both Dom filter for the spatial frequency selectivity and Fan filter for the orientation selectivity [14]. Fig. 5 presents the frequency decomposition. Finally, visual masking refers to the reduction of the visibility of a signal (the target) in the presence of another (masked). Masking model attempts to explain how the presence of one signal affects the detectability of another signal in an image [15]. Different models exist. In this work, we use the masking model proposed by Daly [13]. The visibility threshold elevation is defined as follows: Elevationr;y ðx; yÞ ¼ ð1 þ ðk1 ðk2 jIsbr;y ðx; yÞjÞs Þb Þ1=b
Fig. 2. (a) Compressed image with a quality factor equal to 10, (b) regions of C1 class, (c) regions of C2 class and (d) segmented image.
ð1Þ
530
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Fig. 3. (a) Compressed image, (b) segmented image, (c) region Ri of the segmented image, (d) region Ri of the segmented image and one of its adjacent regions Nij, (e) visibility map obtained for the couple Ri and Nij, (f–j) examples of visibility maps obtained considering the region Ri and other adjacent regions, (k) visibility map obtained for Ri (considering all its adjacent regions) and (l) the global visibility map obtained for the image.
The masking function curve is shown in Fig. 6. To compute the visibility maps the images undergo different processes. First, the CSF filter is applied to each couple of images. Then, for each Cortex transformed image cortex transform, a differential visibility threshold elevation is calculated by applying Daly’s masking model. The result is used as input to a symmetric sigmoid function in the decision step in order to weight the masking effect as shown in Fig. 7. The global visibility map is then obtained by applying this process to all regions Ri in the image. The obtained visibility weights w are given by the following equation:
Fig. 4. Contrast Sensitivity Function (CSF).
w ðElevation ðr; y; x; yÞ; DVTðr; y; x; yÞÞ ¼
where k1=0.0153, k2=392.5, Isbr,y (x,y) is the signal value at the position (x,y) in the subband (r,y). s and b are the parameters depending on the frequency subband.
1 1 þ eðDVTðr;y;x;yÞElevationðr;y;x;yÞÞ
ð2Þ
Elevation(r,y,x,y) and DVT(r,y,x,y) are, respectively, the elevation obtained for the masked signal and the
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
531
Fig. 5. Cortex transform (multi-channel decomposition).
Fig. 9b and c represents the visibility blocking map and the associated 1-D profile, respectively. As can be seen from this example, the blocking visibility map clearly highlights the block frontiers. However, it decreases rapidly in the homogeneous regions. This result clearly shows that the quantification of the blocking effect is well expressed in the visibility map Fig. 10.
Differential visibility threshold elevation (DVT)
2.2. Filtering process
Masked signal contrast
0
Fig. 6. Masking function.
differential visibility threshold elevation obtained at the position (x,y) in the subband (r,y). The obtained results are presented in Fig. 4. Figs. 4(e–j) illustrate the visibility maps (Wji) obtained for a given region Ri by considering its different adjacent regions. At this stage, m visibility maps (Wji) are obtained for a given region Ri of the segmented image (where m describes the number of adjacent regions). The final visibility map obtained for a given region Ri is then computed as follows: W i ðx; yÞ ¼ maxfWji ðx; yÞg;
j ¼ 1...m
ð3Þ
As shown in Fig. 8, the pixel weight visibility differ depending on the considered adjacent region. The underlying idea behind Eq. (3) is to consider, for each pixel, only its higher visibility according to its local neighbourhood (adjacent regions). Fig. 4k and Fig. 8 represent the final visibility map Wi obtained for a given region Ri. One can notice that high visibilities are observed near the boundaries. Finally, the global visibility (V) map is obtained (Fig. 4l) by summing the elementary maps as follows: Xn W i ðx; yÞ ð4Þ Vðx; yÞ ¼ i¼1 where n represents the number of regions in the segmented image. To better illustrate the blocking effect visibility estimation, a synthetic image is considered (Fig. 9a).
One way to reduce the blocking effect is to smoothen the signal across the block boundaries. But this may result in loss of details and blurring. To overcome this side effect, a perceptual recursive filter is used. The basic idea is to exploit the visibility map in order to control the filtering process. Here, we chose the smoothing recursive Deriche filter [16], thanks to its separability and simplicity. Indeed, this filter depends only on one parameter a, which describes the filtering strength. Originally, this smoothing filter combined with a derivative filter has been used for edge detection. The smoothing is applied in both vertical and horizontal directions. For each direction two filters are defined: causal and anti-causal filters. Here, we use only the smoothing part of the Deriche filter defined as follows: y1ðnÞ ¼ k½xðnÞ þ gða 1Þxðn 1Þ þ gy1ðn 1Þ g2 y1ðn 2Þ
ð5Þ
y2ðnÞ ¼ k½gða þ 1Þxðn þ 1Þ g2 xðn þ 2Þ þ 2gy2ðn þ 1Þ g2 y2ðn þ 2Þ k¼
ð1 gÞ2 1 þ 2a g g2
and
g ¼ eðaÞ
ð6Þ ð7Þ
where y1(n) and y2(n) represent the output of the causal and anti-causal filters, respectively. k and g are the two parameters depending on a. To take into account the blocking visibility map, the parameter a is not fixed as in Deriche’s but varies as a function of the strength of blocking effect. The standard deviation of each pixel in the visibility map is then used as a local measure of the blocking effect strength.
532
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Visibility Weights (w)
Differential visibility threshold elevation (DVT)
w
Decision step
Masked signal contrast
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0 DVT (x, y)
Fig. 7. (a) Modified masking model, (b) the visibility weight function.
i
Wj (x, y)
W i (x, y)
Max
Fig. 8. Computation of the visibility map for a given region Ri.
Fig. 9. Simulated results: (a) synthetic blocking effect, (b) blocking visibility image and (c) 1-D profile of (b).
The a function used here is as follows:
b
2
aðx; yÞ ¼ pffiffiffiffiffiffi eð1=2Þðs½Vðx;yÞm=sÞ s 2p
ð8Þ
where s[V(x,y)] is the local standard deviation of the visibility map V5 at the location (x,y). m and s are the Gaussian parameters fixed to 0 and 2.5, respectively. b defines the range of a and its value is fixed to 2.5.
It is worth noticing that only the local standard deviation of the visibility map is used to filter the distorted image. This is mainly due to the fact that the added black backgrounds (used to compute inter-region visibilities) tend to uniformly amplify the visibility coefficient with the increase of the gray levels of the considered regions. This phenomenon is clearly illustrated by the 1-D profile of the visibility map presented in Fig. 9c.
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Using the standard deviations allows one to only consider the distribution of the coefficients around the local means of the visibilities and annihilate the errors induced by using black backgrounds. Another possibility is to use local or global contrast means instead of a black background. However, experimental tests show that this solution yields poor results. 3. Experimental results To test the efficiency of the proposed measure, experimental tests are performed using a set of natural
2.5
2
1.5
1
0.5
0 0
5
10 [V (x, y)] Fig. 10. Curve of (x,y).
15
20
533
images compressed at different bit rates. The experimental procedure is quite simple and does not require the original images or compression information. After computing the visibility map from the labelled image, the restored image is obtained by applying the smoothing part of the Deriche recursive filter using the visibility map. This procedure is applied iteratively and it is stopped when the difference between the average of the visibility map values obtained at the iteration t and t1 is smaller than a threshold T. Better results are obtained for T=2. The proposed method is compared to a recent efficient method proposed by Zhai et al. [17], which has been proven to outperform other similar methods. The first step of this method consists of filtering blocks that have higher visibility than a certain threshold based on HVS experience. Then, a quantization is applied in the DCT domain using the quantization matrix given in the header of the JPEG file. Fig. 11 illustrates the obtained results on a natural test image. Fig. 11a is the compressed test image. Fig. 11b is the filtered image by the Zhai method. Fig. 11c and d are the results of the proposed method obtained for the 1st and 3rd iterations, respectively. For a better visibility, a region is zoomed and shown in Fig. 12. The results obtained with the proposed method for one iteration are quite similar to those with Zhai et al.’s method. But for an increase of iterations better results are achieved with our method. Moreover, we notice that the homogeneous regions are more filtered than the texture regions, which permits to keep the details of image and decrease the visibility of the blocking effect. Indeed, the obtained visibility map confirms a decrease of visibility of blocking effect when the process is iterated.
Fig. 11. (a) Original JPEG compressed (Fq=10), (b) filtered image with Zhai method and (c–d) filtered images with the proposed method for the 1st and 3rd iterations, respectively.
534
A. Chetouani et al. / Signal Processing: Image Communication 25 (2010) 527–534
Fig. 12. (a) Zoomed compressed image with JPEG (Fq=10), (b) zoomed zone corresponding to Fig. 11.a, b, c and d, respectively.
4. Conclusion and perspectives An efficient deblocking method based on a recursive filtering exploiting the visibility map of blocking effect is proposed. The obtained results confirm the relevance of the visibility map in the design of the deblocking filter. The proposed method is compared to a recent efficient method. It is shown that our method provides better results than this method considered as the state-of-art. As perspective work, it would be interesting to adapt this method to video post-processing. In this case the visibility map should incorporate the temporal parameters. This method could also be extended to other block-based image compression techniques. References [1] W.T. Welford, The visual mach effect, Physics Education 3 (2) (1968) 83–88. [2] F. Pan, X. Lin, S. Rahardja, W. Lin, E. Ong, S. Yao, Z. Lu, X. Yang, A locally-adaptive algorithm for measuring blocking artifacts in images and videos, International Symposium on Circuits and Systems 3 (2004) 925–928. [3] Z. Wang, H. Sheikh, A. Bovik, No-reference perceptual quality assessment of jpeg compressed images, In Proceedings International Conference on Image Processing 1 (2002) 477–480. [4] I.H. Jang, N.C. Kim, H.J. So, Iterative blocking artefact reduction using a minimum mean square error filters in wavelet domain, Signal Processing 83 (2003) 2607–2619.
[5] F.X. Coudoux, M.G. Gazalet, P. Corlay, Reduction of blocking effect in DCT-coded images based on a visual perception criterion, Signal Processing: Image Communication 11 (3) (1998) 179–186. [6] B. Zeng, Reduction of blocking effect in DCT-coded images using zero-masking techniques, Signal Processing 79 (2) (1999) 205–211. [7] A. Nosratinia, Enhancement of JPEG-compressed images by re-application of JPEG, Journal of VLSI Signal Processing System (2001) 69–79. [8] S. Singh, V. Kumar, H.K. Verma, Reduction of blocking artefacts in JPEG compressed images, Digital Signal Processing 17 (1) (2007) 225–243. [9] C. Chang, C. Chan, C. Tseng, Removing blocking effects using an artificial neural network, Signal Processing 86 (2006) 2381–2387. [10] A. Zakhor, Iterative procedure for reduction of blocking effects in transform image coding, IEEE Transactions Circuits Systems Video Technology (1992) 91–94. [11] Y. Kim, C.S. Park, S.J. Ko, Fast POCS based post-processing technique for HDTV, IEEE Transactions on Consumer Electronics 49 (4) (2003) 1438–1447. [12] A.B. Watson, H.B. Barlow, J.G. Robson, What does the eye see best?, Nature 3 (1983) 419–422. [13] S. Daly, The visible differences predictor: an algorithm for the assessment of image fidelity, in: A.B. Watson (Ed.), Digital Images and Human Vision, MIT Press, 1993, pp. 179–206 ch 14. [14] A.B. Watson, The cortex transform: rapid computation of simulated neural images, Computer Vision Graphics and Image Processing 3 (1987) 311–327. [15] G. Legge, J. Foley, Contrast masking in human vision 70 (1980) 1458–4471. [16] R. Deriche, Using Canny’s criteria to derive a recursively implemented optimal edge detector, International Journal of Computer Vision (1987) 167–187. [17] G. Zhai, W. Zhang, X. Yang, W. Lin, Efficient image deblocking based on post filtering in shifted windows, IEEE Transactions on Circuits and Systems for Video Technology 18 (1) (2008) 122–126.