- Email: [email protected]
Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream
Signal Processing 73 (1999) 291—295
Fast communication
Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream Eung Tae Kim *, Seung-Jong Choi, Hyung-Myung Kim Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 373-1 Kusong-Dong, Yusong-Gu, Taejon 305-701, South Korea Digital TV Research Laboratory, LG Electronics Inc., 16 Woomyeon-Dong, Seocho-Gu, Seoul 137-140, South Korea Received 13 August 1998; received in revised form 11 December 1998
Abstract A novel error concealment procedure is presented for recovering the lost macroblock while decoding an erroneous MPEG-2 bitstream. The proposed weighted boundary matching algorithm (WBMA) refines the conventional boundary matching algorithm. Different from the conventional BMAs, the boundary pixels of a lost macroblock are compared with those of the previous picture located in the same position. The weighting coefficients are also adaptively adjusted to work robustly in either smooth or abruptly-changing regions. Experimental results show that the proposed algorithm outperforms the conventional algorithms, especially for the interlaced sequences having large motion or complicated textures. 1999 Published by Elsevier Science B.V. All rights reserved. Keywords: Error concealment; MPEG-2 decoder; Weighted boundary matching algorithm
1. Introduction In the MPEG-2 video receiver, transmission errors cannot be avoided in the physical channels such as terrestrial broadcasting channels and ATM networks [5]. To overcome this problem, many researchers have studied the error concealment (EC) techniques [1—4]. Basically, those techniques can be classified into temporal predictive concealment (TPC) and spatial predictive concealment (SPC) [3]. In [3], it has been shown that TPC using
* Corresponding author. Tel. #82-42-869-5440, 3440; fax: #82-42-869-3410; e-mail: [email protected]
the motions of neighboring MBs has the better performance than SPC. However, the paid cost is the overhead memory, since motion vectors of an entire picture should be stored. Another problem is that the assumption of smooth motion vectors is not always true. Recently, a boundary matching algorithm [2,4] was proposed to recover the motion vector of a lost block. In [1], the overlapped motion compensation is combined with the side matching criterion to mask the blocking artifacts. Since the pixels of the lost macroblock are not available, the matching is performed with the boundary pixels surrounding the lost MB. This side matching criterion utilizes the smoothness of pixel values even across block
0165-1684/99/$ — see front matter 1999 Published by Elsevier Science B.V. All rights reserved. PII: S 0 1 6 5 - 1 6 8 4 ( 9 8 ) 0 0 2 4 2 - 4
292
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
boundaries. It works well in the region where no large change in pixel values occurs across the boundary. However, in case neighboring pixel values change abruptly, for example, around edges or corners, the incorrect motion vector would be obtained and the recovered MB by concealment does not fit well. The performance would be seriously degraded for the interlaced sequences, since the spatial correlation is low between fields having different parities. The weighted matching criterion is proposed in this work to search reliable motion vectors. This approach still inherits the boundary matching algorithm. However, different from the conventional BMAs, the boundary pixels of a lost MB are directly compared with those of the matched MB in the previous frame. Furthermore, the weighting coefficients are adaptively adjusted by calculating the spatial correlations across the boundaries to overcome the above mentioned problems.
2. Proposed error concealment technique It is assumed that the locations of damaged blocks are already known via variable length decoding (VLD) process which tests the syntax error
or the semantic rule error in the bitstream [5]. In general, MPEG standard considered the resynchronization problem by inserting several starting codes (especially, the slice unit). If the VLD detects the error code or a syntax error in the bitstream, the error concealment procedure is performed until a next slice start code is detected. Thus, the error propagation can be limited within a slice, even if the pixels in an error-corrupted slice are lost. In this paper, concealment of macroblocks is carried out by recovering the motion vectors of them and then by replacing them with the motion compensated MBs in the previous frame. Let x(i, j) and xˆ (i, j) denote the luminance pixel at location (i, j) in the current frame and the previous frame, respectively. (» ,» ) is the motion vector of the lost block and V W N the block size. Fig. 1 shows the proposed weighted boundary matching algorithm. The bestmatched motion vector is found by minimizing the following equation: min DR (» ,» ), (1) GH V W 4V 4WZ10 where SR is the search range for motion vector. The proposed cost function is defined as DR (» ,» )" DI (» ,» ), (2) GH V W GH V W I
Fig. 1. Weighted boundary matching algorithm.
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
where
follows:
+ ,\ D " W (m) (x(i#p, j!m) GH K N
E+x(i!2, j)x(i!1, j), W (1)" , E+x(i!2, j),
!xˆ (i#» #p, j!m#» )), V W
E+x(i!3, j)x(i!1,j), W (2)" , E+x(i!3, j),
+ ,\ D " W (m) (x(i!m, j#p) GH K N
E+x(i, j#N#1)x(i, j#N), W (1)" , E+x(i, j#N#1),
!xˆ (i!m#» , j#» #p)), V W
E+x(i#N#1, j)x(i#N, j), W (1)" , E+x(i#N#1, j),
!xˆ (i#» #p, j#m#» #N!1)), V W + ,\ D " W (m) (x(i#m#N!1, j#p) GH K N
(5)
(6)
E+x(i#N#2, j)x(i#N, j), W (2)" . E+x(i#N#2, j),
!xˆ (i#m#» #N!1, j#» #p)), V W k"1,2,3,4.
Here, (i, j) refer to the top left position of a lost MB (i.e., they are multiple of N), W I(m) is the weighting coefficient in the kth neighboring block and M the number of compared lines in the boundary. Considering the spatial correlations at the neighboring boundaries in the horizontal and vertical directions, the available adjacent pixels are used to determine the weighting coefficients. First, when M"2, the weighting coefficients in the upper block are represented by E+x(i, j!2)x(i, j!1), W (1)" , E+x(i, j!2),
(4)
E+x(i, j#N#2)x(i, j#N), , W (2)" E+x(i, j#N#2),
+ ,\ D " W (m) (x(i#p, j#m#N!1) GH K N
+ W I(m)"1, K
293
(3)
E+x(i, j!3)x(i, j!1), W (2)" , E+x(i, j!3), where E denotes the mathematical expectation whose value may be estimated by using the sample mean, i.e., ,\+ ) ,. Similarly, the other weights in , L the left, lower and right blocks are defined as
As the spatial correlation between the boundary signals increases, the higher weighting value is assigned. In the interlaced sequence with frame picture format, the pixel values either in odd fields or in even fields are highly correlated. Thus, it is more beneficial to use the matching criterion between the same fields. Hence, the proposed scheme chooses W (1)+0, W (2)+1, W (1)+0 and W (2)+1. Consequently, the total difference of (2) is adjusted by the weights corresponding to the spatial correlation. If there exist the motion vectors of the surrounding blocks, for fast search of the motion vector, the search areas are confined to an appropriate range according to various coding modes. Since the motion vector informations of spatially neighboring blocks are highly correlated, the motion informations of the upper macroblock and the lower macroblock are used in the initial guess as shown in Table 1. Table 1 shows the search range for the various motion types in P-picture. In the case of B-picture, the averaged motion vector of the available neighboring blocks can be used in the initial guess, too. In the proposed scheme, it needs only a little memory to store the motion vectors of a row of macroblocks. The proposed algorithm is summarized as follows.
294
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
Table 1 The search range of motion vector in P-picture (SR: search range) Macroblock
Upper macroblock (C1)
Lower macroblock (C3)
MODE
Motion vector (» ,» ) V W
Motion vector
(» #» )!SR)» V V V ) (» #» )#SR V V (» #» )!SR)» W W W ) (» #» )#SR W W » !SR)» )» #SR V V V » !SR)» )» #SR W W W
(» ,» ) V W Intra
Intra » !SR)» )» #SR V V V » !SR)» )» #SR W W W !2 SR)» )2 SR V !2 SR)» )2 SR W
video sequences were football and flower garden, whose resolutions were 720;480, 60 fields/sec, and YUV 4:2:0 format. The test sequences have been coded with intra-coded (I) and predictive coded (P) pictures with all possible prediction and DCT modes. The bit-rate was 6 Mbits/sec. The size of a GOP was 15, and concealment motion vectors were not used. The search ranges were $15 and $7 for the full-search BMA (FS-BMA) and the proposed algorithm, respectively. To evaluate performances, mean square error (MSE) was measured by MSE" +\ ,\(I (i, j) G H !I (i, j))/MN, where M;N was the frame size, #! I (i, j) the decoded frame without transmission er rors and I (i, j) the decoded frame by concealment #! algorithms with errors. Table 2 summarizes the averaged MSEs of the various error concealment (EC) algorithms. The proposed WBMA significantly outperformed the conventional error concealment algorithms in terms of MSE. Fig. 2 depicts the PSNRs of decoded sequences where Ipicture and P-picture were contaminated with 2—4
Algorithm Step 1. Identify the lost location and available neighboring blocks. Step 2. Compute the weighting coefficients by using Eqs. (3)—(6). Step 3. Determine the initial motion vector and find the motion vector which minimizes the Eq. (1) within the search range $SR. Step 4. Reconstruct the lost block by using the estimated motion vector. 3. Experimental results The proposed algorithm was compared with the conventional EC algorithms such as BMA [4], Modified BMA [2] and temporal predictive concealment (TPC) algorithms [3]. The random bit errors were inserted to MPEG-2 video bitstreams at the slice level according to the given bit error rate (BER) varying from 10\ to 10\. Concealment at the higher level such as the sequence header or the picture header is not the topic of this work. The test
Table 2 Averaged MSE performance of the various error concealment algorithms Method
No EC BMA Full search BMA MBMA TPC WBMA
Football sequence
Flower garden sequence
I
P
Total
I
P
Total
74.68 46.18 39.29 46.98 46.07 27.52
282.04 77.50 102.77 80.36 84.87 61.71
268.22 75.41 98.54 78.13 82.28 59.43
120.57 69.05 73.98 72.44 67.47 19.11
720.45 184.31 369.75 236.71 118.80 74.07
680.46 176.63 350.03 225.76 108.85 70.41
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
295
Fig. 2. PSNR of decoded picture: (a) football sequence, (b) flower garden sequence.
slice errors, respectively. The PSNR was measured by 10 log (255/MSE). The BERs of football and flower garden sequence were 1.2;10\ and 1.6;10\, respectively. The PSNR of the proposed algorithm was much better than the conventional algorithms, especially in the fast moving scene. The PSNR gain by the proposed algorithm was approximately 2—3 dB over the conventional algorithms.
4. Conclusions A novel error concealment algorithm has been presented for the graceful recovery of the missing macroblocks caused by inevitable transmission errors. A new weighted boundary matching criterion was applied to the available boundary pixels surrounding the missing macroblock for searching the good disguising motion vector. The weighting coefficients were continously adjusted according to the
spatial similarities. Simulation showed that the proposed technique outperformed the previous concealment techniques, especially for the macroblock having large motion or complicated textures. References [1] M.J. Chen, L.G. Chen, R.M. Weng, Error concealment of lost motion vectors with overlapped motion compensation, IEEE Trans. on Circuits and Systems for Video Tech. 7 (3) (June 1997) 560—563. [2] J. Feng, K.T. Lo, H. Mehrpour, Error concealment for MPEG video transmissions, IEEE Trans. on Consumer Electronics 43 (2) (May 1997) 183—187. [3] W. Keck, A method for robust decoding of erroneous MPEG-2 video bitstreams, IEEE Trans. on Consumer Electronics 42 (3) (August 1996) 411—421. [4] W.M. Lam, A.R. Reibman, B. Liu, Recovery of lost or erroneously received motion vectors, in: Proc. ICASSP, Vol. 5, April 1993, pp. 417—420. [5] U.S. Advanced Television Systems Committee, ATSC digital television standard, 1995.
Fast communication
Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream Eung Tae Kim *, Seung-Jong Choi, Hyung-Myung Kim Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 373-1 Kusong-Dong, Yusong-Gu, Taejon 305-701, South Korea Digital TV Research Laboratory, LG Electronics Inc., 16 Woomyeon-Dong, Seocho-Gu, Seoul 137-140, South Korea Received 13 August 1998; received in revised form 11 December 1998
Abstract A novel error concealment procedure is presented for recovering the lost macroblock while decoding an erroneous MPEG-2 bitstream. The proposed weighted boundary matching algorithm (WBMA) refines the conventional boundary matching algorithm. Different from the conventional BMAs, the boundary pixels of a lost macroblock are compared with those of the previous picture located in the same position. The weighting coefficients are also adaptively adjusted to work robustly in either smooth or abruptly-changing regions. Experimental results show that the proposed algorithm outperforms the conventional algorithms, especially for the interlaced sequences having large motion or complicated textures. 1999 Published by Elsevier Science B.V. All rights reserved. Keywords: Error concealment; MPEG-2 decoder; Weighted boundary matching algorithm
1. Introduction In the MPEG-2 video receiver, transmission errors cannot be avoided in the physical channels such as terrestrial broadcasting channels and ATM networks [5]. To overcome this problem, many researchers have studied the error concealment (EC) techniques [1—4]. Basically, those techniques can be classified into temporal predictive concealment (TPC) and spatial predictive concealment (SPC) [3]. In [3], it has been shown that TPC using
* Corresponding author. Tel. #82-42-869-5440, 3440; fax: #82-42-869-3410; e-mail: [email protected]
the motions of neighboring MBs has the better performance than SPC. However, the paid cost is the overhead memory, since motion vectors of an entire picture should be stored. Another problem is that the assumption of smooth motion vectors is not always true. Recently, a boundary matching algorithm [2,4] was proposed to recover the motion vector of a lost block. In [1], the overlapped motion compensation is combined with the side matching criterion to mask the blocking artifacts. Since the pixels of the lost macroblock are not available, the matching is performed with the boundary pixels surrounding the lost MB. This side matching criterion utilizes the smoothness of pixel values even across block
0165-1684/99/$ — see front matter 1999 Published by Elsevier Science B.V. All rights reserved. PII: S 0 1 6 5 - 1 6 8 4 ( 9 8 ) 0 0 2 4 2 - 4
292
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
boundaries. It works well in the region where no large change in pixel values occurs across the boundary. However, in case neighboring pixel values change abruptly, for example, around edges or corners, the incorrect motion vector would be obtained and the recovered MB by concealment does not fit well. The performance would be seriously degraded for the interlaced sequences, since the spatial correlation is low between fields having different parities. The weighted matching criterion is proposed in this work to search reliable motion vectors. This approach still inherits the boundary matching algorithm. However, different from the conventional BMAs, the boundary pixels of a lost MB are directly compared with those of the matched MB in the previous frame. Furthermore, the weighting coefficients are adaptively adjusted by calculating the spatial correlations across the boundaries to overcome the above mentioned problems.
2. Proposed error concealment technique It is assumed that the locations of damaged blocks are already known via variable length decoding (VLD) process which tests the syntax error
or the semantic rule error in the bitstream [5]. In general, MPEG standard considered the resynchronization problem by inserting several starting codes (especially, the slice unit). If the VLD detects the error code or a syntax error in the bitstream, the error concealment procedure is performed until a next slice start code is detected. Thus, the error propagation can be limited within a slice, even if the pixels in an error-corrupted slice are lost. In this paper, concealment of macroblocks is carried out by recovering the motion vectors of them and then by replacing them with the motion compensated MBs in the previous frame. Let x(i, j) and xˆ (i, j) denote the luminance pixel at location (i, j) in the current frame and the previous frame, respectively. (» ,» ) is the motion vector of the lost block and V W N the block size. Fig. 1 shows the proposed weighted boundary matching algorithm. The bestmatched motion vector is found by minimizing the following equation: min DR (» ,» ), (1) GH V W 4V 4WZ10 where SR is the search range for motion vector. The proposed cost function is defined as DR (» ,» )" DI (» ,» ), (2) GH V W GH V W I
Fig. 1. Weighted boundary matching algorithm.
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
where
follows:
+ ,\ D " W (m) (x(i#p, j!m) GH K N
E+x(i!2, j)x(i!1, j), W (1)" , E+x(i!2, j),
!xˆ (i#» #p, j!m#» )), V W
E+x(i!3, j)x(i!1,j), W (2)" , E+x(i!3, j),
+ ,\ D " W (m) (x(i!m, j#p) GH K N
E+x(i, j#N#1)x(i, j#N), W (1)" , E+x(i, j#N#1),
!xˆ (i!m#» , j#» #p)), V W
E+x(i#N#1, j)x(i#N, j), W (1)" , E+x(i#N#1, j),
!xˆ (i#» #p, j#m#» #N!1)), V W + ,\ D " W (m) (x(i#m#N!1, j#p) GH K N
(5)
(6)
E+x(i#N#2, j)x(i#N, j), W (2)" . E+x(i#N#2, j),
!xˆ (i#m#» #N!1, j#» #p)), V W k"1,2,3,4.
Here, (i, j) refer to the top left position of a lost MB (i.e., they are multiple of N), W I(m) is the weighting coefficient in the kth neighboring block and M the number of compared lines in the boundary. Considering the spatial correlations at the neighboring boundaries in the horizontal and vertical directions, the available adjacent pixels are used to determine the weighting coefficients. First, when M"2, the weighting coefficients in the upper block are represented by E+x(i, j!2)x(i, j!1), W (1)" , E+x(i, j!2),
(4)
E+x(i, j#N#2)x(i, j#N), , W (2)" E+x(i, j#N#2),
+ ,\ D " W (m) (x(i#p, j#m#N!1) GH K N
+ W I(m)"1, K
293
(3)
E+x(i, j!3)x(i, j!1), W (2)" , E+x(i, j!3), where E denotes the mathematical expectation whose value may be estimated by using the sample mean, i.e., ,\+ ) ,. Similarly, the other weights in , L the left, lower and right blocks are defined as
As the spatial correlation between the boundary signals increases, the higher weighting value is assigned. In the interlaced sequence with frame picture format, the pixel values either in odd fields or in even fields are highly correlated. Thus, it is more beneficial to use the matching criterion between the same fields. Hence, the proposed scheme chooses W (1)+0, W (2)+1, W (1)+0 and W (2)+1. Consequently, the total difference of (2) is adjusted by the weights corresponding to the spatial correlation. If there exist the motion vectors of the surrounding blocks, for fast search of the motion vector, the search areas are confined to an appropriate range according to various coding modes. Since the motion vector informations of spatially neighboring blocks are highly correlated, the motion informations of the upper macroblock and the lower macroblock are used in the initial guess as shown in Table 1. Table 1 shows the search range for the various motion types in P-picture. In the case of B-picture, the averaged motion vector of the available neighboring blocks can be used in the initial guess, too. In the proposed scheme, it needs only a little memory to store the motion vectors of a row of macroblocks. The proposed algorithm is summarized as follows.
294
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
Table 1 The search range of motion vector in P-picture (SR: search range) Macroblock
Upper macroblock (C1)
Lower macroblock (C3)
MODE
Motion vector (» ,» ) V W
Motion vector
(» #» )!SR)» V V V ) (» #» )#SR V V (» #» )!SR)» W W W ) (» #» )#SR W W » !SR)» )» #SR V V V » !SR)» )» #SR W W W
(» ,» ) V W Intra
Intra » !SR)» )» #SR V V V » !SR)» )» #SR W W W !2 SR)» )2 SR V !2 SR)» )2 SR W
video sequences were football and flower garden, whose resolutions were 720;480, 60 fields/sec, and YUV 4:2:0 format. The test sequences have been coded with intra-coded (I) and predictive coded (P) pictures with all possible prediction and DCT modes. The bit-rate was 6 Mbits/sec. The size of a GOP was 15, and concealment motion vectors were not used. The search ranges were $15 and $7 for the full-search BMA (FS-BMA) and the proposed algorithm, respectively. To evaluate performances, mean square error (MSE) was measured by MSE" +\ ,\(I (i, j) G H !I (i, j))/MN, where M;N was the frame size, #! I (i, j) the decoded frame without transmission er rors and I (i, j) the decoded frame by concealment #! algorithms with errors. Table 2 summarizes the averaged MSEs of the various error concealment (EC) algorithms. The proposed WBMA significantly outperformed the conventional error concealment algorithms in terms of MSE. Fig. 2 depicts the PSNRs of decoded sequences where Ipicture and P-picture were contaminated with 2—4
Algorithm Step 1. Identify the lost location and available neighboring blocks. Step 2. Compute the weighting coefficients by using Eqs. (3)—(6). Step 3. Determine the initial motion vector and find the motion vector which minimizes the Eq. (1) within the search range $SR. Step 4. Reconstruct the lost block by using the estimated motion vector. 3. Experimental results The proposed algorithm was compared with the conventional EC algorithms such as BMA [4], Modified BMA [2] and temporal predictive concealment (TPC) algorithms [3]. The random bit errors were inserted to MPEG-2 video bitstreams at the slice level according to the given bit error rate (BER) varying from 10\ to 10\. Concealment at the higher level such as the sequence header or the picture header is not the topic of this work. The test
Table 2 Averaged MSE performance of the various error concealment algorithms Method
No EC BMA Full search BMA MBMA TPC WBMA
Football sequence
Flower garden sequence
I
P
Total
I
P
Total
74.68 46.18 39.29 46.98 46.07 27.52
282.04 77.50 102.77 80.36 84.87 61.71
268.22 75.41 98.54 78.13 82.28 59.43
120.57 69.05 73.98 72.44 67.47 19.11
720.45 184.31 369.75 236.71 118.80 74.07
680.46 176.63 350.03 225.76 108.85 70.41
E.T. Kim et al. / Signal Processing 73 (1999) 291—295
295
Fig. 2. PSNR of decoded picture: (a) football sequence, (b) flower garden sequence.
slice errors, respectively. The PSNR was measured by 10 log (255/MSE). The BERs of football and flower garden sequence were 1.2;10\ and 1.6;10\, respectively. The PSNR of the proposed algorithm was much better than the conventional algorithms, especially in the fast moving scene. The PSNR gain by the proposed algorithm was approximately 2—3 dB over the conventional algorithms.
4. Conclusions A novel error concealment algorithm has been presented for the graceful recovery of the missing macroblocks caused by inevitable transmission errors. A new weighted boundary matching criterion was applied to the available boundary pixels surrounding the missing macroblock for searching the good disguising motion vector. The weighting coefficients were continously adjusted according to the
spatial similarities. Simulation showed that the proposed technique outperformed the previous concealment techniques, especially for the macroblock having large motion or complicated textures. References [1] M.J. Chen, L.G. Chen, R.M. Weng, Error concealment of lost motion vectors with overlapped motion compensation, IEEE Trans. on Circuits and Systems for Video Tech. 7 (3) (June 1997) 560—563. [2] J. Feng, K.T. Lo, H. Mehrpour, Error concealment for MPEG video transmissions, IEEE Trans. on Consumer Electronics 43 (2) (May 1997) 183—187. [3] W. Keck, A method for robust decoding of erroneous MPEG-2 video bitstreams, IEEE Trans. on Consumer Electronics 42 (3) (August 1996) 411—421. [4] W.M. Lam, A.R. Reibman, B. Liu, Recovery of lost or erroneously received motion vectors, in: Proc. ICASSP, Vol. 5, April 1993, pp. 417—420. [5] U.S. Advanced Television Systems Committee, ATSC digital television standard, 1995.