- Email: [email protected]
Motion estimation and compensation technologies for standards conversion
SIGNAL PROCESSING:
IMAGE
COMMUNICATION ELSEVIER
Signal Processing: hnage Communication 6 (1994) 189-190
Preface
Motion estimation and compensation technologies for standards conversion The scanning format Of a video signal is one of the main determinants of overall picture quality. Specifically, it determines su.ch aspects as stationary and dynamic spatial resolution, motion portrayal, ~liasing, scanning structure visibility and flicker. Various formats have been chosen to strike a particular i:ompromise between quality, cost of cameras and displays, required transmission capacity and compatibility with other existing standards (e.g., power mains frequency). Despite sustained efforts over the years to develop worldwide standards for video signals, the number of different video formats has continued to grow. There were three main formats in use a decade ago corresponding to three picture rates: 50 Hz interlaced, 60 Hz interlaced and 24 (or 25)Hz progressive (film). Conversion between these formats was required for Video distribution of film material and for international program exchange. With the advent of videoconferencing, HDTV and high-performance workstations and PCs, many new video formats have appeared. These include low end formats such as CIF and QCIF with smaller picture sizeand lower frame rates, progressive and interlaced HDTV formats at 50 Hz and 60 Hz, and other video formats used on computer workstations and enhanced consumer displays with field rates up to 120Hz. In modern systems, several different formats may co-exist within the system, e.g., a camera format, a transmission format and a display format, each optimized according to its own criteria. In some applications requiring scalability, the same video signal will simultaneously be represented in several formats. Thus the need for standards conversion is increasing, not decreasing! High-quality electronic video standards conversion is a difficult problem for a number of reasons. The most fundamental reason is that the conditions of the sampling theorem are in general not met in video signals. This theorem states that if the spatiotemporal spectrum of a time-varying image signal is limited to a certain fundamental region of spatiotemporal frequency space, it is possible to exactly reconstruct the signal from its samples on a sufficiently dense discrete sampling lattice. If satisfied, standards conversion of arbitrary accuracy could be achieved using suitable linear filters. Since the optical image in general has very high spatial and temporal bandwidth, it is necessary to prefilter the image sequence prior to the vertical and temporal sampling of the scanning process if we wish to respect the sampling theorem. This is difficult to achieve and is not done in any existing camera systems. In any case, such strict temporal prefiltering is undesirable, largely because there is an unknown relationship between spatiotemporal frequencies in the display and those on the retina of the human observer, due to eye motion and tracking. Temporal prefiltering may result in excessive blurring of detailed moving objects that are tracked by the viewer. Thus, we must accept some level of aliasing in the sampled or scanned video signal. In many cases, even though the baseband spatiotemporal spectrum is not confined to a given predetermined region of spatiotemporal frequency space, the replicated versions of the baseband spectrum caused by the scanning process do not overlap. However, the region in which the uncorrupted baseband spectrum lies Elsevier Science B.V.
190
Signal Processing: hnage Communication 6 (1994) 189-190
varies, depending on the motion. In this case the continuous signal can be determined using suitable motion-compensated processing. Thus, motion-compensated interpolation is a method that gives the potential of high quality standards conversion. However, if the replicated versions of the baseband spectrum overlap, then good standards conversion may not be possible, even with motion compensation. This is especially frequent in interlaced signals. Motion-compensated processing has been successfully applied in a number of applications including source coding and noise reduction. As well, a number of schemes for motion-compensated standards conversion have been proposed and shown to give good performance in many situations. However, there remain many problems with robustness and complexity. For example, motion estimation algorithms suitable for coding may not be adequate for standards conversion, because the latter requires motion estimates that more closely reflect the true motion in order to provide high quality. The aim of this special issue is to address some of the remaining concerns. The issue consists of five regular papers and two short communications. The first two papers (Vandendorpe et al., Patti et al.) deal with the problem of motion compensated conversion. The first specifically addresses the problems due to interlace, while the second considers the issue of accelerated motion. The next two papers (de Haan and Biezen, Koivunen and Salonen) consider the problems of motion estimation for use in standards conversion. The first of these .presents methods for accurate motion estimation at relatively low computational complexity, while the second looks at featurebased methods. The last regular paper by Bonse addresses the problem of the accuracy required for motion vector fields based on visual criteria. In the first short communication by Yamauti et al. a practical standards converter using motion compensation is described, showing the extent to which this technology has matured. Finally, Bock considers the problem of motion-adaptive standards conversion when the two signals have similar field rates. These papers make a significant contribution to the problem of motion-compensated standards converSion, but it is clear that much work remains to achieve truly robust, high-quality conversions at low cost. Eric Dubois Gerard de Haan Taiichiro Kurita Guest Editors
IMAGE
COMMUNICATION ELSEVIER
Signal Processing: hnage Communication 6 (1994) 189-190
Preface
Motion estimation and compensation technologies for standards conversion The scanning format Of a video signal is one of the main determinants of overall picture quality. Specifically, it determines su.ch aspects as stationary and dynamic spatial resolution, motion portrayal, ~liasing, scanning structure visibility and flicker. Various formats have been chosen to strike a particular i:ompromise between quality, cost of cameras and displays, required transmission capacity and compatibility with other existing standards (e.g., power mains frequency). Despite sustained efforts over the years to develop worldwide standards for video signals, the number of different video formats has continued to grow. There were three main formats in use a decade ago corresponding to three picture rates: 50 Hz interlaced, 60 Hz interlaced and 24 (or 25)Hz progressive (film). Conversion between these formats was required for Video distribution of film material and for international program exchange. With the advent of videoconferencing, HDTV and high-performance workstations and PCs, many new video formats have appeared. These include low end formats such as CIF and QCIF with smaller picture sizeand lower frame rates, progressive and interlaced HDTV formats at 50 Hz and 60 Hz, and other video formats used on computer workstations and enhanced consumer displays with field rates up to 120Hz. In modern systems, several different formats may co-exist within the system, e.g., a camera format, a transmission format and a display format, each optimized according to its own criteria. In some applications requiring scalability, the same video signal will simultaneously be represented in several formats. Thus the need for standards conversion is increasing, not decreasing! High-quality electronic video standards conversion is a difficult problem for a number of reasons. The most fundamental reason is that the conditions of the sampling theorem are in general not met in video signals. This theorem states that if the spatiotemporal spectrum of a time-varying image signal is limited to a certain fundamental region of spatiotemporal frequency space, it is possible to exactly reconstruct the signal from its samples on a sufficiently dense discrete sampling lattice. If satisfied, standards conversion of arbitrary accuracy could be achieved using suitable linear filters. Since the optical image in general has very high spatial and temporal bandwidth, it is necessary to prefilter the image sequence prior to the vertical and temporal sampling of the scanning process if we wish to respect the sampling theorem. This is difficult to achieve and is not done in any existing camera systems. In any case, such strict temporal prefiltering is undesirable, largely because there is an unknown relationship between spatiotemporal frequencies in the display and those on the retina of the human observer, due to eye motion and tracking. Temporal prefiltering may result in excessive blurring of detailed moving objects that are tracked by the viewer. Thus, we must accept some level of aliasing in the sampled or scanned video signal. In many cases, even though the baseband spatiotemporal spectrum is not confined to a given predetermined region of spatiotemporal frequency space, the replicated versions of the baseband spectrum caused by the scanning process do not overlap. However, the region in which the uncorrupted baseband spectrum lies Elsevier Science B.V.
190
Signal Processing: hnage Communication 6 (1994) 189-190
varies, depending on the motion. In this case the continuous signal can be determined using suitable motion-compensated processing. Thus, motion-compensated interpolation is a method that gives the potential of high quality standards conversion. However, if the replicated versions of the baseband spectrum overlap, then good standards conversion may not be possible, even with motion compensation. This is especially frequent in interlaced signals. Motion-compensated processing has been successfully applied in a number of applications including source coding and noise reduction. As well, a number of schemes for motion-compensated standards conversion have been proposed and shown to give good performance in many situations. However, there remain many problems with robustness and complexity. For example, motion estimation algorithms suitable for coding may not be adequate for standards conversion, because the latter requires motion estimates that more closely reflect the true motion in order to provide high quality. The aim of this special issue is to address some of the remaining concerns. The issue consists of five regular papers and two short communications. The first two papers (Vandendorpe et al., Patti et al.) deal with the problem of motion compensated conversion. The first specifically addresses the problems due to interlace, while the second considers the issue of accelerated motion. The next two papers (de Haan and Biezen, Koivunen and Salonen) consider the problems of motion estimation for use in standards conversion. The first of these .presents methods for accurate motion estimation at relatively low computational complexity, while the second looks at featurebased methods. The last regular paper by Bonse addresses the problem of the accuracy required for motion vector fields based on visual criteria. In the first short communication by Yamauti et al. a practical standards converter using motion compensation is described, showing the extent to which this technology has matured. Finally, Bock considers the problem of motion-adaptive standards conversion when the two signals have similar field rates. These papers make a significant contribution to the problem of motion-compensated standards converSion, but it is clear that much work remains to achieve truly robust, high-quality conversions at low cost. Eric Dubois Gerard de Haan Taiichiro Kurita Guest Editors