- Email: [email protected]
Universal noiseless and noisy data compression techniques — the state of the art
Universal noiseless and noisy data compression techniques
- - t h e s t a t e of
the art B V Cox and R A Lindsay outline the progress that has been made in data compression techniques
The technique of data compression demonstrates desirable properties that assure performance improvements of advanced network architectures such as ISDN, serial communication channels, data storage and data archiving. A global trend in telecommunications networks is the conversion from analogue to digital transmission. The design goal of these new networks is that voice, data, and images will be transmitted in fundamentally the same way, requiring neither prior knowledge as to the type of data being transmitted nor operator intervention. However, the bandwidth requirements for digital transmission are generally high. Spectrum bandwidth utilization is a design parameter for these systems that will benefit from the transmission of compressed data. The proposed Integrated Services Digital Network (ISDN) combines digital voice, data, and images on the same network. Since this network has limited bandwidth for transmission of information, the available bandwidth must be used as efficiently as possible. Keywords: data compression, data communications, universal codin& vector quantization
exchange of information by providing an environment in which central locations are responsible for maintaining databases and shared assets such as printers, mass storage, and high speed computation. Intelligent terminals or workstations access these networks, enabling the functions of a complex information system to be divided physically as well as logically. Networks, such as the telephone system, have been developed for voice transmission. Other computer networks operate with digital data and file transfers. The need for special purpose networks and transmission links will continue in the future. However, there is a growing trend to operate integrated networks of digital voice, data, and images that are not proprietary systems. Standardized integrated digital networks will provide the nucleus for proficiently exchanging information and maintaining databases. ISDN is an architectural concept for a telephone network to carry voice, data, message, video and image traffic from a local area network to a wide area network. Regardless of the size of the network or type of information that is being processed, there will be a requirement for efficient storage and transmission; therefore, compression of voice, data, and images will be a key technology for the networks of the future.
NETWORKING A N D COMPRESSION Computer networking is part of a general trend towards distributed information processing. Modern network architectures permit distributed processing and the Advanced Technology Department, UnisysCommunications Technology Center, 640 North Sperry Way, Salt Lake City, UT 84116, USA
INTRODUCTION TO DATA COMPRESSION The concept of data compression is not new. There has always been a need to increase storage media capacity, reduce transmission costs, or improve the time response of communication systems. Data compression is a phrase
0140-3664/87/040179-10 $03.00 © 1987 Butterworth & Co (Publishers) Ltd vol 10 no 4 august 1987
179
that has broad meaning; unfortunately the term is also application dependent. Data compression has also been referred to by other names in the literature; the most common are data compaction, bandwidth compression, redundancy removal, redundancy reduction, source coding, data rate reduction, adaptive sampling, parameter extraction, and universal coding. Analysis modelling, and coding of information to reduce overall bit rate has been actively studied for 35 years. Data compression is a mechanism or procedure that transforms a signal or data sequence into a new sequence containing the same information, but whose length is smaller than the original. In simple terms, it encodes a string of symbols into a sequence of bits in such a way that the sequence can be deduced uniquely to reproduce the original string or an acceptable replica of the string. These methods are well known; they are subject to active current research in industrial, governmental, and university laboratories; and the performance of these techniques is well documented in the literature. Data compression can be classified into two broad groups: parametric (or model), and waveform representation. Figure I outlines the major categories of data compression. Figure I is not comprehensive, but it is representative of the magnitude of the technological effort and the richness of the available compression algorithms. Waveform representations address the issue of preserving the 'wave shape' of the signal through sampling and quantization procedures. Parametric representations, on the other hand, are concemed with representing the signal as the output of a model that best characterizes the signal generation mechanism. As Figure 1 suggests, data compression techniques have been
i"
Data compression
subject to prodigious scientific inquiry. Figure 1 shows data compression categories and their chronological significance. In 1835, Morse introduced a form of data compression by using a variable-length code. This approach used short codes for the most probable letters of the English alphabet and longer codes for the least probable letters. Most authors attribute the first data compression invention to the Vocoder by Dudley in 19391. The Shannon-Fano code was developed in 1948. Redundancy reduction techniques were developed in space research, sounding rockets and satellite telemetry contexts during the 1960s. Run-length codes were applied to telemetry data during this same time span. Frequency representations were applied to speech and images in the 1960s. Delta modulation and the improved technique of Differential Pulse Code Modulation (DPCM) were invented in 1952. Both delta modulation and DPCM were applied to speech and video signals. Huffman coding was introduced in the 1950s as an algorithm for developing the optimum variable length code over a fixed alphabet with a known probability distribution. However, no advantage is taken of repetition of sequences of symbols. Binary coding techniques were developed in the 1960s, LynchDavisson code in the 1970s, Lempel and Ziv's Universal parsing code in 1978, and in the 1980s the Langdon and Rissanen arithmetic code. Parametric extraction techniques originated in 1950 with the predictive coding investigations, and in the 1960s transform and linear predictive coding developments were reported. Now in the 1980s, information Iossless compression, or source encoding is actively being investigated and applied to speech, data,
J J Waveform J
I Paramricextraction I I
Analysis /Synthesis Voice
LPC
excited
I
i
Other
Waveform representation
Frequency
discrimination
Phase comparison
Homomorphic
I Statistical j
Non ~daptive J
I
I
I [Adaptive[
Incremental Probabilistic Gilbert Huffman Fang mo Shannon/Fan(
Figure 1.
180
(Time, Spacedomain)
I
I Frequency represen Jresentation
Redundancy reduction
I Delta modulation ! 'rensform I PCM LDM iD & 2D) Log PCM Square root DPCM FI R Root-meanADPCM I IR square ADM Subband Fourier zero crossing Filter bank __ Predictors J Prime I Adaptive sampling APC Discrete linear /interpolators Differential Channel Universal J basic vector quantizer Vocoder ATC Adaptive quantizer Lempel/ZIV Chirp Z Feed-forward Tree/incremental Walsh Feed-back Trellis Haar ial Predictive enumerative Discrete cosine e Companding Variable-length Discrete Chebychev tial p--Law tree Slant Log Sequential tree Karhunan-Loeve Laplace N-Gram Hadamard String matching CVSD R Transform Vector quantizer Run length Vector/matrix quantizer Accretive L~east~quare Fractal image (yardstick coding) Ordered run Increased filter Variable-to-block Arithmetic
[ F~,ter,ngJ I~(
Filtering thresholding
I
J
I
I I
analysis
Formant vocoder Channel vocoder Auto-regressive (An) Moving average (MA) ARMA ARIMA Multpulse LPC Stochastically excited LPC
J
Source encoding
Short time spectrum
Phase vocoder
I information Iossy
J Information Iossless I
J
j
Bit Plane Adaptive coefficienl Probabilistic Adaptive Filter Modulator Huffman
~
Data compression classification
computer communications
and image compression. Table 1 summarizes the compression results obtained by the Unisys Communications Technology Center, or reported in the literature for the various compression algorithms.
CLASSIFICATION OF DATA COMPRESSION TECHNIQUES Source coding is the art of conserving storage space, transmission time, and bandwidth by recoding data into a
Table 1.
more compact form. The code may be reversible, in which case it is called 'lossless', or it may entail irreversible (but tolerable) changes, in which case it is called 'lossy'. Lossless coding is essential in applications involving computer programs, encrypted data, and other sources where even minute changes can have enormous consequences; lossy coding is acceptable in such applications as text compression where some 'typos' are tolerable. Lossy compression is most appropriate in applications involving digitized analogue data, such as visual images, audio recordings, seismic soundings, and so on, where
Digital techniques for voice/video/data compression
Type information
Technique
Transmission rate (bit/s)
High-quality sound (Voice) (0/I 5 000 Hz)
PCM sampling Rate 35 kHz 13 bit sampling
450 000
Synthesized-voice
Phase vocoder Adaptive transform (ATC) Channel (FFT) vocoder Subband (4) Homomorphic (Ceptrum) Filtered vocoder LPC vector quantization Format vocoder Voice excited vocoder Pitch synchronized adaptive LPC (10th order) LPC (4th order) Adaptive predictive coding (APC)
7200 7200-32 000" 2400-4800 9600-24 000" 2400-4800
Speech
Video telephone
Television
vol 10 no 4 august 1987
Pixel bit rate (bit/pixel)
800 600-1200 4800-9600 7200 1200-4800 8000 6400-16 000"
Block vector quantization Markov vector quantization Two-stage vector quantization Vector quantization LPC, transform coded residual error Vector quantization LPC, scalar pitch and gain Vector quantization LPC, residual spectral coding Sliding rate design Trellis code Tree encoding Adaptive tree Adaptive Huffman
0.2 kbit/s 0.4-0.8 kbit/s 1.2 kbit/s 2.4-16 kbit/s
Forward error Control PCM PCM
T1 Carrier 1.544 Mbit/s
DPCM-B&W-reduced resolution DPCM-colour-frame storage Required Hadamard slant transform (Colour) PCM Hadamard transform (B&W) Sample PCM colour DPCM-interframe coding (Ishiguro) DPCM-colour-reduced Vertical resolution DPCM-7/8 error code (Golding)
1.5 Mbit/s 4.8 Mbit/s
0.8 kbit/s 2.4 kbit/s 0.8-16 kbit/s 3250-13 000 bit/s 8000 bit/s 8000 bit/s 3200 bit/s
T2 Carrier 6.312 Mbit/s
6 Mbit/s 11 Mbit/s 12 Mbit/s 18-22 Mbit/s 29 Mbit/s 33 Mbit/s
181
Table I continued
Type information
Technique
Transmission rate
DPCM-(Galfield) DPCM-1/2 error code Basic PCM PCM pseudonoise Quantization Frame replenishment Discrete cosine/DPCM Motion-compensated interframe Variable rate hadamard 2D discrete cosine Statistical Adaptive Non-adaptive Hybrid interframe Predictive-interframe Interpolative Transform-interframe Colour non-adaptive 2D DPCM Colour non-adaptive 2D DPCM Slow scan Cosine; DPCM between lines Haar Hadamard Constant area quantization Adaptive 2D DPCM
43 Mbit/s 58.4 Mbit/s 86-92.6 Mbit/s
Colour maps
Colour spatial vector quantization
0.50-2.0*
Time-domain voice coding
Vector/matrix quantization
200-400
CVSD Frame predictive vector Quantization Log DCM Adaptive delta Modulation (ADM) Linear delta Modulation (LDM) Forward error control PCM Non-adaptive delta modulation DPCM Adaptive differential PCM (residual encoder)
16 000-32 000 250
Vector coding
Telemetry
EKG F-16 Flight data
Formant Pattern matching Vector quantization (LPC) Phonemic Diphone Voice excited (LPC) VELP Residual excited (LPC) RELP Adaptive aperture First-order interpolator Zero-order interpolator Zero-order predictor
Pixel bit rate (bit/pixel)
8-5 5-3 2-1 0.5-2.0 0.3-1.5 Mbit/s 1 Mbit/s 1 Mbit/s 6-4 4-3.5 1-0.2 4-2 2-1 3-1 13 Mbit/s 144 kbit/s 0.75-2.0 0.75-2.0 0.75-2.0 0.75-2.0 0.75-2.0
33 000-45-000 8000-16 000 20 000-40 000 10 000 10 000-32 000 30 O00-40 000 9600-16 000 0.6-1.2 kbit/s 0.6-1.2 kbit/s 0.6-1.2 kbit/s 0.075-0.2 kbit/s 0.075-0.2 kbit/s 4.8-16 kbit/s 4.8-16 kbit/s 0.I0 0.30-0.10 0.20-0.50
Cycle-to-Cycle 0th order linear predictor Adaptive Huffman Lempel-Ziv CVSDM
0.001 0.25 0.33 0.31 0.125
*Compression techniques and results developed and reported by Sperry Corporate Technology Center for Communication
182
computer communications
some alteration is undetectable or tolerable to human senses. It is beyond the scope of this paper to attempt a discussion of all of the techniques used historically. As shown in Figure 1, there have been many techniques developed, many of which have not been included; particularly when an algorithm is developed that is specific to a single data type or application. The primary goal of data compression studies is to develop universal techniques that perform optimally on all types of data. The parameter of interest is whether the data is to be compressed with distortion or without distortion. When distortion is 'tolerable' to the user then the amount of acceptable distortion must be defined. The current stateof-the-art algorithms for data compression are the LempelZiv in the lossless application, and vector quantization in the lossy data compression. Both of these algorithms approach the optimum amount of compression achievable without prior knowledge of the data. A brief introduction to each of these algorithms will be given.
state, at both the encoder and the decoder. When an infinite sequence x is fed into E, the encoder emits an infinite sequence y = Yl, Y2, Y3. . . . (y @ B) while going through an infinite sequence of states z = zl, z2, z3 . . . . The decoder traces the same states z by using the decoded message x and the next state function g. This mapping is shown in Figure 2. An encoder E is said to be information lossless if for all zl and all finite Xn -- xl, x2, X3. . . . . Xn, the triple {zl, f (zl, Xn), g (zl, Xn) } uniquely determines Xn. Given the encoder E = (S, A, B, f, g) and the input string Xn, the compression ratio forx n with respect to E is defined by: pE(Xn) = L(yn)/log2cz, where ~ = I A I, Yn = f(zl, Xn), L(yn) = T-,I= ~l(yi), and I(yi) is the length in bits of Yi E B. The minimum of PE(Xn) over the class E(s) of all finitestate information Iossless encoders with [ A I = czand [ 51 < s is denoted by PE(s)(Xn), that is: min PE(s)(Xn) = E E E(s) [PE(xn)].
LOSSLESS COMPRESSION Advanced information systems require compression algorithms that are not source-dependent, so that tandem sources can be transmitted transparently. When a priori knowledge of the data source is not available, and statistical preprocessing techniques are not practical, the problem of data compression becomes more difficult. Lempel and Ziv have developed universal compression techniques that are based on a theory of complexity of finite strings2-4. They defined a new measure of complexity that treats only the record at hand and does not employ a probabilistic source for the data record. From this measure of complexity, Lempel-Ziv proposed a noiseless string matching and parsing approach to data compression. This algorithm addresses many of the traditional problems associated with data compression, in that it dynamically adapts to the redundancy characteristics of the data being compressed. It is not an approximation technique; it does not remove redundancy to some acceptable level, but faithfully reproduces the original datastream. Lempel and Ziv developed theoretical arguments to prove that the compression ratio achieved by the algorithm uniformly approached the lower bounds on the compression ratio attained by algorithms designed to match a statistically specified source. The encoder and decoder work in step to monitor and 'learn' the characteristics of the data and provide an efficient code for compression.
Lempel-Ziv algorithm
Furthermore, let, PE(s) (X) = lim n ~ sup oo PE(s)(Xn),
and p(x) "= lim
s -, ~ PE(s)(x).
This normalizing quantity p(x) that depends solely on the input message x is referred to as the compressibility of x. (See Reference 4 for further detail or proof of the underlying concept.) The Lempel-Ziv compression algorithm has the ability to asymptotically converge on the compressibility of the data message by remembering all past events that have occurred. Continued efforts have focussed on the practical implementation of the Lempel-Ziv algorithm 5. It is necessary to reduce the memory required to a workable size. In theory, the larger the memory, the greater the amount of compression. However, it becomes increasingly difficult to save and search all possible events that have occurred previously in a message. In practice, the algorithm is more easily described as the following: a) Parse a string from the input message source, b) Send a code or index that represents the string, c) Update the dictionary. Step a) is accomplished by searching past events that are stored in a dictionary to find the best possible match of previously seen strings. Originally this was defined as comparing the input source against the previously seen message as a 'sliding window' to find the longest match. This approach was not practical because the search time
The Lempel-Ziv compression algorithm is described as an encoder E defined by the quintuple (S, A, B, f, g) where: 5 = a finite set of states that the encoder and decoder both enter during the compression of a message, A = a finite-set input alphabet, B = a finite set of output words over a finite output alphabet, f = the output function that maps SxA--~ B, and g = the next state function that maps SxA --~ S. Compression of a sequence x = xl, x2, x3 . . . . (x (E A) by the encoder E begins by initializingzl, (z E S), to the same
vol 10 no 4 august 1987
Yi = f (xi, zi ) xi =f-l(yi,zi)
zi+ 1= g ( x i , z
i ) i = 1,2 . . . .
Figure 2. Mapping for finite state information Iossless encoder/decoder
183
increased geometrically as the data was encoded. Simulations also revealed that the pointers to the table were remembering too much about past data and were not efficient. Improvements for searching the dictionary by using tree structures and hashing tables have enabled the algorithm to be implemented in linear time, thereby producing efficient computer and hardware implementations. Step b) has several variations depending on the particular application. The code word that is sent by the encoder is an index to the dictionary. When the dictionary is atree structure, the code is a pointer to a particular node of the tree. When the dictionary is a hashing table, the code is a key to the hashing function or an identifier. Some implementations use both. The codes can be variable in length or fixed in length depending on the searching procedure in a). Some implementations have the next character after the string included in the code to facilitate step c). Dictionary updates are accomplished by rules or heuristic methods which determine how much to remember. These rules can be as simple as adding the next character of the message to the previously parsed string to create a new string. One particular implementation known as the Lempel-Ziv-Welch uses a unique combination of steps b) and c), known as deferred innovation encoding 6. The code-word of step b) does not include the next character of the message, but this character is later derived from the beginning of the next string and used to update a hashing table. This algorithm has been implemented and is in wide use on CS-Net known as Compress. Table 2 shows results of compression for typical data using the Lempel-Ziv and Lempel-ZivWelch algorithms. Experimental results, from computer
Table 2. Lempel-Ziv technique for voice/video/data compression Information type
Compression ratio Lempel-Ziv
EXEC 8 Code Business files English text Speech (3.2 kHz) Fascimile (250 pixels/inch) Geo physical data CAD logic input Database PCM coded satellite photo Pseudo-random digits (Entropy 0.405) Coded fingerprint file
0.67 0.25-0.40 0.40 0.50-0.30 0.10 0.015-0.75 0.25-0.50 0.10 0.39 0.52 0.54
Lempel-Ziv-Welch English text Cobol files Floating point array Formatted scientific data System log data Program source code Object code English prose PL/I source 256 X 256 digitized image 16 levels of grey
184
0.55 0.50-0.16 1.0 0.48 0.38 0.43 0.66 0.50 0.69 0.53
algorithms and hardware simulations, have vedfied the Lempel-Ziv algorithm to be superior in compression and faster in operation than traditional Huffman coding algorithms 6-8. Unisys is completing the construction and testing of custom integrated circuits for lossless data compression using an extension of the Lempel-Ziv algorithm. There have been three chips developed using I IJm GaAs E/D technology. The die sizes are 165 mm 2 containing nearly 10 000 devices each. A single chip is used for compression, another for decompression, and the third is a high speed FIFO for buffer control of the corn pressed data. Additional off-the-shelf memory devices are required to complete a design for compression and decompression. The semiconductor technology and chip architectures have been chosen to minimize size, weight and power while simultaneously maximizing operating speed for special purpose applications. The development has been driven by a need to compress, without loss of information, Synthetic Aperture Radar (SAP,)data. The SAR application requires full military specification in aircraft environments. Compression is achieved by parsing strings from the input source message and sending an identifier which represents the string. In order to parse strings efficiently, accordingto previously seen strings, a hash table is built in memory. The time required to input and output characters from the memory is a limiting factor for operating speed. The amount of memory typically used for the hash table is arranged as a 4096 entries x 12 bit wide. With present technology, the access times are between 15 to 50 ns depending on board layout and chip configurations. The number of access times per character varies according to the hashing function and the length of the string that can be parsed. Because of these variables the operating speed is partially determined by the compression ratio of the data being encoded. This is true of all lossless encoders because of the variable-rate nature of the coding. This variation in the code length requires that data in fixed rate channels have buffering to accommodate local variations of data compressibility. In order to minimize buffer overflow and underflow, analysis of the data source to characterize the overall compression variance is required 7. Other implementations such as packet switching networks, where variable length packets are naturally integrated into the communications protocols, do not require buffering of the data. Communication channels that use Iossless encoders can provide standby users with available bandwidth when compression is achieved by the encoder. This helps to minimize the system integration issues that need to be resolved when implementing a Iossless data compression algorithm into a fixed rate channel. Another important characteristic of all lossless data compression techniques is the propagation of transmission errors through the decoded message. However, this problem is reduced by choosing an appropriate block length to minimize the effects of channel errors. Regardless of the system integration issues that are considered when using a lossless data compression technique, there is considerable benefit derived in reduced costs and increased system throughput.
LOSSY COMPRESSION Much of the data that is transmitted on an information channel is for use by the human sensory system. Minor
computer communications
alterations or infrequent errors to this data are indetectable or tolerable to human senses. Many compression techniques capitalize on this phenomenon. When atechnique is able to reduce the data rate and bandwidth required to send the information by controllingthe distortion without intolerable changes to the data, it is referred to as Iossy compression. Vector quantization is a Iossy technique for reducing the amount of information to be transmitted or stored. This is accomplished by removing information that is perceived as useless in the particular application being considered. The use of vector quantization for the compression of images is generating a lot of interest. A description of vector quantization and examples as applied to image compression will be given below.
Vector quantization Vector quantization is a technique for mapping vectors from a given vector space into a reduced set of vectors within the original vector space or some other representative vector space. The reduced set of vectors, along with the associated mapping, is chosen to minimize error according to some distortion measure. This representative set of vectors is referred to as a code-book and is stored in a memory table. Efficient transmission of vector-quantized data occurs by sending a code-book index location from the memory table, rather then sending the vector itself. The computation required to compute the distortions, thereby finding the code-book entry of minimum distortion, has limited the availability of the technique. Advances in hardware allowing cost-efficient implementations of vector quantization have generated renewed excitement during the last few years. A fundamental result of rate distortion theory is that better overall compression performance can be achieved when encoding a vector (group of scalars), than when encoding the scalars individually9. Vector quantization takes advantage of this theory by compressing groups of scalars, treating each scalar as a vector coefficient. Methods of compression attempt to remove redundancies from data, while causing minimal distortion. Vector quantization uses four properties of vector parameters for redundancy removal: correlation, nonlinear dependency, probability density function shape and vector dimension 10 . Scalar quantization takes advantage of correlations and probability density function shape only. By using the properties of nonlinear dependencies and vector dimensionality, vector quantization is able to outperform scalar quantization, even when compressing totally uncorrelated data. An optimal vector quantizer is designed around a probability distribution, placing the code-vectors in the space according to vector probabilities. Vector probability distributions vary with different data. The Linde-BuzoGray algorithm 11 uses either a known probability distribution, or trains the code-vectors on a select set of training vectors. If the probability distribution is known, the codevectors are placed in the N dimensional vector space according to the probability distribution. Areas of high probability contain a larger population of code-vectors; low probability areas contain a sparse population. If the probability distribution is not known, the code-vectors are distributed according to a select set of training vectors. Iteratively selecting code-vectors to minimize distortion results in a locally optimal set of code-vectors. The algorithm guarantees convergence of a local minimum
vol 10 no 4 august 1987
distortion, but not convergence to an absolute minimum for all vectors. The vector quantization encoding process searches the representative code-vectors and replaces the input vector from the data source with an index. The index represents the codebook vector of minimum distance from the incoming vector. The distance between vector and code-vector is proportional to the amount of degradation that will occur from vector quantization. Distance is most often measured by using a squared error criterion but many others have been discussed in the literature11. As an image compression scheme, vector quantization has both theoretically and experimentally outperformed methods of image compression using scalar quantization. Image data is well suited for vector quantization since the end user of the data is often the human eye. Small changes in the quantization error are not distinguishable when using vectors to quantize, rather than using the sampled scalars. To transmit images on communication channels requires an image bit rate reduction technique. This technique may involve both temporal and spatial compression strategies. Typical frame rates are 15 to 30 frames/s for image motion, and as low as several seconds per frame for still images such as maps and drawings. The amount of data used to digitize a single image frame depends on the spatial resolution of the picture elements (pixels) and on the number of quantization levels defined for the pixels. Image data rates vary depending on the application and quality of the desired image. Time rate compression is accomplished by transmitting fewer image frames with the same amount of data within each frame. A spatial intraframe compressor selectively removes redundant information from each image frame. The overall data rate for image transmission is: BR = FR. RES. PR,
where BR is the bit rate (bit/s), FR is the frame rate (frames/s), RES is the image resolution in pixels/image, and PR is the pixel bit rate in bit/pixel. Trends in image compression have often followed the voice compression field, since many techniques that work well with 1 D voice data are readily adapted to 2D image data. Several motion compensation techniques have been developed to exploit both spatial and temporal image redundancies. Image data rates and distortion van/significantly with compression methods. Both spatial and temporal compression can be used to reduce the bandwidth or storage requirements to a designated level. Further reduction in image data rate is accomplished by reducing the overall bit/pixel value required to represent an image. Vector quantization is a technique used to accomplish this process. Image compression begins by arranging the image into vectors. Blocks of pixels are chosen as components of the vector for quantization. The best results are produced when the components are correlated. Therefore, vectors are most often chosen to be M X N blocks of the image (typical values are 1 < M, N < 64). Another approach is to use a 2D Hilbert scan to correlate the pixels and then 'parse' vectors from the scan. The second approach gives flexibility in selecting vector dimension. After the vectors are arranged from the image data, they are encoded by using indices from the vector codebook table. Decompression is simply a 'table look-up' process to replace the index with the representative vector.
185
Additional gains in image compression have been accomplished by preprocessing of the image and vector data. For example, the mean value of each vector can be predicted or calculated and removed before vector quantization, as developed by Baker 12. This approach, known as Mean Residual Vector Quantization (MRVQ), allows the mean value information to be DPCM encoded, and the vector codebook to represent the residual vector information. Variations for reflection and orientation of the vector data were further investigated by Budge and Chabries 13. Further investigation by Vaughn and Stockham TM improved the subsampling process by bandlimiting the image with a low pass filter. This technique, known as Low Pass Residual Vector Quantization (LPRVQ), was able to remove the aliasing caused by MRVQ. Figure 3 is a block diagram showingthe implementation of LPRVQ. Figures 4 and 5 show, respectively, an original image at 8 bit/pixel and a reconstructed LPRVQ image that was compressed to 0.75 bit/pixel. Continued efforts in image compression by Unisys have lead to the development of a technique for compression of colour maps. Colour maps present an interesting challenge with respect to image compression is. Compression schemes tend to distort fine image details. Printed text, lines, and other details found on maps represent important navigational information which must be preserved. In many situations colour can be distorted without losing much information. The challenge of compressing maps is to preserve the map details while causing minimal distortion to the colours. By combining spatial vector quantizers with colour quantizers, good compression is produced with minimal distortion of both map details and colour. The application of this compression technique, known as Colour-Spatial Vector Quantization (CSVQ), is directed towards navigational maps for advanced aircraft, where quality map data is continually updated on the pilot display. Figure 6 shows the process of CSVQ. An original colour map (sampled at 8 b i t - red, 8 bit -- green, and 8 bit - - blue, foran overall sampling of 24 bit/pixel), is shown in Figure 7. The sampling is at 90 samples/in. A reconstructed map that was compressed using CSVQ at 2 bit/pixel, is shown in Figure 8.
to the intended receiver, orthe distortion is not disturbing to the listener or viewer. However, computer files that have single error are often no longer useful. When a compression algorithm is able to restore the encoded data to its original form, with no degradation, the
Figure4. Original image at 8bit/pixel before transmission at O.75 bit/pixel and reconstruction using LPRVQ
SUMMARY Many techniques have been developed for compression of digital voice, data, and images. Each method takes advantage of specific characteristics or the underlying redundancy of the data. Consideration is also given to the purpose or final use of the data. For example, voice and images do not require perfect replication so long as the introduced distortion does not result in information loss v
+~RC-~TE~
I
~'-~
/
/ I
quant~z'er[-~ Ch. . . . I)-'~ 1 VQ decoder
Figure 5. Reconstructed image after compression to O.75 bft/pixel
1 4-
LO~coPar ~s ilmtge
V
~
Vector quantization
• Vector
= 12 (2 × 2 Pixels, 3 colour)
dimension LP
~
I ........... I - I quantizer]
Figure 3.
lndex
• Vector codebook size (8 bit)
Low-pass~residual vector quantizer - - encoder
and decoder
186
-- [Channel ~ Low-pass,mage ) ) --] ...... truct J
^
~
Figure 6.
Colour-spatial vector quantization
computer communications
efficiency and hardware complexity. Many algorithms that were previously infeasible, are now possible owing to hardware developments. Data compression algorithms are being optimized to trade off the influence of four parameters: bit rate, quality, complexity, and time delay. These parameters are interrelated. Quality of reconstruction falls off as bit rate is reduced; this can be negated if complexity of the coding is increased. The cost of implementation increases as complexity increases, and in many architectural implementations of algorithms, the time delay increases due to processing delays. These issues are being addressed by industry and compression products have already begun to appear.
ACKNOWLEDGEMENTS
Figure 7. Original co/our map (sampled at 8 bit - - red, 8 bit - - green and 8 bit - - blue to give an overall sampling of 24 bit/pixel) before transmission
This research was supported by the Unisys Communications Technology Center in Salt Lake City, Utah, USA under research grants to the following Universities: Brandeis University, Computer Science Department, Waltham, MA, USA; University of Utah, Department of Electrical Engineering, Salt Lake City, UT, USA; Brigham Young University, Department of Electrical Engineering, Provo, UT, USA. Several members of the technical staff of these Universities have contributed to this data compression effort. Professors M Cohn and J Storer of Brandeis University developed the extensions to the LempeI-Ziv algorithm. Professor A Lempel of the Department of Electrical Engineering, Technion-lsrael illustrated the value of the universal algorithm when he was on sabbatical to the Sperry Research Center. Professors T Stockholm and C Rushforth of the University of Utah, along with R Frost and others of the Unisys CTC, developed algorithms for voice and image compression.
REFERENCES I
Figure 8.
Reconstructed map using CSVQ at 2 bit/pixel
algorithm is referred to as a Iossless data compression technique. Other algorithms that introduce a controlled or acceptable amount of distortion are referred to as Iossy data compression. The end use of the data will dictate which compression method is best suited for implementation. Cost of ownership is an important factor in compression algorithm selection. The advantages of reduced storage and transmission charges must be compared to the cost and complexity of the hardware realization. Current hardware operates at higher speeds, allows greater complexity, and is cheaper than ever before. These hardware advances allow complex algorithms to process data before (and after) storage or transmission. Special purpose VLSI chips also allow algorithms to be directly implemented at reduced cost, size, weight, and power. The fact that VLSI chip development costs are dropping very much faster than communications costs has fundamentally changed the trade-off between communication
vol 10 no 4 august 1987
2 3 4 5 6 7 8 9
Lynch, T J 'Data compression techniques and applications' Lifetime Learning Publications, Belmont, CA, USA (1985) Lempel, A and Ziv, J 'On the complexity of finite sequences' IEEETrans. Inf. Theory IT-22 No I (January 1976) pp 75-81 Lempel, A and Ziv, J 'A universal algorithm for sequential data compression' IEEE Trans. Inf. Theory IT-23 No 3 (May 1977) pp 337-343 Lempel, A and Ziv, J 'Compression of individual sequences via variable-rate coding' IEEE Trans. Inf. Theory IT-24 No 5 (September 1978) pp 530-536 Library articles are listed under Data compression Unisys Technology Center for Communications, Salt Lake City, UT, USA Welch, T A 'A technique for high-performance data compression IEEECompuL Mag. (June 1984) pp 8-I 9 Lindsay, R A and Cox, B V 'Universal data compression' Proc. InL Telemetering Conf. Vo121 (October 1985) pp 117-124 Gallager, R G 'Variations on a theme by Huffman' IEEE Trans. Inf. Theory Vol IT-24 No 6 (November 1978) pp 668-674 Gray, R 'Vector quantization' IEEE ASSP Mag. (April 1984) pp 4-28
187
10 Makhoul, J, Roucosand, S and Gish, H 'Vector quantization in speech coding' Proc. IEEEVol 73 No 11 (November 1985) pp 1551-1588 11 Linde, Y, Buzo, A and Gray, R 'An algorithm for vector quantizer design' IEEETrans. Commun. (January 1980) pp 84-95 12 Heckbert, P 'Colour image quantization for frame buffer display' CompuL Graph. Vol 16 No 3 (July 1982) pp 297-367 13 Baker, R 'Vector quantization of digital images' Ph.D. Dissertation Stanford University, CA, USA (June 1984 14 Budge, S et al. 'Data compression of digital images; final report' Contract #5391265 Unisys Corporation, USA (April 1985) 15 Vaughn, V 'Product code vector quantization of images' Masters Thesis University of Utah, UT, USA (June 1987)
188
BIBLIOGRAPHY Barham, S and Love, J 'Voicenet - - packet voice in local area networks' Sperry Tech. Syrup. Track 4-4 (May 19-20 1986) pp 1-6 .Cox, B V 'Data compression for communications engineers' Commun. Int. Vol 11 No 7 (July 1984) pp 67-72 Frost, R and Pulsipher, D 'Bandwidth compression of voice and modem signals using variable rate ATC encoding' Sperry Tech. Syrup. Track 4-5 (May 19-20 1986) pp 1-11 layant, N S and Nell, P Digital encoding of waveforms Prentice Hall, Englewood Cliffs, NJ, USA (1984) Lindsay, R and Vaughn, V 'Image compression for potential use on ISDN' Sperry Tech. Syrup. Track 4-9 (May 19-20 1986) pp 1-9
computer communications
- - t h e s t a t e of
the art B V Cox and R A Lindsay outline the progress that has been made in data compression techniques
The technique of data compression demonstrates desirable properties that assure performance improvements of advanced network architectures such as ISDN, serial communication channels, data storage and data archiving. A global trend in telecommunications networks is the conversion from analogue to digital transmission. The design goal of these new networks is that voice, data, and images will be transmitted in fundamentally the same way, requiring neither prior knowledge as to the type of data being transmitted nor operator intervention. However, the bandwidth requirements for digital transmission are generally high. Spectrum bandwidth utilization is a design parameter for these systems that will benefit from the transmission of compressed data. The proposed Integrated Services Digital Network (ISDN) combines digital voice, data, and images on the same network. Since this network has limited bandwidth for transmission of information, the available bandwidth must be used as efficiently as possible. Keywords: data compression, data communications, universal codin& vector quantization
exchange of information by providing an environment in which central locations are responsible for maintaining databases and shared assets such as printers, mass storage, and high speed computation. Intelligent terminals or workstations access these networks, enabling the functions of a complex information system to be divided physically as well as logically. Networks, such as the telephone system, have been developed for voice transmission. Other computer networks operate with digital data and file transfers. The need for special purpose networks and transmission links will continue in the future. However, there is a growing trend to operate integrated networks of digital voice, data, and images that are not proprietary systems. Standardized integrated digital networks will provide the nucleus for proficiently exchanging information and maintaining databases. ISDN is an architectural concept for a telephone network to carry voice, data, message, video and image traffic from a local area network to a wide area network. Regardless of the size of the network or type of information that is being processed, there will be a requirement for efficient storage and transmission; therefore, compression of voice, data, and images will be a key technology for the networks of the future.
NETWORKING A N D COMPRESSION Computer networking is part of a general trend towards distributed information processing. Modern network architectures permit distributed processing and the Advanced Technology Department, UnisysCommunications Technology Center, 640 North Sperry Way, Salt Lake City, UT 84116, USA
INTRODUCTION TO DATA COMPRESSION The concept of data compression is not new. There has always been a need to increase storage media capacity, reduce transmission costs, or improve the time response of communication systems. Data compression is a phrase
0140-3664/87/040179-10 $03.00 © 1987 Butterworth & Co (Publishers) Ltd vol 10 no 4 august 1987
179
that has broad meaning; unfortunately the term is also application dependent. Data compression has also been referred to by other names in the literature; the most common are data compaction, bandwidth compression, redundancy removal, redundancy reduction, source coding, data rate reduction, adaptive sampling, parameter extraction, and universal coding. Analysis modelling, and coding of information to reduce overall bit rate has been actively studied for 35 years. Data compression is a mechanism or procedure that transforms a signal or data sequence into a new sequence containing the same information, but whose length is smaller than the original. In simple terms, it encodes a string of symbols into a sequence of bits in such a way that the sequence can be deduced uniquely to reproduce the original string or an acceptable replica of the string. These methods are well known; they are subject to active current research in industrial, governmental, and university laboratories; and the performance of these techniques is well documented in the literature. Data compression can be classified into two broad groups: parametric (or model), and waveform representation. Figure I outlines the major categories of data compression. Figure I is not comprehensive, but it is representative of the magnitude of the technological effort and the richness of the available compression algorithms. Waveform representations address the issue of preserving the 'wave shape' of the signal through sampling and quantization procedures. Parametric representations, on the other hand, are concemed with representing the signal as the output of a model that best characterizes the signal generation mechanism. As Figure 1 suggests, data compression techniques have been
i"
Data compression
subject to prodigious scientific inquiry. Figure 1 shows data compression categories and their chronological significance. In 1835, Morse introduced a form of data compression by using a variable-length code. This approach used short codes for the most probable letters of the English alphabet and longer codes for the least probable letters. Most authors attribute the first data compression invention to the Vocoder by Dudley in 19391. The Shannon-Fano code was developed in 1948. Redundancy reduction techniques were developed in space research, sounding rockets and satellite telemetry contexts during the 1960s. Run-length codes were applied to telemetry data during this same time span. Frequency representations were applied to speech and images in the 1960s. Delta modulation and the improved technique of Differential Pulse Code Modulation (DPCM) were invented in 1952. Both delta modulation and DPCM were applied to speech and video signals. Huffman coding was introduced in the 1950s as an algorithm for developing the optimum variable length code over a fixed alphabet with a known probability distribution. However, no advantage is taken of repetition of sequences of symbols. Binary coding techniques were developed in the 1960s, LynchDavisson code in the 1970s, Lempel and Ziv's Universal parsing code in 1978, and in the 1980s the Langdon and Rissanen arithmetic code. Parametric extraction techniques originated in 1950 with the predictive coding investigations, and in the 1960s transform and linear predictive coding developments were reported. Now in the 1980s, information Iossless compression, or source encoding is actively being investigated and applied to speech, data,
J J Waveform J
I Paramricextraction I I
Analysis /Synthesis Voice
LPC
excited
I
i
Other
Waveform representation
Frequency
discrimination
Phase comparison
Homomorphic
I Statistical j
Non ~daptive J
I
I
I [Adaptive[
Incremental Probabilistic Gilbert Huffman Fang mo Shannon/Fan(
Figure 1.
180
(Time, Spacedomain)
I
I Frequency represen Jresentation
Redundancy reduction
I Delta modulation ! 'rensform I PCM LDM iD & 2D) Log PCM Square root DPCM FI R Root-meanADPCM I IR square ADM Subband Fourier zero crossing Filter bank __ Predictors J Prime I Adaptive sampling APC Discrete linear /interpolators Differential Channel Universal J basic vector quantizer Vocoder ATC Adaptive quantizer Lempel/ZIV Chirp Z Feed-forward Tree/incremental Walsh Feed-back Trellis Haar ial Predictive enumerative Discrete cosine e Companding Variable-length Discrete Chebychev tial p--Law tree Slant Log Sequential tree Karhunan-Loeve Laplace N-Gram Hadamard String matching CVSD R Transform Vector quantizer Run length Vector/matrix quantizer Accretive L~east~quare Fractal image (yardstick coding) Ordered run Increased filter Variable-to-block Arithmetic
[ F~,ter,ngJ I~(
Filtering thresholding
I
J
I
I I
analysis
Formant vocoder Channel vocoder Auto-regressive (An) Moving average (MA) ARMA ARIMA Multpulse LPC Stochastically excited LPC
J
Source encoding
Short time spectrum
Phase vocoder
I information Iossy
J Information Iossless I
J
j
Bit Plane Adaptive coefficienl Probabilistic Adaptive Filter Modulator Huffman
~
Data compression classification
computer communications
and image compression. Table 1 summarizes the compression results obtained by the Unisys Communications Technology Center, or reported in the literature for the various compression algorithms.
CLASSIFICATION OF DATA COMPRESSION TECHNIQUES Source coding is the art of conserving storage space, transmission time, and bandwidth by recoding data into a
Table 1.
more compact form. The code may be reversible, in which case it is called 'lossless', or it may entail irreversible (but tolerable) changes, in which case it is called 'lossy'. Lossless coding is essential in applications involving computer programs, encrypted data, and other sources where even minute changes can have enormous consequences; lossy coding is acceptable in such applications as text compression where some 'typos' are tolerable. Lossy compression is most appropriate in applications involving digitized analogue data, such as visual images, audio recordings, seismic soundings, and so on, where
Digital techniques for voice/video/data compression
Type information
Technique
Transmission rate (bit/s)
High-quality sound (Voice) (0/I 5 000 Hz)
PCM sampling Rate 35 kHz 13 bit sampling
450 000
Synthesized-voice
Phase vocoder Adaptive transform (ATC) Channel (FFT) vocoder Subband (4) Homomorphic (Ceptrum) Filtered vocoder LPC vector quantization Format vocoder Voice excited vocoder Pitch synchronized adaptive LPC (10th order) LPC (4th order) Adaptive predictive coding (APC)
7200 7200-32 000" 2400-4800 9600-24 000" 2400-4800
Speech
Video telephone
Television
vol 10 no 4 august 1987
Pixel bit rate (bit/pixel)
800 600-1200 4800-9600 7200 1200-4800 8000 6400-16 000"
Block vector quantization Markov vector quantization Two-stage vector quantization Vector quantization LPC, transform coded residual error Vector quantization LPC, scalar pitch and gain Vector quantization LPC, residual spectral coding Sliding rate design Trellis code Tree encoding Adaptive tree Adaptive Huffman
0.2 kbit/s 0.4-0.8 kbit/s 1.2 kbit/s 2.4-16 kbit/s
Forward error Control PCM PCM
T1 Carrier 1.544 Mbit/s
DPCM-B&W-reduced resolution DPCM-colour-frame storage Required Hadamard slant transform (Colour) PCM Hadamard transform (B&W) Sample PCM colour DPCM-interframe coding (Ishiguro) DPCM-colour-reduced Vertical resolution DPCM-7/8 error code (Golding)
1.5 Mbit/s 4.8 Mbit/s
0.8 kbit/s 2.4 kbit/s 0.8-16 kbit/s 3250-13 000 bit/s 8000 bit/s 8000 bit/s 3200 bit/s
T2 Carrier 6.312 Mbit/s
6 Mbit/s 11 Mbit/s 12 Mbit/s 18-22 Mbit/s 29 Mbit/s 33 Mbit/s
181
Table I continued
Type information
Technique
Transmission rate
DPCM-(Galfield) DPCM-1/2 error code Basic PCM PCM pseudonoise Quantization Frame replenishment Discrete cosine/DPCM Motion-compensated interframe Variable rate hadamard 2D discrete cosine Statistical Adaptive Non-adaptive Hybrid interframe Predictive-interframe Interpolative Transform-interframe Colour non-adaptive 2D DPCM Colour non-adaptive 2D DPCM Slow scan Cosine; DPCM between lines Haar Hadamard Constant area quantization Adaptive 2D DPCM
43 Mbit/s 58.4 Mbit/s 86-92.6 Mbit/s
Colour maps
Colour spatial vector quantization
0.50-2.0*
Time-domain voice coding
Vector/matrix quantization
200-400
CVSD Frame predictive vector Quantization Log DCM Adaptive delta Modulation (ADM) Linear delta Modulation (LDM) Forward error control PCM Non-adaptive delta modulation DPCM Adaptive differential PCM (residual encoder)
16 000-32 000 250
Vector coding
Telemetry
EKG F-16 Flight data
Formant Pattern matching Vector quantization (LPC) Phonemic Diphone Voice excited (LPC) VELP Residual excited (LPC) RELP Adaptive aperture First-order interpolator Zero-order interpolator Zero-order predictor
Pixel bit rate (bit/pixel)
8-5 5-3 2-1 0.5-2.0 0.3-1.5 Mbit/s 1 Mbit/s 1 Mbit/s 6-4 4-3.5 1-0.2 4-2 2-1 3-1 13 Mbit/s 144 kbit/s 0.75-2.0 0.75-2.0 0.75-2.0 0.75-2.0 0.75-2.0
33 000-45-000 8000-16 000 20 000-40 000 10 000 10 000-32 000 30 O00-40 000 9600-16 000 0.6-1.2 kbit/s 0.6-1.2 kbit/s 0.6-1.2 kbit/s 0.075-0.2 kbit/s 0.075-0.2 kbit/s 4.8-16 kbit/s 4.8-16 kbit/s 0.I0 0.30-0.10 0.20-0.50
Cycle-to-Cycle 0th order linear predictor Adaptive Huffman Lempel-Ziv CVSDM
0.001 0.25 0.33 0.31 0.125
*Compression techniques and results developed and reported by Sperry Corporate Technology Center for Communication
182
computer communications
some alteration is undetectable or tolerable to human senses. It is beyond the scope of this paper to attempt a discussion of all of the techniques used historically. As shown in Figure 1, there have been many techniques developed, many of which have not been included; particularly when an algorithm is developed that is specific to a single data type or application. The primary goal of data compression studies is to develop universal techniques that perform optimally on all types of data. The parameter of interest is whether the data is to be compressed with distortion or without distortion. When distortion is 'tolerable' to the user then the amount of acceptable distortion must be defined. The current stateof-the-art algorithms for data compression are the LempelZiv in the lossless application, and vector quantization in the lossy data compression. Both of these algorithms approach the optimum amount of compression achievable without prior knowledge of the data. A brief introduction to each of these algorithms will be given.
state, at both the encoder and the decoder. When an infinite sequence x is fed into E, the encoder emits an infinite sequence y = Yl, Y2, Y3. . . . (y @ B) while going through an infinite sequence of states z = zl, z2, z3 . . . . The decoder traces the same states z by using the decoded message x and the next state function g. This mapping is shown in Figure 2. An encoder E is said to be information lossless if for all zl and all finite Xn -- xl, x2, X3. . . . . Xn, the triple {zl, f (zl, Xn), g (zl, Xn) } uniquely determines Xn. Given the encoder E = (S, A, B, f, g) and the input string Xn, the compression ratio forx n with respect to E is defined by: pE(Xn) = L(yn)/log2cz, where ~ = I A I, Yn = f(zl, Xn), L(yn) = T-,I= ~l(yi), and I(yi) is the length in bits of Yi E B. The minimum of PE(Xn) over the class E(s) of all finitestate information Iossless encoders with [ A I = czand [ 51 < s is denoted by PE(s)(Xn), that is: min PE(s)(Xn) = E E E(s) [PE(xn)].
LOSSLESS COMPRESSION Advanced information systems require compression algorithms that are not source-dependent, so that tandem sources can be transmitted transparently. When a priori knowledge of the data source is not available, and statistical preprocessing techniques are not practical, the problem of data compression becomes more difficult. Lempel and Ziv have developed universal compression techniques that are based on a theory of complexity of finite strings2-4. They defined a new measure of complexity that treats only the record at hand and does not employ a probabilistic source for the data record. From this measure of complexity, Lempel-Ziv proposed a noiseless string matching and parsing approach to data compression. This algorithm addresses many of the traditional problems associated with data compression, in that it dynamically adapts to the redundancy characteristics of the data being compressed. It is not an approximation technique; it does not remove redundancy to some acceptable level, but faithfully reproduces the original datastream. Lempel and Ziv developed theoretical arguments to prove that the compression ratio achieved by the algorithm uniformly approached the lower bounds on the compression ratio attained by algorithms designed to match a statistically specified source. The encoder and decoder work in step to monitor and 'learn' the characteristics of the data and provide an efficient code for compression.
Lempel-Ziv algorithm
Furthermore, let, PE(s) (X) = lim n ~ sup oo PE(s)(Xn),
and p(x) "= lim
s -, ~ PE(s)(x).
This normalizing quantity p(x) that depends solely on the input message x is referred to as the compressibility of x. (See Reference 4 for further detail or proof of the underlying concept.) The Lempel-Ziv compression algorithm has the ability to asymptotically converge on the compressibility of the data message by remembering all past events that have occurred. Continued efforts have focussed on the practical implementation of the Lempel-Ziv algorithm 5. It is necessary to reduce the memory required to a workable size. In theory, the larger the memory, the greater the amount of compression. However, it becomes increasingly difficult to save and search all possible events that have occurred previously in a message. In practice, the algorithm is more easily described as the following: a) Parse a string from the input message source, b) Send a code or index that represents the string, c) Update the dictionary. Step a) is accomplished by searching past events that are stored in a dictionary to find the best possible match of previously seen strings. Originally this was defined as comparing the input source against the previously seen message as a 'sliding window' to find the longest match. This approach was not practical because the search time
The Lempel-Ziv compression algorithm is described as an encoder E defined by the quintuple (S, A, B, f, g) where: 5 = a finite set of states that the encoder and decoder both enter during the compression of a message, A = a finite-set input alphabet, B = a finite set of output words over a finite output alphabet, f = the output function that maps SxA--~ B, and g = the next state function that maps SxA --~ S. Compression of a sequence x = xl, x2, x3 . . . . (x (E A) by the encoder E begins by initializingzl, (z E S), to the same
vol 10 no 4 august 1987
Yi = f (xi, zi ) xi =f-l(yi,zi)
zi+ 1= g ( x i , z
i ) i = 1,2 . . . .
Figure 2. Mapping for finite state information Iossless encoder/decoder
183
increased geometrically as the data was encoded. Simulations also revealed that the pointers to the table were remembering too much about past data and were not efficient. Improvements for searching the dictionary by using tree structures and hashing tables have enabled the algorithm to be implemented in linear time, thereby producing efficient computer and hardware implementations. Step b) has several variations depending on the particular application. The code word that is sent by the encoder is an index to the dictionary. When the dictionary is atree structure, the code is a pointer to a particular node of the tree. When the dictionary is a hashing table, the code is a key to the hashing function or an identifier. Some implementations use both. The codes can be variable in length or fixed in length depending on the searching procedure in a). Some implementations have the next character after the string included in the code to facilitate step c). Dictionary updates are accomplished by rules or heuristic methods which determine how much to remember. These rules can be as simple as adding the next character of the message to the previously parsed string to create a new string. One particular implementation known as the Lempel-Ziv-Welch uses a unique combination of steps b) and c), known as deferred innovation encoding 6. The code-word of step b) does not include the next character of the message, but this character is later derived from the beginning of the next string and used to update a hashing table. This algorithm has been implemented and is in wide use on CS-Net known as Compress. Table 2 shows results of compression for typical data using the Lempel-Ziv and Lempel-ZivWelch algorithms. Experimental results, from computer
Table 2. Lempel-Ziv technique for voice/video/data compression Information type
Compression ratio Lempel-Ziv
EXEC 8 Code Business files English text Speech (3.2 kHz) Fascimile (250 pixels/inch) Geo physical data CAD logic input Database PCM coded satellite photo Pseudo-random digits (Entropy 0.405) Coded fingerprint file
0.67 0.25-0.40 0.40 0.50-0.30 0.10 0.015-0.75 0.25-0.50 0.10 0.39 0.52 0.54
Lempel-Ziv-Welch English text Cobol files Floating point array Formatted scientific data System log data Program source code Object code English prose PL/I source 256 X 256 digitized image 16 levels of grey
184
0.55 0.50-0.16 1.0 0.48 0.38 0.43 0.66 0.50 0.69 0.53
algorithms and hardware simulations, have vedfied the Lempel-Ziv algorithm to be superior in compression and faster in operation than traditional Huffman coding algorithms 6-8. Unisys is completing the construction and testing of custom integrated circuits for lossless data compression using an extension of the Lempel-Ziv algorithm. There have been three chips developed using I IJm GaAs E/D technology. The die sizes are 165 mm 2 containing nearly 10 000 devices each. A single chip is used for compression, another for decompression, and the third is a high speed FIFO for buffer control of the corn pressed data. Additional off-the-shelf memory devices are required to complete a design for compression and decompression. The semiconductor technology and chip architectures have been chosen to minimize size, weight and power while simultaneously maximizing operating speed for special purpose applications. The development has been driven by a need to compress, without loss of information, Synthetic Aperture Radar (SAP,)data. The SAR application requires full military specification in aircraft environments. Compression is achieved by parsing strings from the input source message and sending an identifier which represents the string. In order to parse strings efficiently, accordingto previously seen strings, a hash table is built in memory. The time required to input and output characters from the memory is a limiting factor for operating speed. The amount of memory typically used for the hash table is arranged as a 4096 entries x 12 bit wide. With present technology, the access times are between 15 to 50 ns depending on board layout and chip configurations. The number of access times per character varies according to the hashing function and the length of the string that can be parsed. Because of these variables the operating speed is partially determined by the compression ratio of the data being encoded. This is true of all lossless encoders because of the variable-rate nature of the coding. This variation in the code length requires that data in fixed rate channels have buffering to accommodate local variations of data compressibility. In order to minimize buffer overflow and underflow, analysis of the data source to characterize the overall compression variance is required 7. Other implementations such as packet switching networks, where variable length packets are naturally integrated into the communications protocols, do not require buffering of the data. Communication channels that use Iossless encoders can provide standby users with available bandwidth when compression is achieved by the encoder. This helps to minimize the system integration issues that need to be resolved when implementing a Iossless data compression algorithm into a fixed rate channel. Another important characteristic of all lossless data compression techniques is the propagation of transmission errors through the decoded message. However, this problem is reduced by choosing an appropriate block length to minimize the effects of channel errors. Regardless of the system integration issues that are considered when using a lossless data compression technique, there is considerable benefit derived in reduced costs and increased system throughput.
LOSSY COMPRESSION Much of the data that is transmitted on an information channel is for use by the human sensory system. Minor
computer communications
alterations or infrequent errors to this data are indetectable or tolerable to human senses. Many compression techniques capitalize on this phenomenon. When atechnique is able to reduce the data rate and bandwidth required to send the information by controllingthe distortion without intolerable changes to the data, it is referred to as Iossy compression. Vector quantization is a Iossy technique for reducing the amount of information to be transmitted or stored. This is accomplished by removing information that is perceived as useless in the particular application being considered. The use of vector quantization for the compression of images is generating a lot of interest. A description of vector quantization and examples as applied to image compression will be given below.
Vector quantization Vector quantization is a technique for mapping vectors from a given vector space into a reduced set of vectors within the original vector space or some other representative vector space. The reduced set of vectors, along with the associated mapping, is chosen to minimize error according to some distortion measure. This representative set of vectors is referred to as a code-book and is stored in a memory table. Efficient transmission of vector-quantized data occurs by sending a code-book index location from the memory table, rather then sending the vector itself. The computation required to compute the distortions, thereby finding the code-book entry of minimum distortion, has limited the availability of the technique. Advances in hardware allowing cost-efficient implementations of vector quantization have generated renewed excitement during the last few years. A fundamental result of rate distortion theory is that better overall compression performance can be achieved when encoding a vector (group of scalars), than when encoding the scalars individually9. Vector quantization takes advantage of this theory by compressing groups of scalars, treating each scalar as a vector coefficient. Methods of compression attempt to remove redundancies from data, while causing minimal distortion. Vector quantization uses four properties of vector parameters for redundancy removal: correlation, nonlinear dependency, probability density function shape and vector dimension 10 . Scalar quantization takes advantage of correlations and probability density function shape only. By using the properties of nonlinear dependencies and vector dimensionality, vector quantization is able to outperform scalar quantization, even when compressing totally uncorrelated data. An optimal vector quantizer is designed around a probability distribution, placing the code-vectors in the space according to vector probabilities. Vector probability distributions vary with different data. The Linde-BuzoGray algorithm 11 uses either a known probability distribution, or trains the code-vectors on a select set of training vectors. If the probability distribution is known, the codevectors are placed in the N dimensional vector space according to the probability distribution. Areas of high probability contain a larger population of code-vectors; low probability areas contain a sparse population. If the probability distribution is not known, the code-vectors are distributed according to a select set of training vectors. Iteratively selecting code-vectors to minimize distortion results in a locally optimal set of code-vectors. The algorithm guarantees convergence of a local minimum
vol 10 no 4 august 1987
distortion, but not convergence to an absolute minimum for all vectors. The vector quantization encoding process searches the representative code-vectors and replaces the input vector from the data source with an index. The index represents the codebook vector of minimum distance from the incoming vector. The distance between vector and code-vector is proportional to the amount of degradation that will occur from vector quantization. Distance is most often measured by using a squared error criterion but many others have been discussed in the literature11. As an image compression scheme, vector quantization has both theoretically and experimentally outperformed methods of image compression using scalar quantization. Image data is well suited for vector quantization since the end user of the data is often the human eye. Small changes in the quantization error are not distinguishable when using vectors to quantize, rather than using the sampled scalars. To transmit images on communication channels requires an image bit rate reduction technique. This technique may involve both temporal and spatial compression strategies. Typical frame rates are 15 to 30 frames/s for image motion, and as low as several seconds per frame for still images such as maps and drawings. The amount of data used to digitize a single image frame depends on the spatial resolution of the picture elements (pixels) and on the number of quantization levels defined for the pixels. Image data rates vary depending on the application and quality of the desired image. Time rate compression is accomplished by transmitting fewer image frames with the same amount of data within each frame. A spatial intraframe compressor selectively removes redundant information from each image frame. The overall data rate for image transmission is: BR = FR. RES. PR,
where BR is the bit rate (bit/s), FR is the frame rate (frames/s), RES is the image resolution in pixels/image, and PR is the pixel bit rate in bit/pixel. Trends in image compression have often followed the voice compression field, since many techniques that work well with 1 D voice data are readily adapted to 2D image data. Several motion compensation techniques have been developed to exploit both spatial and temporal image redundancies. Image data rates and distortion van/significantly with compression methods. Both spatial and temporal compression can be used to reduce the bandwidth or storage requirements to a designated level. Further reduction in image data rate is accomplished by reducing the overall bit/pixel value required to represent an image. Vector quantization is a technique used to accomplish this process. Image compression begins by arranging the image into vectors. Blocks of pixels are chosen as components of the vector for quantization. The best results are produced when the components are correlated. Therefore, vectors are most often chosen to be M X N blocks of the image (typical values are 1 < M, N < 64). Another approach is to use a 2D Hilbert scan to correlate the pixels and then 'parse' vectors from the scan. The second approach gives flexibility in selecting vector dimension. After the vectors are arranged from the image data, they are encoded by using indices from the vector codebook table. Decompression is simply a 'table look-up' process to replace the index with the representative vector.
185
Additional gains in image compression have been accomplished by preprocessing of the image and vector data. For example, the mean value of each vector can be predicted or calculated and removed before vector quantization, as developed by Baker 12. This approach, known as Mean Residual Vector Quantization (MRVQ), allows the mean value information to be DPCM encoded, and the vector codebook to represent the residual vector information. Variations for reflection and orientation of the vector data were further investigated by Budge and Chabries 13. Further investigation by Vaughn and Stockham TM improved the subsampling process by bandlimiting the image with a low pass filter. This technique, known as Low Pass Residual Vector Quantization (LPRVQ), was able to remove the aliasing caused by MRVQ. Figure 3 is a block diagram showingthe implementation of LPRVQ. Figures 4 and 5 show, respectively, an original image at 8 bit/pixel and a reconstructed LPRVQ image that was compressed to 0.75 bit/pixel. Continued efforts in image compression by Unisys have lead to the development of a technique for compression of colour maps. Colour maps present an interesting challenge with respect to image compression is. Compression schemes tend to distort fine image details. Printed text, lines, and other details found on maps represent important navigational information which must be preserved. In many situations colour can be distorted without losing much information. The challenge of compressing maps is to preserve the map details while causing minimal distortion to the colours. By combining spatial vector quantizers with colour quantizers, good compression is produced with minimal distortion of both map details and colour. The application of this compression technique, known as Colour-Spatial Vector Quantization (CSVQ), is directed towards navigational maps for advanced aircraft, where quality map data is continually updated on the pilot display. Figure 6 shows the process of CSVQ. An original colour map (sampled at 8 b i t - red, 8 bit -- green, and 8 bit - - blue, foran overall sampling of 24 bit/pixel), is shown in Figure 7. The sampling is at 90 samples/in. A reconstructed map that was compressed using CSVQ at 2 bit/pixel, is shown in Figure 8.
to the intended receiver, orthe distortion is not disturbing to the listener or viewer. However, computer files that have single error are often no longer useful. When a compression algorithm is able to restore the encoded data to its original form, with no degradation, the
Figure4. Original image at 8bit/pixel before transmission at O.75 bit/pixel and reconstruction using LPRVQ
SUMMARY Many techniques have been developed for compression of digital voice, data, and images. Each method takes advantage of specific characteristics or the underlying redundancy of the data. Consideration is also given to the purpose or final use of the data. For example, voice and images do not require perfect replication so long as the introduced distortion does not result in information loss v
+~RC-~TE~
I
~'-~
/
/ I
quant~z'er[-~ Ch. . . . I)-'~ 1 VQ decoder
Figure 5. Reconstructed image after compression to O.75 bft/pixel
1 4-
LO~coPar ~s ilmtge
V
~
Vector quantization
• Vector
= 12 (2 × 2 Pixels, 3 colour)
dimension LP
~
I ........... I - I quantizer]
Figure 3.
lndex
• Vector codebook size (8 bit)
Low-pass~residual vector quantizer - - encoder
and decoder
186
-- [Channel ~ Low-pass,mage ) ) --] ...... truct J
^
~
Figure 6.
Colour-spatial vector quantization
computer communications
efficiency and hardware complexity. Many algorithms that were previously infeasible, are now possible owing to hardware developments. Data compression algorithms are being optimized to trade off the influence of four parameters: bit rate, quality, complexity, and time delay. These parameters are interrelated. Quality of reconstruction falls off as bit rate is reduced; this can be negated if complexity of the coding is increased. The cost of implementation increases as complexity increases, and in many architectural implementations of algorithms, the time delay increases due to processing delays. These issues are being addressed by industry and compression products have already begun to appear.
ACKNOWLEDGEMENTS
Figure 7. Original co/our map (sampled at 8 bit - - red, 8 bit - - green and 8 bit - - blue to give an overall sampling of 24 bit/pixel) before transmission
This research was supported by the Unisys Communications Technology Center in Salt Lake City, Utah, USA under research grants to the following Universities: Brandeis University, Computer Science Department, Waltham, MA, USA; University of Utah, Department of Electrical Engineering, Salt Lake City, UT, USA; Brigham Young University, Department of Electrical Engineering, Provo, UT, USA. Several members of the technical staff of these Universities have contributed to this data compression effort. Professors M Cohn and J Storer of Brandeis University developed the extensions to the LempeI-Ziv algorithm. Professor A Lempel of the Department of Electrical Engineering, Technion-lsrael illustrated the value of the universal algorithm when he was on sabbatical to the Sperry Research Center. Professors T Stockholm and C Rushforth of the University of Utah, along with R Frost and others of the Unisys CTC, developed algorithms for voice and image compression.
REFERENCES I
Figure 8.
Reconstructed map using CSVQ at 2 bit/pixel
algorithm is referred to as a Iossless data compression technique. Other algorithms that introduce a controlled or acceptable amount of distortion are referred to as Iossy data compression. The end use of the data will dictate which compression method is best suited for implementation. Cost of ownership is an important factor in compression algorithm selection. The advantages of reduced storage and transmission charges must be compared to the cost and complexity of the hardware realization. Current hardware operates at higher speeds, allows greater complexity, and is cheaper than ever before. These hardware advances allow complex algorithms to process data before (and after) storage or transmission. Special purpose VLSI chips also allow algorithms to be directly implemented at reduced cost, size, weight, and power. The fact that VLSI chip development costs are dropping very much faster than communications costs has fundamentally changed the trade-off between communication
vol 10 no 4 august 1987
2 3 4 5 6 7 8 9
Lynch, T J 'Data compression techniques and applications' Lifetime Learning Publications, Belmont, CA, USA (1985) Lempel, A and Ziv, J 'On the complexity of finite sequences' IEEETrans. Inf. Theory IT-22 No I (January 1976) pp 75-81 Lempel, A and Ziv, J 'A universal algorithm for sequential data compression' IEEE Trans. Inf. Theory IT-23 No 3 (May 1977) pp 337-343 Lempel, A and Ziv, J 'Compression of individual sequences via variable-rate coding' IEEE Trans. Inf. Theory IT-24 No 5 (September 1978) pp 530-536 Library articles are listed under Data compression Unisys Technology Center for Communications, Salt Lake City, UT, USA Welch, T A 'A technique for high-performance data compression IEEECompuL Mag. (June 1984) pp 8-I 9 Lindsay, R A and Cox, B V 'Universal data compression' Proc. InL Telemetering Conf. Vo121 (October 1985) pp 117-124 Gallager, R G 'Variations on a theme by Huffman' IEEE Trans. Inf. Theory Vol IT-24 No 6 (November 1978) pp 668-674 Gray, R 'Vector quantization' IEEE ASSP Mag. (April 1984) pp 4-28
187
10 Makhoul, J, Roucosand, S and Gish, H 'Vector quantization in speech coding' Proc. IEEEVol 73 No 11 (November 1985) pp 1551-1588 11 Linde, Y, Buzo, A and Gray, R 'An algorithm for vector quantizer design' IEEETrans. Commun. (January 1980) pp 84-95 12 Heckbert, P 'Colour image quantization for frame buffer display' CompuL Graph. Vol 16 No 3 (July 1982) pp 297-367 13 Baker, R 'Vector quantization of digital images' Ph.D. Dissertation Stanford University, CA, USA (June 1984 14 Budge, S et al. 'Data compression of digital images; final report' Contract #5391265 Unisys Corporation, USA (April 1985) 15 Vaughn, V 'Product code vector quantization of images' Masters Thesis University of Utah, UT, USA (June 1987)
188
BIBLIOGRAPHY Barham, S and Love, J 'Voicenet - - packet voice in local area networks' Sperry Tech. Syrup. Track 4-4 (May 19-20 1986) pp 1-6 .Cox, B V 'Data compression for communications engineers' Commun. Int. Vol 11 No 7 (July 1984) pp 67-72 Frost, R and Pulsipher, D 'Bandwidth compression of voice and modem signals using variable rate ATC encoding' Sperry Tech. Syrup. Track 4-5 (May 19-20 1986) pp 1-11 layant, N S and Nell, P Digital encoding of waveforms Prentice Hall, Englewood Cliffs, NJ, USA (1984) Lindsay, R and Vaughn, V 'Image compression for potential use on ISDN' Sperry Tech. Syrup. Track 4-9 (May 19-20 1986) pp 1-9
computer communications