Texture Analysis for Enhanced Color Image Quantization

GRAPHICAL MODELS AND IMAGE PROCESSING

Vol. 59, No. 3, May, pp. 149–163, 1997 ARTICLE NO. IP970428

Texture Analysis for Enhanced Color Image Quantization Jefferey A. Shufelt* School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213-3891 Received September 18, 1995; revised November 4, 1996; accepted March 25, 1997

A traditional problem with color image quantization techniques is their inability to handle smooth variations in intensity and chromaticity, leading to contours in the quantized image. To address this problem, this paper describes new techniques for augmenting the performance of a seminal color image quantization algorithm, the median-cut quantizer. Applying a simple texture analysis method from computer vision in conjunction with the median-cut algorithm using a new variant of a k-d tree, we show that contouring effects can be alleviated without resorting to dithering methods and the accompanying decrease in signal-to-noise ratio. The merits of this approach are evaluated using remotely sensed aerial imagery and synthetically generated scenes.  1997 Academic Press

1. INTRODUCTION

The need for high quality reproductions of color imagery on limited frame buffer resources has been the prime motivation for research in color image quantization. Frame buffers capable of displaying full 24-bit (or greater) color images remain costly, while low end color display systems capable of 8-bit color mapping are becoming increasingly economical alternatives; thus the need for effective color image quantization techniques is not likely to decline in the immediate future. The advent of fast RISC-based workstations and relatively inexpensive memory, however, has made feasible the consideration of more computationally intensive techniques for color image quantization. We wished to run our display applications, currently supported on a 24-bit frame buffer, on several 8-bit color workstations. Our display applications deal primarily with high-resolution multispectral imagery and synthetic ren* The author was supported by an Augmentation Award for Science and Engineering Research Training sponsored by the Army Research Office, Department of the Army, under Contract DAAH04-93-G-0092. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office, the Department of the Army, or the U.S. government.

derings of natural terrain and urban sites, with a variety of interactive modules for querying, segmenting, and manually classifying image pixels. Since our applications require the visual assessment of various natural features, the elimination of contouring effects was a high priority; contours generated by image quantization can be falsely interpreted as scene features rather than quantization artifacts. Given these needs, we desired a quantizer suitable for displaying imagery without loss of resolution, at interactive speeds, producing quantized images of the highest possible quality in terms of precise color representation, and making use of all available color lookup entries. Memory usage was not a major concern, since machines can be cheaply equipped with large amounts of memory. Parameters were also a secondary concern, but the algorithms we employed in this work require only two parameters: the image and the number of colors to which the quantization would be performed. This paper presents a new technique for color image quantization to meet these needs, based on a simple texture analysis scheme employed in computer vision. This approach alleviates contouring effects produced during quantization without using dithering techniques, subject to our requirements of precise color representation and fast quantization. The technique can be implemented efficiently using a new variant of a k-d tree, which is readily suited for use in the seminal median-cut quantizer developed by Heckbert [9]. The technique revolves around the idea that colors in areas of high spatial frequency can be reallocated to areas with low spatial frequency, leveraging the human visual system’s relative insensitivity to precise color representation in areas of high spatial frequency. We begin with a brief discussion of image quantization techniques in Section 2. In Section 3, we discuss the basis for the work described in this paper, Heckbert’s mediancut quantizer, including our specific modifications to support later experiments. In Section 4, we describe an extension to median-cut that addresses contouring effects by texture analysis and discuss experiments with this tech-

149 1077-3169/97 $25.00 Copyright  1997 by Academic Press All rights of reproduction in any form reserved.

150

JEFFEREY A. SHUFELT

nique. We conclude in Section 5, with a discussion of future approaches for improving color image quantization. 2. IMAGE QUANTIZATION TECHNIQUES

A great deal of research has been done on color image quantization. Detailed descriptions of the problem can be found in [6, 9, 10]. In this section, we describe a few basic techniques and their strengths and weaknesses, along with some recent approaches. While the survey here is by no means comprehensive, it is representative of the issues found in the literature. Perhaps the fastest method for color image quantization is uniform quantization, in which an approximately equal number of bits are truncated from each of the three color bands in the original image, so that the resulting image data has only as many bits as can be represented by the color lookup table. Uniform quantization thus generates an evenly spaced set of representative colors for each band. This method typically produces poor quantizations, however, since it assumes that colors are evenly distributed along each color band, which is rarely the case, and it does not address perceived error in any way. An alternative image-independent method was proposed by Spaulding et al. [12], in which a nonlinear quantization function was optimized over all possible colors, using LAB as the color distance metric; this approach requires a computationally expensive gradient descent algorithm to determine the colormap, and is therefore impractical for creating palettes tailored to specific images at interactive speeds. Most methods in use today are adaptive, or imagedependent; they attempt to choose representative colors for an image by analysis of the color distribution of the original image. Of the adaptive methods, popularity quantization is the simplest. The popularity algorithm histograms the color image and chooses the most frequently occurring colors as representatives for the image. While simple to implement and relatively fast, this method functions poorly on images with a wide range of colors or with outliers in the color space. In addition to the median-cut technique described in the next section, there exist other, more complex adaptive quantization techniques. Wan et al. [15] presented an algorithm which iteratively subdivided the color cube by selecting partitions of the color space. The partitioning was based on the criteria of minimizing sum-of-squared error and always dividing the subcube of color space exhibiting the largest weighted variance. This variance-based approach was shown to give slight improvements in mean error at the expense of a slight increase in quantization time over median-cut, but still required dithering in some cases to remove contouring effects. Diaz-Cayeros [5] used standard image remappings to enhance image contrast and then applied histogram equal-

ization to spread image intensities among a fixed subdivision of color space, selecting the center of mass in each subcube as a representative. This basic sequential method for subdivision of color space is presented in a more sophisticated form by Balasubramanian et al. [2], who quantize in a luminance–chrominance space and use a spatial activity measure similar to one used in an earlier work to reduce contouring effects [1]. While these experiments demonstrated a speed improvement over median-cut, much of this improvement was due to subsampling the image prior to quantization, to measure spatial activity and reduce the number of colors to be quantized. 3. MEDIAN-CUT QUANTIZATION

One of the most popular adaptive algorithms is Heckbert’s median-cut quantizer [9]. This algorithm subdivides the color space by repeatedly slicing color cubes with a plane perpendicular to the axis of the cube which exhibits the greatest range of colors. A representative color is then selected for each of the subdivisions. In addition to its adaptive nature, the median-cut quantizer also deals well with outliers in the color space; it is the underlying basis of many recent adaptive quantizers; it achieves quantizations of quality comparable to that of other adaptive methods; and it has well documented implementations capable of quantization at interactive speeds [10]. For these reasons, we chose median-cut as the basis for our experimentation, although our texture analysis method can be applied to any quantizer which represents subdivisions of color space by a tree structure. In the following sections, we give a description of the algorithm and make note of its internal representations, as these are later exploited in texture analysis. Much of our description is based on Rosen’s implementation of median-cut [10]. We briefly describe each phase of the median-cut algorithm; prequantization, colormap selection, and nearest-neighbor remapping. Our new modifications of the Heckbert median-cut algorithm fall into the colormap selection phase, discussed in Sections 3.2 and 3.3. For the experiments described in this section and the following sections, we used 64 test scenes covering a variety of natural and synthesized color imagery: 8 simulated SPOT images, 45 DAEDALUS images, 3 radiosity images, and 8 ray-traced images. The simulated SPOT imagery was generated from the Daedalus Airborne Thematic Mapper Scanner imagery acquired for the 1983 U.S. SPOT Simulation Campaign. The Daedalus multispectral imagery was geometrically and radiometrically processed to replicate SPOT’s spatial and spectral characteristics. For the SPOT and Daedalus imagery, RGB values were taken directly from bands of the images. A typical SPOT image can be seen in Fig. 10 in Section 4.4; Fig. 14 in Section 4.5 is a typical ray-traced image.

COLOR IMAGE QUANTIZATION ENHANCEMENT

FIG. 1. Mean error for test scenes, RGB evaluation.

3.1. Prequantization The first step in the algorithm, prequantization, is used to achieve three goals; histogramming the image to estimate color distribution, reducing of memory requirements, and the elimination of bits which provide visually imperceptible changes in color. Many color images typically carry 24 bits of information, 8 bits per color band. Histogramming a color image into a three-dimensional array which could distinguish all possible colors would require 64 Mb, using 32-bit integers for each bucket of the histogram. Memory requirements on this scale are still impractical for most machines, so prequantization to fewer than 8 bits per band, by truncation of lower-order bits, brings memory requirements to a manageable level. Prequantization is also useful for perceptual reasons as well, since the human eye cannot distinguish all possible colors in a 24-bit image. Many algorithms prequantize to 6 bits per band, under the assumption that since the eye is capable of distinguishing at least 50,000 colors [9], 6 bits per band will provide 262,144 colors for use in representing the image. However, changes in RGB color values do not always correspond to changes in perceived color difference. Since the eye is not equally sensitive to each band, the distribution of colors in RGB color space is also important; 6 bits per band are not sufficient to avoid contouring effects in certain regions of color space. Prequantization can be implemented by using integer bit-shift operations on each band of each pixel to truncate each band to b bits, and histogramming these reduced color values. This requires one scan of the image data, bit-

151

shifting and indexing into the histogram at each pixel of the image. The selection of b involves a tradeoff between space considerations (each increment of b translates into an eight-fold increase in space consumption) and contouring effects; Heckbert has observed that b , 5 guarantees contouring effects, due to the human eye’s discriminatory ability [9]. We applied the median-cut algorithm with b 5 5–7 on our test scenes to evaluate the effects of the choice of b. Figure 1 shows the mean error for the 64 test scenes using the Euclidean RGB distance metric. The horizontal axis has a logarithmic scale indicating the number of colors remaining after prequantization. In almost all cases, increasing b results in a noticeable improvement in mean error. The improvement in increasing b from 5 to 6 is slightly larger than that obtained by going from 6 to 7, but this is to be expected, since the color resolution along each axis of color space doubles with every increment of b. However, this improvement comes at the expense of running time. Figure 2 shows the effect of increasing b on the speed of median-cut quantization. Each of the test scenes was quantized in RGB space, for b 5 5–7. Each test image was quantized 10 times for each value of b; the data points here represent the average of each set of 10 runs. All runs were performed on an unloaded DEC Alpha AXP 3000/400 with 96 Mb of memory running OSF/1, and the times include I/O for writing quantized images and fetching test images to and from AFS, a distributed network file system.

FIG. 2. Prequantization timings for test scenes.

152

JEFFEREY A. SHUFELT

The graph clearly shows the result of increased prequantization resolution on quantization time. With 5-bit quantization, no image takes longer than 5 s, and most images take less than 3 s. As b is increased to 6 and 7, the number of prequantized colors and the quantization time both increase. Three layers of data points can be seen; these correspond to the sizes of the image data, which fell into roughly three classes, 50–55 K, 130–170 K, and 260–300 K. These layers converge as the number of prequantized colors increases; this implies that the asymptotic behavior of quantization time is dominated by the number of prequantized colors, not the image size. We chose b 5 6 as a reasonable compromise between the memory requirements for the histogram and our desire to achieve fast quantization times. With b 5 6, the histogram consumes 1 Mb of space, and basic median-cut quantizations on our test imagery could be done in most cases in under 5 s. 3.2. Colormap Selection: Description In the colormap selection phase of the median-cut algorithm, we attempt to find the set of colors which will best represent the color distribution of the original image. We employ a new variant of a k-d tree for our primary color cube representation; in this section, we briefly describe this variant and sketch its use for colormap selection. A k-d tree [3, 4] is a generalization of a complete binary tree in which each node represents a portion of a k-dimensional search space. An optimized k-d tree [7] is a basic k-d tree in which the expected number of nodes examined by a search on the tree is minimized. In optimized k-d trees, the discriminating axis is chosen by selecting the axis along which the keys exhibit the widest range of values; the partition value is then chosen to be the median of the values along the axis. This strategy allows the partitioning of the tree to more closely fit the distribution of records in the search space, but since it is a balanced tree, each leaf of the tree will contain the same number of colors, without regard to the spread of colors throughout the entire color space. Our new variant, which we call the unbalanced optimized k-d tree, is defined to be an optimized k-d tree without the restriction that the tree be complete; in other words, we allow leaves to reside on more than one level of the tree. This allows dynamic allocation of new nodes to those portions of color space that exhibit the greatest range of color values. The root node of the unbalanced optimized k-d tree represents a color cube, and the records in the color cube are simply the prequantized colors in the original histogram of the color image. Given some desired number of representative colors c, we locate the leaf with the widest spread along any of the red, green, or blue axes, and split the leaf at the median value along the axis of greatest spread,

FIG. 3. Data structures for implementing colormap selection.

adding two children to this leaf, and increasing the total number of leaves by one. This process is iterated until the tree has c leaves. Then, the records within each leaf are averaged to compute the representative color for that leaf, and these c colors constitute the colormap. We note in passing that some implementations have chosen weighted averaging or the median of the leaf records to compute the representative color for a leaf; we chose averaging because of its speed, and because it produced reasonable color representatives. 3.3. Colormap Selection: Implementation Efficient implementation of colormap selection requires the use of several techniques, most of which are extensively detailed elsewhere [10, 11]; we summarize the major points briefly. First, the prequantized colors in the original histogram of the image are stored in an array COL. An array BOX represents the current set of subdivisions, or boxes, in color space as the partitioning proceeds; each entry in this array maintains start and end indices into COL, delimiting the contiguous list of colors that occur within that box. At the end of every iteration, one more box is added to BOX. Each iteration requires a SPLIT operation on the leaf of a tree. These data structures are illustrated in Fig. 3. The first step in a SPLIT requires finding the leaf in the tree with the largest spread of values. To accomplish this,

COLOR IMAGE QUANTIZATION ENHANCEMENT

each box is scanned to find the minimum and maximum values of its color points along each of the red, green, and blue axes. Once the leaf is found (and the box, since there is a one-to-one mapping between the two at any stage of the algorithm), the median is located efficiently by a lineartime bucket sort of the color points in the box. Finally, the leaf and its associated box are split by updating the start and end indices into COL for the two new boxes, which is trivial given the original start and end indices for the parent box and the index of the median. Splitting the leaf just requires the addition of two children to this leaf. At the completion of each SPLIT operation, nleaf (the number of boxes and the number of leaves in the k-d tree) will have increased by one. We note that this SPLIT operation has an obvious inverse, MERGE, which can be readily implemented with the data structures described here, and we omit the details of this operation for brevity. The texture analysis techniques described in Section 4 utilize SPLIT operations on the tree to focus finer partitioning on the colors which occur in image regions exhibiting smooth variations in color frequency, and MERGE to reduce the number of colors used in areas which are not susceptible to contouring effects. 3.4. Nearest-Neighbor Remapping The last phase of the median-cut algorithm involves mapping each prequantized color to the closest representative in the colormap, a process called nearest-neighbor remapping (also referred to as quantization; we will use nearest-neighbor remapping to avoid confusion with the use of quantization to refer to the entire process described in Section 3). Given some prequantized color value, we begin by searching the unbalanced optimized k-d tree until we reach the leaf containing that color value, using the partition axis and median value at every node in the search to decide which child should be visited next. After the leaf (and box) in which the prequantized color resides have been found, the next step is to locate the closest color representative, as it is possible that a closer representative lies in another box. Let the distance between the prequantized color and the color representative for the box be r, and let S be a sphere of radius r centered at the prequantized color. If S is contained completely inside the box, then no representative color outside the box is closer, and the search terminates. If S is not completely contained, then a closer representative may lie in another box, and the search backtracks to visit neighboring boxes. The relevant geometric tests, known as BALL WITHIN BOUNDS and BOUNDS OVERLAP BALL respectively, are presented in detail by Friedman et al. [7]. Efficient mapping of the original image into a quantized one is also discussed elsewhere [10, 11].

153

3.5. Selecting a Color Space In this section, we briefly consider the use of alternate color spaces as a basis for the median-cut algorithm. The basic idea hinges on the observation that distances in RGB space do not represent differences as perceived by the human visual system; hence, the use of alternate spaces which exhibit more perceptual uniformity might allow the median-cut technique to subdivide color space at regions of maximal perceptual difference. This basic idea is not new, and several color spaces have been explored as alternatives to RGB space. Gentile et al. [8] found that CIE L*u*v* color space also provided no significant improvement over RGB space for image dependent and independent quantization, with or without halftoning techniques. On the other hand, Van Dyck and Rajala [13] explored bit allocation schemes for color bands in YIQ, CIE LAB, and AC1C2 space, and found that CIE LAB produced good color fidelity and low perceptual error for high image compression ratios. The difficulty with such spaces for our purposes is quantization speed; nonlinear transformations of RGB space are computationally intensive. Linear models, while easier to manipulate, are already known to be inadequate as representations for uniform chromaticity and intensity. Nonetheless, spaces such as HLS and HSV are widely used because they present a more natural representation of color space and spread colors along axes with more intuitive meaning, such as hue, saturation, and intensity. To evaluate quantization speed in various color spaces, we applied the basic median-cut algorithm in four color spaces: RGB, HLS, HSV, and LAB. We note that singularities in HLS and HSV were handled by assigning these singular colors to specific cells in the histogram; for example, in HSV, when S 5 0, H is then set to zero as well. While this can introduce some bias into the subdivision process since colors cluster at singular cells, it requires no significant computation time. We briefly consider the speed of the quantizations. Figure 4 shows the quantization time for each of the 64 test scenes in each of the four color spaces. As in previous timing experiments, each test image was quantized 10 times for each value of b, and the data points plotted in the graph represent the average of each of these 10 times. All runs were performed using the same machine as described in Section 3.5. The horizontal axis has a logarithmic scale, and indicates the number of colors remaining after the prequantization step. It is clear that RGB quantization is fastest, since it requires no transformation of the pixel values. HLS and HSV quantizations require moderate amounts of arithmetic, and are slower; LAB quantization requires floating point arithmetic and 3 3 3 matrix multiplication for every distinct color, and the timings reflect this slowdown.

154

JEFFEREY A. SHUFELT

FIG. 4. Color space quantization timings for test scenes.

By examining each set of data for a given color space, three layers of points can be seen, as in Fig. 2; the layers here correspond to different image sizes, as they did in the prequantization experiments. The effect is most noticeable for the LAB data, which has three layers at 5, 10, and 18 s. The spacing between layers for the LAB data is quite large, giving another indication that LAB quantization arithmetic has a high time constant. These results show that if interactivity is a prime requirement for a given application, then RGB space is the best option, since it requires no transformation of the input data to another color space. Nearly all of the test images were quantized in less than 5 s in RGB space, including file I/O. Since existing nonlinear transformations are too computationally intensive for real-time usage, one approach might be to compute a transformation lookup table offline for the entire color space, and then use the lookup table for quantization in interactive applications. Such an approach would undoubtedly be memory-intensive, however. For these reasons, we chose to quantize directly in RGB space. 4. TEXTURE ANALYSIS

A traditional problem with quantization techniques is their inability to handle smooth variations in intensity and chromaticity. These smooth variations are represented in quantized images by discrete steps in color space, leading to visible contours in the output image. These contouring effects have traditionally been handled by halftoning or dithering techniques.

Dithering techniques exploit the eye’s ability to perform spatial integration over small areas at large viewing distances. These techniques divide the image into small windows and fill each window with a binary dither pattern, a predetermined pattern of pixel values that approximates the actual color in the window. An n 3 n window can represent n 2 1 1 shades by successively turning on more and more pixels in the window, allowing the eye to spatially integrate the pattern and see the average intensity in the window. Such techniques are used in printing as well, to achieve gray-scale with only white and black pixels, or to achieve full-color by dithering each color band separately [6]. Unfortunately, dithering techniques achieve their gains in intensity resolution at the expense of increased noise. While spatial resolution of high contrast and high frequency patterns is maintained quite well by typical dithering algorithms, the signal-to-noise ratio is lowered by the use of dither matrices to approximate image intensities. For many applications, including remote-sensing and multispectral classification tasks, the degradation of the image introduced by dithering is unacceptable. Our ability to alleviate such effects without recourse to dithering is dependent on the nature of the color distribution in a particular image. Consider a case where image colors are widely distributed throughout the color cube. Given a limited colormap size, any attempt to use more colors to relieve contouring effects must necessarily result in poor approximations of other colors in the image. In many situations, however, the color distribution is limited, due to the color coherence of objects; in other words, adjacent pixels of the same object usually have similar color values. In this case, we can adjust our quantization by using additional colors to relieve contouring effects, while simultaneously using fewer colors for objects with high color coherence (being careful not to introduce new contours or remove actual color boundaries in this process). To make these ideas concrete before describing our technique, consider the test image in Fig. 5. The left half of this image exhibits smooth variation in color from blue to red; the right half is made up of 2 3 2 blocks, in which the lower left element is a random color from the left half of the image, the lower right element is a color kR(0–127), 255, R(0–127)l, the upper left element is a color kR(128– 255), 255, R(128–255)l, and the upper right element is a color kR(128–255), 255, R(0–127)l, where R generates a random integer in its input range. This image was designed with the following properties, which our texture analysis method will exploit: • The image has many different colors (19,586) but nearly all of these occur in a limited region of the RGB color cube (the G 5 255 plane). In other words, the colors have a limited distribution in color space.

COLOR IMAGE QUANTIZATION ENHANCEMENT

155

FIG. 5. Original test image. FIG. 6. Median-cut quantization, no texture analysis. FIG. 7. Median-cut quantization, one-pass texture analysis. FIG. 8. Median-cut quantization, iterative texture analysis.

• Green colors from the same quadrant of the G 5 255 plane never occur next to each other in the image. In this case, we can use fewer slots in the colormap to represent greens from each quadrant, since this will neither eliminate actual color boundaries nor create contouring effects. In

other words, there exist colors which can be compressed safely. • Contouring colors (reds, violets, and blues) are more frequently adjacent to a similar color than to some other color. In particular, red is much more likely to occur next

156

JEFFEREY A. SHUFELT

to itself or to a similar shade of red than to any other particular color. In other words, red has high co-occurrence with itself and with similar colors. Figure 6 shows the result of applying the standard median-cut algorithm to Fig. 5, quantizing down to 200 colors. Many contours can be seen in this quantized image. Figures 7 and 8 show the results of applying the mediancut algorithm with the new texture analysis methods described in the next section (Fig. 7 shows the result of one iteration of texture analysis, and Fig. 8 shows the result of texture analysis after multiple iterations and convergence). The texture analysis scheme was able to exploit the limited color distribution and color compressibility to obtain free slots in the colormap; it then used these slots to allocate extra colors in the reds, violets, and blues, where co-occurrence measures were high. Specifically, the contours in the reds, violets, and blues in Fig. 6 are progressively eliminated as increasing levels of texture analysis are applied in Figs. 7 and 8. In the following sections, we describe the texture analysis schemes used for Figs. 7 and 8 in detail. 4.1. Detecting Contouring Colors Contouring effects are examples of textural phenomena resulting from the discretization of smooth variations in color space. Contours can be qualitatively described as the perceived boundaries, or edges, between homogeneous regions of similar color. Our approach to dealing with contours will exploit this idea by using simple ideas from texture analysis in computer vision to locate contours. We will then adjust our original representative set of colors so that more colors are used in the contoured areas of the image, while using fewer colors in areas of the image where color changes are less noticeable. We will achieve these operations quickly by using SPLIT and MERGE operations on the unbalanced optimized k-d tree described in Section 3.3. We begin by employing a common tool in texture analysis, a co-occurrence matrix. Given n discrete intensity values, a co-occurrence matrix A has dimensions n 3 n. A[i, j] then denotes the number of occurrences of intensity value i with intensity value j in a specific spatial relationship. Co-occurrence statistics have been widely used for texture analysis; some of the early research on co-occurrence matrices and their applications is surveyed in [14]. A co-occurrence matrix is defined by specifying the spatial relationship which the matrix is intended to measure. Since contour effects are defined by adjacency of similar colors, we compute a co-occurrence matrix which measures adjacency. More precisely, if our quantized image contains c colors, then we have a c 3 c matrix A in which A[i, j] counts the number of times that pixels with colors i and j occur in horizontal or vertical adjacency in the image. This

particular co-occurrence matrix can be efficiently implemented by scanning the quantized image and collecting adjacencies in A. Co-occurrence matrices are traditionally used in computer vision as an intermediate source of information for computing textural measures such as image entropy, correlation, and local homogeneity, which can then be used for texture classification and segmentation. However, our goal is not texture classification, but the identification of contouring colors. This can be directly computed from the cooccurrence matrix by identifying two classes of colors in the colormap: • Contouring colors. These colors are suspected of causing contouring effects. Each of these will be SPLIT into two distinct colors, allowing an improved quantization of the contoured image regions. • Compressible colors. These are colors which are children of the same parent in the k-d tree, and which, if compressed into one color, would not produce visible artifacts in the quantized image. We can qualitatively define contouring colors in terms of co-occurrence statistics. First, a color responsible for contouring must appear in a homogeneous region for the contour to be visible; in other words, it must possess high self co-occurrence. Second, two such colors must frequently be adjacent, to form the contour; in other words, these two colors must also have high co-occurrence. Third, these two colors must be in close proximity in the color space; otherwise, they likely represent a real feature in the image rather than an artifact of quantization. We can also qualitatively define compressible colors in terms of co-occurrence statistics. Here, we seek pairs of colors which can be compressed into a single color without noticeable adverse effects in the quantization. Such colors should not form homogeneous regions in which any color approximations would be easily observed; in other words, these colors should have low self co-occurrence. These color pairs should never be adjacent, since their compression into one color might remove real image features; in other words, the colors in the pair should have no cooccurrence. We quantify these criteria in the following manner. Let n be the number of colors in the colormap, and let A be the n 3 n co-occurrence matrix for the image, indexed from 0 to n 2 1. We define the co-occurrence frequency of i with respect to j as A[i, j] Cj (i) 5 n21 if i occurs; 0 otherwise. A[i, k]

O

k50

(1)

COLOR IMAGE QUANTIZATION ENHANCEMENT

Informally, the co-occurrence frequency is the fraction of the time that the neighbor of color i is color j. The self cooccurrence frequency is then the percentage of time that a neighbor of color i is color i, and it can be defined trivially in terms of C. We define the self co-occurrence frequency of i as S(i) 5 Ci (i).

(2)

Let eS and s S be the mean and standard deviation of S(i), i 5 0, . . . , n 2 1, respectively. Let eC( j) and s C( j) be the mean and standard deviation of Cj (i), i 5 0, . . . , n 2 1, respectively. Finally, let dRGB be the Euclidean distance metric in RGB space. Then, two colors i and j are contouring colors if and only if all of the following five criteria are satisfied: 1. S(i) . e S 1 s S 2. S( j) . e S 1 s S 3. Cj (i) . e C( j) 1 s C( j) 4. Ci ( j) . e C(i) 1 s C(i) 5. dRGB (i, j) # t. The first two heuristics test to see that colors i and j have high self co-occurrence, and hence are likely to appear in homogeneous regions in the image. The second pair of criteria tests to see that the colors are frequently adjacent in the image, relative to the average co-occurrence frequency for each color. Finally, the last test checks to see that the colors are near each other in the color space. In our experimentation, we found that lower values of t allowed more contouring colors to remain undetected, while higher values of t began to detect contouring effects where real boundaries existed in the image. All experiments reported in this work use t 5 12, in a 256 3 256 3 256 color cube, which gave reasonable performance across a variety of imagery. This value also coincides with an empirically determined threshold for determining areas of low spatial activity [1]. Two colors i and j are compressible colors if and only if all of the following four criteria are satisfied:

relatively low self co-occurrence, and hence are unlikely to appear in homogeneous regions in the image. The second test is quite strict, and most color pairs do not pass this test. We could have defined the test as a threshold, thus permitting more color pairs to be admitted as compressible colors, but in doing so, we would risk the creation of quantization artifacts where the two colors are adjacent in the image and represent real color changes in the scene. We chose such a strict test to be conservative in admitting compressible colors to avoid creating such artifacts. In practice, we found that many of our synthetic images had no color pairs which passed this test, since similar colors often occur in adjacency on the same object in the synthetic image. One simple way to implement the contouring and compressible color tests would be to apply the heuristics for each pair of colors in the image. A much more efficient implementation, however, first applies the self co-occurrence tests, and then performs the remaining tests on the set of colors which passed the self co-occurrence tests. This obviates the need to check every possible pair of colors. The final step in the detection process is to establish a partial ordering on contouring colors, so that we can address the colors most responsible for contouring artifacts first. We accomplish this by first sorting the pairs (i, j) of contouring colors in decreasing order by the sum of their self co-occurrence frequencies, S(i) 1 S( j), and then sorting each pair in decreasing order by self co-occurrence frequency, producing a list of colors. Since any color can appear in many pairs of contouring colors, it can appear

1. i and j are leaves of the same parent in the k-d tree 2. A[i, j] 5 0 3. S(i) , e S 4. S( j) , e S . The first test ensures that the two colors are near each other in the color space, and more importantly, that we do not attempt to compress a subtree with many colors, restricting our attention to atomic colors. The second test ensures that the colors are never adjacent in the image, and the final pair of tests check to see that each color has

157

FIG. 9. Mean error for test scenes, RGB evaluation.

158

JEFFEREY A. SHUFELT

FIG. 10. SPOT84XS1 test scene. FIG. 11. Median-cut quantization, no texture analysis. FIG. 12. Median-cut quantization, one-pass texture analysis. FIG. 13. Median-cut quantization, iterative texture analysis.

many times in the sorted list; we discard all occurrences but the first, maintaining the ordering. Such an ordering allows us to handle colors with the largest regions first, as these are likely to produce the most visible contouring effects.

4.2. Adjusting the Colormap We are now in a position to adjust the colormap, using our lists of contouring colors and compressible color pairs. The basic strategy is simple; we traverse the lists, pairing

COLOR IMAGE QUANTIZATION ENHANCEMENT

159

FIG. 14. MAN test scene. FIG. 15. Median-cut quantization, no texture analysis. FIG. 16. Median-cut quantization, one-pass texture analysis. FIG. 17. Median-cut quantization, iterative texture analysis.

each contouring color k with a compressible color pair (i, j). We stop the traversal when we have run out of contouring colors or compressible color pairs. For each pair (k, (i, j)), we will split k into two colors, compressing i and j into a single color to hold the number of colors in the colormap at a constant. All that is needed is a SPLIT operation to divide k’s node, producing two new children in the k-d tree, and a MERGE operation to remove i and j ’s leaves and make their parent a leaf. The representative colors for these three new leaves are computed as before, by averaging the color values of the entries in the boxes associated with each leaf; we chose averaging instead of other alternatives such as median or weighted averaging because it is fast,

and provides reasonable color representatives. When the SPLIT and MERGE operations for a list traversal are complete, the new colormap consists of the representative colors for the leaf nodes, as before. The nearest-neighbor remapping process is performed once again for all prequantized colors, to find their closest representatives in the new colormap. Upon completion of this process, all that remains is to remap the original image, using this new colormap. The texture analysis process described here lends itself to an iterative approach; after one pass has been completed, a new co-occurrence matrix can be generated based on the new mapping of the original image, new texture statistics can be computed, contouring and compressible

160

JEFFEREY A. SHUFELT

colors can be detected, and the tree can be operated on again to generate a new colormap. Iteration is terminated when a pass of the texture analysis fails to find either contouring colors or compressible color pairs, or when a fixed number of iterations has been reached. Iterative analysis trades computation time for increased ability to refine the colormap and improve quantization performance on smoothly varying regions in the image. Without a limit on the number of iterations, it is conceivable that the method could SPLIT and MERGE the same pair of color nodes indefinitely; in practice, however, all 64 test images converged in a maximum of 8 iterations. This is primarily due to the fact that e S does not change significantly after each iteration (since the color to be SPLIT does not usually show high-frequency patterns after SPLITting, as a consequence of color coherence of objects), and the self co-occurrence frequency of a newly MERGEd node is bounded by the self co-occurrence frequencies of the nodes which created it, implying that the number of available compressible colors will decrease with each iteration. 4.3. Texture Analysis Evaluation In this section, we evaluate the performance of both the one-pass and iterative texture analysis schemes. Figure 9 plots the mean error for each of the test scenes using no texture analysis, one-pass texture analysis, and iterative texture analysis, using Euclidean distance in RGB space as the pixel-by-pixel comparison metric. The horizontal axis has a logarithmic scale, indicating the number of colors remaining after the prequantization step. Using this metric, we find that all cases fall into one of two categories. In the first case, texture analysis is unable to locate any contouring colors or compressible color pairs, and hence no change is made to the colormap. In the second case, texture analysis for one or more passes changes the colormap, improving the mean error slightly or producing no significant change in mean error. It should be noted that mean error is a coarse measurement of quantization quality. Nonetheless, these results are consistent with our intuition about the texture analysis method. From a pixel-by-pixel mean error standpoint, our main concern is that mean error does not radically increase under texture analysis, which is the case according to the results depicted in Fig. 9. Many of the benefits of texture analysis are expected to manifest themselves in areas of smooth variation. Metrics such as mean error will not capture the anticipated improvements in quantization quality; these improvements will be textural in nature, and hence represent spatial phenomena which by definition cannot be captured well by pixel-to-pixel comparisons. In Sections 4.4 and 4.5, we examine two test images and their quantizations to directly observe the effects of texture analysis.

4.4. Remotely Sensed Imagery: An Example The first example image is SPOT84XS1, a test scene taken from simulated SPOT imagery of Washington, D.C., using bands 3, 2, and 1 as the red, green, and blue planes. Figure 10 depicts the test area. Note the white strip at the top of the test area; this is characteristic of actual SPOT data and results from the simulation of the SPOT satellite’s orbital path. This image contains 43,018 distinct colors. Figures 11 through 13 show the quantizations produced with varying degrees of texture analysis. The quantization produced by median-cut with no texture analysis is shown in Fig. 11. This quantization exhibited a mean error of 8.12 units of RGB distance (in an RGB cube 256 units on a side). The quantization errors are uniformly distributed on land, but contouring effects are clearly visible throughout the river and in the reservoir at the upper left of the image. Quantization with one-pass texture analysis is shown in Fig. 12. This quantization exhibited a mean error of 8.10 units of RGB distance, almost equal to that in the previous quantization. With one pass of texture analysis, some improvements are noticeable; the reservoir has lower error than its surroundings, and portions of the river have lower error. Some portions of the river, particularly the areas near the bridges at the center and lower right of the image, are modeled less well, but these regions of error now correspond to actual homogeneous regions in the image. In other words, these contours actually exist in the image; the failure here is in choosing a good representative for one side of the contour, which is arguably an improvement over the generation of false contours. Quantization with iterative texture analysis is shown in Fig. 13. This quantization exhibited a mean error of 7.69 units of RGB distance, better than either of the previous quantizations. The river and reservoir have now been modeled very well; the errors in these regions are quite low. Further, no significant artifacts have been introduced in other areas of the image; a few scattered pixels show high error, but these pixels appear in areas of rapid variation in chromaticity and intensity, so they are difficult to detect with the naked eye. In fact, it is extremely difficult to distinguish this quantization from the original image in Fig. 10. This example illustrates the effectiveness of iterative texture analysis on remotely sensed imagery, in which colors typically cluster in areas of the color space corresponding to various natural or manmade materials in the scene. In essence, the texture analysis scheme described here reallocates colormap entries among these clusters, using fewer cells in clusters that do not generate contouring artifacts, and using more cells in clusters that do. 4.5. Computer Graphics Imagery: An Example The SPOT84XS1 test scene is representative of remotely sensed imagery containing a variety of natural and man-

COLOR IMAGE QUANTIZATION ENHANCEMENT

made features. Synthetic images, however, provide a different test of the texture analysis techniques. The distributions of colors in remotely sensed imagery typically cluster in certain areas of the color space; multispectral classification algorithms make use of this property to label image regions with material types. Synthetic images often make much wider use of the color palette, making the task of a quantization algorithm more difficult. One example of such synthetic data is MAN, a simple raytraced image shown in Fig. 14. While MAN contains only 2791 colors, a fraction of the 43,018 used by SPOT84XS1, its colors are much more widely distributed throughout the color space, varying smoothly from saturated yellows, blues, and reds to white highlights. Figures 15 through 17 show the differences between the original image and the three quantizations using varying degrees of texture analysis, evaluated by Euclidean distance in RGB space. Figure 15 shows the quantization with no texture analysis. This quantization exhibited a mean error of 8.65 units of RGB distance. Contouring is visible throughout the entire image; quantization of 230 colors is simply insufficient to handle the smooth variations in color that occur on the yellow surface, the reflections, and the pieces of the stick figure. Figure 16 shows the quantization with one-pass texture analysis. This quantization exhibited a mean error of 7.55 units of RGB distance. One pass of texture analysis has reduced the error in the arms and face of the stick figure and the shadow to the right, but this reduction was coupled with an increase in the error in the leg reflections, and more noticeable error in the left shadow and background surface. Figure 17 shows the quantization with iterative texture analysis. This quantization exhibited a mean error of 6.49 units. Repeated passes of texture analysis attempted to refine the background surface and left shadow, but this process resulted in contouring effects with alternating stripes of low and high error. In short, texture analysis is unable to cope with all of the smooth color changes in the scene, and trades increased resolution in one part of color space for reduced resolution in another. This example illustrates the limitations faced by any adaptive quantization technique. Since the image is comprised entirely of smoothly varying regions, the use of additional colors to refine contouring effects in one region is guaranteed to introduce contouring effects in another region. We found that synthetic images often presented a more difficult quantization challenge because of this property. In situations such as this, it seems likely that the only recourse for improving perceived quality is dithering. Nonetheless, even simple forms of texture analysis such as the method described here can produce noticeable quality improvements in image quantization, and appear to be quite useful for quantizing remotely sensed image data. These techniques also serve as an alternative for applica-

161

FIG. 18. Texture-based quantization timings for test scenes.

tions in which spatial resolution must be maintained, ruling out traditional halftoning techniques. 4.6. Performance Timings In this section, we briefly discuss the speed of the various texture analysis methods. Figure 18 shows the quantization times for each of the 64 test scenes. As in the color space selection timings, each test image was quantized 10 times using each of the three levels of texture analysis: none, one-pass analysis, and iterative analysis. Each data point in the figure is the average time of the 10 runs. All runs were performed on the same machine as described in Section 3.1. The horizontal axis has a logarithmic scale, indicating the number of colors remaining after the prequantization step. As expected, the standard quantization with no texture analysis is fastest. One-pass texture quantization incurs a slowdown proportional to the size of the quantized image, which must be scanned on every pass of the texture analysis algorithm. Iterative texture quantization is the slowest method of the three, since it often requires several iterations to reach a point where no more compressible colors are available. In a few cases, iterative analysis runs at the same speed as one-pass analysis; this happens when the first iteration of texture analysis finds no compressible colors. If interactivity and quality are both key requirements, one-pass texture analysis is a reasonable compromise; most of the test images were quantized and subjected to onepass texture analysis in less than 10 s, including I/O. The

162

JEFFEREY A. SHUFELT

time required by iterative analysis limits its applicability for interactive applications, but the technique is well suited to offline high quality quantization. 4.7. A Summary of Texture Analysis Quantitative mean error measurements show that texture analysis can provide improvements in quantization quality. Texture analysis does not reduce mean error substantially in most cases, but this is due to the local nature of the improvements in areas where contouring effects appear. In qualitative assessments of the quantizations produced by one-pass and iterative texture analysis, we found that the quantizations noticeably improved quantization quality, or changed the quality little. For most remotely sensed images, we found that contouring effects were consistently alleviated, and in some cases, almost eliminated completely. We did not observe any undesirable effects from the color compression process, since compressed colors were indistinguishable before compression took place. In synthetic imagery, we generally saw no improvement, but the quantization quality was not degraded. In these cases, it would be useful to have a fast method for computing the likely effectiveness of texture analysis before the actual texture analysis is performed, to save computation time. 5. CONCLUSIONS

In this work, we explored a texture-based color image quantization enhancement technique, subject to the following constraints: • The algorithm is suitable for displaying imagery without negatively impacting the signal-to-noise ratio (eliminating conventional halftoning methods). • The algorithm operates at interactive speeds. • The algorithm produces images of the highest possible quality, in terms of precise color representation and elimination of contouring artifacts. The texture analysis methods described here provide a means for improving the quality of quantized images, without recourse to halftoning techniques. They are particularly applicable to images where colors cluster in color space, and are effective in applications with moderate amounts of interactivity. Texture analysis provides increased quality by alleviating contouring effects, for images in which colors cluster. Iterative analysis is slowest, but provides best image quality; one-pass analysis is a reasonable compromise between speed and quality improvement. The scheme presented in this work uses only a rudimentary analysis of image texture; there has been a great deal of work on texture in the computer vision community, and more rigorous models of texture have been developed.

A careful analysis of contouring effects and quantization errors, in conjunction with a full texture model, could provide improved quality. Such a texture model might also permit selective halftoning, where areas of smooth variation could be halftoned, while sharp boundaries and transitions in the color space could be preserved. Many models of texture are computationally intensive, however, which limits their usefulness for interactive applications. We note that many of the criteria used for the determination of contouring and compressible colors described in Section 4.1 are expressed as a sum of a mean and standard deviation (e 1 s). It may be possible to improve the detection of contouring and compressible colors by formulating these expressions in the form e 1 as, where a is a constant. Although we have not experimented with this approach, we have noted that almost all texture analyses find many more contouring colors than compressible colors, in large part because of the strictness of the compressible color rules. The obvious solution for these rules, a . 0, could allow more contour removal to take place and make use of more contouring color pairs; however, this approach risks the removal of real color boundaries. Adjusting a in each of the rules for optimal performance remains a topic of investigation. Nonetheless, elementary tools from texture analysis in computer vision are suitable for detecting contouring effects in a quantized image, and allow the application of simple heuristics to refine an initial quantization and alleviate these contouring effects. We showed that these methods can provide improved reproductions of gradual intensity and chromaticity changes without generating noticeable degradation in areas of high spatial frequency, and without the need for dithering techniques. We provided qualitative and quantitative results for this approach on a variety of remotely-sensed and synthetic imagery. ACKNOWLEDGMENTS As usual, the members of the Digital Mapping Laboratory provided a congenial and interesting research environment. I thank several people for their contributions to this research effort. Stephen Ford provided a gentle introduction to the science of color measurement. Jeff McMahill generated the ray-traced images used in this report. Steve Gifford provided insights on display application and prequantization concerns, quickly integrated the quantization enhancements into an existing frame buffer emulator for testing purposes, and produced the radiosity images used for evaluation. Paul Heckbert also provided many useful comments on an earlier draft of this report, as well as interesting insights into the quantization problem. Dave McKeown introduced me to the color image quantization problem, commented on earlier drafts of this report, and provided constant encouragement and support. Finally, the anonymous reviewers of an earlier draft of this work contributed many valuable comments and insights.

REFERENCES 1. R. Balasubramanian and J. Allebach, A new approach to palette selection for color images, J. Imaging Technol. 17(6), 1991, 284–290.

COLOR IMAGE QUANTIZATION ENHANCEMENT 2. R. Balasubramanian, C. A. Bouman, and J. P. Allebach, Sequential scalar quantization of color images, J. Electron. Imaging 3(1), 1994, 45–59. 3. J. L. Bentley, Multidimensional binary search trees used for associative searching, Comm. ACM 18(9), 1975, 509–517. 4. J. L. Bentley and J. H. Friedman, Data structures for range searching, ACM Comput. Surv. 11(4), 1979, 397–409. 5. S. Diaz-Cayeros, Quality color composite image display using low cost equipment, Photogramm. Eng. Remote Sens. 56(12), 1990, 1625–1629. 6. J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes, Computer Graphics: Principles and Practice, 2nd ed., Addison–Wesley, Reading, MA, 1990. 7. J. H. Friedman, J. L. Bentley, and R. A. Finkel, An algorithm for finding best matches in logarithmic expected time, ACM Trans. Math. Software 3(3), 1977, 209–226. 8. R. S. Gentile, E. Walowit, and J. P. Allebach, Quantization and multilevel halftoning of color images for near-original image quality, J. Opt. Soc. Am. A 7(6), 1990, 1019–1026.

163

9. P. Heckbert, Color image quantization for frame buffer display, Comput. Graphics 16(3), 1982, 297–304. 10. A. Rosen, Colormap: A Color Image Quantizer, Technical Report 87-845, Department of Computer Science, Cornell University, 1987. 11. J. A. Shufelt, Color Image Quantization Enhancement Techniques, Technical Report CMU-CS-95-190, School of Computer Science, Carnegie Mellon University, 1995. 12. K. E. Spaulding, L. A. Ray, and J. R. Sullivan, Secondary quantization of color images for minimum visual distortion, in Proceedings: SPIE Conference on Human Vision, Visual Processing, and Digital Display IV, Vol. 1913, 1993, pp. 261–269. 13. R. E. Van Dyck and S. A. Rajala, Subband/VQ coding of color images with perceptually optimal bit allocation, IEEE Trans. Circuits Systems Video Technol. 4(1), 1994, 68–82, 101. 14. L. Van Gool, P. Dewaele, and A. Oosterlinck, Texture analysis anno 1983, Comput. Vision Graphics Image Process. 29, 1985, 336–357. 15. S. J. Wan, P. Prusinkiewicz, and S. K. M. Wong, Variance-based color image quantization for frame buffer display, Color Res. Appl. 15(1), 1990, 52–58.