- Email: [email protected]
Using Browsing to Improve Content-Based Image Retrieval
Journal of Visual Communication and Image Representation 12, 123–135 (2001) doi:10.1006/jvci.2000.0453, available online at http://www.idealibrary.com on
Using Browsing to Improve Content-Based Image Retrieval Jesse S. Jin and Ruth Kurniawati School of Computer Science and Engineering, University of New South Wales, Sydney 2052, Australia
and Guangyu Xu and Xuesheng Bai Department of Computer Science, Tsinghua University, Beijing 100084, P. R. China Received February 8, 1999; accepted April 7, 2000
Content-based image retrieval heavily relies on the indexing structure. Current index structures, such as the R-tree, the SS-tree and the X-tree, have a large overlapping area among their nodes, especially at the high (parent) levels of the indexing tree. The overlapping area causes the search engine to examine a large number of nodes and hence it is very inefficient to retrieve at very high levels of the index tree. We develop the SS+ -tree to alleviate the overlapping problem and present a scheme to combine browsing with retrieval in searching for images. Browsing provides a visual tool for a user to quickly narrow the search to a small region and to avoid a large number of high dimensional comparisons. Combined with retrieval, it produces a very efficient content-based retrieval method. °C 2001 Academic Press Key Words: content-based image retrieval; indexing; browsing; multimedia database.
1. INTRODUCTION The advance of image technology has resulted in widespread applications of images to gain more effective representation of visual information and user-friendly human– machine interfaces. It is therefore reasonable to expect that most information systems in the future will require efficient methods for archiving and managing large volume of images and thus advanced imagery databases will be desirable. Images, in a general sense, include natural images, such as photographs, medical images, and satellite images; artificial images, such as computer graphics, paintings, and drawings; and scientific pictures, such as statistical charts and visualization patches. Retrieving images in multimedia databases is an important issue. Current image database systems can be classified, according to the type and nature of the features used for indexing and retrieval, into three categories: attribute-based systems, text-based systems, and content-based systems. The 123 1047-3203/01 $35.00 C 2001 by Academic Press Copyright ° All rights of reproduction in any form reserved.
124
JIN ET AL.
different semantic representations of the features would generally lead to different indexing methods. Attribute-based systems are the legacy of the conventional DBMS in using a set of structured attributes to access objects so that the object handling capability is extended to cope with voluminous and gigantic image objects. Since little special development is required in organizing and retrieving image objects in these systems, many image or pictorial databases in earlier years were attribute-based [1]. Indexing methods used in these systems were usually B+ -trees [2] or inverted files [3]. Chang and Knuii [4] surveyed a variety of attribute-based pictorial database systems and found that the main problem in attributebased systems was that a fixed set of attributes could not take advantage of the rich content of images. Moreover, these attributes limited the scope of the application of databases and provided little room for future and unforeseen usage of the images. Text-based methods annotate images with a text description and structured fields. Queries in these systems use keywords and return partially-matched results ranked by similarity. Indexing methods for text-based image systems include full text scanning access methods, inverted files, and signature file methods [5]. There are a few commercial text-based image databases such as the Kodak Picture Exchange System [6], the PressLink Library [7], and the Time Picture Archive Collection [8]. The advantage of such methods is that they are able to describe abstract concepts. However, the disadvantages are prohibitive. Manually written descriptions fail to reflect the rich content of images completely. Some features in images, such as irregular shapes and jumbled textures, are extremely difficult, if not impossible, to describe in text. Textual descriptions limit the scope of searching to the keywords predetermined by the author of the system or the current application, leaving few avenues for using the data beyond that scope. Description is subjective, and different people may give quite different descriptions for the same image. There is also the common problem of understanding natural language description. Furthermore, descriptive text is entered manually, and therefore, the indexing process is slow. To overcome the above-mentioned problems content-based methods have therefore been developed. The most commonly used content features include shape, color, and texture [9]. These features are extracted automatically in the indexing process when images are entered into a multimedia database. There are a few ongoing developments in content-based image retrieval (CBIR) such as IBM QBIC [10,11], Virage [12], MIT Photobook [13], Chabot [14], and JACOB systems [15]. The indexing structure is a big issue for CBIR systems. Visual features are usually of high dimensions. Conventional access structures such as R-tree [16] are not suitable for retrieving high-dimensional features. A similarity search in a high-dimensional space is an NP problem and, in general, as Minsky and Papert [17] conjectured, will examine a large portion of the data. A number of indexing data structures have been proposed to handle high-dimensional data: the Similarity Search tree (SS-tree) [18], the Telescopic Vector tree (TV-tree) [19, 20], and the X-tree [21]. These new structures all perform better than the R-tree for various reasons: the TV-tree reduces the dimension of the vectors by collapsing the first few dimensions with the same values, the SS-tree utilizes the clustering property of the data, and the X-tree tries to minimize the overlapping between nodes’ bounding boxes (hyperboxes). Current indexing structures have a problem of having a large overlapping area among nodes, causing the retrieval process to inspect a large number of image items. The computational complexity of comparison is also exponential to the dimension.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
125
We have developed an SS+ -tree for indexing high dimensional features and proposed a browsing method in conjunction with retrieval. The techniques have been practiced in developing CBIR-VU, a general purpose image retrieval system [22]. In Section 2 we present our indexing structure and discuss the overlap problem, and in Section 3 we describe our scheme of retrieval using browsing. Finally, there is a conclusion. 2. INDEXING HIGH-DIMENSIONAL FEATURES USING SS+ -TREE The issues of improving an R-tree-based multidimensional access structure include selecting the shape of bounding envelopes, splitting heuristics, the heuristics to reduce overlapping, the updating criteria, and efficient search strategy. These issues are not independent of each other. The splitting heuristic and subtree selection criteria will determine the shape of each node and hence the goodness of the bounding envelope. The goodness of the bounding envelope will also determine the extent of overlapping between nodes. The heuristics used to reduce overlapping will also affect the shape of the nodes. We present our SS+ -tree by addressing four issues: bounding envelope, splitting heuristics, updating criteria, and searching strategy. Overlapping often happens during the updating process, and we address it as an updating criterion. 2.1.
Bounding Envelope
The R-tree family uses bounding rectangles. The main problem with this kind of bounding envelope is that there are large overlapping areas. If a query falls into an overlapping area, such as query a in Fig. 1a, the search has to go through all rectangles containing a (branches c1 and c3 as shown in Fig. 1b). Another problem is that the bounding rectangle is not consistent with the similarity measurement, the distance. For example, query a falls into clusters c1 and c3 in Fig. 1a. Due to the inconsistency, cluster c23 will fail to be the first candidate in the search. For the nearest neighbor queries in Euclidean space, Cleary [23] proved that decomposition into spherical regions is optimal in the sense that it minimizes the expected number of
FIG. 1. Overlapping and inconsistency problem of the bounding rectangle.
126
JIN ET AL.
regions touched. It also keeps the consistency with the distance measurement. To calculate the bounding sphere, the SS-tree [18] uses the centroid and the distance of the farthest element from the centroid as the center and radius of the bounding sphere, respectively, which is not the optimal one with respect to volume and surface area. Because the sphere volume grows exponentially with the dimension, in a high-dimensional space, a small increase in the radius of a sphere will give a large increase in the surface area and the volume of the sphere. In accordance with the analytical results of the R-tree [24,25], we want to minimize the total surface area and the volume of bounding envelopes. The sphere-tree [26] uses the smallest enclosing sphere computed by a variant of the method described by Elzinga and Hearn [27], which at best is dependent exponentially on the dimension. Besides, the sphere-tree has only been tested with two-dimensional spatial data. Computing the smallest enclosing sphere is a classic problem in computational geometry. The time complexity is exponential to the number of dimensions (for approaches using linear programming [27, 28]) or even superexponential [29]. It is not feasible in a high-dimensional space. However, due to the special characteristics of the vector distribution in the feature space, we propose a heuristic method which can calculate the smallest enclosing sphere in O(n). Our heuristic method searches the center of the smallest enclosing sphere using the √ golden ratio (( 5 − 1)/2 = 0.6180339887 . . .), which gives the least number of searches in an unevenly distributed candidate space [30]. In each approximation, it reduces the search area and eliminates a large number of points which cannot be on the surface of the sphere. The algorithm can be described in the following: 1. initialize: c0 to the center of the box; r0 to the maximum dimensional radius of the box; rout to the minimum dimensional radius of the box; 2. find maximum radius r1 of all points to c0 and eliminate all points inside rout ; 3. update: rout = min(r0 , | r1 −2 r0 |), where r0 and r1 are vectors from c0 to r0 and r1 , respectively; r0 = r1 ; c0 = 0.618 r01 , where r01 is the unit vector of r1 ; 4. if c0 has changed, repeat steps 2–4; otherwise, c0 and r1 define the smallest enclosing sphere. A 2D illustration of the algorithm can be found in Fig. 2. The algorithm starts with a guess c0 and r0 , as the sphere center and radius, where c0 is the midpoint of the bounding box
FIG. 2. Calculating the smallest enclosing sphere using elimination.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
127
of all points in a node and r0 is the radius of the maximum dimension. The first iteration finds the maximum radius r1 , as shown in Fig. 2a, and eliminates all points inside the circle (c0 , rout ) because they cannot be on the surface of the smallest enclosing sphere. We obtain an area A1 which is defined by the plane perpendicular to the vector r1 . It is obvious that the center of the smallest enclosing sphere will not be in area A1. We update c0 to a new guess which is on r1 with a length equal to 0.618 of |r1 − r0 |. We then find a new maximum radius r1 and eliminate all points inside the circle (c0 , rout ), as shown in Fig. 2b. This process can be iterated but we have found that after the second iteration, the remaining points are in a small number. They can be fed into Welzl’s smallest enclosing sphere algorithm [31] to obtain the smallest enclosing sphere. 2.2.
Node Split Heuristic
Almost every tree-based multidimensional access structure, e.g., the R-tree and its variants [16, 31, 32], the SS-tree [18], and the X-tree [21], uses a splitting plane that is perpendicular to one of the coordinate axes. Except for the SS-tree, the split criteria used are mainly topological ones, e.g., the minimization of the perimeter, the area, the increase of the node’s area, or the overlapping area. Because the tree structure aims to support similarity searches, we would like nearby vectors to be collected in the same or nearby nodes. This means that a division of data should reflect the clustering property and there should be a lower variance within the nodes. Dividing data into groups while optimizing a statistic feature has been studied extensively in the clustering area. For the SS+ -tree, we chose an unsupervised C-prototype clustering algorithm [33]. The computational complexity for this clustering algorithm is O(cni), where n is the number of data, c is the number of clusters, and i is the maximum number of iterations. In the splitting process, the number of clusters c will be two, the number of data points n is bounded by the node’s maximum capacity plus one, and we can always set an upper bound for the number of iterations i. In this way, we have a greater degree of freedom and the algorithm will seek to minimize the variance in the resulting partitions. Also note that in a Euclidean space, this splitting heuristic will tend to produce spherical-shaped nodes. 2.3.
Updating Index Tree and Reducing Overlapping Area
Updating is an important issue in index trees. To be consistent with our splitting heuristic we insert new data into the cluster with the closest centroid. This strategy will minimize the variance within the clusters to avoid frequent updating. However, updating will happen eventually when one node with a bigger variance eats a smaller node (see Fig. 3a for an illustration—here we use bounding spheres for the nodes, but a similar situation will also occur in bounding boxes). This kind of situation is not desirable since it will be difficult for us to determine which node will be searched during the query process and, also, the total volume or area of the nodes would be significantly larger. To alleviate the problem, the nodes should be adjusted. We have tested an R+ -like downward split propagation to completely remove the overlapping between nodes using the power planes [34]. We finally discarded the strategy since the scope of reorganization resulting from the downward split propagation could involve reorganizing the whole subtree and the fragmentation will be very large since the number of possible power planes of n spheres is n [d/2] , where d is the dimension of the space. We take a simple and effective local updating strategy. We check if a node boundary expands and heavily overlaps its siblings (it will take O(n) time, where n is the number
128
JIN ET AL.
FIG. 3. The node configuration (a) before and (b) after applying the local reorganization rule.
of children within the parent node). If we have this situation, we invalidate the children involved (say k) and perform a generalized k-means clustering process on the grandchildren. Initializing the cluster centers will not be a problem because from our SES calculation we have the points on the sphere. These points serve as the best initial centers for k-means clustering. The result of this rule can be seen in Fig. 3b, where the initial centers are two points on the big circle in Fig. 3a. The reorganization itself, if necessary, will take O(nk) time. Because it involves all grandchildren, measures should be taken so that it occurs infrequently. One suggestion is to put a guard that will decay with the insertions of new data. There are cases where overlapping is unavoidable, in which case it is better not to split the node at all and keep one big node in place of several heavily overlapped ones (similar to the supernodes in the X-trees [21]). 2.4.
Greedy Search Strategy
Similarity searches are generalized k-nearest-neighbor searches. It searches the index tree to find the corresponding nearest-neighbor disk. We propose a branch-and-bound algorithm to reduce the number of nodes being inspected. In a branch-and-bound algorithm, an important issue is how to find the best candidate for the search and what the upper bound is. The selection of the best candidate in our greedy strategy is to go to the cluster whose centroid is the closest to the query point, e.g., C4 in Fig. 4a. We have actually tested several branching criteria, including the minimum distance (the optimistic measure), minimum of the maximum distance (the pessimistic measure), and several other heuristics. The
FIG. 4. A greedy strategy to choose the best candidate in a lower bound (a) and an upper bound (b).
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
129
experiment results suggest that the nearest centroid criterion is the best in terms of minimizing the number of leaf nodes that have to be examined. To set an upper bound, we can safely ignore all nodes whose smallest enclosing sphere does not intersect the current nearest-neighbor disk. We use the furthest distance between the smallest enclosing sphere of a cluster and the query point to check if the cluster is intersected with the nearest-neighbor disk, e.g., C3 in Fig. 4b. Note that the nearest-neighbor disk is decreasing. It starts with a large radius to enclose at least k nearest neighbors. During the search, it gradually reduces. At the end, it is bounded by only k nearest neighbors. 3. RETRIEVAL USING BROWSING Although the SS+ -tree performs better than the other index structures, the overlap and slow comparison problems are not completely resolved. We propose a browsing scheme using the index structure. The idea is very similar to the way we search books in the library. Since the shelves are organized by subjects, we may browse on a particular shelf to find similar books. We have developed an image retrieval system using color, texture, and shape information, as shown in Fig. 5. There are six query methods. A user can submit a query with partial information such as color, shape, or texture by selecting a feature pattern and clicking on 1h, or through user-definition button 4h. A query can also be submitted through a selected image ( 2h) by clicking on an object of interest. The system will then analyze the image and extract features automatically for retrieval. Our system also has a browsing function ( 5h) which will be described in the following section.
FIG. 5. Image retrieval interface. Legend: 1h, query by features; 2h, query by examples; 3h, query by keywords; 4h, query by user definitions; 5h, query with browsing; 6h, query with feedback.
130
JIN ET AL.
FIG. 6. An index tree of texture features (for 259 images).
3.1.
Browsing Using the Index Tree
Careful study of indexing structures of the high-dimensional feature space and the retrieval problems revealed the following. •
An index tree provides information for quick browsing. Figure 6 shows part of the texture index tree in our experiment. The key image at each node has the smallest distance to the center of the cluster; • Cluster centers at high (parent) levels of the index tree have a distinct distance. Table I provides the distance of children in the index tree shown in Fig. 6. The distance between nodes 2.1 and 2.2 is 54.18% of the size of the index space, while the distance between nodes 5.1 and 5.2 is 8.26% and the distance between nodes 6.7 and 6.8 is 2.43%. In other words, the key images at high levels have a high discriminating capability. They provide an effective way to browse images. • Overlapping is low at low (children) levels of the index tree. Due to the isotropic nature of spheres, extending a sphere in one direction will extend it in every direction. At the high level, spheres have to be large enough to cover many points that are not well integrated. This leads to large overlapping at the high level of the index tree. Overlapping is alleviated at low levels. Based on the above discoveries, we combine browsing with a feature-based query by displaying the shape and texture patterns which are close to the center of each cluster in palettes for users to select. Once the user selects a palette, the palette will be updated with patterns which are close to the centers of clusters at the next level. This will function as a browser until the user stops browsing and starts retrieving. We describe our scheme using texture features only. We use a 30-dimensional Gabor vector to represent the texture feature. The Gabor function takes the form: £ ¤ 2 (y−y0 )2 1 (u−u 0 ) g(u, v) = e x a2 + b2 e−2πi[x0 (u−u 0 )+y0 (v−v0 )] , TABLE I Distance between Children Cluster Centers (% in the Vector Space) Level 1.1 2.1 2.2 3.1 3.3 4.1 4.3 4.5 4.7 5.1 5.3 5.10 5.12 6.3 Distance 54.18 18.26 20.08 8.24 12.32 8.26 19.66 38.78 9.62 9.16 14.82 14.06 6.70 2.86
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
131
FIG. 7. Illustration of browsing in the texture tree.
where (x0 , y0 ) and (u 0 , v0 ) are the centers of the filter in the spatial domain and the frequency domain, respectively, and a and b are the radii of the filter. We arrange 30 functions to cover the entire frequency domain [22]. The method can be illustrated by a subtree of 259 images in Fig. 7. We use a palette to display the texture patterns of the current search tree, as shown in Figs. 7a and 7b. The pattern at the top-left corner of the texture palette represents the entire set of images in the left tree of the current node. It is followed by its left child node and right child node, as shown in the same row. The first element in the second row is the pattern for the right tree of the current node, followed by its two children. For example, Fig. 7a gives the palette of node 1.1 in Fig. 6. When the user select a pattern, e.g., node 3.3 in Fig. 6 (highlighted in Fig. 7a), the system will guide the search to the subtree under 3.3 and update the palette, as shown in Fig. 7b. Patterns 4.5, 4.6, 5.9, 5.10, 5.11, and 5.12 (see Fig. 6) are brought up for the user to browse or retrieve, as shown in Fig. 7b. It cuts the search area by restricting search under branch 3.3 only. The process can be iterated until either a leaf node has been reached or the user wants to stop browsing and start retrieving by adding another feature such as shape or color. 3.2.
Performance of Browsing
The time saved using browsing depends on whether the index tree is balanced or unbalanced. It also depends on whether the numbers of images in leaf nodes are similar or not, and this in turn depends on the performance of index tree splitting heuristics. We shall discuss the performance of browsing under the assumption that we have a reasonably good index tree, i.e., the depth difference between the left child and the right child is no more than 1, and the numbers of images among the children are similar. The worst case in browsing is when the fan-out of SS+ -tree is 2, i.e., for a binary tree. A complete binary tree with height h will have 2h+1 -1 nodes. With browsing, only one child will be examined. We will compare at most 2h − 1 2 − (1/2h−1 ) 2 2h − 1 = = ≤ , h h−1 h h−1 h−1 (2 − 1) + (2 − 1) + 1 2 +2 −1 2 + 1 − (1/2 ) 3 where h is the level of node i. In other words, we will compare no more than 2n/3 nodes—the
132
JIN ET AL.
FIG. 8. Query comparison between browsing and qbe (44,818 images, average of 200 queries).
worst case occurs when the browsing node is one level shorter (height h − 2) than the nonbrowsing node and the nonbrowsing node (height h − 1) is full at its leaf level. Therefore, the ratio of saving can be described by the recurrence: S(n) = S(2n/3) + 1. The solution to this recurrence, by the master theorem [35], is S(n) = 2(lgn). We will search much fewer images than those using retrieval methods only. Note this solution is independent of overlapping. In an index tree with heavy overlapping, the number of images searched by a normal retrieval algorithm is significantly larger. We have compared our method with the query-by-example method (qbe) which is the primary method used by many content-based image retrieval systems. We performed queries over the set of 44,818 images. They are collected from various sources, including the vision texture database, Brodatz texture album, MeasTex’s natural and artificial texture, Corel Photo-Paint clipart images and texture patterns, and images collected from the World Wide Web using an image collecting spider. The index tree’s fan-out is 2, and the entire tree has 12 levels. The query images are selected from the set so the query is continued until the image is found. The performance is compared over the total number of nodes touched during the query. Figure 8 gives the comparison on the average of 200 queries. The comparison is drawn along the level of the tree as some queries terminate at nonleaf nodes. It can be seen that at the high level of the index tree, i.e., from levels 1 to 6, structure browsing has a significant saving over the qbe method. This reminds us that structural browsing is a good way to speed up the query. Note that the comparison is made under fan-out equal 2. When the fan-out is larger than 2, the saving is higher. Structural browsing also has an advantage of adapting to the individual user as it is the user who selects the branch. It is also noticed that in refining the query, i.e., searching at the low level of the index tree, structural browsing is not a proper way to choose.
4. CONCLUSION This paper presents a fast and efficient scheme for retrieving images from multimedia databases. We develop SS+ -tree with efficient bounding envelopes and a splitting heuristic to reduce overlapping among the nodes. The method also combines browsing with retrieval to avoid large overlapping at the high level of an index tree. It provides an interactive tool to
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
133
incorporate users’ judgments. It will be able to handle a large image database with millions of items. Our research is focused on the nature of the retrieval mechanism. There is no similar method for comparison. Experiments on a large collection of images are carried out and the results are compared with those of other methods in general.
REFERENCES 1. N. S. Chang, and K. S. Fu, Picture query languages for pictorial data-base systems, IEEE Comput. 14, 1981, 23–33. 2. D. Comer, The ubiquitous B-tree, ACM Comput. Surveys 11, 1979, 121–137. 3. G. Salton and M. J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, New York, 1983. 4. S. K. Chang and T. L. Kunii, Pictorial data-base systems, IEEE Comput. 14, 1981, 13–21. 5. C. Faloutsos and S. Christodoulakis, Signature files: an access method for documents and its analytical performance evaluation, ACM Trans. Office Inform. Systems 2, 1984, 267–288. 6. J. Larish, Kodak’s still picture exchange for print and film use, Adv. Imaging 10, 1995, 38–39. 7. M. Martucci, Digital still marketing at PressLink, Adv. Imaging 10, 1995, 34–36. 8. L. Bielski, The image database of the future begins, Adv. Imaging 10, 1995, 26–91. 9. T. Caelli and D. Reye, On the classification of image regions by colour texture and shape, Pattern Recognition 26, 1993, 461–470. 10. M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, Query by image and video content: The QBIC system, IEEE Comput. 28, 1995, 23–32. 11. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin, The QBIC project: querying images by content using color, texture and shape, in Proc. of SPIE: Storage and Retrieval for Image and Video Databases, Vol. 1908, pp. 173–187, 1993. 12. R. Jain, NSF workshop on visual information management systems, SIGMOD Record 22, 1993, 57–75. 13. Pentland, R. W. Picard, and S. Sclaroff, Photobook: content-based manipulation of image databases, in Proc. of SPIE: Storage and Retrieval Image and Video Databases II, San Jose, Vol. 2185, 1995. 14. V. E. Ogle and M. Stonebraker, Chabot: retrieval from a relational database of images, IEEE Comput. 28, 1995, 40–48. 15. M. La Cascia and E. Ardizzone, JACOB: just a content-based query system for video databases, in Proceedings, ICASSP-96, Atlanta, GA, 1996. 16. Guttman, R-trees: A dynamic index structure for spatial searching. in Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, Boston, MA, June 1984, pp. 47–57. 17. M. Minsky and S. Papert, Perceptrons: An Introduction to Computational Geometry, MIT Press, Cambridge, MA, 1969. 18. D. A. White and R. Jain, Similarity indexing with the SS-tree, in Proceedings of the Twelfth IEEE International Conference on Data Engineering, New Orleans, LA, February 1996, pp. 516–523. 19. H. V. Jagadish, Indexing for retrieval by similarity, in Multimedia Database Systems: Issues and Research Directions (V. Subrahmanian and S. Jajodia, Eds.), Springer-Verlag, Berlin, 1996. 20. K. I. Lin, H. V. Jagadish, and C. Faloutsos, The TV-tree: An index structure for high-dimensional data, J. Very Large Databases 3, 1994, 517–549. 21. S. Berchtold, D. A. Keim, and H. P. Kriegel, The X-tree: An index structure for high-dimensional data, in Proceedings of the 22th International Conference on Very Large Data Bases, Bombay, India, 1996, pp. 28–39. 22. J. S. Jin, R. Kurniawati, and G. Xu, A scheme for intelligent image retrieval in multimedia databases. J. Visual Comm. Image Representation 7, 1997, 369–377. 23. J. G. Cleary, Analysis of an algorithm for finding nearest neighbors in Euclidean space, ACM Trans. Math. Software 5, 1979, 183–192. 24. I. Kamel and C. Faloutsos, On packing R-trees, in Proceedings of the Second International Conference on Information and Knowledge Management, Washington DC, November 1993, pp. 490–499. 25. U. Pagel, H. W. Six, H. Toben, and P. Widmayer, Towards an analysis of range queries performance in spatial data structures, in Proceedings of the Twelfth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, Washington, DC, May 1993, pp. 214–221.
134
JIN ET AL.
26. P. J. M. v Oosterom, Reactive Data Structures for Geographic Information Systems, Oxford University Press, New York, 1993. 27. Elzinga and D. Hearn, The minimum covering sphere problem, Management Sci. 19, 1972, 96–104. 28. N. Megiddo, Linear-time algorithms for linear programming in R3 and related problem, SIAM J. Comput. 12, 1983, 759–776. 29. Welzl, Smallest enclosing disks (Balls and Ellipsoids), in Proceedings of New Results and New Trends in Computer Science, Lecture Notes in Computer Science, Vol. 555, pp. 359–370, Springer-Verlag, Berlin/New York, 1991. 30. M. Schroeder, Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise, W H Freeman, New York, 1991. 31. N. Beckmann, H. P. Kriegel, R. Schneider, and B. Seeger, The R∗ -tree: an efficient and robust access method for points and rectangles, in Proceedings of the ACM SIGMOD International Conference on Management of Data, Atlantic City, NJ, May 1990, pp. 322–331. 32. T. Sellis, N. Roussopoulos, and C. Faloutsos, The R+ -tree: A dynamic index for multi-dimensional objects, in Proceedings of the Thirteenth Conference on Very Large Databases, Los Altos, CA, September 1987, pp. 507–518, Morgan Kauffman, San Mateo, CA. 33. H. Frigui and R. Krishnapuram, A robust algorithm for automatic extraction of an unknown number of clusters from noisy data, Pattern Recognition Lett. 17, 1996, 1223–1232. 34. Aurenhammer, Voronoi diagrams—A survey of a fundamental geometric data structure, ACM Comput. Surveys 23, 1991, 345–405. 35. T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms, MIT Press, Cambridge, MA, 1990.
JESSE JIN graduated with a Ph.D. from University of Otago, New Zealand. Currently, he is the Director of the Visual Information Processing Laboratory and the Head of Information Engineering Group in the School of Computer Science and Engineering, University of New South Wales. He has received many grants from both academic sources and industry sources including Motorola, SiliconGraphics, and CyberView. His research interests include multimedia, the Internet, and image processing.
RUTH KURNIAWATI graduated with a B.Sc. in computer science from Bandung Institute of Technology, Indonesia, and a Ph.D. from the School of Computer Science and Engineering, University of New South Wales. She is now working at Microsoft in Seattle. Her research interests include the Internet, multimedia, and highdimensional indexing.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
135
GUANGYU XU graduated with a B.Sc.(Hons.) from the Department of Control and Automation, Tsinghua University, in 1963. He was a visiting scholar at Purdue University working with Professor K. S. Fu from 1982 to 1984, and a visiting professor at Beckman Institute, University of Illinois at Urbana-Chamgain from December 1994 to June 1995. Currently, he is a professor in the Department of Computer Science, Tsinghua University, and a senior member of IEEE. He has received many research grants and is the author of many papers and four books. His research interests include computer vision and multimedia information systems.
XUESHENG BAI graduated with a Ph.D. from the Department of Computer Science, Tsinghua University, in 1998. He has been an assistant professor in the department since then. His research interests include content-based visual information retrieval, video multimedia, and image processing.
Using Browsing to Improve Content-Based Image Retrieval Jesse S. Jin and Ruth Kurniawati School of Computer Science and Engineering, University of New South Wales, Sydney 2052, Australia
and Guangyu Xu and Xuesheng Bai Department of Computer Science, Tsinghua University, Beijing 100084, P. R. China Received February 8, 1999; accepted April 7, 2000
Content-based image retrieval heavily relies on the indexing structure. Current index structures, such as the R-tree, the SS-tree and the X-tree, have a large overlapping area among their nodes, especially at the high (parent) levels of the indexing tree. The overlapping area causes the search engine to examine a large number of nodes and hence it is very inefficient to retrieve at very high levels of the index tree. We develop the SS+ -tree to alleviate the overlapping problem and present a scheme to combine browsing with retrieval in searching for images. Browsing provides a visual tool for a user to quickly narrow the search to a small region and to avoid a large number of high dimensional comparisons. Combined with retrieval, it produces a very efficient content-based retrieval method. °C 2001 Academic Press Key Words: content-based image retrieval; indexing; browsing; multimedia database.
1. INTRODUCTION The advance of image technology has resulted in widespread applications of images to gain more effective representation of visual information and user-friendly human– machine interfaces. It is therefore reasonable to expect that most information systems in the future will require efficient methods for archiving and managing large volume of images and thus advanced imagery databases will be desirable. Images, in a general sense, include natural images, such as photographs, medical images, and satellite images; artificial images, such as computer graphics, paintings, and drawings; and scientific pictures, such as statistical charts and visualization patches. Retrieving images in multimedia databases is an important issue. Current image database systems can be classified, according to the type and nature of the features used for indexing and retrieval, into three categories: attribute-based systems, text-based systems, and content-based systems. The 123 1047-3203/01 $35.00 C 2001 by Academic Press Copyright ° All rights of reproduction in any form reserved.
124
JIN ET AL.
different semantic representations of the features would generally lead to different indexing methods. Attribute-based systems are the legacy of the conventional DBMS in using a set of structured attributes to access objects so that the object handling capability is extended to cope with voluminous and gigantic image objects. Since little special development is required in organizing and retrieving image objects in these systems, many image or pictorial databases in earlier years were attribute-based [1]. Indexing methods used in these systems were usually B+ -trees [2] or inverted files [3]. Chang and Knuii [4] surveyed a variety of attribute-based pictorial database systems and found that the main problem in attributebased systems was that a fixed set of attributes could not take advantage of the rich content of images. Moreover, these attributes limited the scope of the application of databases and provided little room for future and unforeseen usage of the images. Text-based methods annotate images with a text description and structured fields. Queries in these systems use keywords and return partially-matched results ranked by similarity. Indexing methods for text-based image systems include full text scanning access methods, inverted files, and signature file methods [5]. There are a few commercial text-based image databases such as the Kodak Picture Exchange System [6], the PressLink Library [7], and the Time Picture Archive Collection [8]. The advantage of such methods is that they are able to describe abstract concepts. However, the disadvantages are prohibitive. Manually written descriptions fail to reflect the rich content of images completely. Some features in images, such as irregular shapes and jumbled textures, are extremely difficult, if not impossible, to describe in text. Textual descriptions limit the scope of searching to the keywords predetermined by the author of the system or the current application, leaving few avenues for using the data beyond that scope. Description is subjective, and different people may give quite different descriptions for the same image. There is also the common problem of understanding natural language description. Furthermore, descriptive text is entered manually, and therefore, the indexing process is slow. To overcome the above-mentioned problems content-based methods have therefore been developed. The most commonly used content features include shape, color, and texture [9]. These features are extracted automatically in the indexing process when images are entered into a multimedia database. There are a few ongoing developments in content-based image retrieval (CBIR) such as IBM QBIC [10,11], Virage [12], MIT Photobook [13], Chabot [14], and JACOB systems [15]. The indexing structure is a big issue for CBIR systems. Visual features are usually of high dimensions. Conventional access structures such as R-tree [16] are not suitable for retrieving high-dimensional features. A similarity search in a high-dimensional space is an NP problem and, in general, as Minsky and Papert [17] conjectured, will examine a large portion of the data. A number of indexing data structures have been proposed to handle high-dimensional data: the Similarity Search tree (SS-tree) [18], the Telescopic Vector tree (TV-tree) [19, 20], and the X-tree [21]. These new structures all perform better than the R-tree for various reasons: the TV-tree reduces the dimension of the vectors by collapsing the first few dimensions with the same values, the SS-tree utilizes the clustering property of the data, and the X-tree tries to minimize the overlapping between nodes’ bounding boxes (hyperboxes). Current indexing structures have a problem of having a large overlapping area among nodes, causing the retrieval process to inspect a large number of image items. The computational complexity of comparison is also exponential to the dimension.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
125
We have developed an SS+ -tree for indexing high dimensional features and proposed a browsing method in conjunction with retrieval. The techniques have been practiced in developing CBIR-VU, a general purpose image retrieval system [22]. In Section 2 we present our indexing structure and discuss the overlap problem, and in Section 3 we describe our scheme of retrieval using browsing. Finally, there is a conclusion. 2. INDEXING HIGH-DIMENSIONAL FEATURES USING SS+ -TREE The issues of improving an R-tree-based multidimensional access structure include selecting the shape of bounding envelopes, splitting heuristics, the heuristics to reduce overlapping, the updating criteria, and efficient search strategy. These issues are not independent of each other. The splitting heuristic and subtree selection criteria will determine the shape of each node and hence the goodness of the bounding envelope. The goodness of the bounding envelope will also determine the extent of overlapping between nodes. The heuristics used to reduce overlapping will also affect the shape of the nodes. We present our SS+ -tree by addressing four issues: bounding envelope, splitting heuristics, updating criteria, and searching strategy. Overlapping often happens during the updating process, and we address it as an updating criterion. 2.1.
Bounding Envelope
The R-tree family uses bounding rectangles. The main problem with this kind of bounding envelope is that there are large overlapping areas. If a query falls into an overlapping area, such as query a in Fig. 1a, the search has to go through all rectangles containing a (branches c1 and c3 as shown in Fig. 1b). Another problem is that the bounding rectangle is not consistent with the similarity measurement, the distance. For example, query a falls into clusters c1 and c3 in Fig. 1a. Due to the inconsistency, cluster c23 will fail to be the first candidate in the search. For the nearest neighbor queries in Euclidean space, Cleary [23] proved that decomposition into spherical regions is optimal in the sense that it minimizes the expected number of
FIG. 1. Overlapping and inconsistency problem of the bounding rectangle.
126
JIN ET AL.
regions touched. It also keeps the consistency with the distance measurement. To calculate the bounding sphere, the SS-tree [18] uses the centroid and the distance of the farthest element from the centroid as the center and radius of the bounding sphere, respectively, which is not the optimal one with respect to volume and surface area. Because the sphere volume grows exponentially with the dimension, in a high-dimensional space, a small increase in the radius of a sphere will give a large increase in the surface area and the volume of the sphere. In accordance with the analytical results of the R-tree [24,25], we want to minimize the total surface area and the volume of bounding envelopes. The sphere-tree [26] uses the smallest enclosing sphere computed by a variant of the method described by Elzinga and Hearn [27], which at best is dependent exponentially on the dimension. Besides, the sphere-tree has only been tested with two-dimensional spatial data. Computing the smallest enclosing sphere is a classic problem in computational geometry. The time complexity is exponential to the number of dimensions (for approaches using linear programming [27, 28]) or even superexponential [29]. It is not feasible in a high-dimensional space. However, due to the special characteristics of the vector distribution in the feature space, we propose a heuristic method which can calculate the smallest enclosing sphere in O(n). Our heuristic method searches the center of the smallest enclosing sphere using the √ golden ratio (( 5 − 1)/2 = 0.6180339887 . . .), which gives the least number of searches in an unevenly distributed candidate space [30]. In each approximation, it reduces the search area and eliminates a large number of points which cannot be on the surface of the sphere. The algorithm can be described in the following: 1. initialize: c0 to the center of the box; r0 to the maximum dimensional radius of the box; rout to the minimum dimensional radius of the box; 2. find maximum radius r1 of all points to c0 and eliminate all points inside rout ; 3. update: rout = min(r0 , | r1 −2 r0 |), where r0 and r1 are vectors from c0 to r0 and r1 , respectively; r0 = r1 ; c0 = 0.618 r01 , where r01 is the unit vector of r1 ; 4. if c0 has changed, repeat steps 2–4; otherwise, c0 and r1 define the smallest enclosing sphere. A 2D illustration of the algorithm can be found in Fig. 2. The algorithm starts with a guess c0 and r0 , as the sphere center and radius, where c0 is the midpoint of the bounding box
FIG. 2. Calculating the smallest enclosing sphere using elimination.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
127
of all points in a node and r0 is the radius of the maximum dimension. The first iteration finds the maximum radius r1 , as shown in Fig. 2a, and eliminates all points inside the circle (c0 , rout ) because they cannot be on the surface of the smallest enclosing sphere. We obtain an area A1 which is defined by the plane perpendicular to the vector r1 . It is obvious that the center of the smallest enclosing sphere will not be in area A1. We update c0 to a new guess which is on r1 with a length equal to 0.618 of |r1 − r0 |. We then find a new maximum radius r1 and eliminate all points inside the circle (c0 , rout ), as shown in Fig. 2b. This process can be iterated but we have found that after the second iteration, the remaining points are in a small number. They can be fed into Welzl’s smallest enclosing sphere algorithm [31] to obtain the smallest enclosing sphere. 2.2.
Node Split Heuristic
Almost every tree-based multidimensional access structure, e.g., the R-tree and its variants [16, 31, 32], the SS-tree [18], and the X-tree [21], uses a splitting plane that is perpendicular to one of the coordinate axes. Except for the SS-tree, the split criteria used are mainly topological ones, e.g., the minimization of the perimeter, the area, the increase of the node’s area, or the overlapping area. Because the tree structure aims to support similarity searches, we would like nearby vectors to be collected in the same or nearby nodes. This means that a division of data should reflect the clustering property and there should be a lower variance within the nodes. Dividing data into groups while optimizing a statistic feature has been studied extensively in the clustering area. For the SS+ -tree, we chose an unsupervised C-prototype clustering algorithm [33]. The computational complexity for this clustering algorithm is O(cni), where n is the number of data, c is the number of clusters, and i is the maximum number of iterations. In the splitting process, the number of clusters c will be two, the number of data points n is bounded by the node’s maximum capacity plus one, and we can always set an upper bound for the number of iterations i. In this way, we have a greater degree of freedom and the algorithm will seek to minimize the variance in the resulting partitions. Also note that in a Euclidean space, this splitting heuristic will tend to produce spherical-shaped nodes. 2.3.
Updating Index Tree and Reducing Overlapping Area
Updating is an important issue in index trees. To be consistent with our splitting heuristic we insert new data into the cluster with the closest centroid. This strategy will minimize the variance within the clusters to avoid frequent updating. However, updating will happen eventually when one node with a bigger variance eats a smaller node (see Fig. 3a for an illustration—here we use bounding spheres for the nodes, but a similar situation will also occur in bounding boxes). This kind of situation is not desirable since it will be difficult for us to determine which node will be searched during the query process and, also, the total volume or area of the nodes would be significantly larger. To alleviate the problem, the nodes should be adjusted. We have tested an R+ -like downward split propagation to completely remove the overlapping between nodes using the power planes [34]. We finally discarded the strategy since the scope of reorganization resulting from the downward split propagation could involve reorganizing the whole subtree and the fragmentation will be very large since the number of possible power planes of n spheres is n [d/2] , where d is the dimension of the space. We take a simple and effective local updating strategy. We check if a node boundary expands and heavily overlaps its siblings (it will take O(n) time, where n is the number
128
JIN ET AL.
FIG. 3. The node configuration (a) before and (b) after applying the local reorganization rule.
of children within the parent node). If we have this situation, we invalidate the children involved (say k) and perform a generalized k-means clustering process on the grandchildren. Initializing the cluster centers will not be a problem because from our SES calculation we have the points on the sphere. These points serve as the best initial centers for k-means clustering. The result of this rule can be seen in Fig. 3b, where the initial centers are two points on the big circle in Fig. 3a. The reorganization itself, if necessary, will take O(nk) time. Because it involves all grandchildren, measures should be taken so that it occurs infrequently. One suggestion is to put a guard that will decay with the insertions of new data. There are cases where overlapping is unavoidable, in which case it is better not to split the node at all and keep one big node in place of several heavily overlapped ones (similar to the supernodes in the X-trees [21]). 2.4.
Greedy Search Strategy
Similarity searches are generalized k-nearest-neighbor searches. It searches the index tree to find the corresponding nearest-neighbor disk. We propose a branch-and-bound algorithm to reduce the number of nodes being inspected. In a branch-and-bound algorithm, an important issue is how to find the best candidate for the search and what the upper bound is. The selection of the best candidate in our greedy strategy is to go to the cluster whose centroid is the closest to the query point, e.g., C4 in Fig. 4a. We have actually tested several branching criteria, including the minimum distance (the optimistic measure), minimum of the maximum distance (the pessimistic measure), and several other heuristics. The
FIG. 4. A greedy strategy to choose the best candidate in a lower bound (a) and an upper bound (b).
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
129
experiment results suggest that the nearest centroid criterion is the best in terms of minimizing the number of leaf nodes that have to be examined. To set an upper bound, we can safely ignore all nodes whose smallest enclosing sphere does not intersect the current nearest-neighbor disk. We use the furthest distance between the smallest enclosing sphere of a cluster and the query point to check if the cluster is intersected with the nearest-neighbor disk, e.g., C3 in Fig. 4b. Note that the nearest-neighbor disk is decreasing. It starts with a large radius to enclose at least k nearest neighbors. During the search, it gradually reduces. At the end, it is bounded by only k nearest neighbors. 3. RETRIEVAL USING BROWSING Although the SS+ -tree performs better than the other index structures, the overlap and slow comparison problems are not completely resolved. We propose a browsing scheme using the index structure. The idea is very similar to the way we search books in the library. Since the shelves are organized by subjects, we may browse on a particular shelf to find similar books. We have developed an image retrieval system using color, texture, and shape information, as shown in Fig. 5. There are six query methods. A user can submit a query with partial information such as color, shape, or texture by selecting a feature pattern and clicking on 1h, or through user-definition button 4h. A query can also be submitted through a selected image ( 2h) by clicking on an object of interest. The system will then analyze the image and extract features automatically for retrieval. Our system also has a browsing function ( 5h) which will be described in the following section.
FIG. 5. Image retrieval interface. Legend: 1h, query by features; 2h, query by examples; 3h, query by keywords; 4h, query by user definitions; 5h, query with browsing; 6h, query with feedback.
130
JIN ET AL.
FIG. 6. An index tree of texture features (for 259 images).
3.1.
Browsing Using the Index Tree
Careful study of indexing structures of the high-dimensional feature space and the retrieval problems revealed the following. •
An index tree provides information for quick browsing. Figure 6 shows part of the texture index tree in our experiment. The key image at each node has the smallest distance to the center of the cluster; • Cluster centers at high (parent) levels of the index tree have a distinct distance. Table I provides the distance of children in the index tree shown in Fig. 6. The distance between nodes 2.1 and 2.2 is 54.18% of the size of the index space, while the distance between nodes 5.1 and 5.2 is 8.26% and the distance between nodes 6.7 and 6.8 is 2.43%. In other words, the key images at high levels have a high discriminating capability. They provide an effective way to browse images. • Overlapping is low at low (children) levels of the index tree. Due to the isotropic nature of spheres, extending a sphere in one direction will extend it in every direction. At the high level, spheres have to be large enough to cover many points that are not well integrated. This leads to large overlapping at the high level of the index tree. Overlapping is alleviated at low levels. Based on the above discoveries, we combine browsing with a feature-based query by displaying the shape and texture patterns which are close to the center of each cluster in palettes for users to select. Once the user selects a palette, the palette will be updated with patterns which are close to the centers of clusters at the next level. This will function as a browser until the user stops browsing and starts retrieving. We describe our scheme using texture features only. We use a 30-dimensional Gabor vector to represent the texture feature. The Gabor function takes the form: £ ¤ 2 (y−y0 )2 1 (u−u 0 ) g(u, v) = e x a2 + b2 e−2πi[x0 (u−u 0 )+y0 (v−v0 )] , TABLE I Distance between Children Cluster Centers (% in the Vector Space) Level 1.1 2.1 2.2 3.1 3.3 4.1 4.3 4.5 4.7 5.1 5.3 5.10 5.12 6.3 Distance 54.18 18.26 20.08 8.24 12.32 8.26 19.66 38.78 9.62 9.16 14.82 14.06 6.70 2.86
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
131
FIG. 7. Illustration of browsing in the texture tree.
where (x0 , y0 ) and (u 0 , v0 ) are the centers of the filter in the spatial domain and the frequency domain, respectively, and a and b are the radii of the filter. We arrange 30 functions to cover the entire frequency domain [22]. The method can be illustrated by a subtree of 259 images in Fig. 7. We use a palette to display the texture patterns of the current search tree, as shown in Figs. 7a and 7b. The pattern at the top-left corner of the texture palette represents the entire set of images in the left tree of the current node. It is followed by its left child node and right child node, as shown in the same row. The first element in the second row is the pattern for the right tree of the current node, followed by its two children. For example, Fig. 7a gives the palette of node 1.1 in Fig. 6. When the user select a pattern, e.g., node 3.3 in Fig. 6 (highlighted in Fig. 7a), the system will guide the search to the subtree under 3.3 and update the palette, as shown in Fig. 7b. Patterns 4.5, 4.6, 5.9, 5.10, 5.11, and 5.12 (see Fig. 6) are brought up for the user to browse or retrieve, as shown in Fig. 7b. It cuts the search area by restricting search under branch 3.3 only. The process can be iterated until either a leaf node has been reached or the user wants to stop browsing and start retrieving by adding another feature such as shape or color. 3.2.
Performance of Browsing
The time saved using browsing depends on whether the index tree is balanced or unbalanced. It also depends on whether the numbers of images in leaf nodes are similar or not, and this in turn depends on the performance of index tree splitting heuristics. We shall discuss the performance of browsing under the assumption that we have a reasonably good index tree, i.e., the depth difference between the left child and the right child is no more than 1, and the numbers of images among the children are similar. The worst case in browsing is when the fan-out of SS+ -tree is 2, i.e., for a binary tree. A complete binary tree with height h will have 2h+1 -1 nodes. With browsing, only one child will be examined. We will compare at most 2h − 1 2 − (1/2h−1 ) 2 2h − 1 = = ≤ , h h−1 h h−1 h−1 (2 − 1) + (2 − 1) + 1 2 +2 −1 2 + 1 − (1/2 ) 3 where h is the level of node i. In other words, we will compare no more than 2n/3 nodes—the
132
JIN ET AL.
FIG. 8. Query comparison between browsing and qbe (44,818 images, average of 200 queries).
worst case occurs when the browsing node is one level shorter (height h − 2) than the nonbrowsing node and the nonbrowsing node (height h − 1) is full at its leaf level. Therefore, the ratio of saving can be described by the recurrence: S(n) = S(2n/3) + 1. The solution to this recurrence, by the master theorem [35], is S(n) = 2(lgn). We will search much fewer images than those using retrieval methods only. Note this solution is independent of overlapping. In an index tree with heavy overlapping, the number of images searched by a normal retrieval algorithm is significantly larger. We have compared our method with the query-by-example method (qbe) which is the primary method used by many content-based image retrieval systems. We performed queries over the set of 44,818 images. They are collected from various sources, including the vision texture database, Brodatz texture album, MeasTex’s natural and artificial texture, Corel Photo-Paint clipart images and texture patterns, and images collected from the World Wide Web using an image collecting spider. The index tree’s fan-out is 2, and the entire tree has 12 levels. The query images are selected from the set so the query is continued until the image is found. The performance is compared over the total number of nodes touched during the query. Figure 8 gives the comparison on the average of 200 queries. The comparison is drawn along the level of the tree as some queries terminate at nonleaf nodes. It can be seen that at the high level of the index tree, i.e., from levels 1 to 6, structure browsing has a significant saving over the qbe method. This reminds us that structural browsing is a good way to speed up the query. Note that the comparison is made under fan-out equal 2. When the fan-out is larger than 2, the saving is higher. Structural browsing also has an advantage of adapting to the individual user as it is the user who selects the branch. It is also noticed that in refining the query, i.e., searching at the low level of the index tree, structural browsing is not a proper way to choose.
4. CONCLUSION This paper presents a fast and efficient scheme for retrieving images from multimedia databases. We develop SS+ -tree with efficient bounding envelopes and a splitting heuristic to reduce overlapping among the nodes. The method also combines browsing with retrieval to avoid large overlapping at the high level of an index tree. It provides an interactive tool to
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
133
incorporate users’ judgments. It will be able to handle a large image database with millions of items. Our research is focused on the nature of the retrieval mechanism. There is no similar method for comparison. Experiments on a large collection of images are carried out and the results are compared with those of other methods in general.
REFERENCES 1. N. S. Chang, and K. S. Fu, Picture query languages for pictorial data-base systems, IEEE Comput. 14, 1981, 23–33. 2. D. Comer, The ubiquitous B-tree, ACM Comput. Surveys 11, 1979, 121–137. 3. G. Salton and M. J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, New York, 1983. 4. S. K. Chang and T. L. Kunii, Pictorial data-base systems, IEEE Comput. 14, 1981, 13–21. 5. C. Faloutsos and S. Christodoulakis, Signature files: an access method for documents and its analytical performance evaluation, ACM Trans. Office Inform. Systems 2, 1984, 267–288. 6. J. Larish, Kodak’s still picture exchange for print and film use, Adv. Imaging 10, 1995, 38–39. 7. M. Martucci, Digital still marketing at PressLink, Adv. Imaging 10, 1995, 34–36. 8. L. Bielski, The image database of the future begins, Adv. Imaging 10, 1995, 26–91. 9. T. Caelli and D. Reye, On the classification of image regions by colour texture and shape, Pattern Recognition 26, 1993, 461–470. 10. M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, Query by image and video content: The QBIC system, IEEE Comput. 28, 1995, 23–32. 11. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin, The QBIC project: querying images by content using color, texture and shape, in Proc. of SPIE: Storage and Retrieval for Image and Video Databases, Vol. 1908, pp. 173–187, 1993. 12. R. Jain, NSF workshop on visual information management systems, SIGMOD Record 22, 1993, 57–75. 13. Pentland, R. W. Picard, and S. Sclaroff, Photobook: content-based manipulation of image databases, in Proc. of SPIE: Storage and Retrieval Image and Video Databases II, San Jose, Vol. 2185, 1995. 14. V. E. Ogle and M. Stonebraker, Chabot: retrieval from a relational database of images, IEEE Comput. 28, 1995, 40–48. 15. M. La Cascia and E. Ardizzone, JACOB: just a content-based query system for video databases, in Proceedings, ICASSP-96, Atlanta, GA, 1996. 16. Guttman, R-trees: A dynamic index structure for spatial searching. in Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, Boston, MA, June 1984, pp. 47–57. 17. M. Minsky and S. Papert, Perceptrons: An Introduction to Computational Geometry, MIT Press, Cambridge, MA, 1969. 18. D. A. White and R. Jain, Similarity indexing with the SS-tree, in Proceedings of the Twelfth IEEE International Conference on Data Engineering, New Orleans, LA, February 1996, pp. 516–523. 19. H. V. Jagadish, Indexing for retrieval by similarity, in Multimedia Database Systems: Issues and Research Directions (V. Subrahmanian and S. Jajodia, Eds.), Springer-Verlag, Berlin, 1996. 20. K. I. Lin, H. V. Jagadish, and C. Faloutsos, The TV-tree: An index structure for high-dimensional data, J. Very Large Databases 3, 1994, 517–549. 21. S. Berchtold, D. A. Keim, and H. P. Kriegel, The X-tree: An index structure for high-dimensional data, in Proceedings of the 22th International Conference on Very Large Data Bases, Bombay, India, 1996, pp. 28–39. 22. J. S. Jin, R. Kurniawati, and G. Xu, A scheme for intelligent image retrieval in multimedia databases. J. Visual Comm. Image Representation 7, 1997, 369–377. 23. J. G. Cleary, Analysis of an algorithm for finding nearest neighbors in Euclidean space, ACM Trans. Math. Software 5, 1979, 183–192. 24. I. Kamel and C. Faloutsos, On packing R-trees, in Proceedings of the Second International Conference on Information and Knowledge Management, Washington DC, November 1993, pp. 490–499. 25. U. Pagel, H. W. Six, H. Toben, and P. Widmayer, Towards an analysis of range queries performance in spatial data structures, in Proceedings of the Twelfth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, Washington, DC, May 1993, pp. 214–221.
134
JIN ET AL.
26. P. J. M. v Oosterom, Reactive Data Structures for Geographic Information Systems, Oxford University Press, New York, 1993. 27. Elzinga and D. Hearn, The minimum covering sphere problem, Management Sci. 19, 1972, 96–104. 28. N. Megiddo, Linear-time algorithms for linear programming in R3 and related problem, SIAM J. Comput. 12, 1983, 759–776. 29. Welzl, Smallest enclosing disks (Balls and Ellipsoids), in Proceedings of New Results and New Trends in Computer Science, Lecture Notes in Computer Science, Vol. 555, pp. 359–370, Springer-Verlag, Berlin/New York, 1991. 30. M. Schroeder, Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise, W H Freeman, New York, 1991. 31. N. Beckmann, H. P. Kriegel, R. Schneider, and B. Seeger, The R∗ -tree: an efficient and robust access method for points and rectangles, in Proceedings of the ACM SIGMOD International Conference on Management of Data, Atlantic City, NJ, May 1990, pp. 322–331. 32. T. Sellis, N. Roussopoulos, and C. Faloutsos, The R+ -tree: A dynamic index for multi-dimensional objects, in Proceedings of the Thirteenth Conference on Very Large Databases, Los Altos, CA, September 1987, pp. 507–518, Morgan Kauffman, San Mateo, CA. 33. H. Frigui and R. Krishnapuram, A robust algorithm for automatic extraction of an unknown number of clusters from noisy data, Pattern Recognition Lett. 17, 1996, 1223–1232. 34. Aurenhammer, Voronoi diagrams—A survey of a fundamental geometric data structure, ACM Comput. Surveys 23, 1991, 345–405. 35. T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms, MIT Press, Cambridge, MA, 1990.
JESSE JIN graduated with a Ph.D. from University of Otago, New Zealand. Currently, he is the Director of the Visual Information Processing Laboratory and the Head of Information Engineering Group in the School of Computer Science and Engineering, University of New South Wales. He has received many grants from both academic sources and industry sources including Motorola, SiliconGraphics, and CyberView. His research interests include multimedia, the Internet, and image processing.
RUTH KURNIAWATI graduated with a B.Sc. in computer science from Bandung Institute of Technology, Indonesia, and a Ph.D. from the School of Computer Science and Engineering, University of New South Wales. She is now working at Microsoft in Seattle. Her research interests include the Internet, multimedia, and highdimensional indexing.
BROWSING IN CONTENT-BASED IMAGE RETRIEVAL
135
GUANGYU XU graduated with a B.Sc.(Hons.) from the Department of Control and Automation, Tsinghua University, in 1963. He was a visiting scholar at Purdue University working with Professor K. S. Fu from 1982 to 1984, and a visiting professor at Beckman Institute, University of Illinois at Urbana-Chamgain from December 1994 to June 1995. Currently, he is a professor in the Department of Computer Science, Tsinghua University, and a senior member of IEEE. He has received many research grants and is the author of many papers and four books. His research interests include computer vision and multimedia information systems.
XUESHENG BAI graduated with a Ph.D. from the Department of Computer Science, Tsinghua University, in 1998. He has been an assistant professor in the department since then. His research interests include content-based visual information retrieval, video multimedia, and image processing.