- Email: [email protected]
A structural analyzer for regularly arranged textures
COMPUTER
GRAPHICS
AND
IMAGE
PROCESSING
18,
259-278 (1982)
A Structural Analyzer for Regularly Arranged Textures TAKASHI MATSUYAMA,KINJIROSABURI,ANDMAKOTONAGAO Depurtment
of Electrical
Engineering,
Kyoto
University,
Sakyo,
Kyoto
606, Japan
Received November 17, I980 A method for the structural analysis of regularly arranged textures is presented. Given a picture of texture, we first extract texture elements by a region growing procedure, using the knowledge that a texture contains many regions with similar properties corresponding to texture elements. In the process of describing the spatial arrangement among the extracted texture elements, we at first extract “regularity vectors” from a set of relative positional vectors between the texture elements. A regularity vector means a relative vector which appears quite often in an arrangement of texture elements. Using the regularity vectors, we can estimate locations of “missing elements” and can remove erroneous elements which result from errors of picture processing programs. Then, we try to find out the “simplest” description of the arrangement of the texture elements in terms of the regularity vectors. This method is applicable to a wide variety of regular patterns such as printed patterns on clothes and tile-works on walls and pavements, where complex hierarchical and multi-element textures are described as overlays of several lattice arrangements. 1. INTRODUCTION
Texture provides an important feature to characterize and discriminate regions. Because of the importance of textural properties in various applications, many methods have been developed to extract textural features [l], which, in general, can be classified into two categories: statistical and structural methods. In the statistical approach, texture is characterized by a set of statistics which are extracted from a large ensemble of local properties representing interpixel relationships. Generally speaking, the statistical methods are useful for very fine textures which do not contain any obvious “texture elements,” nor regular spatial arrangements. On the other hand, the structural methods are based on the model that texture is made of a set of elements arranged with some regular placement rule. Thus, to describe the structure of texture, we have to characterize the elements and the placement rules. An element consists of a set of pixels characterized by a set of properties. The simplest element is a pixel and its attribute is a gray level. Tomita et al. [2] extracted sets of connected pixels with similar gray levels as elements, and characterized them by size, area, directionality, and shape. Carlton and Mitchell [3] regarded local extrema of gray levels as elements and characterized them by their magnitudes. Several methods have been proposed to characterize placement rules among texture elements: distribution density of elements [4], adjacency probabilities of texture elements [2], a graph-like language to describe arrangements of lines and polygons [5], a tree grammar to express the spatial structure of pixels in an N X N window [6], and so on. The distribution density and adjacency probabilities, however, are too simple features to be used for the description of spatial arrangements. Recently, Davis [7] proposed a method to characterize the spatial structures of textures in terms of the magnitude and direction of relative positional vectors 259
0146-664X/82/030259-20$02.00/0 Copyright 0 1982 by Academic Press, Inc. All tights of reproduction in any form reserved.
260
MATSUYAMA.
SABURI,
AND
NAGAO
between texture elements and their k-nearest neighbors. This method, however. can only handle square or rectangular grid arrangements, and is not applicable 1~~1 hierarchical textures where texture elements make groups to form “high-1evcl” texture elements. In this paper we will present a new method to describe structures of regular!y arranged patterns. This method is designed to describe a variety of spatial arrangements of texture elements, such as hierarchical textures and textures with different types of elements, which often appear in printed patterns on clothes and tile-works on walls and pavements. Figure 1 shows the block diagram of our analysis procedure to describe structures of textures. The analysis process consists of the following four steps: (I) Extraction of texture elements. A set of texture elements are extracted from an input picture by a region growing procedure. For the effective extraction of texture elements, the segmentation program utilizes the knowledge that a texture contains many regions with similar properties corresponding to texture elements. INPUT
PICTURE
'I Extraction 0f Texture
Elements 4
v Extraction
I Estimating
Of Regularity
Vectors
v Description Spatial of
Locations of
Missing
Elements
of
Arrangements
Texture
Elements
Texture
Generation a Two-Dimensional of
of
Segmentation Image
Of
Texture
Texture FIG.
I, Block
diagram
of the analysis
process.
Regipns
STRUCTURAL
ANALYZER
FOR REGULAR
TEXTURES
261
To describe the spatial arrangement among the texture elements, we at first extract “regularity vectors” from a set of relative positional vectors between the elements. A regularity vector is (informally) defined as a relative vector which appears quite often in an arrangement of elements. The spatial structure of texture is to be described by using these regularity vectors as descriptive terms. (3) Estimating locations of ‘missing elements”. The first process of extracting texture elements is a simple (bottom-up) picture processing procedure, so that some elements are sometimes left unextracted because of noise. This process estimates locations of such “missing elements” by using the regularity vectors. This feedback (top-down) analysis compensates for the imperfection of extracting texture elements by the simple picture processing procedure. (4) Describing a spatial structure among the texture elements. Although a regular pattern contains many regularity vectors, most of them are redundant. Here, we select a small set of regularity vectors with which the spatial arrangement can be described in the simplest way. By using the selected regularity vectors, the arrangement is described as an overlay of several two-dimensional lattices. In order to show the availability of the structural description given by the above analysis process, we will also present two applications: (1) Regeneration of two-dimensional images of textures from their structural descriptions (2) Segmentation of texture regions by using their structural descriptions Several experimental results will be given to show the performance of our method. The experiments were performed on some artificially generated patterns and a few natural textures. (2) Extraction
of regularity vectors.
2. EXTRACTION
OF TEXTURE
ELEMENTS
Figure 2 shows an artificially generated texture pattern. The size of the picture is 128 X 128 and the gray scale is quantized to 256 gray levels. The average gray levels of the background area and texture elements are 120, 40 (dark elements), and 200
FIG. 2. An artificially generated texture pattern.
262
MATSUYAMA.
SABURI,
AND
NAGAO
(bright elements), respectively, where Gaussian noise with standard deviation 20 1x added. Moreover, the locations of the texture elements fluctuate around ideal positions. We consider a texture element as a set of connected pixels with similar gray leveli. The most straightforward way to extract texture elements is to segment a picture into regions and to consider those regions with similar properties as texture elements. In the segmentation, however, there exists a very difficult problem of selecting a gray level threshold. Although many threshold-determination methods have been proposed, none of them give completely satisfactory results if they rely only on the properties of picture data. In our case, fortunately, we can utilize the knowledge that a picture of texture contains many texture elements. That is, for extracting texture elements, we should segment the picture so that many regions with similar properties may be produced. We iterate a region growing process by increasing a threshold step by step until the whole picture area is merged into one region. Every time regions are merged, we memorize a distribution of feature vectors of the regions in the feature space, and check whether or not a cluster with a large population (in the feature space) is generated. Since ordinary clustering methods take much computation time, we utilized a simplified clustering method to detect a cluster in the feature space. The feature space is partitioned into a set of “rectangular” blocks with the same volume. Then, we only have to memorize the number of feature vectors in each block at each cycle of region growing. The algorithm of extracting texture elements is as follows: STEP 1. Apply “edge preserving smoothing” [8] to an input picture in order to remove noise and sharpen blurred edges. STEP 2. INITIAL SEGMENTATION. Merge neighboring pixels whose gray level differences are very small. STEP 3. Calculate several properties of the resultant regions. (The properties used to characterize a region are average gray level, area size, direction of principal axis, elongatedness, and compactness.) STEP 4. Merge pairs of adjacent regions with the least gray level difference, and calculate feature vectors of the new generated regions. Then, memorize the distribution of feature vectors in the feature space. STEP 5. Iterate STEP 4 until all regions in the picture are merged into one region. STEP 6. Let Bij (i = 1, . . . , N, j = 1,. . . , M) denote the number of feature vectors in the i th block in the feature space at the jth cycle of region growing. Calculate B, = maxB,,. i.J
Then, regard as texture elements the regions whose feature vectors are contained in the Ith and its neighboring blocks at the Jth cycle of region growing. When a texture consists of texture elements of several different types, our method extracts texture elements of each class individuaIIy by iterating STEP 6. Figure 3a and b show the texture elements extracted from the picture shown in Fig. 2. In Fig. 3a, an element has been left unextracted because of noise. This
STRUCTURAL
ANALYZER
FOR REGULAR
TEXTURES
263
FIG. 3. Texture elements extracted from the picture in Fig. 2; an element in (a) has been left unextracted because of noise.
“missing element” will be located by using regularity vectors between the already extracted elements. 3. EXTRACTION
OF REGULARITY
VECTORS
In what follows, we will consider that all texture elements belong to the same class and will present a method to describe their spatial arrangement. The method to combine such “within-class” descriptions will be given in Section 5.4. The first step of processing for describing the spatial arrangement is to extract regularity vectors from a set of relative positional vectors between the elements. Here, we consider a regularity vector as a relative vector which appears quite often in an arrangement of elements. In Fig. 4, a, b,c, etc., can be considered as regularity vectors, while a relative vector between elements E, and E2a cannot, because it occurs only once in this arrangement. (All arrangements we consider are those in the “finite” two-dimensional space.) All spatial relationships necessary for the description are represented in the set of regularity vectors, even if elements are arranged in a hierarchical way. For example, it seems to be natural to describe the arrangement in Fig. 5 as having a two-layered structure; one describes a triangular arrangement among three neighboring elements, and the other a two-dimensional lattice of local triangles. Since the vectors required
E16
17
0
0
FIG.
Ek2El* 0
E&3E,
E#E20
0
0
2
4. A two-dimensional lattice arrangement
264
MATSUYAMA.
SARURI.
0 l
0
0
FIG.
5. A hierarchical
NA
0 0
0
AND
0 0
l
0 0
0
arrangement
l 0
0
of elements;
0
l E2i
a, b, and c are regularity
vectors.
for describing both the triangle and the lattice often appear in this arrangement, they are always extracted as regularity vectors. Practically, positions of texture elements, which have been calculated by analyzing a picture of texture, often fluctuate around ideal positions, so that there exist no relative vectors which coincide completely with others. Therefore, we apply clustering in the two-dimensional “vector space,” where a point (x, r) denotes a vector whose components are x and y (Fig. 6). Then, we define regularity vectors as central points of clusters with large populations in the vector space. All relative vectors in the same cluster are represented by a regularity vector. The clustering procedure to extract regularity vectors is as follows: STEP 1. Plot N(N - 1)/2 relative vectors between all pairs of elements into the vector space, and memorize starting and ending elements for each relative vector. STEP 2. Apply the furthest neighbor clustering technique [9] using the absolute value of the shortest relative vector as the threshold. (I) Calculate distances between all pairs of different clusters in the vector space, where the distance D(i, j) between ith and jth clusters is defined by
where (xik, Y,~) and (x,,, y,,) (k = 1, . . . , N,, I = 1, . . . . N,) denote points (relative
picture
ve*r
space
FIG. 6. Transformation
from
the picture
space
space to the vector
space
STRUCTURAL
ANALYZER
FOR REGULAR
265
TEXTURES
FIG. 7. See text.
vectors) in i th and jth clusters, respectively, and Ni and Nj denote the numbers of points in the clusters, respectively. (II) Let i* and j* denote a pair of clusters whose distance takes the minimum value. If D(i*, j*) is less than the absolute value of the shortest relative vector, then merge the i*th and j*th clusters into one cluster and go to (I). Else go to STEP 3. STEP 3. Reject those clusters whose populations are less than a threshold. STEP 4. For each detected cluster, exchange, if needed, the starting and ending elements of the relative vectors included in it so that the y component of the center of the gravity of the cluster becomes nonnegative. Then, define the center of the gravity as a regularity vector, and list the pairs of elements connected by the relative vectors in the cluster. By the furthest neighbor clustering method, the distance between any pair of points in a resultant cluster cannot exceed the length of the shortest relative vector. This property is necessary for getting meaningful regularity vectors. That is, in Fig. 7, if v, and v, were included in the same cluster, E, and E3 would be regarded as located at the same position from E,. This is obviously nonsense or will result in contradictions in the structural description. Since two similar relative vectors are sometimes plotted in opposite directions around the origin of the vector space, we define the minimum between tx,k
-
xj,)2
+
(Yik
-
Yj,j2
and
JtXik
+
xj/)2
+
(Yik
+
Yji12
as a distance between (xik, yik) and (xj,, y,,). By this clustering, we have a set of regularity vectors {v,, v,, . . . , v,} and n sets of elements, each of which is composed of pairs of elements connected by a regularity vector. For example, in Fig. 4, we have a as a regularity vector and a set’of pairs of elements whose spatial relationships are represented by a, {(E,, E2), ( E2, E3), (4, Ed, (4 Ed, (6 ET)>.. . ,(E,9, E,)}. To describe the structure of the arrangement among elements, we only have to perform symbolic manipulations on a set of primitive symbols {E,, E,, E,, . . . , EN}, among which n types of binary relations {v,, v,, . . . , v,} are defined. 4. ESTIMATING
LOCATIONS
OF MISSING
ELEMENTS
In general, a picture of texture is corrupted by noise, and moreover the sizes of and the spaces between texture elements are very small. Therefore, it often happens
266
MATSUYAMA,
l
FIG.
SABURI, AND NA
l
l
l
l
8. Estimating locations of missing elements.
that some elements are left unextracted and some false elements are extracted. But, if most of the elements have been correctly extracted, we can estimate the positions of the missing elements by using regularity vectors. STEP 1. Calculate regularity vectors from the already extracted elements. STEP 2. Suppose that v denotes some regularity vector, then shift an element by v (or -v). If there exist no elements at a shifted position, while another element is found at the position shifted by 2v ( - 2v) (Fig. 8), then examine the area around that position in the original picture more carefully to see whether a new element can be found. STEP 3. Apply the above process to all combinations of the elements and regularity vectors. If new elements are extracted, return to STEP 1 and recalculate the regularity vectors. When searching for a missing element at an estimated postion, we can use a model (template) of the texture elements, that is, the averaged image of the already extracted elements. Therefore, we only have to apply a “template matching” method around the estimated position. The missing element in Fig. 3(a) has been successfully extracted by this process. On the other hand, if some false elements have been extracted due to errors made by the picture processing programs, the clustering process for regularity vectors will reject them. Since false elements usually appear at random in an arrangement, the number of relative vectors similar to those from/to the false elements is usually small (Fig. 9). As a result, the relative vectors from/to the false elements are not recognized as regularity vectors, and no relationships by regularity vectors are defined over the false elements. Thus, the regions extracted by the region growing method are authorized as texture elements only when regularity vectors are defined over them.
.. .
FIG.
.
.
.
.
.
l
.
.
l
. l
34 . .
.
.
.
9. Removing a false element (E’) (see text).
STRUCTURAL 5. DESCRIBING
ANALYZER SPATIAL
FOR
REGULAR
ARRANGEMENTS
TEXTURES OF
267
ELEMENTS
We have a set of regularity vectors and corresponding sets of element pairs. There exist many different ways of describing a given arrangement of elements in terms of regularity vectors, so that we have to incorporate a criterion from which we determine the most suitable description. It is natural to consider that the simplest description is the best. In our method, we try to describe the structure of the arrangement by using as small a number of regularity vectors as possible. 5.1. One-Dimensional Repetitive Pattern If the arrangement under consideration is a one-dimensional repetitive pattern (Fig. lo), we can describe it with a single regularity vector a which connects any two neighboring elements. That is, this arrangement can be described as a repetitive sequence of elements defined by a. Let us denote this sequence by a[ E,, E,, . . . , E,,]. On the other hand, if we use 2a as a relation between the elements (2a is also a regularity vector), the total set of the elements is divided into two disjoint repetitive sequences, 2a[ E,, E3, E,, E,, Es] and 2a[E,, Ed, E6, E,, E,,], and then we must describe the relation between these two sequences by a. Comparing the above two descriptions, it is clear that the former is simpler and more suitable. Thus, the first step of describing the spatial arrangement is as follows: STEP Dl. Remove from a set of regularity vectors those vectors that are multiples of other regularity vectors. If we have a single vector left, the arrangement under consideration is a one-dimensional repetitive pattern, which can be described by that vector. In Fig. 10, a relation E, and E3 defined by 2a is considered as redundant because these elements can be connected by two relations defined by a (that is, (E,, E2) and (E,, E3)). Thus, we need not perform numerical computations. 5.2. Two-Dimensional
Lattice
Extending the above idea to two-dimensional arrangements, the simplest arrangement, a lattice (Fig. 1 l(a)), can be described as a repetitive sequence defined by a of a set of repetitive sequences defined by b, that is a[b[E,, E2, E3, Ed, EJ, b[E,, E,, Es, E,, E,,], b[E,,, E,,, E,3, E,,, EJ]. Here we have assumed that 1aI> ) b 1and that the elements are grouped into a set of repetitive sequences by a shorter vector and then these sequences are grouped by a longer vector. Selecting a and b from a set of regularity vectors (there are many other diagonal regularity vectors in this pattern) requires some computation and is affected by the overall shape of the arrangement. For example, see Fig. 1 l(b), where the shape of the arrangement is changed while keeping the same local spatial relationships between elements as Fig. 1l(a). In this case, vector a can no longer give a simple description. We have to use c and describe it as c[b[ E,, E2, E3, E4, Es], b[ E6, E,, Es, E,, E,,],
‘NE,,,Em Em 4, UlConsidering the above discussion, the following algorithm gives the most suitable pair of regularity vectors to describe a lattice.
FIG. 10. One-dimensional
repetitive
pattern.
268
MATSUYAMA.
SABURI. AND NA(i.40
FIG. I I. Describing structures of two-dimensional lattices (see text).
STEP D2. Take any pair of vectors, v, and VI, from a set of regularity vectors. (Of course, redundant vectors have already been removed by STEP Dl in the previous section.) STEP D3. Count the number of elements, Njj, which are contained in both of the repetitive sequences defined by vi and 7 at the same time. STEP D4. Select a pair of vectors as generating vectors for the lattice such that N,j is maximum among all combinations of regularity vectors. For the lattice shown in Fig. 11(b), if we use a and b, only nine elements (Fig 1 l(c)) are included in both of the one-dimensional repetitive sequences defined by a and b. Therefore, the description of the arrangement would become complex if we used these two vectors (a and b). On the other hand, b and c generate repetitive sequences which contain all 15 elements. Thus, if the arrangement under consideration is really a lattice, the above algorithm finds a pair of regularity vectors which gives the simplest description. 5.3. Hierarchical
Arrangement
Next, suppose that the arrangement is a two-dimensional repetitive pattern of some local arrangement of elements. For example, see Fig. 12, where triangular arrangements of three elements are arranged on a lattice. Then, we have v,, v,, v,, v,, etc., as regularity vectors. If we apply the above algorithm to this pattern, we have v, and v, as the most suitable pair of regularity vectors, because all elements are contained in both the repetitive sequences defined by v3 and v, at the same time. Using v, and vq, the total set of the elements is partitioned into three disjoint subsets, each of which is described as a lattice generated by v, and v,. Thus, we can describe this arrangement as an overlay of three lattices, and the displacements between them are represented by v, and v,. Generally speaking, using the above algorithm, we can describe any hierarchical arrangement where a cluster of n elements is placed on a lattice as an overlay of the same n lattices. The displacements among these lattices is determined by the following method.
STRUCTURAL
ANALYZER
l l
FIG.
FOR
l
l
l
12. A hierarchical
269
TEXTURES
l
l l
REGULAR
l l
l
arrangement
l
of elements.
STEP D5. If a pair of regularity vectors selected by STEP D4 divides the total set of elements into n disjoint subsets, then describe each subset as a lattice generated by these vectors, and (1) find an upper-left element ei for each lattice (i = 1, . . . , n); (2) find the “upper-left-most” element e* among {e,, e2,. . . , e,}; (3) find regularity vectors connecting e* and ei, and make them displacement vectors between lattices. From the psychological point of view, this representation sometimes does not coincide with human interpretation. That is, we would like to represent the arrangement shown in Fig. 12 as a lattice of local triangular arrangements of three elements. This is because human beings tend to group neighboring elements into a cluster and consider it as a “high-level” element. In order to get such a natural description of the hierarchical arrangement; STEP D6. Compare the absolute values between the regularity vectors generating the lattices and those describing displacements between the lattices. If the former are substantially larger than the latter, merge elements located at the same positions in the lattices into a cluster, and regard it as a “high-level” element. In this case an element itself has an inner structure of its own (see Fig. 13). So far, we have implicitly assumed that a unique pair of regularity vectors are extracted by STEP D4, but in some cases this is not the case. We have only one
:
t
FIG. 13. Reorganization triangles.
by Gestalt
clustering:
overlay
--\ ‘0 \b
l , . -4
I’
)‘\
1.
./
./
/‘5’\
,‘a-
I
\T :/
\o \_ /l ,’
\
,
I’;‘,
\
I
of three lattices
I
\t
,/ l
,I
,;\
,
,-,)
\. -. l / ’
\
,I,,\
‘\O y/ .-
is transformed
1
\
I
\
\
‘\O 0; .-/
to lattice
of local
270
MATSliYAMA. l
S.4BURI.
0
00
a
90
FIG.
l
AND
NACiAO e
@ e
e*
14. A one-dimensional
l
hierarchical
a o
l
*
arrangement.
regularity vector for the arrangement of a one-dimensional repetitive pattern of local arrangements as shown in Fig 14. Then, we describe it as an overlay of onedimensional repetitive sequences. (If needed, “Gestalt clustering” will be performed as in the case of Fig. 13). On the other hand, if a local arrangement of elements in a hierarchical arrangement itself is regular, our algorithm gives more than one pair of regularity vectors. For example, three and six pairs of regularity vectors will be detected in the arrangements shown in Fig. 15(a) and (b), respectively, i.e., (v,,vz), (vl,v3), and (v2,v3), and @,,v2), (v1,v3), @,,v4), (v2,v3), (v2,v4), and (v3,v4). These regular hierarchical arrangements can be described as three or four level repetitions of one-dimensional repetitive sequences. That is, STEP D7. If we have more than one pair of regularity vectors in STEP D4 and if the arrangement can be represented as an overlay of lattices, using any pair of regularity, then we sort such regularity vectors according to their absolute values (from short to long). Let {v,, v,, . . . , vk} denote this sequence of the selected regularity vectors. Then, we first make one-dimensional repetitive sequences defined by v,, and align these sequences in one-dimensional repetitive sequences using v,, and so on. 5.4. Discussion
The flow chart in Fig. 16 summarizes our method of describing spatial structures of regularly arranged patterns which have been discussed in the previous sections. Here, we will discuss some extensions of our method to more complex and less regular arrangements.
l
l
l
l l
l l
0
l
0
l
0
l
l
l
a 0
0
0
0
l l
a l
b
a FIG.
15. Regular
hierarchical
arrangements
l a
l
0 l
l
STRUCTURAL
ANALYZER
FOR REGULAR regularity
Extract
relative
from
between
TEXTURES
271
vectors
vectors
elements
Estimate
locations
missing
of
elements
STEPl:Remve
those
vectors
are
that
other
reSulerity multiples
regularity
of
vectors
One-dimensional repetitive (Fig.
STEPZ-4:Find
a pair
regularity the in
the
ST'7:Regular
such
elements
lattice
vectors
.
by the is
maximum
hierarchical
One-dimensional
1
arrangement (Fig
that contained
generated
two regularity
15)
FIG.
hierarchical pattern(Fig.
16.
Flow
chart for describing structures of regularly arrangedpatterns.
0 0
)
of
vectot-s
number of
pattern 10
.
l
. l
.
0
. 0
.
l
. l
.
l
. 0
0
. l
.
.
.
FIG. 17. Texture composed of two different kinds of elements.
14
)
272
MATSUYAMA,
SABURI. AND ru’AGA0
18. Eleven regular wall patterns (from [IO])
FIG.
When we are going to describe a texture composed of elements of different types, we first describe the spatial arrangements among elements of a single class, respectively, and then combine such “within-class” descriptions into an overall description by using relative vectors between upper-left elements of the descriptions. For example, see Fig. 17, where two types of elements are placed on lattices, respectively. The relation between the two lattices (within-class descriptions) can be described with a relative vector which connects the upper-left elements of the two
FIG.
arranged.
.
.
.
.
b
b
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
*
.
.
b
0
.
@
.
@
.
.
.
.
l
.
.
.
.
.
.
l
.
.
.
.
.
.
.
.
.
.
.
.
@
.
0
.
0
0
0
.
.
.
.
l
.
.
b
.
.
.
.
.
.
.
.
.
.
.
.
.
.
b
.
.
.
.
19. Two-dimensional
@.@bQ..
lattice with missing elements; missing elements (@- 0)
are reguiarly
STRUCTURAL ANALYZER FOR REGULAR TEXTURES
E4
E7
E8
l
l
l
E9
E10
E11
E12
E13
l
l
l
l
l
E14
El5
El6
El7
El8
El9
l
l
l
l
l
l
EZl l
E l
22
213
E20 l
E23 l
FIG. 20. Irregular-shapedarrangement;overallshapeof the arrangementis irregular.
lattices, i.e., v. Since these lattices have the same structure and the relative vector between them is much smaller than the vectors generating the lattices, we can consider each pair of neighboring elements of different classes as a high-level element and describe the overall arrangement as a lattice of the new elements.
Though all that our method can describe are regular repetitive patterns, it may be applicable to most artificially generated textures. Figure 18 shows eleven different designs for regular wall patterns, where the two-dimensional plane is covered with regular polygons without producing any spaces or overlaps [lo]. Though some of them seem to be very complex and difficult to describe, all of them can be described by our method as overlays of two-dimensional lattices of several kinds of elements. Our method in substance decomposes arrangements into two-dimensional lattices. Therefore, if some elements at the grid points are randomly missing, it can not give any simple descriptions even if it selects the most suitable regularity vectors. But if the missing elements themselves are arranged regularly, we first generate pairs of “normal” and “inverse” elements at the positions of missing elements, and describe it as an overlay of lattices of normal and inverse elements. For example, in Fig. 19, we generate the pairs of 0 (normal element) and 0 (inverse element) at positions, a, b, c, d, e, f, g, h, and i, resulting in complete lattices of 0 and 0. (The method described in Section 4 can find out these positions.) We assume that if a pair of 0 and 0 are located at the same place, we have nothing in the real two-dimensional space. Then, the arrangement can be described as an overlay of the lattices of 0 and 0. When an overall shape of an arrangement is irregular as shown in Fig. 20, we incorporate “NIL” elements (*) so that the description may become simple. As a result the arrangement in Fig. 20 is described as b[a[*, *, E,, E2, E3, *, *],
a[*, E,, E,, E,, E,, E8, *I, a[*, E,, E,,,, E,,, E12, E,,, *I, alL J% E16,E,,, E,,, &, Ed a[*, *, J%,,&, Ez3,*, *Il. 6. APPLICATIONS OF STRUCTURAL DESCRIPTIONSAND SOME EXPERIMENTAL RESULTS
We have shown the method of describing the structures of texture patterns. Figure 21 shows the structural description of the texture shown in Fig. 2, where two arrangements of different structures are connected by a displacement vector.
274
MATSUYAMA,
SABURl, AND KAGAO
FIG. 21. The structural description of the texture shown in Fig. 2; i,[6$] and $[7$?] denote six and seven one-dimensional repetitive sequences of the elements $ and $ defined by q, and ii,, respectively.
Here, we will show two application programs which use structural descriptions of textures given by the above mentioned analysis process. 6.1. Generating Two-Dimensional Structural Descriptions
Images of Textures from Their
Since a structural description of a texture contains (1) models (averaged two-dimensional images) of texture elements, (2) the average gray level of the background area (the regions which were not recognized as texture elements), and
FIG. 22. Image of texture synthesized from the description in Fig. 21.
STRUCTURAL
ANALYZER
a
FOR
REGULAR
TEXTURES
275
b
FIG. 23. Other examples of the analysis; (a) artificially generated hierarchical pattern, (b) natural texture D103 in [I I]. Pictures on the upper row show the original pictures and those on the bottom row the generated pictures. Some elements in the natural picture failed to be extracted due to the strong irregularity in the shape of the elements and in the spatial arrangement.
(3) descriptions of the spatial arrangements of the texture elements, it is very easy to generate a two-dimensional image of a texture from its structural description. Figure 22 shows the two-dimensional image generated from the description in Fig. 21. Figure 23 shows some examples of experiments made on other pictures. In the natural texture of Fig. 23(b), due to strong irregularities both in the shape of the elements and in the spatial arrangement, some elements in the marginal area have failed to be extracted. 6.2. Segmentation of Texture Regions Tsuji and Tomita [4] proposed a method of segmenting texture regions by structural analysis. Given texture elements in a picture, they put squares of size L X L onto the centers of gravity of all elements and defined a texture region as a
276
MATSUYAMA.
SABURI, AND NAGAG
m
r---i---I /@joj L---I--.--J
r
-
I
.
1
L
-
L--
m
FIG.
-7
- .
-I-
- - --l . j-w
I 1
-
-I
~
24. Detecting a texture region by the method proposed by Tsuji (41 (see text).
connected area covered by the squares (Fig. 24(a)). Here L denotes the maximum among the minimum distances between an element and the other elements. As mentioned in their paper, however, the resultant texture area sometimes becomes disjoint. When “vertical” and “horizontal” intervals between the elements are quite different, their method cannot give a correct texture region (Fig. 24(b)). Moreover, if the shape of the elements is very elongated, the square cannot cover each element (Fig. 24(c)). Neither can their method handle hierarchical textures. In our method, we move a template of a texture element according to the structural description and define the “locus-area” as a texture region. In a onedimensional repetitive pattern, an element is repeatedly shifted by the regularity vector (Fig. 25(a)). In a two-dimensional lattice, we trace an outer boundary of the lattice with the template, and define the region enclosed by the locus of the template
STRUCTURAL
FIG. 25. Determination
ANALYZER
FOR REGULAR
TEXTURES
277
of texture regions by moving a template of a texture element (see text).
as a texture region (Fig. 25(b)). In the case of hierarchical textures, we first make a template of a “high-level” element, and the texture region is formed in the same way as in ordinary arrangements. When a picture contains multiple arrangements of different structures as shown in Fig. 21, we first perform the above operation on each arrangement respectively, and overlay the generated texture regions onto the two-dimensional picture space. Then,
FIG. 26. Texture regions generated from the structural description in Fig. 21; there are three different texture regions. The third texture region resulted from the overlay of the other two texture regions.
278
MATSUYAMA.
SABURI, AND NAGAO
we sometimes have new texture regions where different textures are overlapped. Figure 26 shows the texture regions generated from the description in Fig. 21, where we have three different texture regions. 7 CONCLUSION
We have discussed the method of describing structures of regular textures. In the process of extracting texture elements, the feedback analysis procedure worked fairly well to repair errors of the segmentation process. In the natural textures, however, some elements were still left unextracted. This is because the condition for estimating locations of missing elements is rather strict: a missing element should be surrounded by already extracted ones. If we relax the condition, we will detect more missing elements. To describe the spatial structure among the texture elements, we introduced the concept of the regularity vectors which represent local regular spatial relationships between the texture elements. The analysis program selects the most suitable set of regularity vectors, with which the spatial structure of the texture is described as an overlay of lattices. Although the textures which our method can describe are confined to regular ones, it is applicable to a wide class of arrangements from simple one-dimensional repetitive patterns to complex two-dimensional hierarchical arrangements. This method may be used for the structural description of a variety of regular textures such as printed patterns on clothes and tile-works on walls and pavements. REFERENCES I. R. M. Haralick, Statistical and Structural Approaches to Texture, Proc. 4th IJCPR, pp. 45-69, 1978. 2. F. Tomita, Y. Shirai, and S. Tsuji, Classification of Textures by a Structural Analysis, Proc. 4th IJCPR, pp. 556-568, 1978. 3. S. G. Carlton, and 0. R. Mitchell, Image Segmentation Using Texture and Gray Level, Proc. of the IEEE Conf. on Image Processing and Pattern Recognition, pp. 387-391, 1977. 4. S. Tsuji, and F. Ton&a, A structural analyzer for a class of textures, Cornpurer Graphics and Image Processing
2, 1973, 216-231.
5. L. Carlucci, A formal system for texture languages, Pattern Recognition 4, 1972, 53-72. 6. S. Y. Lu and K. S. Fu, A syntactic approach to texture analysis, Computer Graphics Processing
and
Image
and
Image
7, 1978, 303-330.
7. L. S. Davis, Computing the spatial structure of cellular textures, Computer Processing 11, 1979, 111-122. 8. M. Nagao and T. Matsuyama, Edge preserving smoothing, Computer Graphics
Graphics and Image
Processing
9, 1979, 394-407.
R. 0. Duda and P. E. Hart, Pattern Classification and Scene Analysis, 1973. 10. L. Fejes-Toth, Regular Figures, Pergamon, Elmsford, N.Y., 1964. 11. P. Brodatz, Textures, Dover, New York, 1966. 9.
pp. 233-235, Wiley. New York,
GRAPHICS
AND
IMAGE
PROCESSING
18,
259-278 (1982)
A Structural Analyzer for Regularly Arranged Textures TAKASHI MATSUYAMA,KINJIROSABURI,ANDMAKOTONAGAO Depurtment
of Electrical
Engineering,
Kyoto
University,
Sakyo,
Kyoto
606, Japan
Received November 17, I980 A method for the structural analysis of regularly arranged textures is presented. Given a picture of texture, we first extract texture elements by a region growing procedure, using the knowledge that a texture contains many regions with similar properties corresponding to texture elements. In the process of describing the spatial arrangement among the extracted texture elements, we at first extract “regularity vectors” from a set of relative positional vectors between the texture elements. A regularity vector means a relative vector which appears quite often in an arrangement of texture elements. Using the regularity vectors, we can estimate locations of “missing elements” and can remove erroneous elements which result from errors of picture processing programs. Then, we try to find out the “simplest” description of the arrangement of the texture elements in terms of the regularity vectors. This method is applicable to a wide variety of regular patterns such as printed patterns on clothes and tile-works on walls and pavements, where complex hierarchical and multi-element textures are described as overlays of several lattice arrangements. 1. INTRODUCTION
Texture provides an important feature to characterize and discriminate regions. Because of the importance of textural properties in various applications, many methods have been developed to extract textural features [l], which, in general, can be classified into two categories: statistical and structural methods. In the statistical approach, texture is characterized by a set of statistics which are extracted from a large ensemble of local properties representing interpixel relationships. Generally speaking, the statistical methods are useful for very fine textures which do not contain any obvious “texture elements,” nor regular spatial arrangements. On the other hand, the structural methods are based on the model that texture is made of a set of elements arranged with some regular placement rule. Thus, to describe the structure of texture, we have to characterize the elements and the placement rules. An element consists of a set of pixels characterized by a set of properties. The simplest element is a pixel and its attribute is a gray level. Tomita et al. [2] extracted sets of connected pixels with similar gray levels as elements, and characterized them by size, area, directionality, and shape. Carlton and Mitchell [3] regarded local extrema of gray levels as elements and characterized them by their magnitudes. Several methods have been proposed to characterize placement rules among texture elements: distribution density of elements [4], adjacency probabilities of texture elements [2], a graph-like language to describe arrangements of lines and polygons [5], a tree grammar to express the spatial structure of pixels in an N X N window [6], and so on. The distribution density and adjacency probabilities, however, are too simple features to be used for the description of spatial arrangements. Recently, Davis [7] proposed a method to characterize the spatial structures of textures in terms of the magnitude and direction of relative positional vectors 259
0146-664X/82/030259-20$02.00/0 Copyright 0 1982 by Academic Press, Inc. All tights of reproduction in any form reserved.
260
MATSUYAMA.
SABURI,
AND
NAGAO
between texture elements and their k-nearest neighbors. This method, however. can only handle square or rectangular grid arrangements, and is not applicable 1~~1 hierarchical textures where texture elements make groups to form “high-1evcl” texture elements. In this paper we will present a new method to describe structures of regular!y arranged patterns. This method is designed to describe a variety of spatial arrangements of texture elements, such as hierarchical textures and textures with different types of elements, which often appear in printed patterns on clothes and tile-works on walls and pavements. Figure 1 shows the block diagram of our analysis procedure to describe structures of textures. The analysis process consists of the following four steps: (I) Extraction of texture elements. A set of texture elements are extracted from an input picture by a region growing procedure. For the effective extraction of texture elements, the segmentation program utilizes the knowledge that a texture contains many regions with similar properties corresponding to texture elements. INPUT
PICTURE
'I Extraction 0f Texture
Elements 4
v Extraction
I Estimating
Of Regularity
Vectors
v Description Spatial of
Locations of
Missing
Elements
of
Arrangements
Texture
Elements
Texture
Generation a Two-Dimensional of
of
Segmentation Image
Of
Texture
Texture FIG.
I, Block
diagram
of the analysis
process.
Regipns
STRUCTURAL
ANALYZER
FOR REGULAR
TEXTURES
261
To describe the spatial arrangement among the texture elements, we at first extract “regularity vectors” from a set of relative positional vectors between the elements. A regularity vector is (informally) defined as a relative vector which appears quite often in an arrangement of elements. The spatial structure of texture is to be described by using these regularity vectors as descriptive terms. (3) Estimating locations of ‘missing elements”. The first process of extracting texture elements is a simple (bottom-up) picture processing procedure, so that some elements are sometimes left unextracted because of noise. This process estimates locations of such “missing elements” by using the regularity vectors. This feedback (top-down) analysis compensates for the imperfection of extracting texture elements by the simple picture processing procedure. (4) Describing a spatial structure among the texture elements. Although a regular pattern contains many regularity vectors, most of them are redundant. Here, we select a small set of regularity vectors with which the spatial arrangement can be described in the simplest way. By using the selected regularity vectors, the arrangement is described as an overlay of several two-dimensional lattices. In order to show the availability of the structural description given by the above analysis process, we will also present two applications: (1) Regeneration of two-dimensional images of textures from their structural descriptions (2) Segmentation of texture regions by using their structural descriptions Several experimental results will be given to show the performance of our method. The experiments were performed on some artificially generated patterns and a few natural textures. (2) Extraction
of regularity vectors.
2. EXTRACTION
OF TEXTURE
ELEMENTS
Figure 2 shows an artificially generated texture pattern. The size of the picture is 128 X 128 and the gray scale is quantized to 256 gray levels. The average gray levels of the background area and texture elements are 120, 40 (dark elements), and 200
FIG. 2. An artificially generated texture pattern.
262
MATSUYAMA.
SABURI,
AND
NAGAO
(bright elements), respectively, where Gaussian noise with standard deviation 20 1x added. Moreover, the locations of the texture elements fluctuate around ideal positions. We consider a texture element as a set of connected pixels with similar gray leveli. The most straightforward way to extract texture elements is to segment a picture into regions and to consider those regions with similar properties as texture elements. In the segmentation, however, there exists a very difficult problem of selecting a gray level threshold. Although many threshold-determination methods have been proposed, none of them give completely satisfactory results if they rely only on the properties of picture data. In our case, fortunately, we can utilize the knowledge that a picture of texture contains many texture elements. That is, for extracting texture elements, we should segment the picture so that many regions with similar properties may be produced. We iterate a region growing process by increasing a threshold step by step until the whole picture area is merged into one region. Every time regions are merged, we memorize a distribution of feature vectors of the regions in the feature space, and check whether or not a cluster with a large population (in the feature space) is generated. Since ordinary clustering methods take much computation time, we utilized a simplified clustering method to detect a cluster in the feature space. The feature space is partitioned into a set of “rectangular” blocks with the same volume. Then, we only have to memorize the number of feature vectors in each block at each cycle of region growing. The algorithm of extracting texture elements is as follows: STEP 1. Apply “edge preserving smoothing” [8] to an input picture in order to remove noise and sharpen blurred edges. STEP 2. INITIAL SEGMENTATION. Merge neighboring pixels whose gray level differences are very small. STEP 3. Calculate several properties of the resultant regions. (The properties used to characterize a region are average gray level, area size, direction of principal axis, elongatedness, and compactness.) STEP 4. Merge pairs of adjacent regions with the least gray level difference, and calculate feature vectors of the new generated regions. Then, memorize the distribution of feature vectors in the feature space. STEP 5. Iterate STEP 4 until all regions in the picture are merged into one region. STEP 6. Let Bij (i = 1, . . . , N, j = 1,. . . , M) denote the number of feature vectors in the i th block in the feature space at the jth cycle of region growing. Calculate B, = maxB,,. i.J
Then, regard as texture elements the regions whose feature vectors are contained in the Ith and its neighboring blocks at the Jth cycle of region growing. When a texture consists of texture elements of several different types, our method extracts texture elements of each class individuaIIy by iterating STEP 6. Figure 3a and b show the texture elements extracted from the picture shown in Fig. 2. In Fig. 3a, an element has been left unextracted because of noise. This
STRUCTURAL
ANALYZER
FOR REGULAR
TEXTURES
263
FIG. 3. Texture elements extracted from the picture in Fig. 2; an element in (a) has been left unextracted because of noise.
“missing element” will be located by using regularity vectors between the already extracted elements. 3. EXTRACTION
OF REGULARITY
VECTORS
In what follows, we will consider that all texture elements belong to the same class and will present a method to describe their spatial arrangement. The method to combine such “within-class” descriptions will be given in Section 5.4. The first step of processing for describing the spatial arrangement is to extract regularity vectors from a set of relative positional vectors between the elements. Here, we consider a regularity vector as a relative vector which appears quite often in an arrangement of elements. In Fig. 4, a, b,c, etc., can be considered as regularity vectors, while a relative vector between elements E, and E2a cannot, because it occurs only once in this arrangement. (All arrangements we consider are those in the “finite” two-dimensional space.) All spatial relationships necessary for the description are represented in the set of regularity vectors, even if elements are arranged in a hierarchical way. For example, it seems to be natural to describe the arrangement in Fig. 5 as having a two-layered structure; one describes a triangular arrangement among three neighboring elements, and the other a two-dimensional lattice of local triangles. Since the vectors required
E16
17
0
0
FIG.
Ek2El* 0
E&3E,
E#E20
0
0
2
4. A two-dimensional lattice arrangement
264
MATSUYAMA.
SARURI.
0 l
0
0
FIG.
5. A hierarchical
NA
0 0
0
AND
0 0
l
0 0
0
arrangement
l 0
0
of elements;
0
l E2i
a, b, and c are regularity
vectors.
for describing both the triangle and the lattice often appear in this arrangement, they are always extracted as regularity vectors. Practically, positions of texture elements, which have been calculated by analyzing a picture of texture, often fluctuate around ideal positions, so that there exist no relative vectors which coincide completely with others. Therefore, we apply clustering in the two-dimensional “vector space,” where a point (x, r) denotes a vector whose components are x and y (Fig. 6). Then, we define regularity vectors as central points of clusters with large populations in the vector space. All relative vectors in the same cluster are represented by a regularity vector. The clustering procedure to extract regularity vectors is as follows: STEP 1. Plot N(N - 1)/2 relative vectors between all pairs of elements into the vector space, and memorize starting and ending elements for each relative vector. STEP 2. Apply the furthest neighbor clustering technique [9] using the absolute value of the shortest relative vector as the threshold. (I) Calculate distances between all pairs of different clusters in the vector space, where the distance D(i, j) between ith and jth clusters is defined by
where (xik, Y,~) and (x,,, y,,) (k = 1, . . . , N,, I = 1, . . . . N,) denote points (relative
picture
ve*r
space
FIG. 6. Transformation
from
the picture
space
space to the vector
space
STRUCTURAL
ANALYZER
FOR REGULAR
265
TEXTURES
FIG. 7. See text.
vectors) in i th and jth clusters, respectively, and Ni and Nj denote the numbers of points in the clusters, respectively. (II) Let i* and j* denote a pair of clusters whose distance takes the minimum value. If D(i*, j*) is less than the absolute value of the shortest relative vector, then merge the i*th and j*th clusters into one cluster and go to (I). Else go to STEP 3. STEP 3. Reject those clusters whose populations are less than a threshold. STEP 4. For each detected cluster, exchange, if needed, the starting and ending elements of the relative vectors included in it so that the y component of the center of the gravity of the cluster becomes nonnegative. Then, define the center of the gravity as a regularity vector, and list the pairs of elements connected by the relative vectors in the cluster. By the furthest neighbor clustering method, the distance between any pair of points in a resultant cluster cannot exceed the length of the shortest relative vector. This property is necessary for getting meaningful regularity vectors. That is, in Fig. 7, if v, and v, were included in the same cluster, E, and E3 would be regarded as located at the same position from E,. This is obviously nonsense or will result in contradictions in the structural description. Since two similar relative vectors are sometimes plotted in opposite directions around the origin of the vector space, we define the minimum between tx,k
-
xj,)2
+
(Yik
-
Yj,j2
and
JtXik
+
xj/)2
+
(Yik
+
Yji12
as a distance between (xik, yik) and (xj,, y,,). By this clustering, we have a set of regularity vectors {v,, v,, . . . , v,} and n sets of elements, each of which is composed of pairs of elements connected by a regularity vector. For example, in Fig. 4, we have a as a regularity vector and a set’of pairs of elements whose spatial relationships are represented by a, {(E,, E2), ( E2, E3), (4, Ed, (4 Ed, (6 ET)>.. . ,(E,9, E,)}. To describe the structure of the arrangement among elements, we only have to perform symbolic manipulations on a set of primitive symbols {E,, E,, E,, . . . , EN}, among which n types of binary relations {v,, v,, . . . , v,} are defined. 4. ESTIMATING
LOCATIONS
OF MISSING
ELEMENTS
In general, a picture of texture is corrupted by noise, and moreover the sizes of and the spaces between texture elements are very small. Therefore, it often happens
266
MATSUYAMA,
l
FIG.
SABURI, AND NA
l
l
l
l
8. Estimating locations of missing elements.
that some elements are left unextracted and some false elements are extracted. But, if most of the elements have been correctly extracted, we can estimate the positions of the missing elements by using regularity vectors. STEP 1. Calculate regularity vectors from the already extracted elements. STEP 2. Suppose that v denotes some regularity vector, then shift an element by v (or -v). If there exist no elements at a shifted position, while another element is found at the position shifted by 2v ( - 2v) (Fig. 8), then examine the area around that position in the original picture more carefully to see whether a new element can be found. STEP 3. Apply the above process to all combinations of the elements and regularity vectors. If new elements are extracted, return to STEP 1 and recalculate the regularity vectors. When searching for a missing element at an estimated postion, we can use a model (template) of the texture elements, that is, the averaged image of the already extracted elements. Therefore, we only have to apply a “template matching” method around the estimated position. The missing element in Fig. 3(a) has been successfully extracted by this process. On the other hand, if some false elements have been extracted due to errors made by the picture processing programs, the clustering process for regularity vectors will reject them. Since false elements usually appear at random in an arrangement, the number of relative vectors similar to those from/to the false elements is usually small (Fig. 9). As a result, the relative vectors from/to the false elements are not recognized as regularity vectors, and no relationships by regularity vectors are defined over the false elements. Thus, the regions extracted by the region growing method are authorized as texture elements only when regularity vectors are defined over them.
.. .
FIG.
.
.
.
.
.
l
.
.
l
. l
34 . .
.
.
.
9. Removing a false element (E’) (see text).
STRUCTURAL 5. DESCRIBING
ANALYZER SPATIAL
FOR
REGULAR
ARRANGEMENTS
TEXTURES OF
267
ELEMENTS
We have a set of regularity vectors and corresponding sets of element pairs. There exist many different ways of describing a given arrangement of elements in terms of regularity vectors, so that we have to incorporate a criterion from which we determine the most suitable description. It is natural to consider that the simplest description is the best. In our method, we try to describe the structure of the arrangement by using as small a number of regularity vectors as possible. 5.1. One-Dimensional Repetitive Pattern If the arrangement under consideration is a one-dimensional repetitive pattern (Fig. lo), we can describe it with a single regularity vector a which connects any two neighboring elements. That is, this arrangement can be described as a repetitive sequence of elements defined by a. Let us denote this sequence by a[ E,, E,, . . . , E,,]. On the other hand, if we use 2a as a relation between the elements (2a is also a regularity vector), the total set of the elements is divided into two disjoint repetitive sequences, 2a[ E,, E3, E,, E,, Es] and 2a[E,, Ed, E6, E,, E,,], and then we must describe the relation between these two sequences by a. Comparing the above two descriptions, it is clear that the former is simpler and more suitable. Thus, the first step of describing the spatial arrangement is as follows: STEP Dl. Remove from a set of regularity vectors those vectors that are multiples of other regularity vectors. If we have a single vector left, the arrangement under consideration is a one-dimensional repetitive pattern, which can be described by that vector. In Fig. 10, a relation E, and E3 defined by 2a is considered as redundant because these elements can be connected by two relations defined by a (that is, (E,, E2) and (E,, E3)). Thus, we need not perform numerical computations. 5.2. Two-Dimensional
Lattice
Extending the above idea to two-dimensional arrangements, the simplest arrangement, a lattice (Fig. 1 l(a)), can be described as a repetitive sequence defined by a of a set of repetitive sequences defined by b, that is a[b[E,, E2, E3, Ed, EJ, b[E,, E,, Es, E,, E,,], b[E,,, E,,, E,3, E,,, EJ]. Here we have assumed that 1aI> ) b 1and that the elements are grouped into a set of repetitive sequences by a shorter vector and then these sequences are grouped by a longer vector. Selecting a and b from a set of regularity vectors (there are many other diagonal regularity vectors in this pattern) requires some computation and is affected by the overall shape of the arrangement. For example, see Fig. 1 l(b), where the shape of the arrangement is changed while keeping the same local spatial relationships between elements as Fig. 1l(a). In this case, vector a can no longer give a simple description. We have to use c and describe it as c[b[ E,, E2, E3, E4, Es], b[ E6, E,, Es, E,, E,,],
‘NE,,,Em Em 4, UlConsidering the above discussion, the following algorithm gives the most suitable pair of regularity vectors to describe a lattice.
FIG. 10. One-dimensional
repetitive
pattern.
268
MATSUYAMA.
SABURI. AND NA(i.40
FIG. I I. Describing structures of two-dimensional lattices (see text).
STEP D2. Take any pair of vectors, v, and VI, from a set of regularity vectors. (Of course, redundant vectors have already been removed by STEP Dl in the previous section.) STEP D3. Count the number of elements, Njj, which are contained in both of the repetitive sequences defined by vi and 7 at the same time. STEP D4. Select a pair of vectors as generating vectors for the lattice such that N,j is maximum among all combinations of regularity vectors. For the lattice shown in Fig. 11(b), if we use a and b, only nine elements (Fig 1 l(c)) are included in both of the one-dimensional repetitive sequences defined by a and b. Therefore, the description of the arrangement would become complex if we used these two vectors (a and b). On the other hand, b and c generate repetitive sequences which contain all 15 elements. Thus, if the arrangement under consideration is really a lattice, the above algorithm finds a pair of regularity vectors which gives the simplest description. 5.3. Hierarchical
Arrangement
Next, suppose that the arrangement is a two-dimensional repetitive pattern of some local arrangement of elements. For example, see Fig. 12, where triangular arrangements of three elements are arranged on a lattice. Then, we have v,, v,, v,, v,, etc., as regularity vectors. If we apply the above algorithm to this pattern, we have v, and v, as the most suitable pair of regularity vectors, because all elements are contained in both the repetitive sequences defined by v3 and v, at the same time. Using v, and vq, the total set of the elements is partitioned into three disjoint subsets, each of which is described as a lattice generated by v, and v,. Thus, we can describe this arrangement as an overlay of three lattices, and the displacements between them are represented by v, and v,. Generally speaking, using the above algorithm, we can describe any hierarchical arrangement where a cluster of n elements is placed on a lattice as an overlay of the same n lattices. The displacements among these lattices is determined by the following method.
STRUCTURAL
ANALYZER
l l
FIG.
FOR
l
l
l
12. A hierarchical
269
TEXTURES
l
l l
REGULAR
l l
l
arrangement
l
of elements.
STEP D5. If a pair of regularity vectors selected by STEP D4 divides the total set of elements into n disjoint subsets, then describe each subset as a lattice generated by these vectors, and (1) find an upper-left element ei for each lattice (i = 1, . . . , n); (2) find the “upper-left-most” element e* among {e,, e2,. . . , e,}; (3) find regularity vectors connecting e* and ei, and make them displacement vectors between lattices. From the psychological point of view, this representation sometimes does not coincide with human interpretation. That is, we would like to represent the arrangement shown in Fig. 12 as a lattice of local triangular arrangements of three elements. This is because human beings tend to group neighboring elements into a cluster and consider it as a “high-level” element. In order to get such a natural description of the hierarchical arrangement; STEP D6. Compare the absolute values between the regularity vectors generating the lattices and those describing displacements between the lattices. If the former are substantially larger than the latter, merge elements located at the same positions in the lattices into a cluster, and regard it as a “high-level” element. In this case an element itself has an inner structure of its own (see Fig. 13). So far, we have implicitly assumed that a unique pair of regularity vectors are extracted by STEP D4, but in some cases this is not the case. We have only one
:
t
FIG. 13. Reorganization triangles.
by Gestalt
clustering:
overlay
--\ ‘0 \b
l , . -4
I’
)‘\
1.
./
./
/‘5’\
,‘a-
I
\T :/
\o \_ /l ,’
\
,
I’;‘,
\
I
of three lattices
I
\t
,/ l
,I
,;\
,
,-,)
\. -. l / ’
\
,I,,\
‘\O y/ .-
is transformed
1
\
I
\
\
‘\O 0; .-/
to lattice
of local
270
MATSliYAMA. l
S.4BURI.
0
00
a
90
FIG.
l
AND
NACiAO e
@ e
e*
14. A one-dimensional
l
hierarchical
a o
l
*
arrangement.
regularity vector for the arrangement of a one-dimensional repetitive pattern of local arrangements as shown in Fig 14. Then, we describe it as an overlay of onedimensional repetitive sequences. (If needed, “Gestalt clustering” will be performed as in the case of Fig. 13). On the other hand, if a local arrangement of elements in a hierarchical arrangement itself is regular, our algorithm gives more than one pair of regularity vectors. For example, three and six pairs of regularity vectors will be detected in the arrangements shown in Fig. 15(a) and (b), respectively, i.e., (v,,vz), (vl,v3), and (v2,v3), and @,,v2), (v1,v3), @,,v4), (v2,v3), (v2,v4), and (v3,v4). These regular hierarchical arrangements can be described as three or four level repetitions of one-dimensional repetitive sequences. That is, STEP D7. If we have more than one pair of regularity vectors in STEP D4 and if the arrangement can be represented as an overlay of lattices, using any pair of regularity, then we sort such regularity vectors according to their absolute values (from short to long). Let {v,, v,, . . . , vk} denote this sequence of the selected regularity vectors. Then, we first make one-dimensional repetitive sequences defined by v,, and align these sequences in one-dimensional repetitive sequences using v,, and so on. 5.4. Discussion
The flow chart in Fig. 16 summarizes our method of describing spatial structures of regularly arranged patterns which have been discussed in the previous sections. Here, we will discuss some extensions of our method to more complex and less regular arrangements.
l
l
l
l l
l l
0
l
0
l
0
l
l
l
a 0
0
0
0
l l
a l
b
a FIG.
15. Regular
hierarchical
arrangements
l a
l
0 l
l
STRUCTURAL
ANALYZER
FOR REGULAR regularity
Extract
relative
from
between
TEXTURES
271
vectors
vectors
elements
Estimate
locations
missing
of
elements
STEPl:Remve
those
vectors
are
that
other
reSulerity multiples
regularity
of
vectors
One-dimensional repetitive (Fig.
STEPZ-4:Find
a pair
regularity the in
the
ST'7:Regular
such
elements
lattice
vectors
.
by the is
maximum
hierarchical
One-dimensional
1
arrangement (Fig
that contained
generated
two regularity
15)
FIG.
hierarchical pattern(Fig.
16.
Flow
chart for describing structures of regularly arrangedpatterns.
0 0
)
of
vectot-s
number of
pattern 10
.
l
. l
.
0
. 0
.
l
. l
.
l
. 0
0
. l
.
.
.
FIG. 17. Texture composed of two different kinds of elements.
14
)
272
MATSUYAMA,
SABURI. AND ru’AGA0
18. Eleven regular wall patterns (from [IO])
FIG.
When we are going to describe a texture composed of elements of different types, we first describe the spatial arrangements among elements of a single class, respectively, and then combine such “within-class” descriptions into an overall description by using relative vectors between upper-left elements of the descriptions. For example, see Fig. 17, where two types of elements are placed on lattices, respectively. The relation between the two lattices (within-class descriptions) can be described with a relative vector which connects the upper-left elements of the two
FIG.
arranged.
.
.
.
.
b
b
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
*
.
.
b
0
.
@
.
@
.
.
.
.
l
.
.
.
.
.
.
l
.
.
.
.
.
.
.
.
.
.
.
.
@
.
0
.
0
0
0
.
.
.
.
l
.
.
b
.
.
.
.
.
.
.
.
.
.
.
.
.
.
b
.
.
.
.
19. Two-dimensional
@.@bQ..
lattice with missing elements; missing elements (@- 0)
are reguiarly
STRUCTURAL ANALYZER FOR REGULAR TEXTURES
E4
E7
E8
l
l
l
E9
E10
E11
E12
E13
l
l
l
l
l
E14
El5
El6
El7
El8
El9
l
l
l
l
l
l
EZl l
E l
22
213
E20 l
E23 l
FIG. 20. Irregular-shapedarrangement;overallshapeof the arrangementis irregular.
lattices, i.e., v. Since these lattices have the same structure and the relative vector between them is much smaller than the vectors generating the lattices, we can consider each pair of neighboring elements of different classes as a high-level element and describe the overall arrangement as a lattice of the new elements.
Though all that our method can describe are regular repetitive patterns, it may be applicable to most artificially generated textures. Figure 18 shows eleven different designs for regular wall patterns, where the two-dimensional plane is covered with regular polygons without producing any spaces or overlaps [lo]. Though some of them seem to be very complex and difficult to describe, all of them can be described by our method as overlays of two-dimensional lattices of several kinds of elements. Our method in substance decomposes arrangements into two-dimensional lattices. Therefore, if some elements at the grid points are randomly missing, it can not give any simple descriptions even if it selects the most suitable regularity vectors. But if the missing elements themselves are arranged regularly, we first generate pairs of “normal” and “inverse” elements at the positions of missing elements, and describe it as an overlay of lattices of normal and inverse elements. For example, in Fig. 19, we generate the pairs of 0 (normal element) and 0 (inverse element) at positions, a, b, c, d, e, f, g, h, and i, resulting in complete lattices of 0 and 0. (The method described in Section 4 can find out these positions.) We assume that if a pair of 0 and 0 are located at the same place, we have nothing in the real two-dimensional space. Then, the arrangement can be described as an overlay of the lattices of 0 and 0. When an overall shape of an arrangement is irregular as shown in Fig. 20, we incorporate “NIL” elements (*) so that the description may become simple. As a result the arrangement in Fig. 20 is described as b[a[*, *, E,, E2, E3, *, *],
a[*, E,, E,, E,, E,, E8, *I, a[*, E,, E,,,, E,,, E12, E,,, *I, alL J% E16,E,,, E,,, &, Ed a[*, *, J%,,&, Ez3,*, *Il. 6. APPLICATIONS OF STRUCTURAL DESCRIPTIONSAND SOME EXPERIMENTAL RESULTS
We have shown the method of describing the structures of texture patterns. Figure 21 shows the structural description of the texture shown in Fig. 2, where two arrangements of different structures are connected by a displacement vector.
274
MATSUYAMA,
SABURl, AND KAGAO
FIG. 21. The structural description of the texture shown in Fig. 2; i,[6$] and $[7$?] denote six and seven one-dimensional repetitive sequences of the elements $ and $ defined by q, and ii,, respectively.
Here, we will show two application programs which use structural descriptions of textures given by the above mentioned analysis process. 6.1. Generating Two-Dimensional Structural Descriptions
Images of Textures from Their
Since a structural description of a texture contains (1) models (averaged two-dimensional images) of texture elements, (2) the average gray level of the background area (the regions which were not recognized as texture elements), and
FIG. 22. Image of texture synthesized from the description in Fig. 21.
STRUCTURAL
ANALYZER
a
FOR
REGULAR
TEXTURES
275
b
FIG. 23. Other examples of the analysis; (a) artificially generated hierarchical pattern, (b) natural texture D103 in [I I]. Pictures on the upper row show the original pictures and those on the bottom row the generated pictures. Some elements in the natural picture failed to be extracted due to the strong irregularity in the shape of the elements and in the spatial arrangement.
(3) descriptions of the spatial arrangements of the texture elements, it is very easy to generate a two-dimensional image of a texture from its structural description. Figure 22 shows the two-dimensional image generated from the description in Fig. 21. Figure 23 shows some examples of experiments made on other pictures. In the natural texture of Fig. 23(b), due to strong irregularities both in the shape of the elements and in the spatial arrangement, some elements in the marginal area have failed to be extracted. 6.2. Segmentation of Texture Regions Tsuji and Tomita [4] proposed a method of segmenting texture regions by structural analysis. Given texture elements in a picture, they put squares of size L X L onto the centers of gravity of all elements and defined a texture region as a
276
MATSUYAMA.
SABURI, AND NAGAG
m
r---i---I /@joj L---I--.--J
r
-
I
.
1
L
-
L--
m
FIG.
-7
- .
-I-
- - --l . j-w
I 1
-
-I
~
24. Detecting a texture region by the method proposed by Tsuji (41 (see text).
connected area covered by the squares (Fig. 24(a)). Here L denotes the maximum among the minimum distances between an element and the other elements. As mentioned in their paper, however, the resultant texture area sometimes becomes disjoint. When “vertical” and “horizontal” intervals between the elements are quite different, their method cannot give a correct texture region (Fig. 24(b)). Moreover, if the shape of the elements is very elongated, the square cannot cover each element (Fig. 24(c)). Neither can their method handle hierarchical textures. In our method, we move a template of a texture element according to the structural description and define the “locus-area” as a texture region. In a onedimensional repetitive pattern, an element is repeatedly shifted by the regularity vector (Fig. 25(a)). In a two-dimensional lattice, we trace an outer boundary of the lattice with the template, and define the region enclosed by the locus of the template
STRUCTURAL
FIG. 25. Determination
ANALYZER
FOR REGULAR
TEXTURES
277
of texture regions by moving a template of a texture element (see text).
as a texture region (Fig. 25(b)). In the case of hierarchical textures, we first make a template of a “high-level” element, and the texture region is formed in the same way as in ordinary arrangements. When a picture contains multiple arrangements of different structures as shown in Fig. 21, we first perform the above operation on each arrangement respectively, and overlay the generated texture regions onto the two-dimensional picture space. Then,
FIG. 26. Texture regions generated from the structural description in Fig. 21; there are three different texture regions. The third texture region resulted from the overlay of the other two texture regions.
278
MATSUYAMA.
SABURI, AND NAGAO
we sometimes have new texture regions where different textures are overlapped. Figure 26 shows the texture regions generated from the description in Fig. 21, where we have three different texture regions. 7 CONCLUSION
We have discussed the method of describing structures of regular textures. In the process of extracting texture elements, the feedback analysis procedure worked fairly well to repair errors of the segmentation process. In the natural textures, however, some elements were still left unextracted. This is because the condition for estimating locations of missing elements is rather strict: a missing element should be surrounded by already extracted ones. If we relax the condition, we will detect more missing elements. To describe the spatial structure among the texture elements, we introduced the concept of the regularity vectors which represent local regular spatial relationships between the texture elements. The analysis program selects the most suitable set of regularity vectors, with which the spatial structure of the texture is described as an overlay of lattices. Although the textures which our method can describe are confined to regular ones, it is applicable to a wide class of arrangements from simple one-dimensional repetitive patterns to complex two-dimensional hierarchical arrangements. This method may be used for the structural description of a variety of regular textures such as printed patterns on clothes and tile-works on walls and pavements. REFERENCES I. R. M. Haralick, Statistical and Structural Approaches to Texture, Proc. 4th IJCPR, pp. 45-69, 1978. 2. F. Tomita, Y. Shirai, and S. Tsuji, Classification of Textures by a Structural Analysis, Proc. 4th IJCPR, pp. 556-568, 1978. 3. S. G. Carlton, and 0. R. Mitchell, Image Segmentation Using Texture and Gray Level, Proc. of the IEEE Conf. on Image Processing and Pattern Recognition, pp. 387-391, 1977. 4. S. Tsuji, and F. Ton&a, A structural analyzer for a class of textures, Cornpurer Graphics and Image Processing
2, 1973, 216-231.
5. L. Carlucci, A formal system for texture languages, Pattern Recognition 4, 1972, 53-72. 6. S. Y. Lu and K. S. Fu, A syntactic approach to texture analysis, Computer Graphics Processing
and
Image
and
Image
7, 1978, 303-330.
7. L. S. Davis, Computing the spatial structure of cellular textures, Computer Processing 11, 1979, 111-122. 8. M. Nagao and T. Matsuyama, Edge preserving smoothing, Computer Graphics
Graphics and Image
Processing
9, 1979, 394-407.
R. 0. Duda and P. E. Hart, Pattern Classification and Scene Analysis, 1973. 10. L. Fejes-Toth, Regular Figures, Pergamon, Elmsford, N.Y., 1964. 11. P. Brodatz, Textures, Dover, New York, 1966. 9.
pp. 233-235, Wiley. New York,