- Email: [email protected]
Fuzzy relaxation approach for inexact scene matching
Fuzzy relaxation approach for inexact scene matching Heggere S Ranganath and Laure J Chipman
A graph theoretic approach for matching imperfectly segmented images with stored scene models is presented. The segmentation errors addressed are missing objects, extra objects, mismeasured relations, m&measured attributes, split objects, and merged objects. By combining enhanced fuzzy relaxation and association graph techniques, the mthod integrates global inter-object relations and local object attributes to obtain more reliable matching. Problems of oversegmentation and undersegmentation are handled by modifying the association graph to include nodes involving merged regions and objects. Keywords: fuzzy relaxation, scene matching, mentation, association graph
overseg-
The ability to match two scenes by matching their images is needed in a variety of computer vision tasks, including object location and automatic navigation. Graph theoretic scene matching methods have the advantage that they are capable of dealing with the problems caused by differences in imaging geometries (location of sensors, viewing angles, etc.) noise, and the limitations of image segmentation algorithms better than template matching methods. In this methods, a scene is represented or modelled by a graph or a set of graphs defined on the same set of vertices, with vertices representing objects (scene components) and arcs representing relationships between them. Thus the problem of matching scenes becomes that of matching two graphs. To develop a graph-based method for matching imperfectly segmented images to scene models, one must understand the types of segmentation errors and their effect on graph representations of images. This is discussed in the next section. Details of the graphical representation of scenes which is the basis for the method presented here are then given, and previous research which has focused on inexact matching of scenes is reviewed. Our method of inexact
matching, using fuzzy relaxation applied to association and an enhancement which graphs, is presented, handles cases of oversegmentation and undersegmentation is outlined. Results of simulations are given using this approach. SEGMENTATION
Due to noise, bad imaging conditions, and limitations or procedures used, the following five types of segmentation errors are possible. Mismeasured attributes: after segmenting the image. each region is characterized by a real-valued attribute vector. In the presence of corrupting noise, attributes of regions in the image may not exactly match with attributes of the corresponding objects in the scene. This may lead to the incorrect classification of some regions. Missing objects: one or more objects may not be visible in the segmented image. This error is possible when the object is not clearly visible due to glare, shadows, or occlusion. Nodes corresponding to the missing objects will not be present in image graphs. False objects: extraneous marks or shadows may be segmented as regions which do not correspond to any scene objects. This error results in extra nodes in the image graphs. Fragmented objects or oversegmentation: this error occurs when noise-induced spurious edges fragment a single object into more than one region. Merged objects or undersegmentation: during segmentation, two or more objects may be combined to produce a single region. This type of error is possible when noise blurs edges. This first of the five types of errors does not alter the structure of the image graphs and is relatively easy to handle. The remaining types alter the structure of the image graphs and are relatively difficult to handle. GRAPH
Computer Science Department, University ville, Huntsville, AL 35899. USA P~XV recrived:
of Alabama
in Hunts-
I July 1991: revised paper received: 29 January I992
0262~8856/92/009631-10 vol IO no 9 november 1992
0
ERRORS
REPRESENTATION
OF SCENES
Over the years, several methods for representing scenes by graphs have been developed’-‘. Our approach is to model a scene by a set of graphs {G,, G7, 1992 Butterworth-Heinemann
Ltd 631
. . ., G,},
all graphs defined on the same set of vertices {V,, Vt, , . ., If,+,}. Each object or component in the scene is represented by a vertex. With vertex V,, we associate an attribute vector X(V, = [x, (Vk) x2 (V,) . . . x, (V,)]. The components of X( V,) are real-valued measurements taken on the object represented by V,. For example, nl(Vk) may be the area and xr(Vk) may be the circularity measure of Vk. Each graph describes a particular relation among the vertices. For example, GI may describe the adjacency relation, and G2 may describe the reflectance relation among the scene objects. Some obvious attributes that can be used to describe objects are length, height, area, perimeter, intensity, and texture. Attributes such as circularity measure, rectangularity measure, and length-to-height ratio are also useful. Proper selection of attributes is very important, and is generally problem dependent. Some attributes such as perimeter are quite susceptible to noise and shortcomings of boundary finding algorithms. For example, if the boundary of a blob has a break, the perimeter finding algorithm may follow both the inside and the ousdie of the blob’s boundary. The perimeter value thus obtained will be approximately twice the actual value. If possible, those attributes which are less sensitive to noise, imaging conditions, and segmentation errors must be selected. The selection of relations is also determined by the type of scenes being processed. Greene has investiand gated the usefulness of several topological topology-like relations for scene mode11ing4. For aerial images which consist of regions (fields, lakes, woods, etc.) and lines (roads, rivers, etc.), adjacency and inclusion are sensible choices. For images which consist of unconnected blobs, relations such as left-of and above are suitable. Greene’s work shows that topologylike relations such as more-textured-than and brighterthan are very useful in scene modelling. Each arc may be assigned a weight to indicate the strength of the relation. For example, the adjacency relation between V, and Vi., which have 50 pixels on their common boundary IS much stronger than the adjacency relation between Vi and V,, which have only five pixels on their common boundary. During segmentation, the probability of missing the relation between Vi and V, is less than the probability of missing the relation between V; and V,. Figure 1 shows the drawback of using binary, ‘either-or’ relations. Slight errors in segmentation can cause mismatches in relations. Figure 2 shows the benefit .of real-valued relations. When segmentaion errors or other problems cause differences between observed and expected relations, the use of real-valued relations allows the similarity between relations to be conveyed.
EVALUATION
OF EXISTING
METHODS
Several researchers have studied the scene matching problem from a graph theoretic point of view. Ullmann and others have developed conventional graph isomorphism algorithms which can only be used for matching symbolically identical graphs with no attributes5-‘. Matching methods based on isomorphism can handle problems of missing or extra objects, but not both. Association graph methods can provide more 632
Adjacent A
Not adjacent
6
OQ a
Included- In
0
Not included -in
A
A
Above
Not above
0
b
B
B 8
0
0
A
B
C
Brighter
A
- than
0 Not brighter-
B
0 B
A
than
@
@
A
B
d
Figure I. ~is~~v~nt~ges rehtions
of us&g livery,
‘either-or’,
genera1 solutions, so they are useful for scene matching applications in which exact isomorphism cannot be found”-‘“. In an association graph, a node represents a mapping between a region in the segmented image and an expected object, and an arc between two nodes represents compatibility between the two mappings. The matching can be accomplished by finding the largest fully connected subgraph in the association graph. This method has been used by Yang et al.” to deal with the problem of oversegmentation, but in that work there was no attempt to assign merit values to the association graph nodes, or to the compatibilities between nodes. So if there are missing objects or extra regions there is no assurance that the largest clique actually represents the best match. If there are two cliques of the same size, there is no criterion for choosing between them. Several improvements to the basic association graph method can be made. A weight can be assigned to each possible region-mapping (association graph node) indicating the quality of mapping. Similarly, each arc is assigned a weight. A high arc weight indicates the region-object mappings represented by the nodes connected by the arc are highly compatible. When weights are assigned to nodes and arcs of the association graph, the essence of the problem is to facilitate image And vision computing
A 0% adjacent - to0
A I% adjacent-to B B I% adjacent- to A
a
B 0%
m
adjacent-
B
A 37% adjacentto 6 B 85 % adjacent - to A
00B A
A IiT
A 10%
from
A B
8
A lB3O from B
6
brighter-than
B
A-
B
A
ways. They assume that the input image contains a true matching region for each scene object. When the image segmentation is not error free, this assumption may not be true. Also, they use probabilistic relaxation and make no reference to searching the association graph for the best clique to determine the best match in which all region-object mappings are mutually compatible. In their work, oversegmentation is handled by running the relaxation algorithm until some mapping is highly favoured, then removing that node from consideration, re-intializing, and running the relaxation process again. Probablistic relaxation has several shortcomings when applied to scene matching. Initial node weights (probabilities) are assigned with the assumption that each region maps to one and only one object. Therefore, it becomes necessary to create a ‘null’ object to handle extra regions in the segmented image. The determination of the probability that a region maps to the null object as opposed to any existing object is a puzzling problem. Also, dealing with cases where a region maps to several objects and several regions map to a single object is not simple. On the other hand, fuzzy relaxation can handle the above situations without major modifications. In the remainder of this paper, our enhancement to the assocation graph method for the purpose of matching imperfectly segmented images with stored models is presented.
I% brighter - than B
($2& A
A
OQ A
A 34 % adjacent -to B B 100% adjacent-toA
b
to
@ B
d
ENHANCED ASSOCIATION FOR SCENE MATCHING
An improved association graph method, capable of dealing with the problems of mismeasured attributes and relations, missing objects, and extra regions, is described in the steps below. Oversegmentation and undersegmentation are dealt with in the next section. Step 1 - Compute
the process of finding cliques and to determine which clique indicates the best match, considering the weights. The use of a relaxation algorithm furthers both goals. Davis has applied discrete relaxation to association graphs in a boundary matching application’“. His use of discrete relaxation is intended to reduce the size of the assocation graph by deleting nodes and arcs, thus reducing the number of cliques to be evaluated. After relaxation, Davis evaluates all cliques of the association graph by a cost formula: the cost of a clique is determined by the degree of poorness of region-object niappings, the incompatibiIity among mappings, and the penalties incurred for missing objects or regions. Our approach differs from his in two important ways: the relaxation process used here updates the node weights, hence incorporating contextual information into them; and the result of the relaxation process is used to simplify the evaluation of the merit of cliques in the association graphs. No cost formula is used; after eliminating nodes with low weights after relaxation, the node weights are simply added to determine the merit of a clique. Price et ai. “-“. and others, have applied relaxation to the scene matching problem. The approach presented here is different from their method in many vol10 no 9 november
1992
GRAPH METHOD
initial node weights
Let the model consist of K objects (O,, O1_,. . .1 Ok) and the image consist of L regions (R,, RZ, . . ., R,). The total number of region-object mappings possible is KL. The mapping weight for the pair (Ri, Oi), S(i.j)‘“‘. is computed as: q j, j)(O) =
n - E(i, j) n
where:
equation (Z), II is the number of attributes, and IV, is a weighting constant used for scaling and indicating the relative importance of the kth attribute. Some researchers set S(i, j)(O) to 1 or 0 indicating whether R, can map to 0; or not based on the similarity between their attribute vectors. For scene matching applications, assigning a value between 0 and 1 is more meaningful. Computing the similarity between two patterns as shown in equations (1) and (2) is common in pattern recognition. If S(i, j)‘“’ is less than a predetermined threshold T, it is assumed that R, cannot map to 0,. Each possible mapping is represented by a node in In
633
the association graph. If R, can map to Oj, node (i, j) is created in the association graph-and S(i, i)“’ becomes its initial weight. Step 2 - Compute coefficients
Step 3 - Update node weight using suitable updating relaxation rule
The node weight updating is different from the most formula. Many variations rules exist 19*20.The most
j,
[
or
rule described in this section widely used fuzzy relaxation of fuzzy relaxation updating widely used formula is:
my(S(h, k)(” C(i, j; h, k))] (3)
After updating ail node weights, they are normalized. Normalization can be accomplished by forcing the sum of node weights to remain constant as shown below: S(i, qi,
j)““’
j)““’
I.
K
i=I
)=I
c c qi, I)@)
=r
(4) i &(i, r=l ,=I
i)““”
These rules works as desired only if each object has its corresponding region in the input image, and the distortion is mainly structure preserving. A node which is intially weak, but has proper relations with other strong nodes, is strengthened by these rules. Similarly, a node which is out of context (has inconsistent relations with other nodes) is weakened by the updating rule. Initial node weight has no control over the final outcome when the context is strong. Since segmentation errors alter the structure and relations to some extent, the updating rule must integrate initiat node weight and the relational consistency between nodes to achieve better mapping. A rule with the desired property is presented by the following equation: S(i,
j)('+')=
In equation 634
Result,
Result,
arc weights or compatibility
A weight proportional to the compatibility between mappings (i, i) and (h, k) is assigned to the arc connecting the nodes. The value of this arc weight C(i,i; h, k), is based on similarity of relations between regions Ri and Rt,, and objects Oj and Ok. If binary relations are used, it may be computed as the ratio of the number of relations between Ri and Rh which match with the relations between Oj and Ok to the total number of relations. If real-valued relations are used, the arc weights are computed in a similar fashion to the node weights. In this paper, we define the value of C(i, j; i, k) to be 0, so that we do not consider the mappings of one region to two different objects to be compatible. Likewise C(i, i; h, j) is defined as 0.
St6 i) fr+‘) =;
Original association graphs
aS(i, j)(O) +
(5) CYis in the range [O, a]. An alternate
o/ ‘.’
B,c
Figure 3. Results of applying the relaxation updating rule uf equation (5) to two graphs with values on nodes (B, b) and (B, c) switched. With cy = 0, the reversal of node weights has no effect on the outcome. With CY= 0.15, the initial node weights affect the result
form, with (Yin the range [O, 11, is given by: S(i, i)““’
= aS(i, i)@’ +
(I--a)[ii,jmXbx(S(h, k)(‘)C(i,j;h,k)l)l(6) k=l When cy is zero, equation (6) reduces to equation (3), the more commonly used updating rule”“. It is instructive to compare the new node weight updating rule in equations (5) or (6) with the previously used rule in equation (3). Figure 3 illustrates the advantage of an updating rule that allows initial node weight to affect the outcome of the relaxation process. By changing the value of cy, the relative importance of local node properties and global inter-node relations can be controlled. Figure 4 demonstrates that a different value of a can control the outcome of the relaxation process by giving a different relative importance to initial node weights versus relational compatibilities. Step 4 - Test for relaxation
termination
condition
The number of iterations needed for the fuzzy relaxation algorithm to converge depends on initial node and arc weights, and on the radius of the association graph. In fuzzy relaxation, the process changes the weights of each node during each iteration. When scaled so that the sum of node weights is constant, as more iterations are run, the differences between previous and current node weights become smaller and smaller.
Result, a=0.15
Result, a 0.03 q
Figure 4. Effect of dtfferent values of cr on the outcome of the relaxation process. At a: = 0. I5 the higher importance of the initial node weights allows node ( B,c) to be favoured. At cy = 0.03, the arc weights are more in~p~rtant, so node (B, bl is ,fav~L~red image and vision computing
So,
the
termination
condition
is that
1S(i, j)(r+rr-
S(i, j)“’ I<6, where 6 is a predefined small number. However. this can be wasteful, since the order of the weights may not change after the first few iterations. If the termination condition is not satisfied, control is passed to step 3. Step 5 - Evaluation of the result The result of the relaxation process is a set of updated weights on nodes of the association graph. Since each updated weight now reflects the contextual support for its mapping, we may evaluate the result by first thresholding the node weights to discard nodes with low weights. and then finding the cliques in the association graph. The clique with the remaining highest sum of node weights should represent the best mapping of observed regions to model objects. There are several algorithms in the literature for which can be paralclique finding” -‘j. An algorithm lelized is the following: Assume all nodes are in the same clique. Check pairs of nodes; if there is no arc between them, then they must be in separate cliques. Split the potential clique into two parts; one with vertex x and not y. and the other with vertex y and not X. Each part contains all of the other vertices of the original potential clique. Recursively perform (2) on each potential clique. When no more splitting is needed, all the remaining potential cliques are actual cliques. The maximal cliques arc the ones that are no subsets of any other cliques. PROCEDURE FOR MERGED REGIONS
HANDLING
SPLIT
AND
Oversegmentation can be handled by mapping multiple regions to one object. Our approach is to create a merged region consisting of these multiple regions. The merged region is then considered as any other region, having a set of attribute values and relations to other regions. Likewise, undersegmentation amounts to mapping one region to multiple objects. We create merged objects to handle this case. The procedure is as follows: 1 Compute the initial node weights for the regions and objects. 2 Determine candidate regions/objects for which we will attempt to merge corresponding objects/regions. If all of a region’s node weights are below a threshold, this indicates that none of the objects map well to that region. If this is the case, consider pairs of objects. If a pair of objects are adjacent (or meet some ‘closeness’ criterion), have similar region-wide attribute values, and have region-wide attribute values that match well with the region having all low node weights, and the merged object does not already exist, then create a merged object, and compute its node weight with the region. If the merged object already exists, simply compute its node weight with the region. Similarly, check node weights for each object. If an object has all low node weights, attempt to create a merged region which will map well to the object.
vol 10 no 9 novemher
1992
2a Process for creating a merged object or region: Calculate attribute values for the merged object or region. For region-wide attributes. the new values are weighted averages of the values for the two objects being merged. For area, the new value is the sum of the areas of the two objects. Add a row and column for the new object to the matrices representing the relations. and estimate values of the new object’s relations with other objects. Angle relations are estimated by simply averaging the angle relations of the two constituent objects. The real-valued adjacency relation (the percentage of an object’s pixels that border on another object) is computed by determining the perimeter of the merged object (the sum of the perimeters minus twice the length of their Then, the percentage ot shared boundary). border pixels with other objects is calculated based on the new perimeter. Since the new object is simply added to the list of objects, it can also be considered for merging with other objects, thus allowing for merges of three or more objects. 2b Compute node weight of the merged object with the region that prompted the attempt to find merged objects. Since the node weights are represented in a matrix with rows corresponding to objects and columns corresponding to regions. if the merged object is being newly added to the list of objects, all the other node weights for the merged object should be initialized to zero. After all plausible merged objects/regions have been created and their initial node weights computed, find the values of the arcs in the association graph. Arcs are not allowed between nodes that include the same object or the same region; e.g. there is no arc between node (RI, (01, 02)) and (R2, 02). Arcs from merged objects/regions are computed in the same way as other arcs, since the estimated relation values for the merged objects are included in the relation matrices. just as for any other objects. Perform the relaxation process on the association graph which includes nodes involving merged objects or regions. A slight change in the relaxation updating rule is necessary. Since two nodes including the same region are defined as incompatible. nodes involving regions that belong to merge nodes have fewer terms possible in the sum in the updating rule, which is an unfair disadvantage. Nodes including only a region that is not involved in a merge node have possible terms in the sum from every region. To make up for missing terms. a merge compensation factor h, is included in the updating rule. The value of S(i. j)“) is then counted h, times in the sum. So, the new updating rule is given by: S(i, j)‘l’+” = tuS(i, j)“‘) + (I-+[
h,S(i, ;)“I +
,$ r+r~(S(h, I, I [ it, An example
k)“’ c(i, j; h, k))
of this is shown in Table
111
1. Assuming
(7)
there
635
Table 1. Missing terms due to merged regions S(i, i)
Regions
affecting
sum
b,
1
2
3
SC,i)
x
x
x
S(2, i) S(3> i) S((l, 21, i)
x x
x x
x
(1.
X
X
x
x
2) 2 2 1 3
are four regions, 1, 2, 3 and (1, 2), the table shows that for nodes which map region 1 (or 2) to some object, there can only be terms in the sum in the updating rule from nodes involving regions 1, 2, and 3. However, since nodes involving region 3 could be compatible with nodes involving any of the regions, they have four possible terms. Nodes mapping region (1, 2) to any object can have terms from only region 3 and region (1, 2). The factors bj compensate for these missing terms by counting the value of S(i, j) multiple times. SIMULATION
RESULTS
The fuzzy relaxation process tested on hypothetical example cases of imperfect segmentation
described above was scenes. The following were tested:
scene is missing an object and Case 1 The observed also contains a spurious region. Case 2 The observed scene contains an oversegmented region. Case 3 The observed scene contains an undersegmented region. Case 4 The observed scene contains oversegmentation and undersegmentation errors, so that the desired result is to have a set of regions map to a set of objects as a whole. Case 5 The observed scene contains an oversegmented region that has been split three ways.
(i 1
b
(ii)
(iii)
(IV)
Figure 5. Scene models and inexact segmentations used in examples. (a) scene model: observed scene, case I: missing object and extra region. 5, (b) scene model: observed scenes, (i) case 2, split region, (ii) case 3, merged region, (iii) case 4, split and merged regions, (iv) case 5, three-way split region
Table 2. Simulation
results,
Case 1
Attributes of intensity, area, and circularity were used to describe the objects in Case 1. Intensity, area, and texture were used for Cases 2-5. The value of Wk was chosen so that each element of the sum shown in equation (2) would be in the range [0, 11. The node weights were then determined by the formula given in equation (1). This results in node weights between 0 and 1, with 1 representing a perfect match and 0 representing the worst match possible, given the attributes as measured. For Case 1, the real-valued relation of ‘angle’ is used, defined as the angle that a line between the centroids of the two regions makes with the horizontal. For Cases 2-5, relations of ‘angle’ and ‘adjacency’ were used, with adjacency defined as the percentage of a region’s boundary that is adjacent to the other region. The scene models and inexact segmentations are shown in Figure 5. The results of the relaxation procedure are shown in Tables 2-6. Table 2 represents the case in which there is an extra region and a missing object, but no oversegmented objects. The desired result is to have no region map to Object 3, and to have Region 5 map to no object. In the final result, Region 3 is the best mapping for Object 3, but it is a better mapping for Object 4, which is the 636
image and vision computing
correct
n~apping.
Because
port. all the node weights
of a lack of contextual supfor nodes involving Region 5
become low. For Cases 1 and 4, the intensity value for Object 2 was set to 0.1. For Cases 2 and 3, in which the hypothetical segmentation has merged Objects 1 and 2, the intensity for Object 2 was set to 0.5, a more plausible value. since ihe intensities of Objects 1 and 2 would probably be similar if the two objects were merged. The results for Case 2, in which Object I is split into Regions I and 5, are shown in Table 3. The process correctly mapped Region 6, which includes Regions I and 5, to Object 1. Results for LY= 0.2 and (Y=0.25 are shown, Table 4 shows the results for Case 3, in which Objects 1 and 2 are merged into Region 1. Again, the process correctly mapped Region I to Object 5, which includes Objects I and 2. The mapping of Region 1 to Object 1 also attained a high value, since Region I matches quite well to Object 1 alone, and its centroid is close to the centroid of Object 1, which leads to high arc values for the node (Rl, 01). Table 5 contains the results for Case 4, in which Objects I and 2 are merged into Region 1, and Object 1 is split into Regions 1 and 2. The mapping desired in this case is Region 5 (Regions 1 and 2) to Object 5 (Objects 1 and 2). At LY= 0.2, the mapping (RS, 01) is wrongly favoured. At cy = 0.25, the correct mapping (RS. OS) prevails. Table 6 contains the results for Case 5, in which Object 1 has been split into three regions, Regions I, 5,
Table 4. Simulation
results,
Table 5. Simulation
results, Case 4 __- - . _-
iwmsity
krca
Case 3
Texture
REtto!
1.83
0.7
0.8
*wte
0.23
0.5
0.5
1
1.82
t.0
0.0
0.25
0.1
0.2
11
D
37 355
2.06
0.63
0.77
21 217
0 335
3i 175 155
0
2.02
0.M
0.1
41 190 150 2‘5
2.22
0.9
0.05
51
0.28
0.02
0.35
0
‘I 0
Initiak n&e rei*ts .. .. . .. .. ...__....__
10
D
‘I
0
0
2) 170
0
Sl 192
265
ull,o2)
0.53
(Rl,o3f
0.53
(R1,041
0.28
0 345
5
0.97
G?2,01)
0.57
(12.02)
0.36
'
0.88
CR2,01)
a.34
m2.m)
0.00
~RS,Ol)
0.33
fRS.02)
0.77
wt3.03)
0.21
'(13,cA)
O.Pl
w3,m)
0.00
Derired mappings --
-
12 2.5 0
0
Relstion
Adjscency
2
3
4
5
0.09
0
12
3
0
*
II0
0
0
0
0
0
210
0
0.2*
3) 0
0
0
0.2 0
31 0
0.6
0
410
0
1
0
0
510
0
0
0
0
r,pra = 0.25
I.009
1.004
1.010
1.006
0.948
0.943
0.556
0.575
0.957
0.953
llergc Dbject m=Ol*O* *
350
0 105
Pinat mle weights . . . ...... .. ..-....
0.95
0
3
21 0.24 0
n,phs = 0.2 ul1,00
2
65 165
Adjacency Relation I
Ile,rti‘m
lnitist weight too 10" to consider
-
and 6. This example illustrates the way that more than two regions may be merged into one. Regions are merged pairwise, and the new regions are added to the list of regions, so that they are considered for future merges. In this example, Region 7 is the merge of Regions 1 and 5; Region 8 is the merge of Regions 6 and 1, Region 9 is the merge of Regions 6 and 5: and Region 10 is the merge of Regions 8 (6 and 1) and 5. Of all nodes mapping to Object 1, the node (RIO. 01) has the highest weight after relaxation. In each of the examples with (Y=0.25, the clique with the highest node sum after relaxation represents the correct mapping. The scene in Figure 6 is an example of an actual scene on which this approach was tested. This is an image of a space shuttle simulator panel, annotated with object numbers for clarity. The image was segmented by the segmentation procedure provided with a commercially available image processor (Perceptics 9200e). The segmented image contained a total of 50 regions, one missing object (number 7), and several oversegmented objects (numbers 1, 3, 11, and 13). In cases where there are many extra regions expected, it is better to take the objects of the stored representation
Table 6. Simulation
results,
Initial de weights _..._......._...._..
Case 5 Find node weights ..........._..._.. RLpha = 0.2
Upha
= 0.25
0.656
0.674
0.920
0.920
0.721
0.733
0.945
0.940
0.719
0.733
0.91
0.942
0.941
(R5.01) 0.73
0.621
0.634
(11,011 0.80
Table 6 (continued)
1
(Pl,OZ) 0.57 (R1,03) 0.39 (R1,04) 0.45 (R2.01) 0.43 W2.02)
0.90
(R2,03) 0.33 (R2.04) 0.87 (13.01) 0.57 (R3.02) 0.22
l(R3,03)
0.88
as the ‘units’ in relaxation and regions as ‘classes’. This is to avoid extra regions making spurious contributions to node weights. So, the updating rule used on this example is as follows:
(R3.04) 0.34 (R4.01) 0.33 (R4.02) 0.91
S(i,
j)(“+ ’ ) zz
aS(i,
j)(“) +
(R4,03) 0.27 VR4.04)
(R5.02) 0.69 (R5,M)
(1-a)[‘: [l&(S(h, k)(r)c(i,j;h,k))]] (8) m k=l
(R5.03) 0.31 0.57
(R6.01) 0.70
0.609
0.621
0.940
0.934
0.930
0.923
0.871
0.859
1.179
1.140
(R6.02) 0.69 (R6.03) 0.33 (R6,OL) 0.57 Merge region RI=RS+Rl (R7.01) 0.86 CRT.021 0.00 (R7.03) 0.00 (R7.04) 0.00 Merge region RB=R6+Rl (R8.01) 0.84 (R8,OZ) 0.00 (R8.03) 0.00 (R8.04) 0.00 Merge region RP=Rb+R5 (R9.00
0.75
(R9.02) 0.00 (R9,03) 0.00 (R9.04) 0.00 Merge region RlO=R8+R5 '(R10.01) 0.90 CRlO,OZ) 0.00 (R10,03) 0.00 (Rl0,04) 0.00 l
638
Desired
mappings
- Initial weight too low to consider
Figure 6. Space shuttle simulator scene
panel used as example
image and vision computing
also be used in applications in which classes,
a set of units taking
into
unrelated
needs
account
to machine
to be mapped both
local
vision
to a set of
and contextual
information.
REFERENCES 1
Pavlidis,
T
Structural
Pattern
New
(1977)
Springer-Verlag. 2
‘1’ ‘Structural
Pavlidis, &rammers’,
in
Picforial New
3
obtained
by
It!fhrtnatiotz
is
regions
analogous
to
and objects
reversed.
compensation required
factor
since
h,
(7)
with
the
roles
of
In this case, an analogous
for
merged
undersegmentation
is
objects
Trans.
Greene,
The
of the
did not occur in this
7. The
relaxation
figure
is annotated
around
the
objects
(I. 3. I I. and 13).
regions
~~versegmented
regions
that
are shown
with
thick
mapped
to
Every
object
in
more
corresponding included
regions.
In
oversegmented
to printing
in the winning
each case, the merged to the expected
7
was not
cases,
on the control
merged
region
obJcct than
some
represented
better
did any region
8
extra but
9
We
have
association
graph.
of mismeasured segmentation.
local
match IO
for
.
scene
applied
to an
and relations, over-
between
mination value for
objects
1979)
of CY. Other or
object
for
merging
One
possibility
is to perform
first
assuming
no
then to determine the results
is another
make
the method
In this for
or
graph,
the primitives
for merging process.
in real-time
process. using
It should
relaxation
can be used
in other
may be line
vol 10 no 9 novemher
1902
It could
Artif:
Vol
Itirrll..
6
structures Vol
f_cll..
isomor-
and maximal 4 No
4 (1976)
feature
Proc.
matching
through
Vol
SPIE.
I81
(April
using
associa-
IEEE
matching
Tratl.s.
using
PAMI.
relaxation
Vol
I
No
I
0 D and Berthod,
and reducing optimization Vol of
ambiguit)
M ‘Improving
approach’.
images
using
label-
IEEE
I No 4 (19X1) pp 112-3’3 0 D and Price, K E ‘Semantic
aerial
consis-
in stochastic
stochastic
Tratw. descriplabeling’,
IEEE
Trrrtts. PA MI, Vol 3 No 6 (1YXl ) pp 633-642
K E: ‘Hierarchical matching using relaxaC’orupill. Visioti. Graph. R 1ttiag’e Prow.s.s.,
Vol IX
3D images to models
ltmgc R Vi.sion C‘omprr~. , Vol 7 No 3
Price, tion’.
the
to an associa-
segments.
for testing
isomorphism’.
pp 60-72
An
tion
is need to
in which
Procrxs.
L S ‘Shape
PAM/,
as the primitives
applications
for
iso-
pp 3 l-33
pp 135-143
Faugeras,
situations.
applied
or curve
ing:
and
that
?
pp I40-I40
tency
implemen-
be noted
subgraph
I ( 1976)
program
graphs
relational
cliques’.
Faugeras,
procedure
much research
for
7.3 No
assembly’.
R C ‘Robust
(1979)
by examining
Parallel
we have used regions
the matching
Itl/it.
techniques‘.
be investigated.
under-segmentation,,
candidates
useful
should
No
pp 129-306
matching
Davis,
candidates
the relaxation
area in which
paper.
same procedure, tion
over-
of the relaxation
tation
determining
line
controlled
tion graphs’.
support
into the automatic deteron node weights and the best
methods
‘9
Yang, B, Snyder, W E and Bilbro, G L ‘Matching
is needed
of thresholds
Vol
- A heuristic
directed
o\crsegmcnted
observed
of good contextual
I7
for graph
Vol
ht.,
Barrow, H G and Burstall, R M ‘Subgraph
Belles,
of good
and
Vol
c’ottmutt. A (‘M, Vol 23 No I ( 196-I) Ambler, A P, Barrow, H G, Brown, C M, Burstall, R M and Popplestone, R .J ‘A versatile system for
(1989)
research
region
Unger, S H ‘GIT of
algorithm
algorithm
J. A CM,
maximal
application,
the importance
model
efficient
1. ACM,
pp x3-x-I
under-
and
iso-
IO
J R ‘An
Ullmann,
cliques’.
missing
the mappings. Further
pp 105-l
phism.
used in the relaxation
to the scene matching
between
II
the problems
addresses
rule
the importance
and
to inexact rule
and
updating
a balance
matches
regions
paper attributes
regions. The
is well-suited
as it allows
approach
relaxation
This
object
extra
objects. process
a new
a fuzzy
for
C C ‘An
for graph isomorphism’.
No 3 (1975)
using
I
.l. A C.44. Vol 20 No
D G and Gottleib,
computer
prcaented
procedure
graphs’.
Mittal, H B ‘A fast backtrack iaomorphism’. Itlfo. Proccxs.
pairs
CONCLUSION matching.
No
(19X9)
backtrack
of directed
morphism’,
in
by itself.
I3
I ( 1970)
(l9X8)
panels
regions.
recognition’. Vol
& C)~hcrn..
in Huntsville
A T ‘A
Corneil, No
matched. Those that were matched to merged regions con-
were or
of Alabama
Berztiss,
algorithm
outlines
that
Sysr. Man
error-correcting
pattern
3 (197.1)
was correctly
oversegmented misting of two were
process
syntactic
R Suw ktzo~~lc~dgc rc~~~rcsett~utiott for \,i.siott .s):stem.s. PhD dissertation, Univer-
morphism 6
results
Figure
sity 5
not
example.
for
( 198.1) pp 38-62 4
equation
graph
Springer-Verlag.
Systems,
( 1980) pp 86-103 Tsai, W H and Fu, K S ‘Subgraph
csperr This
and
and K S Fu teds),
York
IEEE
merging
Recopifion,
descriptions
K Chang
S
isomorphisms
Figure 7. Object assignments o~,c~rsegrtletlrc,ri regions
York
34 No
I (1086)
pp 66-75
Price, K E ‘Relaxation comparison’.
IiCl?E
matching
Trrrt7.s. PAMI,
techniques Vol
-
A
7 No
5
639
19
20
640
(1985) 617423 Rosenfetd, A, Hummel, R A and Sucker, S W ‘Scene labeling by relaxation operations’, IEEE Trans. Syst. Man & Cybern., Vol6 No 6 (1976) pp 420-423 Hummel, R A and Rosenfeld, A ‘Relaxation processes for scene labeling’, IEEE Trans. Syst. Man & Cybern., Vol 8 No 10 (1978) pp 76.5-768
21
22
23
Balas, E and Yu, C ‘Finding a maximum clique in an arbitrary graph’, SIAM J. Cornput., Vol 15 No 4 (1986) pp 1054-1068 Bron, C and Kerbosch, J, ‘Finding all cliques of an undirected graph’, Common. ACM., Vol 16 No 9 (1973) pp 575-577 Lau, H T Algorithms on Graphs, TAB Books, Blue Ridge Summit, PA (1989) pp 59-69
image and vision computing
A graph theoretic approach for matching imperfectly segmented images with stored scene models is presented. The segmentation errors addressed are missing objects, extra objects, mismeasured relations, m&measured attributes, split objects, and merged objects. By combining enhanced fuzzy relaxation and association graph techniques, the mthod integrates global inter-object relations and local object attributes to obtain more reliable matching. Problems of oversegmentation and undersegmentation are handled by modifying the association graph to include nodes involving merged regions and objects. Keywords: fuzzy relaxation, scene matching, mentation, association graph
overseg-
The ability to match two scenes by matching their images is needed in a variety of computer vision tasks, including object location and automatic navigation. Graph theoretic scene matching methods have the advantage that they are capable of dealing with the problems caused by differences in imaging geometries (location of sensors, viewing angles, etc.) noise, and the limitations of image segmentation algorithms better than template matching methods. In this methods, a scene is represented or modelled by a graph or a set of graphs defined on the same set of vertices, with vertices representing objects (scene components) and arcs representing relationships between them. Thus the problem of matching scenes becomes that of matching two graphs. To develop a graph-based method for matching imperfectly segmented images to scene models, one must understand the types of segmentation errors and their effect on graph representations of images. This is discussed in the next section. Details of the graphical representation of scenes which is the basis for the method presented here are then given, and previous research which has focused on inexact matching of scenes is reviewed. Our method of inexact
matching, using fuzzy relaxation applied to association and an enhancement which graphs, is presented, handles cases of oversegmentation and undersegmentation is outlined. Results of simulations are given using this approach. SEGMENTATION
Due to noise, bad imaging conditions, and limitations or procedures used, the following five types of segmentation errors are possible. Mismeasured attributes: after segmenting the image. each region is characterized by a real-valued attribute vector. In the presence of corrupting noise, attributes of regions in the image may not exactly match with attributes of the corresponding objects in the scene. This may lead to the incorrect classification of some regions. Missing objects: one or more objects may not be visible in the segmented image. This error is possible when the object is not clearly visible due to glare, shadows, or occlusion. Nodes corresponding to the missing objects will not be present in image graphs. False objects: extraneous marks or shadows may be segmented as regions which do not correspond to any scene objects. This error results in extra nodes in the image graphs. Fragmented objects or oversegmentation: this error occurs when noise-induced spurious edges fragment a single object into more than one region. Merged objects or undersegmentation: during segmentation, two or more objects may be combined to produce a single region. This type of error is possible when noise blurs edges. This first of the five types of errors does not alter the structure of the image graphs and is relatively easy to handle. The remaining types alter the structure of the image graphs and are relatively difficult to handle. GRAPH
Computer Science Department, University ville, Huntsville, AL 35899. USA P~XV recrived:
of Alabama
in Hunts-
I July 1991: revised paper received: 29 January I992
0262~8856/92/009631-10 vol IO no 9 november 1992
0
ERRORS
REPRESENTATION
OF SCENES
Over the years, several methods for representing scenes by graphs have been developed’-‘. Our approach is to model a scene by a set of graphs {G,, G7, 1992 Butterworth-Heinemann
Ltd 631
. . ., G,},
all graphs defined on the same set of vertices {V,, Vt, , . ., If,+,}. Each object or component in the scene is represented by a vertex. With vertex V,, we associate an attribute vector X(V, = [x, (Vk) x2 (V,) . . . x, (V,)]. The components of X( V,) are real-valued measurements taken on the object represented by V,. For example, nl(Vk) may be the area and xr(Vk) may be the circularity measure of Vk. Each graph describes a particular relation among the vertices. For example, GI may describe the adjacency relation, and G2 may describe the reflectance relation among the scene objects. Some obvious attributes that can be used to describe objects are length, height, area, perimeter, intensity, and texture. Attributes such as circularity measure, rectangularity measure, and length-to-height ratio are also useful. Proper selection of attributes is very important, and is generally problem dependent. Some attributes such as perimeter are quite susceptible to noise and shortcomings of boundary finding algorithms. For example, if the boundary of a blob has a break, the perimeter finding algorithm may follow both the inside and the ousdie of the blob’s boundary. The perimeter value thus obtained will be approximately twice the actual value. If possible, those attributes which are less sensitive to noise, imaging conditions, and segmentation errors must be selected. The selection of relations is also determined by the type of scenes being processed. Greene has investiand gated the usefulness of several topological topology-like relations for scene mode11ing4. For aerial images which consist of regions (fields, lakes, woods, etc.) and lines (roads, rivers, etc.), adjacency and inclusion are sensible choices. For images which consist of unconnected blobs, relations such as left-of and above are suitable. Greene’s work shows that topologylike relations such as more-textured-than and brighterthan are very useful in scene modelling. Each arc may be assigned a weight to indicate the strength of the relation. For example, the adjacency relation between V, and Vi., which have 50 pixels on their common boundary IS much stronger than the adjacency relation between Vi and V,, which have only five pixels on their common boundary. During segmentation, the probability of missing the relation between Vi and V, is less than the probability of missing the relation between V; and V,. Figure 1 shows the drawback of using binary, ‘either-or’ relations. Slight errors in segmentation can cause mismatches in relations. Figure 2 shows the benefit .of real-valued relations. When segmentaion errors or other problems cause differences between observed and expected relations, the use of real-valued relations allows the similarity between relations to be conveyed.
EVALUATION
OF EXISTING
METHODS
Several researchers have studied the scene matching problem from a graph theoretic point of view. Ullmann and others have developed conventional graph isomorphism algorithms which can only be used for matching symbolically identical graphs with no attributes5-‘. Matching methods based on isomorphism can handle problems of missing or extra objects, but not both. Association graph methods can provide more 632
Adjacent A
Not adjacent
6
OQ a
Included- In
0
Not included -in
A
A
Above
Not above
0
b
B
B 8
0
0
A
B
C
Brighter
A
- than
0 Not brighter-
B
0 B
A
than
@
@
A
B
d
Figure I. ~is~~v~nt~ges rehtions
of us&g livery,
‘either-or’,
genera1 solutions, so they are useful for scene matching applications in which exact isomorphism cannot be found”-‘“. In an association graph, a node represents a mapping between a region in the segmented image and an expected object, and an arc between two nodes represents compatibility between the two mappings. The matching can be accomplished by finding the largest fully connected subgraph in the association graph. This method has been used by Yang et al.” to deal with the problem of oversegmentation, but in that work there was no attempt to assign merit values to the association graph nodes, or to the compatibilities between nodes. So if there are missing objects or extra regions there is no assurance that the largest clique actually represents the best match. If there are two cliques of the same size, there is no criterion for choosing between them. Several improvements to the basic association graph method can be made. A weight can be assigned to each possible region-mapping (association graph node) indicating the quality of mapping. Similarly, each arc is assigned a weight. A high arc weight indicates the region-object mappings represented by the nodes connected by the arc are highly compatible. When weights are assigned to nodes and arcs of the association graph, the essence of the problem is to facilitate image And vision computing
A 0% adjacent - to0
A I% adjacent-to B B I% adjacent- to A
a
B 0%
m
adjacent-
B
A 37% adjacentto 6 B 85 % adjacent - to A
00B A
A IiT
A 10%
from
A B
8
A lB3O from B
6
brighter-than
B
A-
B
A
ways. They assume that the input image contains a true matching region for each scene object. When the image segmentation is not error free, this assumption may not be true. Also, they use probabilistic relaxation and make no reference to searching the association graph for the best clique to determine the best match in which all region-object mappings are mutually compatible. In their work, oversegmentation is handled by running the relaxation algorithm until some mapping is highly favoured, then removing that node from consideration, re-intializing, and running the relaxation process again. Probablistic relaxation has several shortcomings when applied to scene matching. Initial node weights (probabilities) are assigned with the assumption that each region maps to one and only one object. Therefore, it becomes necessary to create a ‘null’ object to handle extra regions in the segmented image. The determination of the probability that a region maps to the null object as opposed to any existing object is a puzzling problem. Also, dealing with cases where a region maps to several objects and several regions map to a single object is not simple. On the other hand, fuzzy relaxation can handle the above situations without major modifications. In the remainder of this paper, our enhancement to the assocation graph method for the purpose of matching imperfectly segmented images with stored models is presented.
I% brighter - than B
($2& A
A
OQ A
A 34 % adjacent -to B B 100% adjacent-toA
b
to
@ B
d
ENHANCED ASSOCIATION FOR SCENE MATCHING
An improved association graph method, capable of dealing with the problems of mismeasured attributes and relations, missing objects, and extra regions, is described in the steps below. Oversegmentation and undersegmentation are dealt with in the next section. Step 1 - Compute
the process of finding cliques and to determine which clique indicates the best match, considering the weights. The use of a relaxation algorithm furthers both goals. Davis has applied discrete relaxation to association graphs in a boundary matching application’“. His use of discrete relaxation is intended to reduce the size of the assocation graph by deleting nodes and arcs, thus reducing the number of cliques to be evaluated. After relaxation, Davis evaluates all cliques of the association graph by a cost formula: the cost of a clique is determined by the degree of poorness of region-object niappings, the incompatibiIity among mappings, and the penalties incurred for missing objects or regions. Our approach differs from his in two important ways: the relaxation process used here updates the node weights, hence incorporating contextual information into them; and the result of the relaxation process is used to simplify the evaluation of the merit of cliques in the association graphs. No cost formula is used; after eliminating nodes with low weights after relaxation, the node weights are simply added to determine the merit of a clique. Price et ai. “-“. and others, have applied relaxation to the scene matching problem. The approach presented here is different from their method in many vol10 no 9 november
1992
GRAPH METHOD
initial node weights
Let the model consist of K objects (O,, O1_,. . .1 Ok) and the image consist of L regions (R,, RZ, . . ., R,). The total number of region-object mappings possible is KL. The mapping weight for the pair (Ri, Oi), S(i.j)‘“‘. is computed as: q j, j)(O) =
n - E(i, j) n
where:
equation (Z), II is the number of attributes, and IV, is a weighting constant used for scaling and indicating the relative importance of the kth attribute. Some researchers set S(i, j)(O) to 1 or 0 indicating whether R, can map to 0; or not based on the similarity between their attribute vectors. For scene matching applications, assigning a value between 0 and 1 is more meaningful. Computing the similarity between two patterns as shown in equations (1) and (2) is common in pattern recognition. If S(i, j)‘“’ is less than a predetermined threshold T, it is assumed that R, cannot map to 0,. Each possible mapping is represented by a node in In
633
the association graph. If R, can map to Oj, node (i, j) is created in the association graph-and S(i, i)“’ becomes its initial weight. Step 2 - Compute coefficients
Step 3 - Update node weight using suitable updating relaxation rule
The node weight updating is different from the most formula. Many variations rules exist 19*20.The most
j,
[
or
rule described in this section widely used fuzzy relaxation of fuzzy relaxation updating widely used formula is:
my(S(h, k)(” C(i, j; h, k))] (3)
After updating ail node weights, they are normalized. Normalization can be accomplished by forcing the sum of node weights to remain constant as shown below: S(i, qi,
j)““’
j)““’
I.
K
i=I
)=I
c c qi, I)@)
=r
(4) i &(i, r=l ,=I
i)““”
These rules works as desired only if each object has its corresponding region in the input image, and the distortion is mainly structure preserving. A node which is intially weak, but has proper relations with other strong nodes, is strengthened by these rules. Similarly, a node which is out of context (has inconsistent relations with other nodes) is weakened by the updating rule. Initial node weight has no control over the final outcome when the context is strong. Since segmentation errors alter the structure and relations to some extent, the updating rule must integrate initiat node weight and the relational consistency between nodes to achieve better mapping. A rule with the desired property is presented by the following equation: S(i,
j)('+')=
In equation 634
Result,
Result,
arc weights or compatibility
A weight proportional to the compatibility between mappings (i, i) and (h, k) is assigned to the arc connecting the nodes. The value of this arc weight C(i,i; h, k), is based on similarity of relations between regions Ri and Rt,, and objects Oj and Ok. If binary relations are used, it may be computed as the ratio of the number of relations between Ri and Rh which match with the relations between Oj and Ok to the total number of relations. If real-valued relations are used, the arc weights are computed in a similar fashion to the node weights. In this paper, we define the value of C(i, j; i, k) to be 0, so that we do not consider the mappings of one region to two different objects to be compatible. Likewise C(i, i; h, j) is defined as 0.
St6 i) fr+‘) =;
Original association graphs
aS(i, j)(O) +
(5) CYis in the range [O, a]. An alternate
o/ ‘.’
B,c
Figure 3. Results of applying the relaxation updating rule uf equation (5) to two graphs with values on nodes (B, b) and (B, c) switched. With cy = 0, the reversal of node weights has no effect on the outcome. With CY= 0.15, the initial node weights affect the result
form, with (Yin the range [O, 11, is given by: S(i, i)““’
= aS(i, i)@’ +
(I--a)[ii,jmXbx(S(h, k)(‘)C(i,j;h,k)l)l(6) k=l When cy is zero, equation (6) reduces to equation (3), the more commonly used updating rule”“. It is instructive to compare the new node weight updating rule in equations (5) or (6) with the previously used rule in equation (3). Figure 3 illustrates the advantage of an updating rule that allows initial node weight to affect the outcome of the relaxation process. By changing the value of cy, the relative importance of local node properties and global inter-node relations can be controlled. Figure 4 demonstrates that a different value of a can control the outcome of the relaxation process by giving a different relative importance to initial node weights versus relational compatibilities. Step 4 - Test for relaxation
termination
condition
The number of iterations needed for the fuzzy relaxation algorithm to converge depends on initial node and arc weights, and on the radius of the association graph. In fuzzy relaxation, the process changes the weights of each node during each iteration. When scaled so that the sum of node weights is constant, as more iterations are run, the differences between previous and current node weights become smaller and smaller.
Result, a=0.15
Result, a 0.03 q
Figure 4. Effect of dtfferent values of cr on the outcome of the relaxation process. At a: = 0. I5 the higher importance of the initial node weights allows node ( B,c) to be favoured. At cy = 0.03, the arc weights are more in~p~rtant, so node (B, bl is ,fav~L~red image and vision computing
So,
the
termination
condition
is that
1S(i, j)(r+rr-
S(i, j)“’ I<6, where 6 is a predefined small number. However. this can be wasteful, since the order of the weights may not change after the first few iterations. If the termination condition is not satisfied, control is passed to step 3. Step 5 - Evaluation of the result The result of the relaxation process is a set of updated weights on nodes of the association graph. Since each updated weight now reflects the contextual support for its mapping, we may evaluate the result by first thresholding the node weights to discard nodes with low weights. and then finding the cliques in the association graph. The clique with the remaining highest sum of node weights should represent the best mapping of observed regions to model objects. There are several algorithms in the literature for which can be paralclique finding” -‘j. An algorithm lelized is the following: Assume all nodes are in the same clique. Check pairs of nodes; if there is no arc between them, then they must be in separate cliques. Split the potential clique into two parts; one with vertex x and not y. and the other with vertex y and not X. Each part contains all of the other vertices of the original potential clique. Recursively perform (2) on each potential clique. When no more splitting is needed, all the remaining potential cliques are actual cliques. The maximal cliques arc the ones that are no subsets of any other cliques. PROCEDURE FOR MERGED REGIONS
HANDLING
SPLIT
AND
Oversegmentation can be handled by mapping multiple regions to one object. Our approach is to create a merged region consisting of these multiple regions. The merged region is then considered as any other region, having a set of attribute values and relations to other regions. Likewise, undersegmentation amounts to mapping one region to multiple objects. We create merged objects to handle this case. The procedure is as follows: 1 Compute the initial node weights for the regions and objects. 2 Determine candidate regions/objects for which we will attempt to merge corresponding objects/regions. If all of a region’s node weights are below a threshold, this indicates that none of the objects map well to that region. If this is the case, consider pairs of objects. If a pair of objects are adjacent (or meet some ‘closeness’ criterion), have similar region-wide attribute values, and have region-wide attribute values that match well with the region having all low node weights, and the merged object does not already exist, then create a merged object, and compute its node weight with the region. If the merged object already exists, simply compute its node weight with the region. Similarly, check node weights for each object. If an object has all low node weights, attempt to create a merged region which will map well to the object.
vol 10 no 9 novemher
1992
2a Process for creating a merged object or region: Calculate attribute values for the merged object or region. For region-wide attributes. the new values are weighted averages of the values for the two objects being merged. For area, the new value is the sum of the areas of the two objects. Add a row and column for the new object to the matrices representing the relations. and estimate values of the new object’s relations with other objects. Angle relations are estimated by simply averaging the angle relations of the two constituent objects. The real-valued adjacency relation (the percentage of an object’s pixels that border on another object) is computed by determining the perimeter of the merged object (the sum of the perimeters minus twice the length of their Then, the percentage ot shared boundary). border pixels with other objects is calculated based on the new perimeter. Since the new object is simply added to the list of objects, it can also be considered for merging with other objects, thus allowing for merges of three or more objects. 2b Compute node weight of the merged object with the region that prompted the attempt to find merged objects. Since the node weights are represented in a matrix with rows corresponding to objects and columns corresponding to regions. if the merged object is being newly added to the list of objects, all the other node weights for the merged object should be initialized to zero. After all plausible merged objects/regions have been created and their initial node weights computed, find the values of the arcs in the association graph. Arcs are not allowed between nodes that include the same object or the same region; e.g. there is no arc between node (RI, (01, 02)) and (R2, 02). Arcs from merged objects/regions are computed in the same way as other arcs, since the estimated relation values for the merged objects are included in the relation matrices. just as for any other objects. Perform the relaxation process on the association graph which includes nodes involving merged objects or regions. A slight change in the relaxation updating rule is necessary. Since two nodes including the same region are defined as incompatible. nodes involving regions that belong to merge nodes have fewer terms possible in the sum in the updating rule, which is an unfair disadvantage. Nodes including only a region that is not involved in a merge node have possible terms in the sum from every region. To make up for missing terms. a merge compensation factor h, is included in the updating rule. The value of S(i. j)“) is then counted h, times in the sum. So, the new updating rule is given by: S(i, j)‘l’+” = tuS(i, j)“‘) + (I-+[
h,S(i, ;)“I +
,$ r+r~(S(h, I, I [ it, An example
k)“’ c(i, j; h, k))
of this is shown in Table
111
1. Assuming
(7)
there
635
Table 1. Missing terms due to merged regions S(i, i)
Regions
affecting
sum
b,
1
2
3
SC,i)
x
x
x
S(2, i) S(3> i) S((l, 21, i)
x x
x x
x
(1.
X
X
x
x
2) 2 2 1 3
are four regions, 1, 2, 3 and (1, 2), the table shows that for nodes which map region 1 (or 2) to some object, there can only be terms in the sum in the updating rule from nodes involving regions 1, 2, and 3. However, since nodes involving region 3 could be compatible with nodes involving any of the regions, they have four possible terms. Nodes mapping region (1, 2) to any object can have terms from only region 3 and region (1, 2). The factors bj compensate for these missing terms by counting the value of S(i, j) multiple times. SIMULATION
RESULTS
The fuzzy relaxation process tested on hypothetical example cases of imperfect segmentation
described above was scenes. The following were tested:
scene is missing an object and Case 1 The observed also contains a spurious region. Case 2 The observed scene contains an oversegmented region. Case 3 The observed scene contains an undersegmented region. Case 4 The observed scene contains oversegmentation and undersegmentation errors, so that the desired result is to have a set of regions map to a set of objects as a whole. Case 5 The observed scene contains an oversegmented region that has been split three ways.
(i 1
b
(ii)
(iii)
(IV)
Figure 5. Scene models and inexact segmentations used in examples. (a) scene model: observed scene, case I: missing object and extra region. 5, (b) scene model: observed scenes, (i) case 2, split region, (ii) case 3, merged region, (iii) case 4, split and merged regions, (iv) case 5, three-way split region
Table 2. Simulation
results,
Case 1
Attributes of intensity, area, and circularity were used to describe the objects in Case 1. Intensity, area, and texture were used for Cases 2-5. The value of Wk was chosen so that each element of the sum shown in equation (2) would be in the range [0, 11. The node weights were then determined by the formula given in equation (1). This results in node weights between 0 and 1, with 1 representing a perfect match and 0 representing the worst match possible, given the attributes as measured. For Case 1, the real-valued relation of ‘angle’ is used, defined as the angle that a line between the centroids of the two regions makes with the horizontal. For Cases 2-5, relations of ‘angle’ and ‘adjacency’ were used, with adjacency defined as the percentage of a region’s boundary that is adjacent to the other region. The scene models and inexact segmentations are shown in Figure 5. The results of the relaxation procedure are shown in Tables 2-6. Table 2 represents the case in which there is an extra region and a missing object, but no oversegmented objects. The desired result is to have no region map to Object 3, and to have Region 5 map to no object. In the final result, Region 3 is the best mapping for Object 3, but it is a better mapping for Object 4, which is the 636
image and vision computing
correct
n~apping.
Because
port. all the node weights
of a lack of contextual supfor nodes involving Region 5
become low. For Cases 1 and 4, the intensity value for Object 2 was set to 0.1. For Cases 2 and 3, in which the hypothetical segmentation has merged Objects 1 and 2, the intensity for Object 2 was set to 0.5, a more plausible value. since ihe intensities of Objects 1 and 2 would probably be similar if the two objects were merged. The results for Case 2, in which Object I is split into Regions I and 5, are shown in Table 3. The process correctly mapped Region 6, which includes Regions I and 5, to Object 1. Results for LY= 0.2 and (Y=0.25 are shown, Table 4 shows the results for Case 3, in which Objects 1 and 2 are merged into Region 1. Again, the process correctly mapped Region I to Object 5, which includes Objects I and 2. The mapping of Region 1 to Object 1 also attained a high value, since Region I matches quite well to Object 1 alone, and its centroid is close to the centroid of Object 1, which leads to high arc values for the node (Rl, 01). Table 5 contains the results for Case 4, in which Objects I and 2 are merged into Region 1, and Object 1 is split into Regions 1 and 2. The mapping desired in this case is Region 5 (Regions 1 and 2) to Object 5 (Objects 1 and 2). At LY= 0.2, the mapping (RS, 01) is wrongly favoured. At cy = 0.25, the correct mapping (RS. OS) prevails. Table 6 contains the results for Case 5, in which Object 1 has been split into three regions, Regions I, 5,
Table 4. Simulation
results,
Table 5. Simulation
results, Case 4 __- - . _-
iwmsity
krca
Case 3
Texture
REtto!
1.83
0.7
0.8
*wte
0.23
0.5
0.5
1
1.82
t.0
0.0
0.25
0.1
0.2
11
D
37 355
2.06
0.63
0.77
21 217
0 335
3i 175 155
0
2.02
0.M
0.1
41 190 150 2‘5
2.22
0.9
0.05
51
0.28
0.02
0.35
0
‘I 0
Initiak n&e rei*ts .. .. . .. .. ...__....__
10
D
‘I
0
0
2) 170
0
Sl 192
265
ull,o2)
0.53
(Rl,o3f
0.53
(R1,041
0.28
0 345
5
0.97
G?2,01)
0.57
(12.02)
0.36
'
0.88
CR2,01)
a.34
m2.m)
0.00
~RS,Ol)
0.33
fRS.02)
0.77
wt3.03)
0.21
'(13,cA)
O.Pl
w3,m)
0.00
Derired mappings --
-
12 2.5 0
0
Relstion
Adjscency
2
3
4
5
0.09
0
12
3
0
*
II0
0
0
0
0
0
210
0
0.2*
3) 0
0
0
0.2 0
31 0
0.6
0
410
0
1
0
0
510
0
0
0
0
r,pra = 0.25
I.009
1.004
1.010
1.006
0.948
0.943
0.556
0.575
0.957
0.953
llergc Dbject m=Ol*O* *
350
0 105
Pinat mle weights . . . ...... .. ..-....
0.95
0
3
21 0.24 0
n,phs = 0.2 ul1,00
2
65 165
Adjacency Relation I
Ile,rti‘m
lnitist weight too 10" to consider
-
and 6. This example illustrates the way that more than two regions may be merged into one. Regions are merged pairwise, and the new regions are added to the list of regions, so that they are considered for future merges. In this example, Region 7 is the merge of Regions 1 and 5; Region 8 is the merge of Regions 6 and 1, Region 9 is the merge of Regions 6 and 5: and Region 10 is the merge of Regions 8 (6 and 1) and 5. Of all nodes mapping to Object 1, the node (RIO. 01) has the highest weight after relaxation. In each of the examples with (Y=0.25, the clique with the highest node sum after relaxation represents the correct mapping. The scene in Figure 6 is an example of an actual scene on which this approach was tested. This is an image of a space shuttle simulator panel, annotated with object numbers for clarity. The image was segmented by the segmentation procedure provided with a commercially available image processor (Perceptics 9200e). The segmented image contained a total of 50 regions, one missing object (number 7), and several oversegmented objects (numbers 1, 3, 11, and 13). In cases where there are many extra regions expected, it is better to take the objects of the stored representation
Table 6. Simulation
results,
Initial de weights _..._......._...._..
Case 5 Find node weights ..........._..._.. RLpha = 0.2
Upha
= 0.25
0.656
0.674
0.920
0.920
0.721
0.733
0.945
0.940
0.719
0.733
0.91
0.942
0.941
(R5.01) 0.73
0.621
0.634
(11,011 0.80
Table 6 (continued)
1
(Pl,OZ) 0.57 (R1,03) 0.39 (R1,04) 0.45 (R2.01) 0.43 W2.02)
0.90
(R2,03) 0.33 (R2.04) 0.87 (13.01) 0.57 (R3.02) 0.22
l(R3,03)
0.88
as the ‘units’ in relaxation and regions as ‘classes’. This is to avoid extra regions making spurious contributions to node weights. So, the updating rule used on this example is as follows:
(R3.04) 0.34 (R4.01) 0.33 (R4.02) 0.91
S(i,
j)(“+ ’ ) zz
aS(i,
j)(“) +
(R4,03) 0.27 VR4.04)
(R5.02) 0.69 (R5,M)
(1-a)[‘: [l&(S(h, k)(r)c(i,j;h,k))]] (8) m k=l
(R5.03) 0.31 0.57
(R6.01) 0.70
0.609
0.621
0.940
0.934
0.930
0.923
0.871
0.859
1.179
1.140
(R6.02) 0.69 (R6.03) 0.33 (R6,OL) 0.57 Merge region RI=RS+Rl (R7.01) 0.86 CRT.021 0.00 (R7.03) 0.00 (R7.04) 0.00 Merge region RB=R6+Rl (R8.01) 0.84 (R8,OZ) 0.00 (R8.03) 0.00 (R8.04) 0.00 Merge region RP=Rb+R5 (R9.00
0.75
(R9.02) 0.00 (R9,03) 0.00 (R9.04) 0.00 Merge region RlO=R8+R5 '(R10.01) 0.90 CRlO,OZ) 0.00 (R10,03) 0.00 (Rl0,04) 0.00 l
638
Desired
mappings
- Initial weight too low to consider
Figure 6. Space shuttle simulator scene
panel used as example
image and vision computing
also be used in applications in which classes,
a set of units taking
into
unrelated
needs
account
to machine
to be mapped both
local
vision
to a set of
and contextual
information.
REFERENCES 1
Pavlidis,
T
Structural
Pattern
New
(1977)
Springer-Verlag. 2
‘1’ ‘Structural
Pavlidis, &rammers’,
in
Picforial New
3
obtained
by
It!fhrtnatiotz
is
regions
analogous
to
and objects
reversed.
compensation required
factor
since
h,
(7)
with
the
roles
of
In this case, an analogous
for
merged
undersegmentation
is
objects
Trans.
Greene,
The
of the
did not occur in this
7. The
relaxation
figure
is annotated
around
the
objects
(I. 3. I I. and 13).
regions
~~versegmented
regions
that
are shown
with
thick
mapped
to
Every
object
in
more
corresponding included
regions.
In
oversegmented
to printing
in the winning
each case, the merged to the expected
7
was not
cases,
on the control
merged
region
obJcct than
some
represented
better
did any region
8
extra but
9
We
have
association
graph.
of mismeasured segmentation.
local
match IO
for
.
scene
applied
to an
and relations, over-
between
mination value for
objects
1979)
of CY. Other or
object
for
merging
One
possibility
is to perform
first
assuming
no
then to determine the results
is another
make
the method
In this for
or
graph,
the primitives
for merging process.
in real-time
process. using
It should
relaxation
can be used
in other
may be line
vol 10 no 9 novemher
1902
It could
Artif:
Vol
Itirrll..
6
structures Vol
f_cll..
isomor-
and maximal 4 No
4 (1976)
feature
Proc.
matching
through
Vol
SPIE.
I81
(April
using
associa-
IEEE
matching
Tratl.s.
using
PAMI.
relaxation
Vol
I
No
I
0 D and Berthod,
and reducing optimization Vol of
ambiguit)
M ‘Improving
approach’.
images
using
label-
IEEE
I No 4 (19X1) pp 112-3’3 0 D and Price, K E ‘Semantic
aerial
consis-
in stochastic
stochastic
Tratw. descriplabeling’,
IEEE
Trrrtts. PA MI, Vol 3 No 6 (1YXl ) pp 633-642
K E: ‘Hierarchical matching using relaxaC’orupill. Visioti. Graph. R 1ttiag’e Prow.s.s.,
Vol IX
3D images to models
ltmgc R Vi.sion C‘omprr~. , Vol 7 No 3
Price, tion’.
the
to an associa-
segments.
for testing
isomorphism’.
pp 60-72
An
tion
is need to
in which
Procrxs.
L S ‘Shape
PAM/,
as the primitives
applications
for
iso-
pp 3 l-33
pp 135-143
Faugeras,
situations.
applied
or curve
ing:
and
that
?
pp I40-I40
tency
implemen-
be noted
subgraph
I ( 1976)
program
graphs
relational
cliques’.
Faugeras,
procedure
much research
for
7.3 No
assembly’.
R C ‘Robust
(1979)
by examining
Parallel
we have used regions
the matching
Itl/it.
techniques‘.
be investigated.
under-segmentation,,
candidates
useful
should
No
pp 129-306
matching
Davis,
candidates
the relaxation
area in which
paper.
same procedure, tion
over-
of the relaxation
tation
determining
line
controlled
tion graphs’.
support
into the automatic deteron node weights and the best
methods
‘9
Yang, B, Snyder, W E and Bilbro, G L ‘Matching
is needed
of thresholds
Vol
- A heuristic
directed
o\crsegmcnted
observed
of good contextual
I7
for graph
Vol
ht.,
Barrow, H G and Burstall, R M ‘Subgraph
Belles,
of good
and
Vol
c’ottmutt. A (‘M, Vol 23 No I ( 196-I) Ambler, A P, Barrow, H G, Brown, C M, Burstall, R M and Popplestone, R .J ‘A versatile system for
(1989)
research
region
Unger, S H ‘GIT of
algorithm
algorithm
J. A CM,
maximal
application,
the importance
model
efficient
1. ACM,
pp x3-x-I
under-
and
iso-
IO
J R ‘An
Ullmann,
cliques’.
missing
the mappings. Further
pp 105-l
phism.
used in the relaxation
to the scene matching
between
II
the problems
addresses
rule
the importance
and
to inexact rule
and
updating
a balance
matches
regions
paper attributes
regions. The
is well-suited
as it allows
approach
relaxation
This
object
extra
objects. process
a new
a fuzzy
for
C C ‘An
for graph isomorphism’.
No 3 (1975)
using
I
.l. A C.44. Vol 20 No
D G and Gottleib,
computer
prcaented
procedure
graphs’.
Mittal, H B ‘A fast backtrack iaomorphism’. Itlfo. Proccxs.
pairs
CONCLUSION matching.
No
(19X9)
backtrack
of directed
morphism’,
in
by itself.
I3
I ( 1970)
(l9X8)
panels
regions.
recognition’. Vol
& C)~hcrn..
in Huntsville
A T ‘A
Corneil, No
matched. Those that were matched to merged regions con-
were or
of Alabama
Berztiss,
algorithm
outlines
that
Sysr. Man
error-correcting
pattern
3 (197.1)
was correctly
oversegmented misting of two were
process
syntactic
R Suw ktzo~~lc~dgc rc~~~rcsett~utiott for \,i.siott .s):stem.s. PhD dissertation, Univer-
morphism 6
results
Figure
sity 5
not
example.
for
( 198.1) pp 38-62 4
equation
graph
Springer-Verlag.
Systems,
( 1980) pp 86-103 Tsai, W H and Fu, K S ‘Subgraph
csperr This
and
and K S Fu teds),
York
IEEE
merging
Recopifion,
descriptions
K Chang
S
isomorphisms
Figure 7. Object assignments o~,c~rsegrtletlrc,ri regions
York
34 No
I (1086)
pp 66-75
Price, K E ‘Relaxation comparison’.
IiCl?E
matching
Trrrt7.s. PAMI,
techniques Vol
-
A
7 No
5
639
19
20
640
(1985) 617423 Rosenfetd, A, Hummel, R A and Sucker, S W ‘Scene labeling by relaxation operations’, IEEE Trans. Syst. Man & Cybern., Vol6 No 6 (1976) pp 420-423 Hummel, R A and Rosenfeld, A ‘Relaxation processes for scene labeling’, IEEE Trans. Syst. Man & Cybern., Vol 8 No 10 (1978) pp 76.5-768
21
22
23
Balas, E and Yu, C ‘Finding a maximum clique in an arbitrary graph’, SIAM J. Cornput., Vol 15 No 4 (1986) pp 1054-1068 Bron, C and Kerbosch, J, ‘Finding all cliques of an undirected graph’, Common. ACM., Vol 16 No 9 (1973) pp 575-577 Lau, H T Algorithms on Graphs, TAB Books, Blue Ridge Summit, PA (1989) pp 59-69
image and vision computing