Knowledge-based picture understanding of weather charts

(~111 3203:84 S3(~1~ .1~) Pergamon Pre,,s Lid ~2, 1984 Pilltern Ret2ognilll~n Sl~icl'v

Pattern Rcco~nithm Vol. 17. No. I. pp 109 123. 1984 Printed in Great Britain

KNOWLEDGE-BASED PICTURE UNDERSTANDING OF WEATHER CHARTS R I N - I C H I R ( ) T A N I ( i t ~ ( ' H I . M A S A ( ) Y O K O T A , E I J I K A W A ( ; I ~ C H I and TUNE()TAMATI Department of Information Systems, Graduate School of Engineering Sciences, Kyushu University, Sakamoto 33. Kasuga, Fukuoka-ken, 816 Japan

(Received 10 February 1983; in revisedform 16 March 1983; receit~edfor publication 6 May 1983) Abstract--This paper describes a knowledge-based weather chart understanding system named WERP, which is working as a picture processing part of our Information understanding System Of BAsic weather Report (1SOBARI. WERP is designed to extract necessary information from a weather chart for generating weather report sentences explaining the chart. This system is based on a structural model of the weather charts. Here, we study what problems are involved in weather chart understanding, how they are solved and how an actual system is organized. Also, picture-processing techniques for weather chart processing and some experimental studies are considered. Knowledge-based image analysis Weather chart analysis Representation of picture meaning

Picture processing Isobar line analysis

1. I N T R O D U C T I O N

1.1. Background of this research Picture understanding is one of the main topics in computer science and many researchers have devoted considerable time to this difficult problem. In their researches, a picture is observed from several points of view. In some cases a picture is regarded as a random field, or in other cases, as a two dimensional formal language, etc. From the viewpoint of human communication, a picture is a representation of information which human beings interchange with each other, while natural language is an alternative to pictorial representation. As natural language and pictures {or pictorial patterns) share the same function of communication, information represented in a picture can be explained in natural language, and vice versa. In some cases, of course, it is not easy to express pictorial information in natural language, and the opposite, in other cases, is difficult. However, generally, pictorial patterns and natural language are mutually transformable to some extent. According to this idea, a picture is judged to be understood when the contents of the picture have been explained in natural language, i.e. when the information necessary for generating the linguistic explanation has been extracted from the picture. Language understanding can be treated in the reverse way. In addition, the correctness of the understanding can be examined by the translated representation. In order to apply our idea to a practical system, we have developed an understanding system for basic weather reports named ISOBAR? ~2~ which accepts both pictorial inputs {weather chartsl and linguistic ones (weather

report sentences). It is one of the characteristics of this system that natural language and pictures are represented by a common meaning representation called Semantic Table "1 and that they are mutually translated through the medium of this representation. The configuration of ISOBAR is presented in the Appendix. In this paper we present a part of this system, i.e. the picture understanding component, which translates a picture, i.e. weather chart, into the common meaning representation. 1.2. Introduction to this research In picture understanding, the most difficult problems include how to extract meaningful data from the enormous amounts of information in a picture and how to transform or arrange the extracted information into a more comprehensible form. "Meaningfulness" generally depends on the context in which the picture is presented. Therefore, any picture understanding system requires a model of the context that is both accurate and adequate. In other words, without any knowledge of the objects in the picture, it is almost impossible to understand what is represented. In the case of our weather chart understanding system, it must have some knowledge of weather charts and related factors, especially meteorological phenomena. To construct a successful system, it is of primary importance that the knowledge be well organized. The system presented here, named W E R P (WEather chart Recognition Program), ~3~ accepts weather charts covering Japan and adjacent Far East Asian areas and translates them into a Semantic Table. W E R P is designed according to a structural model of 109

PR 17:I-H

Picture understanding Binary image analysis

110

RIN-ICHIRO TAN1GI_CHI, MASAOYOKOT.% EIJI KAWAGtCHI and Tt .~Eo TAMATI

I1

Fig. 1. Examples of weather chart for input.

Table 1. Constituent and Attribute (a) List of Constituents Symbol

Constituent

Pictorial expression

Ct C2 C3 C, C~ C6 C7

Low pressure High-pressure Typhoon Tropical depression Warm front Cold front Stationary front

Cs

Occluded front

C9

Isobar

Cto

Local weather

l_ H T TD

(b) List of Attributes Symbol

Attribute

At A2 A3 A, As A6 AT As

Location Pressure Number of typhoon Direction Area Wind force Weather Wind direction

(c) Cross-relation between Constituents and Attributes

Ct C2 C3 C, C5 C6 C7 Cs Co C1o

At

A2

0 0 0 0 0 0 0 0 0 0

0 0 0 0

0

A3

A,

0 0 0 0 0 0

A~

A6

A7

0 0 0 0 0 0 0 0 0 0

0

0

As

Knowledge-based picture understanding

I1 t

o-~ o

t

e~ e-,

b £

Front P i c t u r e

Local Weather Picture

Mop Picture

%. oj~J6 L 1006

~020

Character Picture

Isobar P i c t u r e

Fig. 2. Structural model of weather chart.

weather charts and is regarded as a series of information extracting filters driven by a knowledge source. The knowledge source is employed for deciding the strategic parameters of each processing filter and also for high-level processing of the extracted information. In the following discussions we describe what problems lie in weather chart understanding, how they are solved and how the actual system is organized. 2. KNOWLEDGE-BASED WEATHER CHART UNDERSTANDING

2.1. Representation of weather chart meaning (12) W E R P is a special-purpose system which is expected to recognize weather charts covering Japan and adjacent Far East Asian areas. Figure 1 shows examples of our weather charts. The format is based on Tenkizu Shusei (The Collection of Weather Charts) published by Nihon Kisho Kyokai (Japan Meteorological Bureau), which is popular in Japanese newspapers and magazines. The weather charts represent weather conditions of everyday interest and can be regarded as a concise world of meteorological phenomena. In order to describe the semantic contents of the weather charts, we introduce two basic notions. They are : Constituents of the world of the weather charts and Attributes of the Constituents. We postulate 10 kinds of Constituents in all, and each one has its own Attribute value and corresponds to a specific pattern in

a weather chart. Eight kinds of Attributes of Constituents are extracted. Constituents and Attributes are tabulated in Table 1. The meaning of a weather chart is represented in a "'cross-relation table" of Constituents and Attributes. We call such a "crossrelation table" the Semantic Table of a weather chart. The aim of W E R P is to construct the Semantic Table for an input weather chart. In ISOBAR, the supersystem of WERP, Sentence Generator synthesizes weather report sentences from the Semantic Table which is the output of WERP. Here we must draw attention to the fact that one weather chart cannot generate an entire weather report, which ISOBAR deals with. This is because a single weather chart is a static, synchronic picture that cannot represent temporal changes in information. A weather report includes such information, and a typical example is the traveling of a typhoon. Therefore, the Semantic Table of a weather chart must be a part of that of the "weather report. In ISOBAR, outputs of W E R P are fed into a Semantic Analyzer, which then analyzes the differences in meanings among some successive weather charts and extracts changes of weather conditions with time. 2.2. Model of weather charts The control structure of W E R P is based on a structural model Of weather charts, which is built into the system. A weather chart is regarded as a superposition of five kinds of subpictures : the local weather

112

RIN-ICHIRO T-XNIGUf'Ht, MAS~O YOKOTA. EIJI K xxw(;t CHI and Tt NEo.T X~,IATI

picture, front picture, map picture, isobar picture and character picture (see Fig. 2). Those subpictures are categorized according to both their meanings and pictorial features. The local weather picture indicates weather and wind conditions in principal cities. The front picture shows the occurrence of four kinds of fronts such as cold, warm, occluded and stationary fronts. The map picture shows longitudes, latitudes and coast lines, which give positional information on a weather chart and do not directly convey information concerning meteorological phenomena. The isobar picture shows isobar lines drawn every 4 mbar. The character picture indicates characters and numerals, which indicate occurrences of several kinds of Constituents, such as low-pressures, and specify pressure values and typhoon numbers. WERP decomposes an input weather chart into these five subpictures and extracts meteorological information from each of them. These subpictures are not semantically independent, because one subpicture bears information additional or supplemental to that of the other subpictures. A typical example is the relation between the isobar picture and the character picture. To understand the pressure value of an isobar line, it is necessary to link a pressure value represented in the character picture with the correct isobar line in the isobar picture. This kind of correlation is made after the information of each subpicture has been extracted. Therefore, the structure of WERP must also be hierarchical. 2.3. Knowledge organized in WERP The goal of WERP is to recognize a weather chart and to describe its semantic contents. Therefore, it is necessary for WERP to have knowledge, represented in a proper form (declarative, procedural, etc.), of the weather charts and related factors (especially meteorological phenomena). The knowledge incorporated into WERP is categorized into two types: Knowledge of the Syntax of Chart (KSC) and Knowledge of Meteorological Phenomena (KMP). KSC is based on the syntax of the weather charts (see Table 2) and is concerned with the pictorial characteristics of weather charts. It affects the strategy of WERP operations, especially insofar as it reduces to a minimum the computing efforts required. On the other hand, KMP can be regarded as the meteorological common sense of the system and indicates normalcy of weather attributes. K M P is employed mainly in higher-level processing modules which process information extracted in lower-level processings. It is used for correction of errors, clarification of ambiguities, recovery of missing information, etc. The following are examples of KSC and KMP utilized in WERP. (KSC-1) Every weather symbol is enclosed with a circle of constant size. (KSC-2) The location of each weather symbol is fixed in a chart without considering minor de-

viation. There are 15 locations in all. (KSC-6) The locations of coast lines, latitudes and longitudes are fixed, but they are partly erased or interrupted by other component patterns. (KMP-I) Pressure values range between 800 and 1200. (KMP-2) Typhoon numbers range between 1 and 99. (KMP-4) Isobar lines are ideally simply closed, except when they are cut offby the frame of a chart. 3. TECHNICAL ASPECTS

This system consists of several processing modules. which are organized with many kinds of pictureprocessing techniques. The modules are designed to maintain their modularity and the update of algorithms does not affect the operations of other processing modules. Some of the picture-processing algorithms contrived for the system are described in this section. These algorithms are knowledged-based and are given strategic parameters by the knowledge source. 3.1. Searching and i&nt(fication of symbols Image processing cannot avoid dealing with a large amount of information. This type of difficulty arises in searching for primitive symbols in image data. It is time-consuming to search for the symbols over the whole area of the image data. Therefore, it is desirable to pick out candidate regions and to bound the searching areas, based on features of the regions, if possible. For this purpose, image data is processed on an image pyramid, t'~l At a high level of the pyramid (low-resolution level), features are calculated and when the values of the features lie within ranges indicating a high possibility of the symbol occurring, the system searches for the symbols in the corresponding regions at a low level (high-resolution level). In weather chart processing, a typical example is searching for and identifying front symbols. As front symbols contain a certain size of black area, the average brightness (or density of black pixels) of regions including front symbols is darker (or greater) than that of regions not including them. Therefore, when the average brightness of a region is darker than a certain brightness threshold, it is judged to have front symbols. In order to choose such candidate regions and to search for symbols containin~ a black area of a certain size, operations are executed at a higher level in the pyramid. After the approximate positions of the front symbols have been evaluated, identification is achieved at a lower level. After find the symbols, the system is required to identify them. Strictly speaking, it is difficult to separate searching and identification completely, because, to search for a symbol, the system must examine whether an object pattern belongs to the category of the symbol. In WERP, however, the searching and

Knowledge-based picture understanding

113

3.2. Elimination of constant patterns

if!iiiiiil!iiil

2[.

"

.

%"

.

.

.

.

•

•

•

Fig. 3. Chains of target points• identification processes can be separated, because symbols are categorized so as to have different pictorial features and to be found almost without identification. There are various kinds of techniques for symbol identification. One is template matching. Although it is simple, it works effectively because some of symbols in the weather charts are normalized in shape and size• Similarity between an unknown pattern and a template pattern is defined as follows: I P,( x, Y) -

o¢lx, y ) = 1 - i=1

N

q, I (1)

where u(x, y) is the similarity, (x, y) are coordinates of the reference point in the picture, Pi(X, Y) is the darkness value (1 : black, 0: white) of the i-th pixel in the local pattern, q~ is the darkness value of the i-th pixel in the template pattern and N is the number of pixels of the template. As this similarity can be calculated exclusively by additions and subtractions (except for one division), its computing cost is less than the welt-known RMS-type similarity. identification consists of finding the template pattern which makes the similarity maximum. The similarity must be evaluated around symbol positions detected in the searching process, because the searching process does not always give the optimum positions for the best matched templates. In this case, the computing effort increases. Therefore, to reduce computing cost, the similarity is calculated in multiple stages. First of all, similarity is evaluated on a small number of particular pixels which reflect the structure of the pattern. If the similarity is greater than a certain threshold, the similarity is recalculated on the whole pattern to get the precise similarity. However. if it is beneath the threshold, the pattern is judged to be dissimilar to the template and the precise similarity is not calculated.

Weather charts have patterns which are almost invariant in any weather chart. They are longitudes, latitudes and coast lines, which do not bear information about meteorological phenomena but only positional information. Once the mapping relation is computed between a coordinate system on the earth and that on the image array, such constant patterns are no longer necessary for understanding. In fact, they should be removed from the charts, because they cause difficulty and ambiguity in interpreting other patterns. However, as a scheme for removing them, simple subtraction of standard patterns from an input weather chart does not work effectively. Because minor shifting and local reduction of patterns are inevitable in any input process, simple subtraction tends to partially erase other important patterns as well. In WERP, curve-following with guidance is introduced in order to cope with such minor shifting and reduction. First of all, chains of "'target points", which are regarded as representative points of constant patterns, are decided by preliminary inspection (see Fig. 3). Elimination by curve-following is a repetition of segment-by-segment erasing between two adjacent "'target points", just like inchworm movement. The procedure of an actual elimination scheme is as follows. 1. A chain of"target points" are set up along the expected line of the longitude/latitude or the coast. 2. Two adjacent "target points" are selected. Suppose that the procedure is at the j-th stage and that the two target points are Pj and Pj+ 1. 3. Two points on a line which are respectively near to Pj and to Pj+ 1 are extracted. The points are denoted by Qj and Qj+~ respectively. Qj was given at the previous stage and Qj+l must be found near Pj+~ at this stage. 4. The system calculates parameters to judge whether the line between Qj and Q j+ : is a part of a constant pattern corresponding to the line between Pj and Pj+ 1- The parameters are such factors as parallelism, linearity and possibility defined below.

, I,

PJ PJ+~~ • ~

(parallelism factor)(2)

I IC2jQj+ 11 ~] = 8 ' 7

(iinearity factor) (3) (possibility factor)

(4)

where ~ and *-', are vector and curve notation, respectively. The linearity factor ;' comes from the fact that each "'target point" is located so close to the next point that every constant pattern can be treated as a sequence of linear segments. 5. If the possibility factor exceeds a certain thresh-

114

RIN-ICHIRO TANIGUCHI, MASAOYOKOT.% EIJI KA',VAGLCHIand TLNEO T ~,MATI

J

~ s (a) Broken Isobar Lines

lS

(b) Restored Isobar Lines

Fig. 4. Restoration of broken isobar lines. old, the line between Qi and Q~+ 1 is judged to be a part of the constant pattern and is erased. 6. Steps 1 to 5 are repeated along all the chains of "target points". Usually an input chart is not exactly in the same position as the standard one and the target points often do not match the constant patterns. When this is the case, the deviation of the input chart is detected and positions of target points are recalibrated. The chains of"target points" are a representation of knowledge of geography around Japan. 3.3. Recovery of broken isobar lines In a weather chart, only several principal isobar lines are explicitly given their pressure values and it is necessary to infer the pressure values of the other isobar lines from those of the principal isobar lines, high-pressures, low-pressures, etc. When isobar lines drawn in a weather chart are broken for some reason, it is necessary to recognize the connective relation among the broken isobar lines and to restore them into complete closed isobar lines before the inference of pressure values. As a misunderstanding of the connective relations causes a failure in pressure inference, the recovering of broken isobar lines is an important problem. We human beings can restore broken isobar lines such as Fig. 4(a} into complete isobar lines such as Fig. 4(b) by interpolation. That is, we first recognize the connective relations among the broken isobar lines and then restore the isobar lines to be connected, considering the characteristics of isobar lines and global pictorial features. The restoration process thus cannot work successfully without an understanding of the characteristics of isobar lines. These characteristics are as follows. A. Each isobar line is simply closed, except when it is cut by the frame of a chart. B. Each isobar line is smooth, except in the places where atmospheric pressure changes steeply, as in the vicinity of a cold front. C. Isobar lines are drawn at constant pressure intervals [these intervals are hereafter denoted by d(mbar)].

D. A part of an isobar line is erased when a pictorial component which has higher priority occupies the same position as the isobar line. E. Some, though not all, principal isobar lines are explicity given their pressure value. Corollaries of A and C are as follows. A'. Isobar lines do not cross each other. C'. When there is no isobar line between two isobar lines, the difference between their pressure values is less than or equal to d mbar. With respect to D, in addition, isobar lines are partly erased during the recognition process when overlapping pictorial components are removed by the system. In the recognition of connective relations among broken isobar lines, characteristics A and B take the most important role. The practical procedure consists of two major parts: the clustering of break points to be connected and their reconnection in each cluster. A cluster is formed by iterating to merge into the cluster a break point whose distance from one member of the cluster is less than a certain threshold. The merit of this clustering is the reduction of complexity and computing effort. Before describing the reconnection of the break points, we must introduce two concepts, connectability and smoothness of connection, which are important for the,reconnection procedure. (1) Connectability. If a straight line between two break points crosses another isobar line 2 x n times (17

\

Fig. 5. Smoothness of connection.

Knowledge-based picture understanding

I 15

4. If the connection causes a crossing, it is abandoned, and the system tries again to connect candidate pairs, starting with the candidate pair which has the next highest priority after the abandoned connection. 5. Repeat process 1 to 4 until there are no remaining connectable pairs. If any break points are left unconnected in a cluster, they are marked as non-connectable points. The recovering process can be regarded as a procedural knowledge concerning the characteristics of isobar lines. Fig. 6. Adjacency of isobar lines. 3.4. Pressure inference is a positive integer or zero), the two points are said to be connectable. (2) Smoothness of connection. This factor indicates the smoothness of a connection of two break points. The smoothness v is calculated based on 01 and 0 2, which are angles between the straight line connecting the two break points and tangents at both ends of the line (see Fig. 5). v = max (cos 01, cos 02) (-l_
(5)

The smaller v is, the smoother the connection is judged to be. Making use of these two concepts, reconnection in each cluster can be achieved in the following way. 1. Examine connectability of all pairs of break points and then pick up connectable pairs as candidates for reconnection. 2. Sort the candidates according to their smoothness. The smoother the connection is, the higher priority it has. 3. Connect candidate pairs with a straight line according to their priority, examining whether the line currently connected crosses the ones previously connected.

l

For inferring pressure values, characteristic C' in 3.3, which mentions the difference in pressure values of isobar lines, is the most essential. In such a situation, i.e. when there are no isobar lines between two isobar lines, they are said to be adjacent, and each of them is constrained by its adjacent isobar lines. The unknown pressure values of isobar lines are calculated by extrapolation using known pressures of some principal isobar lines. Therefore, establishing adjacency relations among isobar lines is the first step to the solution of this problem. The actual procedure is as follows. 1. Consider a picture with isobar lines as a binary picture; isobar lines are black and the background is white. Then, segment the picture into black and white connected components (Fig. 6). 2. Identify adjacency relations between isobar lines and background components. In the case of Fig. 6, A and a for example are adjacent. We denote the relation adj(A, a). 3. When two different isobar lines are adjacent to the same background component, the two isobar lines are judged to be adjacent. In Fig. 6, adj{B, a), adj(B, b) ~ ADJ(a, b),

where ADJ indicates adjacency of isobar lines. After the adjacency relations of isobar lines are established, the relations of their pressure values must be considered. The relations of pressure values in a weather chart is classified into three types as follows. Type I. For two adjacent isobar lines ai and a t, one of the following three formulae holds. p(ai} = p(aj) + d p(ai) = p(aj)

(7)

p(ai) = p(aj) - d

where p(aO indicates the pressure value of isobar line a~ and d is the pressure interval introduced in C of 3.3. Type 2. Where isobar lines are in a nesting relation, as in Fig. 7, p(a~÷l) = p(ai} + d

Fig. 7. Nesting of isobar lines,

(6)

or

(I < i -< n - 1)

(8)

116

R|N-ICHIRO TANIGLCHk MASAOYOKOTA, EIJI KA~3.AGUCHIand TtNEO T~,MATI

(a) Broken isobar lines.

(b) Complete isobar lines corresponding to (a).

J

(c) Broken isobar lines which have same adjacency relation as (a),

(d) Complete isobar lines corresponding to (c).

Fig. 8. Ambiguity in pressure inference.

p(ai+l)=p(ai)--d

(1 < i - - < - n - - 1)

(9)

holds. In these formulae, the sign is " + " if a high-pressure occurs at the center and " - " if a low-pressure does. This reflects the fact that atmospheric pressure distribution usually forms a crest or trough, but not undulations. T y p e 3. In a weather chart, some isobar lines are explicitly given their pressure values. Such isobar lines are the bases for extrapolation of pressure values. Their pressure relations are expressed as p(ai) = constant.

(10)

Based on the consideration of pressure value relations, the practical procedure for pressure inference is 1. Identify the relations of pressure values between adjacent isobar lines. 2. Select an alternative from the possible relations for a pair of adjacent isobar lines (Type 1-2) which is consistent with the relations of the other pairs. 3. Calculate p(ai) of each line for the selected relations. The method described above is designed for complete isobar lines, which should be available after the restoration process mentioned in (3). In fact, the restoration process sometimes leaves some break points unconnected and does not always produce complete isobar lines. However, this method can be

applied to incomplete isobar lines to some extent. Let us consider a simple example like Fig. 8. Figure 8(a) includes an incomplete isobar line, which is seen fully restored in Fig. 8(b). In Fig. 8(b), the pressure value relations are described as follows. p(a) = p(b) + d

(11)

p(a) = p(b) p(a) = p(b) - d p(b) = p(c) + d

(12)

p(b) = p(c) p(b) = p(c) - d

On the other hand, in Fig. 8(a), isobar a and c are judged to be adjacent, because they are both diagrammatically adjacent to the same background component. Therefore, the pressure relations of these isobar lines are regarded as follows. p(a) = p ( b ) + d

(13)

p(a) = p ( b ) p(a)=p(b)-

d

p(b) = p(c) + d p(b) = p(c) p(b) = p(c) - d p(a) = p(c) + d

(14)

Knowledge-based picture understanding

(15)

p(a) = p(c) p(a) = p(c) - d

The pressure relations of Fig. 8(a) are more constrained than those of Fig. 8(b), because there is an additional relation (15), compared to the relationship

of (b). Let us consider another example, such as Fig. 8(c) and Fig. 8(d), for comparison (incomplete and complete). The pressure relationships for Fig. 8(c) and Fig. 8(d) are the same as that for Fig. 8(~). Thus, in this case, no problem occurs. However, it is a problem that there is no difference in adjacency relation between Fig. 8(a) and Fig. 8(c), whose respective restored isobar lines are different from each other in their adjacency. To solve the contradiction, we should ignore relations between isobar lines which are adjacent to the same incomplete isobar line, because such relations are ambiguous. In the example mentioned above, this would mean ignoring the pressure relation between isobar a and c in Fig. 8(a) and in Fig. 8(c). If this is done, Fig. 8(a) will be dealt with correctly, but Fig. 8(c) will not, because in Fig. 8(c), the adjacent relation between a and c is not a side effect of the incompleteness of isobar line b and the adjacent relation is necessary for the inference. In such a case, the result of the inference may be ambiguous because some relations are deleted. However, it includes true pressure distribution. When other constraints given by other isobar lines are strong, the pressure inference procedure described here can lead to correct inference. 4. EXPERIMENTAL SYSTEM 4.1. Image data representation

One of the most difficult problems in image processing is the immensity of information represented in an image. In the case of WERP, an input image is a two dimensional binary array with a size of 1024 x 1024 pixels. In view of the nature of ordinal computer programming, it is convenient to represent one pixel in at least 1 byte. Therefore, at least 1 Mbyte is required for representation of the array. As this necessitates a

Picture

_-

117

large amount of computing effort, it is desirable to represent image data in a more compact form, if possible. One way is to encode the 2-dimensional image array into another compact representation and to implement image-processing algorithms on the representation. A typical example is Freeman's chain code. (5t However, most of coded representations sacrifice flexibility and applicability of image-processing techniques. On the other hand, direct processing on 2 dimensional image arrays has such flexibility and applicability. However, there is a drawback in that the memory requirement for data storage and the computing time become much larger with any increase in the image array size. The problem of memory requirement is being solved by recent developments in memory devices and computer architecture, but the problem of computing effort has not been solved in principle. One solution is to use a parallel array processor, such as CLIP4. (6~ However, it is difficult and expensive to construct a sufficiently large processor of this kind. To solve the problem, in WERP, the concept of Image Pyramid ~4>is introduced. High resolution of the image data is not required in all phases of image processing. For example, the brightness variation in a small local region does not significantly influence global features, such as the average brightness, spatial frequency spectrum in a low frequency range, etc. Therefore, the image data is processed on a resolution level which is appropriate for each image-processing algorithm. The resolution level is decided according to a strategy based on the knowledge source. Mapping from a lower level to an upper level of the image plane is usually a repetition of logical AND of 2 x 2 lower level pixels. However, in some special cases logical OR is employed for the mapping. 4.2. Structure of W E R P WERP works as a series of"information extracting filters" driven by a "knowledge source." Figure 9 illustrates the filter model of WERP. Corresponding to one part of the weather chart model described in 2.2, each filter module extracts the information which each subpicture conveys. As the picture passes through a

- - - - - e K n ° w l e d g , Soucce

IEF:Inf°rmati°n~~~'~ Extracting Filter ~-~High-level Processor Interpretation Fig. 9. Model of WERP.

118

RIN-ICHIROTANIGUCHI,MASAOYOKOT.,~,EIJI KAWAGLCHIand TL NEOTAYLaTI

series of filters, each subpicture is separated and the required information is extracted. Therefore, the picture processed in each successive filter becomes a less information-rich form. Each filter module decides the strategies of its operation according to parameters supplied by the knowledge source. The strategies include (1) selection of the resolution level at which the module processes the image data, (2) focus control, in other words, control of area to which the module pays attention, and (3) control of the retrial process when it fails to acquire the expected infortnation. When an input picture, a composition of the five subpictures, is given to WERP, the first module separates the local weather picture, extracts local weather information such as weather and wind conditions, and transfers the remaining picture to the next step. The second module separates the front picture and acquires front information such as type and location. The third module receives the remaining image data, i.e. a composition of map, isobar and character pictures, and removes the map picture. The fourth module is a separator of isobar pictures and extracts isobar-line information such as location, length, etc. After the picture is processed in the fourth module, what remains should be a character picture, but usually some uninterpretable fragments remain in the picture. Therefore, the fifth module must verify and extract the character picture. Then, the information extracted by the filters is processed in higher-level modules. Operations in these modules are as follows. I. Verification of extracted information and correction of errors. 2. Recovering of lost information. 3. Indication of ambiguous interpretation and clarification of ambiguities where possible. 4. Rearrangement and reinterpretation of combined information. Finally, the results of the higher-level processing are arranged in the form of a Semantic Table, and output from WERP. The detailed structure of WERP will be described in a later section. 4.3. Hardware contiguration Before the detailed structure of WERP is considered, we will look at the computer system on which our experiments were carried out. Figure 10 shows the configuration of the system, which has two computers, an E-800/7 and a M-240H. The former has image I/O devices and the latter has a large magnetic disk system. Picture input and pre-processing are executed on the E-800/7 and actual procedures are developed and executed on the M-240H. In addition, a database of weather charts is constructed on the disk. 4.4. Implementation and experimental results Figure II shows the detailed structure of WERP. The organization of WERP is based on a structural model of weather charts and consists of six expert

-•K F7-1

PDS

I

H.S.BUS

|

:HANNEL--q HITAC-MZ40H [

Fig. 10. Hardware configuration of WERP.

modules, which are WESI, WISI, FSI, CCE, ICR and SYA. In Fig. 11, double arrows indicate the sequence in which the expert modules are activated and arrows in each module indicate the sequence in which procedures in each module are executed. Arrows into or from each module show information supplied by the knowledge source or acquired by each procedure. Before WERP is applied to an input weather chart, the chart is processed in the Picture Pre-Processor, which performs thresholding of an input image into a binary picture, noise reduction, compression and accumulation of compressed data in the picture database. A noise-reduced binary picture is fed to WERP. The operations of each module are as follows. (I) WESI (Weather Symbol ldent!lier ). WESI is designed to identify weather symbols in a weather chart and to store the results in a Local Weather Table (LWT). The locations of weather symbols are invariant and their expected positions are indicated by the knowledge source. After decision on the exact positions, parameters for coordinate transformation from the coordinate system of a standard weather chart to that of the input weather chart are calculated, based on the distances between expected positions of weather symbols and their detected positions. (Currently, only translation can be handled.} These parameters are referred to in each processing module. Then, each weather symbol is identified by template matching. In this procedure, the proportion of the black pixels in each weather symbol is examined in order to narrow down candidate templates to be examined. After each weather symbol has been identified, it is extracted by border following~7}and, if there is a wind symbol attached to it, the wind symbol is extracted as well.

Knowledge-based picture understanding

119

(Pre-Processing Part) PPP

~ESI

"U' Search -, (TSD) --

KSC-I KSC-2

Extract -,-(Boundary Trace)

KSC-3 KSC-I

Recog!ition----'(TSD)

/

FSI

LWT

k

WISI Thinning (Hilditch's)

Decision of Candidate Regions~

+

Search (TSD)

/

(TSD)

"-'-FST

I I

Recognition ~ - - K S C - 4 (Curve Following)----~LWT

KSC-5

.N e c o g¼... / nltlOn~

.~.

SYA

CCE

_ Recognition of FrontLine

Thinning (Hilditch's)

FST

Elimination of Constant Component

KMP-2 ~ KSC-6 KMP-3 IT -----

Tracing Front Lines~-~

KMP-I NCT - - - - ' -

FST

-

IT

KMP-2

~-

KMP-3 ~

1oR Tracing Isobar~ (Curve F o l l o w i n g ) ~

IT KSC-7

+ ~ C ~ KSC-B Character Recognition'~KSC-9 (Curve Following) - ~ N C T

> ST

+

Recovery of Broken Isobars

> IT

Recognize Pressure Value

> ST

Inference of Pressure Value

> ST

Coordibate Transformation

ST

I

I SEMANTICTABLE of WEATHERCHART

Fig. 11. Configuration of WERP. Experimental results are shown in Fig. 12. Figures and tables in Fig. 12 show processed charts and information extracted from the input weather chart by each expert module. (2) W I S I ( Wind Symbol IdentUier ). If wind symbols are detected in WESI. WIS1 is activated. As wind symbols are the only symbols that are connected to weather symbols, it is not difficult to extract them. A wind symbol indicates wind scale by the number of branches which protrude from the axis of the wind symbol. Different wind scales correspond to topologically different wind symbols, except that wind scale 1 and 2 are topologically the same. Discrimination between them is achieved by comparing the lengths of the branches. Recognized wind scales are recorded in LWT.

(3) FSI (Front Symbol ldent!tier ). As described in 3.1, a special strategy is introduced in order to search for and identify front symbols. In an actual scheme of front symbol processing, the whole area is equally partitioned into 64 square regions and the number of black pixels in each region is calculated. The resolution of each region is 32 x 32 and one pixel corresponds to 4 x 4 pixels at the lowest level, where the total area is represented as a 1024 x 1024 array. Mapping from the low level to the high level is accomplished by logical AND of 4 x 4 pixels at the lowest. Candidate regions for the existence of front symbols are regions which have more black pixels than a certain threshold. After the selection, the positions of front symbols in each candidate region are evaluated, searching for clusters containing a certain size of black

120

RIN-ICHIROTANIGLCHL MASAOYOKOTA,EIJI K.~,xx.~,(Jt('HI and Tt yE()T~MATI

Table 2. Syntax of weather chart Component

Pictorial pattern

Weather symbol Wind symbol High pressure Low pressure Tropical depression

Information from the component Location, variety of weather Location, direction, force

O ~ • ",:~.--';7x~H

"~

L Location, central pressure

T.D

Typhoon

T

Front symbol Isobar Coast line Latitude, longitude Numeral

Location, variety of shape, direction Locus, pressure value

~

No information about weather

t

Pressure value, typhoon number

@l ..... 9

Content of L~I"

Location Tokyo Osaka Fukuoka Ni~gata Sapporo Naha Chichi~ima Taipei Shanghai Pekin9 Seoul Chang(hun Khabarovsk Vladivostok Poronaisk

Weather Degree of Wind Wind Force Direction Clear 3 62 Clear 1 60 Cloudy 2 30 Cloudy 2 35 Fair Z 57 Cloudy

Z

Fair Cloudy Fair Clear Cloudy Clear Fair Clear Cloudy

2 2 2 1 3 2 2 6 3

Content of FST

14 7

Ig 50 59 17 13 40 59 63

Direction:O-63 Clockwise Content of IT

Shape Coordinate Attitude 27 T 72110? T 144~152 27 HD 228,159 4 4 HD 2811108 T 790~118 29 T 842~191 28 28 T 897,261 T 949,327 28 T:Triangle, HD:Half Disk Attitude:O-63 Content of NCT

Type

Coordinate

Attitude

I 0 2 0 I 0 0 6 9 9 6 L L

247,394 265.4~ 278,407 296,412 706,584 723,587 741,591 759.59~ 8531707 873,713 888.720

56 55 ~ 56 60 62 60 61 59 60

58

732,638

6~

855,759

57

Coordinate of End-points (321 56) ( 55 I01} (404 57) ( 52 224} (548 59) (565 545) C739 61) (981 381) (. 51 309) ( 80 320) C 49 368) (155 467} (240 390) (103 330} (.308 423) (499 566) C980 451} (644 641) 060 478) (197 543) C 48 489} (521 982) (.213 510) (199 533) (.980 555) (736 571) C498 577) (733 982}

C697 580) (976 915)

Length 351 416 568 399 34 163 154 451 5t7 80 870 28 273 467 528

C 45 633} (978 651) C843 699} (441 723) C898 730) (826 759) (468 763)

(361 980) (977 806} (835 738) (427 709) (874 748) (.784 720) (572 889)

543 598 46 21 25 60 166

C849 769) O29 979)

(798 673)

322

(.232 979)

283

(611 766) (560 603)

Fig. 12 (al. Experimental results (extracted information).

171

Knowledge-based picture understanding

12I

2

Original Picture J

Cr~

After Extraction of Weather and Wind Symbols

°

/.

/

\

After Elimination of Constant Patterns

J

Candidates for Isobar Lines

k. L ~006

Extracted Characters and Numerals

Recovered Isobar Lines

Fig. 12 (b). Experimental results (processed pictorial data). area. The search operation is also executed on the same level (i.e. one pixel corresponds to 4 × 4 pixels of the lowest).

Then. the identification process is carried out on the lowest level in the pyramid. Each front symbol is normalized in shape and size. but can be freely rotated.

122

RIN-ICHIRO TANIGUCHI, MASAOYOKOT,, EIJI KA~XAGUCHIand TtxEo T ~ t ~ r t

Therefore, templates must be prepared for all directions. However, as the direction of a front symbol is constrained by nearby front symbols [see Table l(a)], candidate templates can be narrowed down. The result is entered in the FST (Front Symbol Table). (4) CCE (Constant Component Eliminator ). After the picture is processed in FSI, the processed picture no longer has regional patterns, in other words, it is a line drawing picture. Therefore, in CCE, in order to simplify further processings, skeletonization ~s} is first applied to the processed data. Then, fronts, which connect front symbols, are erased, by testing whether a straight path between two front symbols exists or not. If such a path exists, it is judged to be a part of a front and is erased. However, at the ends of a front there is no such connecting path and they remain undeleted. In this case, the system estimates the ends of the front, based on the body of the line. After eliminating fronts, constant components are erased by the method described in 3.2. (5) ICR (Isobar and Character Recognizer). After the elimination of constant components, it is expected that only isobar lines and characters remain in the pictorial data. However, the algorithms do not work perfectly and small portions of other patterns remain undeleted. Therefore, the system must discriminate isobar lines, characters and the others. Standard characters are described in Freeman's chain code (in the knowledge source) and the system examines whether each portion of line drawing pattern can be matched with the standard patterns. The practical procedure for the discrimination is as follows. 1. Find an end point of a line drawing pattern and trace it until a singular point is found. 2. If the length from the end point to the singular point is smaller than a certain threshold, the system examines whether the pattern is a character or not. 3. If it is a character, the identification result is stored in the Numeral and Character Table (NCT). Otherwise, the path from the end point to the singular point is deleted. 4. If the length is greater than the threshold, it is a candidate for an isobar line and features (length, starting point, etc.) are recorded in the IT (Isobar Table). 5. Repeat steps 1 to 4 until all end points are examined. 6. If there exist paths without end points, choose one appropriate point on each path and execute 1 to 4 for each path. After candidates for isobar lines are extracted, the system tries to resolve ambiguities and contradictions in isobar lines. In this case, the longest path has the highest priority and the system selects isobar lines to maintain good continuity. (6) SYA (Syntax Analyzer). After each pictorial component has been recognized, acquired information

is verified, reconstructed and rearranged in SYA. The main purposes of this module are translating a string of numerals into a numerical value, recovering broken isobar lines, inferring the pressure value of each isobar line, etc. In ICR, some short segments of isobar lines are interpreted as the digit 1. They are deleted here with the aid of the knowledge which specifies the ranges of pressure values and typhoon numbers. This is a typical example of the verification process in this module. In isobar line processing, it is not necessary to deal with the pictorial data at the highest resolution level. The procedures are executed at the level where each pixel corresponds to 4 x 4 pixels at the highest resolution level. The mapping relation is logical OR of the 16 pixels. Details of the procedures are described in 3.3 and 3.4. As Iocational information acquired in previous modules is represented in an x - y coordinate system on an image plane, SYA transforms it into a representation in the longitude and latitude system. Finally, SYA rearranges the information and outputs a Semantic Table as the result of the interpretation of an input weather chart. 5. DISCUSSIONS AND CONCLUSIONS

Our experimental approach to a weather chart understanding system has been described in this paper. As can be seen from the experimental results, the system is working almost perfectly. This is because a structural model of weather charts and a knowledge of weather charts and meteorology (i.e. KSC's and KMP's) are well organized in the system. This organization of the structural model is facilitated by the fact that weather charts are somewhat restricted images. However, the concept of a modular control structure driven by a knowledge source is applicable to other picture understanding systems. In the current version ofWERP, a pressure inference procedure has not yet been fully incorporated into the system. Under simulation tests, the procedure is being examined to determine whether it can infer pressure values correctly when broken isobars are not completely restored. In any case, the inference procedure operates quite well when isobars are completely restoredJ 9} Therefore, a complete version of WERP will be finished soon. At the same time, we are developing a weather chart retrieving system based on the semantic contents of the charts. This topic is not discussed here because it is a little outside our present focus and because the system is not yet well developed. (A brief introduction to this system is provided in Reference 10.) In such a picture retrieving system, the computing cost of picture processing becomes a serious problem. The present WERP also has a computing cost problem, but we have already begun to improve picture-processing algorithms in order to reduce computing time.

Knowledge-based picture understanding

1.

2. 3. 4. 5.

REVERENCES E. Kawaguchi, M. Yokota, T. Endo and T. Tamati, An understanding system of natural language and pictorial pattern in the world of weather report, Proc. 6th IJCAI. Tokyo, Japan, p. 469 (19791. E. Kawaguchi, M. Yokota, T. Endo, R. Taniguchi and T. Tamati, An information understanding system of basic weather report, Trans. IECE Japan E64, 71 (19811. R. Taniguchi, M. Yokota, E Kawaguchi and T. Tamati, Knowledge based recognition system of weather chart, Proc. 5th ICPR, Miami, USA, Vol. I, p. 333 (1980). S. Tanimoto and T. Pavlidis, A hierarchical data structure for picture processing, Comput. Graphics Image Process. 4, 104 (1975). H. Freeman. On the encoding of arbitrary geometric configurations, IRE Trans. Electron. Comput. EC-10, 260

123

(1961). 6. M.J.B. Duff, CLI P4 : a large scale integrated circuit array parallel processor, Proc. 3rd IJCPR. Coronado, USA, p. 728 (1976~. 7. A. Rosenfeld, Adjacency in digital pictures, Inf. Control 26, 24 (1974). 8. C. J. Hilditch, Linear skeletons from square cupboards, Machine Intelli.qence IV. B. Mertzer and D. Michie, eds., p. 403. University Press, Edinburgh (1969). 9. R. Taniguchi, E. Kawaguchi and T. Tamati, Isobar processing in weather chart recognition, National Conl~ention Record. Institute of Television En~lineers Japan. p. 21 (1981 t. 10. R. Taniguchi, M. Yokota, E. Kawaguchi and T. Tamati, Understanding and retrieving system of weather chart, Proc. 6th ICPR. Munich, Germany (1982).

APPENDIX : CONFIGURATION OF ISOBAR

(Chart) Weather~ Chart / #

I

Pictorial

PreProcessor

I

Chart I Generator

~Sentence ~(Sentence) IGenerator /WeatherReport\ Sentence

Semantic I Picture Anal yzer and " Recognizeri -[ Synthesizer WERP I

Qo::tion /

Semantic l c Sturucture Generator

SyntacticI Analyzer

I

About the Author--RIN-ICHIROTANIGUCH!was born in Tokyo, Japan, on 25 November 1955. He received the B.E and M.E. degrees from Kyushu University, Fukuoka, Japan, in 1978 and 1980, respectively. Since 1980 he has been working as a Research Associate at the Department of Information Systems, Interdisciplinary Graduate School of Engineering Sciences, Kyushu University. His current research interests include picture recognition, image processing, picture database, etc. About the Author--MAsAo YOKOTAwas born in Miyazaki, Japan, on 12 September 1949. He received the B.E. from Kyushu Institute of Technology, Fukuoka, Japan, in 1972, and the M.E. and D.E. from Kyushu University, Fukuoka, Japan, in 1974 and 1982, respectively. Currently he is working as a Lecturer at Kyushu University Hospital. He has been studying automatic processing of natural language and linguistic processing of biomedical information. About the Author--EUl K AWA~;t'CHIwas born in Kumamoto, Japan, on 27 November 1940. He received the B.E., M.E and D.E. degrees in 1964, 1966 and 1971, respectively, from Kyushu University, Fukuoka, Japan. Currently he is working as an Associate Professor at the Interdisciplinary Graduate School of Engineering Sciences, Kyushu University. His main technical interest pertains to speech recognition systems, digital picture processing, computer systems, etc.

About the Author--Tt ~EoTA~tATIwas born in Tokyo, Japan, in 1922. He received the B,S. and D.E. degrees in Electrical Engineering from Kyushu University, Fukuoka, Japan, in 1945 and 1961, respectively. Since 1945 he has been with the Department of Computer Science and Communication Engineering, Kyushu University. He is presently a Professor in the Department of Information Systems, Interdisciplinary Graduate School of Engineering Sciences in the same university. He is currently interested in the areas of pattern recognition, natural language understanding, machine translation and mathematical linguistics.