Towards color picture processing

cmlPUTEIl G1W'JIICS ,\:>;0 mAGE PIlOCESS1:>;G

(1973) 2, 347-354

Towards Color Picture Processing TAKAYASU ITO CCII/ral Research Laboratory, Milsubishi Elcctric Corporatioll, Al1lagasaki, Japan

COllllllunicated by A. Rosenfeld

Reccived July 23, 1973 Picture processing by computer has been of considerahle interest in computer science. So far, most efforts in picture processing have been concemed with monochromatic pictures. But, as everyone realizes, actual scenes and pictures all contain color, and color information plays important roles in human perception and recognition of scencs and pictures. This suggests the importance of using color information in picture processing by computer. In this papcr we discuss the technical feasibility and effectiveness of color picture processing, and we describc some research efforts toward color picturc processing in our laboratory.

1. nASIC PROnLEMS IN COLOR PICTURE rnOCESSING

There are several approaches to color picture processing. But we are interested in a general purpose approach from the standpoint of computer science. In this section, we discuss some basic problems and a conceptual framework for color picture processing by computer. Framework of Color-Pictllre-Processing 1'eclmology

We think that the framework of color-picture-processing technology consists of (a) (b) (c) (d)

computer-oriented chromatics, color picture classification and recognition techniques, color pattern design, and color pattern display,

in addition to the standard picture-processing technology. Within this framework, there are various approaches to color pichue processing, including (1) (2) (3) (4)

information-theoretic models, semantic-information-processing models, psychological models, and physiological models.

Among these approaches, we arc interested in research based on sematicinformation-processing and information-theoretic models. \Ve are currently working on these models in our laboratory from the standpoint of computer science. COl'rrighl © 1973 by Academic Press. Inc. All ri~hts reproduction in an)' fonn rC'servt·d.

or

347

ITO

348

Semantic-Information-Processing Models

A semantic-information-processing model of color pichlres must have the following functions: (1) analysis of color information, (2) local feature extraction by effective use of color, (3) construction of color picture data bases, (4) numerical measurement, analytic description, and syntactic representation of color pictures, (5) structural analysis of color pictures, and (6) assigning semantics to color pictures. Thus, a semantic-information-processing model is expected to playa principal role in color picture processing by computer. But the evaluation of color information, and of pictures as information sources, should be made from the information-theoretic standpoint.

Information-Theoretic Models From an information-theoretic standpoint, color picture processing can be modelled as shown below: Inrormalion source or color pictures !----Optical informolion

r---Eleclrical signal

r---Structured information Feature ell/actor r - - - L o c o l semantic base

This inlormalion-Iheorelic model will provide compuler-orienled chroma'ics.

0

basis for

This information-theoretic model will provide a basis for computeroriented chromatics.

COLOR-PICTURE PHOCESSING

349

2. A LABORATORY Fon COLon-PICTURE pnOCESSING

In order to develop color-picture-processing technology, we need a laboratory for color-picture processing so that we can examine the basic problems and techniques. This laboratory should have a flexible and expandable organization with respect to (i) hardware features, (ii) software features, and (iii) systems organization.

We require this laboratory to have the following facilities: (A) Input Materials color film color pictures color scenes (B) Input Devices color film reader color FSS for reading color print/hardcopy materials color TV camera for color-scene analysis (C) Output Devices color CRT display color graphic display color film writer color-picture hardcopy unit color curve plotter (D) Color Analysis Software noise filtering and adjustment of device fluctuations analysis of color into color quantities (hue, lightness, chroma) with adjusting capability for (1) sensor characteristics (2) material characteristics (3) environmental characteristics , (4) film-processing effects (5) I/O device characteristics (logical, electrical, and mechanical features) . coding of color information (E) Color-Picture-Processing Software data base management system for color-picture files data structuring software for color pictures construction and transformation of color pattern spaces preprocessing software using local processing local feature extraction local classification algorithms numerical measurement procedures

ITO

350

colored representation of pictures color-picture-processing'languages and their compilers (F) Semantic-Information-Processing System conversational list processing languages and their compilers extensible languages and their compilers color association system QA system based on color picture data bases semantic interpretation of color pictures by an inferential programming system (G) Computer System For Color Picture Processing parallel processor for color pattern space flexible microprogrammed implementation of picture processors which involve data structuring in microprogrammed firmware computer network for color pattern processing. 3. ON COMPUTER-ORIENTED

CHnO~fATICS

Computer acquisition of color pictures becomes the first requirement for color picture processing by computer. As is seen from our infommtiontheoretic model in Section 2, we have color information in the following ways: (i) (ii) (iii) (iv) (v)

optical waves electrical wave signals digitized electrical signals coded information semantic information.

In order to develop color-picture-processing technology, we must establish a method of computer analysis of color information along the above lines. We emphasize the importance of "computer-oriented chromatics" as the first step of this process. In this section, we discuss some aspects of computer-oriented chromatics.

Information Quantity of Color Pictures First, we should know how much information is contained in color pictures, and how much information is available in color picture processing. We assume that (m X n) digitized color pictures are presented for computer analysis. Suppose that each point is separated into k colors by color filters or color sensors, and each color has d-Ievel quantization. Then each pichlre may be thought to have the following quantity of infonnation, depending on the treatment of color information: (1) Monochromatic Case

QI =d X

III X n

(2) Frame-Wise Chromatic Case Each picture is separated by k filters, and picture processing does not closely incorporate the relation among the k colors.

Q2

=k

X

d

X 111 X

n

COLOR-PICTURE PROCESSING

351

(3) Point-Wise Chromatic Case Each point of the picture is separated into k colors, and picture processing utilizes the point-wise color information.

03 =d k

X 111 X n

Thus we have the following relation:

01:02:03 = l:k:d k - l • As is seen from this analysis, color pictures contain a large amount of information, compared with monochromatic pictures. This suggests the following; (1) Color information provides high-quality pictures from the standpoint of picture processing. (2) Color information allows easy and efficient picture processing methods by effective use of color. (3) Color infonnation is useful for semantic interpretation and description. From a classification-theoretic point of view, the capacity C of a decision function can be defined as C=2D, where D is the freedom of dependence of the variables in a decision functions (or the dimension of variables in a linear decision functions). Then the capacities corresponding to 0.,02' and 03 are in the proportion

C(01):C(02):C(03) = 1:logk: (k - l)1og d. Color Ouantities and Color Representation

The nature of "color" has been studied from various standpoints, and a field called "chromatics" or "color science" has been established. But, so far, chromatics is concerned with such properties of color as (1) optical information, (2) color information by human sensing, and (3) color displaying and representation. But in color picture processing, we need to modify traditional chromatics so as to make it appropriate for computer analysis of color information. Color analysis and synthesis procedures are hoped to have the following aspects. Analysis of color quantities. As is well known, the ideoretinal color of objects is their color under a white-light source. Object color is specified physically by colored light partially subtracted from white light. The following three quantities are used to specify the colors of objects: (1) hue, which is specified by the hue circle (or color circle), (2) lightness which is the value of brightness or darkness of hue, and (3) chroma, which represents the intensity of hue. There are several methods for specifying these quantities. Among them, the Munsell method is most popular. The relations among the Munsell method

352

ITO

and other methods (Ostwald method, C.I.E., etc.) are known, so that we may use other methods whenever appropriate for a specified purpose. Color specification method. After the analysis of object colors, we have to represent color information as a dat'l structure for computers. For this purpose, we must introduce a color space, in which we can perform a systematic treatment of hue, lightness, and chroma. Two different approaches are proposed for this purpose. (1) Color Appearance System This is the method of comparing a given material with the material standard specified by a color chip. The Munsell and Ostwald methods belong to this category. (2) Color Mixing System This method is based on a mixing experiment using three basic standard lights; color quantities can be represented numerically by physical and psychological experiments. The C.LE. standard is a typical system based on this approach.

Among the various methods, the Munsell method and the C.LE. standard are commonly used as the standard methods of color specification.

Primary Factors of Analysis of Color Information The following factors affect computer analysis of color information: (1) characteristics of input devices-film reader, TV camera, FSS, etc, (2) characteristics of color filter and color sensing material, (3) properties of light source and light quantity, (4) properties and conditions of objects, (5) environmental conditions - background color and size of field of vision, (6) boundary conditions in color separation, (7) mechanical and electrical conditions, and (8) film characteristics, and characteristics of display and hardcopy systems.

Other Aspects of Computer-Oriented Chromatics There are various problems involved in developing pattern processing algorithms for color pictures on the basis of the above ideas. Among them we would like to mention the importance of the following topics: (1) Distance measures and information quantity in a color space. (2) Color data bases, including specification of input devices, data mode, sampling mode, environmental conditions, and characteristics of color picture media. (3) Color association models based on the data structure of color information. (4) Color mixing procedures for color design. (5) Color arrangement procedures under various specifications of color harmony and environmental conditions.

COLOR-PICTUHE PROCESSING

353

4. AN ATTEMPT AT COLOR PICTURE PROCESSING BY cmlPUTER

An attempt at color picture processing by computer is being developed along the above lines in the Central Research Laboratory, Mitsubishi Electric Corporation. So far, we have developed the computer system shown below for this research purpose. Several new units are to be installed in this system, including (1) color TV camera, (2) new film reader, and (3) several I/O peripherals for the PDP-ll, PDP-I5, and MELCOM7500. Icolor filml reader

I Film

writer

IColor display Icolcomp with I color pen I IToblet with pen

I p

~ U N I

B

1------ US

PDP -11/20 Arithmetic unit MT DISC Line printer PDP-15 DISC Line printer

DEC tope t _________ ~

(MELCOM75001

This system is being developed for color picture processing along the lines envisaged in the above sections. The research and development of color picture processing has just started and must be carried out consistently to yerify the effectiveness of color information. As of April 1973, we have carried out the following experimental and fundamental studies using this system. (i) List Processing and Inferential Programming Systems (a) Resolution theorem provers. (b) Compiler for extensible languages-simulator for Compilercompiler machine. (ii) Picture Processing Software (a) Simple compiler for a symbolic manipulation language. (b) Handwritten character recognizer. (c) Various picture preprocessing routines. (iii) Color Picture Analysis (a) Color specification programs for film data, based on the Munsell renotation system and color chips called JACOL COLOR CARDS M. (b) An attempt to derive experimental equations for the color information of each picture element.

This research is now being continued in connection with the Pattern Information Processing Project of the Ministry of International Trade and Industry of Japan. ACKNOWLEDGMENTS

The research and development of color picture processing has been carried out through the cooperation of Dr. T. Ito (principal investigator) and a number of research groups, including:

354

ITO

(1) for the inferential programming system, Mr. A. Fusaoka, Mr. S. Ando, and Mr. H. Nakajima; (2) for color picture processing software, Mr. T. Narikawa, Mr. M. Fukushima, and Miss K. Suzuki; (3) for picture processing compilers and hardware aspects, Mr. K. Akita, Mr. J. Shibayama, and Mrs. K. Itoh; as well as others (Mr. Sato, Mr. Fukada, Mr. Honda, Mr. Isoda, Mr. Takeo). We thank them for their cooperation.