Visual Perception of Robot Environment – Construction Methods and Means

Copyright © IFAC A rtificial Inte ll ige nce. Leni ngrad . USS R 1983

J

VISUAL PERCEPTION OF ROBOT ENVIRONMENT CONSTRUCTION METHODS AND MEANS V. I. Rybak, A. I. Boldyrev and A. V. Khomenok InstitUle of Cybernetics. Uk raine Academ y of Sciences. Kiev. USSR

Abstract. Methods for investigating and constructing the robot visual systems are discussed. These methods are developed for creation of adaptive robots, automatization of assembly, inspection and preorganizing of environment the robots are ~~ operating in • .,' Specific features of robot visual perception and their employment for 3-D objects recognition are analysed.

., V

Results of research in hardware and software of CAD system for the adaptive robots are presented. Keywords. Robots; environment control; adaptive systems; computer hardware; computer software; artificial intelligence. THE TASK OP ROBOT VISUAL PERCEPTION

rable differences in the nature of objects to be observed necessitating extensive studies in the field. The principal differences consist in the following:

Prom the very beginning of robots application in industry, i.e. since 1962, robots with artificial intelligence elements, or adaptive robots, are considered as one of the most promising tools for improving the industry flexibility. The key problem for development of such robots is the perception of environment, especially the visual perception as the most informative, accessible and potentially applicable to a vast variety of task in industry. The main goal of visual perception is a picture processing, object recognition and determination of the object's position in a working space, inspection. The picture processing and pattern recognition as an independent field of studies appeared earlier than the robot's visual perception problem. The picture processing and pattern recognition process is traditionally subdivided into the following stages: input of an image; segmentation of the image into homogeneous regions corresponding to some patches of the object surface; object identification. The process is based on the set of local procedure results and directed onto successive decreasing of information to be processed. Although obtained solutions can be an ~llent source of methods and alg~ithms when constructing systems >"of robot visual perception there are conside-

- 3-D objects data acquisition and analysis; - incompleteness of input data due not only to the picture noise but to the fact that objects overlap as well; - analysis of dinamically changeable working environment during 1s time period response (Agin, 1980) characteristic of a visual perception system; - interaction between the visual system and other robot's systems, and the operator. METHODS OF VISUAL SYSTEMS INVESTIGATION AND CONSTRUCTION The investigations in the adaptive' robots and corresponding visual systems are carried out at V.M.Glushkov Institute of Cybernetics Ac.Sci.Ukr. SSR on the basis of computer simulation and implementation of results in real systems. The algorithm debugging, testing, and estimation of limits of their applications are conducted on the imitation models by si283

284

V. I . Rybak, A.I. Bol dyr ev a nd A. V. Khornenok

mulation techniques. The software is verified on real sensors and manipulators. From this point of view we support the Project (Industrial Robot,1982) published by the Japanese group of CAM-I (Computer Aided ManufacturingInternational). The project involves development of the system-prototype for geometric modelling and control of movements of the robots. Such projects are now under construction in a number of scientific centers. The main goal of these projects is to provide users with the possibility to investigate and debug the robot control program by simulation of robot's movement on the display according to a given mathematical model. The results are further applied for the real manipulator control. Such approach permits to exclude a damage of equipment because of mistakes in programs and to construct manipulators, to estimate their service zone, to check different structures in different modes of operation and for different trajectories of movements. To our mind it is necessary to use the same approach for the development of systems of robots adaptation to their environment. The diagram of our research complex for adaptive robots construction is given in the Fig.1. Horizontal and vertical dotted lines separate in the Figure the hardware dependent and independent parts of the research complex. On the left side are blocks for investigations into control systems, on the right are the blocks for investigations into perception systems. Even though the mentioned above distinctions of the visual perception task from the picture processing and pattern recognition complicate the image analysis they allow using some specific features of robots and conditions of its functioning to reach simplification. The majority of visual perception systems are intended to the photometric data processing in spite of impossibility in a common case to get the space parameters of objects on the basis of a 2-D image. Such systems deal with some simplified cases. Results are also available on objects recognition from rangefinder data only (Navatia, 1977; Shirai, 1972). To the best of our belief both sources of data should be analysed. Taking into consideration results of processing of photometer and rangefinder data which complement each other nicely we shall obtain more reliable results than each of them taken separately

(Glushkov, 1976). The possibility to control by light sources, by visual sensor parameters and to manipulate objects permits of active influence on perception process (Glushkov, Rybak, 1976) and considerable decrease in difficulties that present themselves. The interaction between the robot and operator during the teaching and recognition processes may exert a significant influence on the quality of processing results and algorithms complexity. The results are displayed in the form convenient for the operator and he in his turn makes the right decisions in ambiguous situations, corrects parameters of processing procedures and achieves the required effect. The above-mentioned order of hierarchical picture processing was used in the first experimental adaptive robot projects (Feldman, 1969; Joutavi, 1971). Results of processing of some higher level data could not be used for correction of mistakes that are inevitable due to the local nature of working procedures. This drawback is removed in the heterarchical mode of processing largely (Winston, 1972). From our viewpoint the idea of processing control with employment of input data should be developed towards the analysis by synthesis when the results are tested by the bottom-up and top-down initial and synthetized data comparison (Gimelfarb). As may be seen from Fig.1 the robot languages play an important part in the research project. Robot languages determine not only the efficiency of investigations but also the practical value of results as a convenience of operator-robot interaction, complexity and time of reprogramming are now included in a number of commercial parameters. Analysis of the known robot languages shows that the most successful of them have such properties of common languages as modularity, high depth of subroutine imbeddedness, a broad spectrum of data types, of arithmetic and logic operations, the feasibility of do-looping, of conditional and unconditional branching. Among the specific characteristics of great importance are the feasibility of parallel processing, of language extenSion through introduction of new functions and operators, the availability of control commands for special equipment and sensors, for the dynamic interaction between models of all components of a robotic system. Decision-making capability based on the more or less complex models is in-

Visua l Pe r ception of Robot Environment

to all robot languages. Among them it is advisable to distinguish three levels: the robot-dependent, robot-independent and the task-oriented level. The first-level languages are strictly oriented towards specific manipulator and sensor models for whose its are intended. The second-level robot languages interact with a variety of manipulator and sensor models as well as manipulation objects and environment. The third-level robot languages include the information about tools, special devices, mechanical and other features of the equipment. They are intended for task description in a usual technological manner.

heren~

By xirtue of a relative simplicity ~the manipulator structure the deve~ppment of adequate models,.o~ ~on­ tr81 systems and language fac~l~t~es presents no special problems. Not so with sensors. The development of a "good" 3-D object model is a key problem in the visual perception design. It is clear that the model has to take into account the specific features of robot's visual perception and to satisfy such conditions as universality, constructiveness, invariance, uniqueness and economy. In spite of the efforts of many scientific centers and intensive investigations the specific indu~trial robot applications with 3-D obJects perception are existent only. The successful research project demonstrations integrate many different control and sensor functions. But they cannot satisfy industry needs on many aspects. Robots with technical vision systems are considered to be closer to implementation in practice (Businaro, 1982). The most important parameters of technical visual systems are the low cost, reliability, high speed of image processing and simplicity in handling. Such results can be attained at the existing level of hardware and software by the hardware implementation of image processing algorithms and the development of a special visual systems for specific applications only. From this po~nt of view it is expedient and prof~table to build the applied perception systems with the help of CAD systems using a universal set of modules. Therefore the research complex project shown in Fig.1 is intended not only to test new principles and algorithms but also to develop industrial robots adaptation and control systems. It is evident that this complex may be used as CADs are traditionally used and thereby to close the process

285

beginning from the designing of machine parts to the execution of such operations as inspection. loading of equipment and assembly by robots. VISUAL PERSEPTION SYSTEM HARDWARE The computer complex SM-4 - Electronica-60 provides the hardware support for our research project. The system hardware architecture is shown i~ Fig.2. The standard set of SM-4 devices has been complemented by developed modules for the image and distance data input and by the module for displaying the input data and intermediate results. SM-4 is connected with two microcomputers Electronica-60 through an interface of parallel exchange. One of the microcomputers is used for the manipulator control and the other is a core of the technical vision system. Computer vision system is accessible for industrial applications just now. The image input module (Rybak, 1980) permits an introduction of 512 x 512 pixels with 256 grey scale levels in the real time during 1/25 s. The module is programmed to interrogate several TV cameras and to execute various modes of exchange with the computer along channels of different capacity. A laser rangefinder (Rybak, 1976) was developed on the basis of the image input module for measuring distance from the TV target plane to object surfaces. The laser beam is spread out to a width of a strip with a cylindrical lens. The rangefinder measures the distance within the limits from 1 m to 2 m with a relative error no more than 1%. The image output module is intended for interaction between the operator and the complex and permits the realtime generation of a 512 x 512 pixel image with 256 grey scale levels on the TV screen. It is possible to read and to store any window of 8 x 8 elements simultaneously and to read any slice of a picture. The color picture display is possible too provided the image storage is extended. An angular manipUlator with seven degrees of freedom (including the grasp) is connected to the microcomputer Elektronica-60 through an interface developed at our institute. VISUAL PERCEPTION SYSTEM SOFTWARE Visual perception system software development is based on the models of

V. I . Rybak, A.I. Bo ldy r ev and A. V. Khomenok

286

robot parts. The objects with plane and quadric surfaces are now chosen for investigations because they make up about 80% of industry output. l~thematical and information object models are developed (Boldyrev, 1979) that satisfied the above-formulated requirements and are oriented towards recognition. The models are represented as a hierarchical structure. The first level of hierarchy is formed of surfaces with parameters corresponding to them. The second level is a set of patches composing the object surfaces. Objects models are on the third level of hierarchy. Geometry of patches is represented in a logic-analitical mode and photometric characteristics are described by the Lambert law. The plane object model is specified by a parameter set including values as the object area, perimeter, number of holes, magnitudes of semiaxes of the approximating normed ellipse, etc. The geometry and kinematics manipulator model includes the lengths of links, initial positions of coordinat systems prescribed to the degree of mobility of the links, mode of transferences of the links, range of transferences and transformation matrioes. These models are at the root of the algorithms and programs to obtain initial data, to process and use the data in the manipulator control (Rybak, 1982). Routines are realized as the modules that may be corrected, extended and combined in different configurations to suit some specific applications. THE STATE-OF-THE-ART The set of system modules and their interrelatlons are illustrated in Fig.3. Modules of object space oharacteristics by stereo vision. 3-D object identification and manipulator geometric modelling as well as robot languages and system monitor are under development. Some algorithms such as adaptive thresholding at a given number of brightness levels. selection of homogeneous regions of the binare and half-tone images. computation of their parameters have implemented in pipeline hardware. Algorithms realization into special LSI chips and development of the problem-oriented architecture for implementation of decision rules of pattern recognition are the next step

of perception systems creating for industrial applications. REFERENCES Agin, G.I. (1980). Computer vision systems for industrial inspection and assembly. Computer. 13, N 5, 11-20. Boldyrev, A.I. (1979). Construction of Model of 3- D Bodies Bounded by Plane and Quadric Surfaces for Space Scenes Description. Kiev, "Inst.of Cybern.A.S. Ukr. SSR" (in Russian). Businaro, V.L., L.Francione and others. (1982). Computer assisted vision in Fiat automobile works of assembly of body parts and software inspections of mechanical components. Proceedigws "Robots in Automotive Industry rnt ' I Con:t'. BirmiIlgh8Jil, U. K. , 163-172. CAM-I proposes standards in robot software (1982). Industrial Robot. ~, N 4, 252-255. Feldman,.A., G.M.Feldman, G.Falk and others (1969). The Stanford hand-eye project. 1st Int,J.Conf. of Artificial IntellIgence. Washington. Gimelfarb, G.L., V.I.Rybak, and V.I. Shulga. Getting of 3-D scene description making up from the simple polyhedrons of their image. Kibernetica, N 3, 73-76. (in Russian). Glushkov, V.M., V.I.Rybak (1976). Problems of construction of active environment perception robot systems. Proceedings 6th AIIUnion S~poSium on t~heo~ and Princip~ of Robot and MBn~Uia­ tor Arran~ement. Section 2. 0liatti, 2 -32. (in Russian). Joutaki, M., K.Sato and T.Nagata (1971). ETL robot-1: artificial intelligence robot. JEE Ja an Eleotronic En ineeri 51,

•

Navatia, R., T.O.Binford (1977). Description and recognition of ourved objects. Artificial Intelli~encef 8. N 1~ t~-98. Ryba , V• • , A.I. 0 yrev, V.S.Lanbin and others (1976). Hardware for modelling of the "Hand-Eye" type robot. Proceedi~s 6th AII-Union Symp. of the eo~ and princiXles of Robot andnIpulator rra~ement. Sect,2. Toiiattl, 14~7. (in Russ~an). Rybak, V.I., V.S.Lanbin and others (1980). T.V.Image input Module for robot environment description system. in: The Theo!! of Robots and Artificial Intel~gence. Kiev. "Inst. of Cybern. A.S.Ukr. SSR", 47-51, (in Russian),

Visual Perception of Robot Environment Ryba~,

V.I. (1982). Investigation and development of visual analysis and environment decription systems of autonomous manipulation robots. Kibernetika, N 5, 95-100. (in Russian). Shirai Y. (1972). Recognition of polyhedrons with a rangefinder. Pattern Recognition, 4, N 3,

243-250.

Winston, P.H. (1972). The MIT robot. Machine Intelligence, 7, EdinburgJl, "Univ.Press" , 431-463 •

..• ..:.}f

287

N

0:> 0:>

FMS

Robot oontrol

r -- m---'

®l

I®

1

1 1

I

1

ITeaching I. box

Environment perception

.

- (!)-

-

l 1

I

I

Teaching box , I-I (tor general)

j~

. (tor

+>

r- I I

1'1

...

1'1

III

.. '"

'd

.... '"

I:l

'd

1'1

'd

+>

,0

0

Il:

'd

~

!l1 0

,I

fT\

(!)

,

....I:l1

'I 11 11

~

I

'"

<: H

1'1

Qj

I

Ul

~

r::r

11

III ;>;'

11

:>

L ___ _ _ ~

... 1'1

II 11 .....

Teaohing box (tor the spect. robot)

- I

..r'..

11-.\' . .. '

11 1

Illsensor 11 model

H

r =- .::----:. . -=- :;-<

tJj

o

t-'

Robot geometry model

Object geometry models

Scene model

0.

'<

Sensor models

'1

III

<:

~

III

III

'd

11

1'1

IL-------

III

L _______ _

'" ...o III

'd

1

---, I I

,0

o

'"I:l '"

---T-----

,--

Qj

:>

Qj

<:

'd

~

o

I

'"

I:l

'"

Ul

Il:

.L __

-.J

III ;:l 0.

;><:

::r o ::3

III ;:l

o

;>;'

Sensors sensor data

Pig.1.

A prototype structure ot robot sottware. - teaching system; 2 - language system; J - simulator system.

Sensors data

289

Visual Percep tion of Robot Environ ment

! SM - 4

CPU &:

RAM

U -

.. 'V.

BUS

Plotter CAloIAC

Special hardwar e

Blektro nica-60 - 2

Blektro nica-60 -

S1mbol d1apla,.

S1mbol displa,.

Contr. deTice

Technic al visual system

Manipu....- - - . 1

lat~r

Pig.2. Project hardwar e

archite oture

IV \0

o

Hardware

.1 TV-camera __

~Hietogrammator

~

I'quantization Adaptive

I··

Software

I Rangefinder I

TV-oamera

!

!

Image data input

\Rangefinder data input

I

I TV-camera & rangefinder calibration I T

I

I

I I I

~

I

1

T

I

1

j

I

I Region

1

t parameters calculation

l

Simbolic display

J

I

I I

[contour lines parameters caloulation

I I I

I False

I I

I

I

Interface

r•

Q!

er

'"

l I l

t

I

states

(without limitat i ons)

I

I Global lines extraction

I

I Models

I

data base

t:C

o

""'0'<

l

'"

;:l

0-

I

:> <:

I

;><:

;:r

J

Clustering

o

~1

!

I

I:neighbour Nearest ~

calculation ~

I

I

'1

t

~

H

<:

l

I

1 Optimal robot trajectory calculation

Fig.)

1

,

t Object

:>

ID

IRegion parameters calculation

J

1

I

l

Graph matching

Decision tree

I

I I

1

contours excluding

I

I

';:t. Iluanipula tor Icontroller

H

I

• I Homogeneous regions extraction I

I

I

A

"'"

I

regions extraction

&

I Contour lines extraotion

I I

I

f

Local oontours T

I

Raster half-tone display

<:

&

;>;"

1

I A

I Image rangefinder data eimulation

I

I

Homogeneous regions extraction

1

Depth data getting by rangefinder

I

Local contoure & regions extraotion

I

1

1I

Depth data getting by stereovision

I

ID ;:l

o

;>;"

1

I

IRobot trajeotory oalculation (with plane limitations) I

El

I