- Email: [email protected]
Knowledge Representation Approaches in Sensor Fusion
( :op\rigilr
©
11-' .\(
10t h I rit't1l1i.tI \ \ '()ll d ( :tllI .L:lt""'"
\ llIll ilh, FK(; , I'IK,
KNOWLEDGE REPRESENTATION APPROACHES IN SENSOR FUSION L. Pau '/'1'111111('((1
('II !i'I'I'lty
"I
li"llIlIlnl"
li lrlg, ;.,8,
li}\ - ]Sll!i
J..\IIglir.
/) 1'111111111,
crease s , the f eatu r e number l\F will incrcC!.se and the r ecogn it ion systcn i :l terms o f data fl o \o.'s , and classification tir.les . T he key r cquire Qe~t is th u s t o improve the ove r a ll f e ature ext r ac tion and :...;clecti cl: , so that NF .:1!ld NT i nc r e as e at a slo~e r rate than ~S. ove r~h e lm
This pape r r e\"icv,;S some k:--:'o ',·,.'lt.?dye r eF res02 i( tati o J1 ap? r c~ches d e vot e d t o the senso r f us i on i)r oblem , as e rlc ou~te r e d wh en~ve r images , sig~als , t e xt must be combined tc l?r0vide the in!:-'ut to a cont r o ller o r to a!1 Inferen ce proced~re . The basic steps in volved i~ the derivati on of the kno ~ ledge repr e sentati o n scheme, are : loc~te a re p resent at i o n based on e x ogencous A. corl t e xt informati on B. compare two repr e s elltations to filld out if they refer t o the same object/en tity C. merging s e ns o r based f eatures fr o m th e va ri ous r ep res entations o f th e same o bj ec t int o a new set of feat u r es o r attributes D. aggregati!lg the r ev r csen tati on s into a joint fused r ep r e sentati on , usually mo re abstract than each o f the sensor r e l a t e d represent~ ti on s The importallce o f sensor fusi o n steDs first from the fact that it is gene rally correct to assune that improvements in cont r ol law simplicit! and robustness , as well as better classi fication r e sults , can be achieved by combining div e rse informa tion sources . The second e l e ment , is that e.g . spatially distributed s en sing , o r ot herwise di verse sensing , does inde e d re qu ire fusion as well .
Mo re o v e r , b~ca~sc most r ecognit i ol! proceSSeS a r ~ hiera r c hical , they carlIlot proceed to the ne x t 10v e l o f !-lattern representati o n until old l1.ultis en sor data have bCe!l fus ed t oget h e r with flew sens o r d ata , Ar:1ong the difficulties involved , the critic.} l o n e s a r e the :ollowi~g : 1.
senso r div e rsity (nature , location , access , acqu isiti on delays , speed) : Exampl e 1 . 1 : r usion of imay e ry , si gna ls , bina r y data and ve rbal r epo rts ; Examp l e 1 , 2 : Distribu ted sensor netwo rks; Example 1 . 3 : Corrununications to o n -l ine vs o ff-lin e sens o rs; Example 2 .4: Diff e renc es between real - tiDe and s l o w-r esponse senso r s .
2,
diversity i n s enso r r.ata r cp r e sentu.tions made j o intly availabl~ f o r f ea ture e xtraction: Example 1 . 5 : Graphic , syntactic , graph theor etic o r statistica l r eprese~ tati ons .
KEYWORDS Artificial intelli gence , s en s o r fusi on , visi on , kn o wledge r epresentation , r eg istrati o n , sensors , signal processing .
3.
conf lict between ove r a ll re cogn ition tine and feature complexi ty , as detcruined by the NF feature extracto r s ;
4.
diffe r ences in handling o f dYlla~ ic inf o r ma tion , especially of sensor outputs , changing feature s o~ patter:l re p r esentat ions .
INTRODUCTI ON
This pape r emphasizes k no wledg e representation for sensor fusion , and not specific applications . Said app lications are r,los tly in tu r get classifica ti on [10 ,1 2,22J , F.W [17J , industrial visi on [7 , 13 ,24 J , and napping [2 0 , 25J ,
In this paper will be p res ented a number of "p p roaches "nd techniqu e s by which multisensor data can be fUged to impro ve the feature selecti on and the o v e rall pe rf o rman ce of a classification or interpretation system [7 , 22 , 2·1, LSJ . ~t is :;er.0rally assumed that the latter is made ou t by :
THE SENSOR DATA FCSIGr; PROCE SS
NS separate sensing devic e s o r k~ o wledg e s ou rces i E 5, i = 1, ... , NS p r ov idi ~g pat tc r~ r epresentations
In o rd e r to :
NF classificati on f ea t u r e s, derived fr o@ the NS sensors
(a)
fuse multisens o r nata
(b)
gene rate an updat e d r epresentat i on (X~)i+1 fr o l!l ( X ). for an unk n o wn patte rn o r entity w , wi tff' tepresentation rut. r based on the sen sors in sensing set S , a ~umber of basic steps must be accomplished in the ove rall fUSlon an d represe n tation p ~ ocess :
NT processing and classification stages in a multilevel r ecogn ition p r oce dure , using a t most NF feat u r es , The ir:lpor tance of i nro ri .. dtion fusion stens first fr o m the fact that it is generally co rr ect to as sune that i mprovements in terms of classification error probability , rejection rate , and interp r eta t i on robustness, can on ly be achieved ut tl.e ex pense of addit i onal independent features delivered by ~ore separate sensors . On the other hand , as the nunber NS of sensors in-
323
A.
~ocate a re p resent a ti on x . based on the set of exogcneous va l ues z (x') characterising -r:.he acquisi ti.un o r thle 6utput o f sen sor j E S; i n othe r words , i!lcor.d.ng sensor o r knowledge data must be chp.cked as to re ferring to the same ~ [ l, 3J
B.
Associ'·' ::i ng Tho representati on s Us ( 1) and Vs (2)
:\~
I
L to find out if they refer to the same entity (u ) ~ (v); this assumes the location protlem A. tu be solved, and the i~tersectlo11 of (Z,(Cl);JES(l) and z~(v);J -S(2)} not to be ~ amp t y [ 7 , 13 , 24, 25· J ~
c.
P
featun:· .. t.-:-:xiliUL,_· ':·....'c..:Lr~: E (1); dun (:";,) =....: structural 0"r~l.rh, ',.,·It.h L(,dcs 1,.;hlCrr clr-t f03t~rt's f.; th0 arc 13bels ~rL list uf f~at~rc !:,dcs il1 th0 vectur X~, ~ith the
X
vl L
(X) 1
~:~~~~i~j-~ra~.::,
Merging features from two re~rese~tations x S(1 ) a~d xS(~) of the sane ~atter~ into a
set x , \..;nere S=S (l)u S (2); thlS assumes the c0m~a~iso~ problem 9. to be solved. [9,21,22,24J
__ ti;:.n here-c·r ',·;it:-: 1.·_:1,. :5
;~o\\,·
D.
two representatio~s (Xs(llil 3!:d (X s (2)i) l~to a new higher level ~e~res e~ta=lGn (X ),+] , where S~S(l) ~ S(2); this
:.;
1 is:.
(X) 1
~,~
solved f~~r
S
feat~re
merging prublem C. to
~bcv~
st~~s
r_: 1:..:s ..'::' ::i
are also
E.
association process attribute data fusion
r
KI~u~led0~ tyt;;<~
: ']st
call~d:
L~s~d
~~'5tc~s
tee. -:I:c 5'_':.5 .. r
,-
,1
chi
C~;:-,SiS"':..L·:;". :'
tvct~:1."'''';s,
·.,·itr: i..!.:.s:riL:..i:...·...;G I t . ·,.~··.. :::::).i.:
5iI.l~ L·· ,. :~.:.. r, :t:·h ....·s . .;1...., tI1ct·vfl..'Yl..- I r~ . ·s,. :.t ::lTI~.ll·.: ~~T:....:"; !_YGC(.·SS tv _. . . '--;.(.lrL·ss ti ',: r:.:. ··.·:1· ~:.;l . rv-
~1d("'1---.:.:.;tC.
(Prc,blcr.l K) used in the data bast;'. This dilt,], b3S0 is assumed to be divided into files or objects cG!:sistirlq of the required informatio~ arld retrievable with the proper entry keys as specified iri the above problem statenents.
~,n'::5 ·,,:'r.t.J.ti(j;-~ issu!~ th!.-~:'l: rh, -..;.t Llll '",·ht:>rL· :l:n:l: rl·.: . ri...-s0:·lt.J.ti ·,11 ll:\'l:ls t)'le uf ser:suY Jd:~.
t1L,:-.
ISSUES
In specific implementations, the issues are the following: 1. The main goal is to require better discriminating features, e.g. features giving the highest correct recogIlition rates with lowest false ali3.rms and reJections [9J. 2. The next goal is to avoid having to use high
performance and high cost (single) sensors and computers
[10,17J.
3. The third goal is to achieve sensor diversity and thus integrity, to cope with sensor failures or data link failures and destruction [17,25J 4. The fourth goal 1S to achieve higher recogni-
tion, or segmentation, or understanding speeds, through parallel sensor data pre-processing [2,4,12,14,15J. 5. Sensor diversity offers segmentation, charac-
terization, and feature extraction clues when retrieving a~d processing features fro~ other sensors. This is the cross-fertilization effect at the query and understanding level [12,15J
6.
Sensor fusion allows to avoid stereo sensors a~d ~rocessing, if one sensor or several sellsors of different types in combination provide range [1,7,13,14J.
In terms of hierarchical processing it turns out that different sensors provide each higher level ico~s which are quite sinilar in terms of their fornal re~resentation. This allows us to ex~ect that a proger knowledge representation framework 1,o;ill make it possible to combine such icons, even if they originate in different inforwation sources. At the saDe tiwe, this weans that the selection and structuring of the knowledge representa-
is highly critical for the performance of
any sensor fusion system.
hi<2r.:,1.rcti·-'::~11
Notation: object/ process / pattern to be fied and classified 1
i E
identi-
dimension of a vector space E. for the lileasurement vector Z., i E S 1 sensor/information c~annel index
vector space, with its
nor~
:'";...;si'~.!.
St'::_::>,
Thrce
rc:.:~'rcsc:;talic'n ll'\'·~'ls ':'l.re cU:·lsid~'rl~d: 1: The lu~ost 10\'~1 f the rCl"Yi..,s,_::tatiG:: is () vecter spa.ce .. (i) uf dimc1;siun d , fl·.:, th,-t t is by controllinG tRe illumir:c.:'l.tiot, or ,1r:"th'."r LevE~l
contcxt/cnvirol:me~t3l
r~orcnetcr.
SX.:1ml..lle 1: I:". a tYl- iC.:J.I d=3 diLll::.si . Jl"dl S('·'...:II'_·, observed through a l~scr se:~sor, ass0ciJt~d tu a 2-D CCD array: Xl: 1st cuordir·;ate of l_c-,i:,t (I>Ci.ssivc·) x : 2:":d c()ordin.i3.te of i_vi:-.t Ci..lassi\'e) 2 x3: 3rd cocrdirlate (ra:-.c:L') ((lctive) x : o.nplitudc ef Si,:!"L.].} (LJrt..:y 1(:\-'e1) (j~~ssivc) 4 x5: freque~cy of las~r sot:rce (activ~) x6: wavelength of DeaSl:rLQe~t x (p~ssjve) 4 x : time (rassi\'c) 7 x8: estimated reflectivity (active) Xg: irrodi.:1tion field (active) Level 2: If X is the feature \'(:ctor 11. :.:: (i) :.:-rovided by L(~.(;el 1, the: L~':t:l 2 r~·l. rf:s'--::~·.+'::'.J.ti, 1S the list L.
(X)
dof1~cd
.s fallows:
10t G(i) ~e the: feature grai~h, ~hich sr~ci fies all feature ty~0S ~hich can be e!lCGU~t ered i~ £(i), and ostablishes the rolatio::s between those which are feasible/allGwable the: first sllblist ot L. (X) is the list L)f fcat~res ~resent i~ X.l the secol:d sublist coAtai~s all the labl:ls attac~ed to arcs c~c ~Gde ef ~hich ~t least is in the first s~blist.
ExanFle 2: If a F-ict~re is Cc.:·;.sid~red all c:lenents of which are nade ef ~riDitive shafEs, then G(i) is the gra~h reFrese~tatic~ of the correspondi~g picture descri~tio~ la~g~age f~~ objects
KNOWLEDGE REPRESENTATION (PROBLEM K)
'\\'t:
,~Y"-
th'...: ':-'D()\'t:
four problem, one of the most iml-)crt3::t cunsiderati0ns is the knowledge repr0sc~til
I~lPLEMENTATION
, .(!
tc bl:
c:.::alysis d3.ta fusion
=:1 ,'1.11 th(:sc:
d.
.:-;-
process and pusitional data fusion
c. u.
tio~
1:
b~
[1,6,14,16,19,21,22,23J.
alisnme~t
data
,~::-::. l"""-.':'~::;::..:l
L::r!":l.
~ggr~gatlng
assumes the T~~
!:="rE:dic~1t(:
allo~ed/expected
in the
sce~e.
The primitive shapes are: straight seSMe~~s, curved segme~ts, regio~s of specific share. The labels of graph G. are: length of these rrinitives, as observea in the sce~e and registered through coordinates in X., as well as other feature attributes (e.g. cu}vature, area, orientation, etc.). The values of these labels are 0Jk. Level 3:
The third level is a set of rules re-
presenting the context in which the scene is;
KIl()\dt'd gt' Rt'prt'st' IlI ;t1ioll .-'l.ppro ;lcht·s ill Sell sor Fusioll if M is the predicate form rep r esentation of the context graffiJ'lar , the" the result N. (X) o f the le vel 3 knu~ l~dge represen t a ti o n ~i l t be the list of predicate rules f~ om ~1 co~t3in i ng feat~res e n-
ca l match cannot be perfect , and ~ ill require two isomorphic distance measures d and d (2) b~ Slll S tween sensor ou tputs in S(l) and 5(2) : r
,. .',\1.::Itere d ir, L . (X) , \..:ith the cc r respor.ding attri -
b~t~ l~bels '~ flthe rcl e s . Mo re
..,
n~'E reS 0: .t t::C t.:~r ou .:;h tnL: St[;1a.r.tic sr alTU':1a r ~1 . , ~1 is s~iiused re~r0S0;1t ~ d b)' ~ set af logical ~r~di c~teS
'.1
~hich
c 0 rr es~o~d
t~ ,
~h~
r~writi~g
r~lcs
i::
.
. 'i ' Rt.:l~ r 1 : ',,' \ , ' d th 1.1:0,,1 "1 v..' , ': are C0ncate:!2:ted lIsts o f tt:?rmlr:.al a:"'.c ::0;: t~r l.\ ir.al sY:-:1bc.'1 1s fr,,', l:l tht! la:".guQge ~ ger:er.Jt es .
The tt!rr.1ir'.J.l symbcls u t this LlI.-:;:U.3.Ljt i nc lude all ft:ac.:.res o f . .;( j ) , 'JL: l_I.~1t;: me ;1te d b .... cthL'r symbols re -
t o th ~ CO~t0 x t . Thn list N ( X) ~ill be defi~ed by : ~ fi r sE s~blist o f.rules rl ~ s~ch that at 10J.st . . i::e fl..:J.ture ln L . ( X) 15 er.countt2 red in V ; this r:1>.:Clr:S '::1at L ~·ill be u:~ified ....·ith the ~redicJtc rcpres er~ ti~g r . cl secor.Q s"..lbllst , made ot the labels b of s~ch r~l~s ~hich are u~ificd ~ith L . (:;t. l~ted
Ou tput of problem K: The represelltation of the fused sens o r data is Xs = (Xi ' Li (X) , Ni (X) ; i E S) , covering ,Ill sen so r s In S . LOC ,\TI ON OF A REPRESENTATION (PROBLHl A) This process is equivalent to scanning the exo gcneous IJa r ameter space E of z. to locate by search a domaln Z contal~lng z~( w ) ,and to gene rate all assoclatJd rep r esentat10rls x . . This re quires a distance measure 6 between the vector z. ( :d and the set Z. ; il conpleted location will bJ characterised by 6(z. (w) , Z.) = O. In nost ca ses , the distance measute 6 iise l f may incorporate contextual info r mation , although this should be avo i d ed . On ly representations with e xogeneous va lues z. in Z will be considered hereafter . ASS()CIl,TION OF TlvO REPRESENTATIONS
(PR O ElLE~1
gr~~ . h
J
cODpariso~s
are possible :
statistical ma t ch o f sensor outputs z. , i mply ing and isomo r phisr:: betv.'een f eatu r es fk st r uctura l ma t ch fuzzy match
St atist i cal match (8 1 ) uS( l ) and v (2) re f e r t o t h e same en ti ty S ~(u) = w (v) if there e xi sts an isomorphism I be twe en t he sen s or outputs in S (l) and S (2) . Th is isomorph i sm i s an i nve r tib l e one - to - one co rres pondence . As ment i o ned i n t he above e x a mp les, t h e s tatisti -
r0~rt.:s t.: : ~~2ti0 ::S ;
req~ir0
~
G i ~ S (l))
lar I
nat ct lns
th~
cU ~i~rls o~
dist ~~ce
d_
b0t~~e!1
~lll
the
t~c:" grCl~ ' hs
."cd :: ( \. S (2) ) :
-( uS(l)) = -(\'S(2))<=>d c lC(t.: S (1)) ' C ('·S(2))) ~ .:::'5 m.) tchi LS d is t.:mccs d:=; ,
the
~os t
t YF iCJ.l o n es
ar c : (a)
Syrrunc t r ic differc!lc0 bet'. .;eL':', sub..;r.:l.i-hs t..... f ar:d l;(\"S(:2) ) , arid .:l cl)mbil~ato ri.J.l
G(u S(l))
search am0r:~ Jll ~ ossib l ~ subgr.J.lhs t o f i !ld , if a~y , d - close SL:.b~rd~~ls . It should !:ut be ~ x f~cted that the fl:ll g ra!Jhs ~ ill ~atch . (b) Lir.€a r combir:atio:L o f the nt.:.mbe:r o f fa CeS , cut sets , o ricntati ons , degrees a f the nod es , in the graphs G(u S(l)) ' G( v S (2)) ' Fuzzy ~Iatch (B3) This is typlcally the cas~ ~hc~ the sensor measu re merlts o r r 0I) res e ~tati o rls S(l) a~d S(2) are of such divers e natures that !)either a statistical nor a structural match are l"'o ssiblc . A fuzzy r c la ti o :lshiF R may the!) helF ill some instances : (a) either uS( l ) and v (2' have jOlnt observables S z. , and a olfferellce tcpresentation S ca~ be dJfined , such that a fuzzy r e lationship may be applied to i t . (b)
B)
This assumes first that these candidate represen tation from two different sensor famil i es have a non - enpty Joint location defined as Z12 = {Z ; j£S(l)} {Z. j£S(2)}. I f no t , the loca tlons arJ different , a ~d the rep r esentations u (I) ' v (2) cannot refer to the saDe entity S S to.! (ul = u.., (v) .
B2 B3
I
~atch (E:) case, th0 re~resc~t~ti o :.s uS(I)' \"S(2)3re terms uf strL:.ct-..:al ..jrurhs 0: h omoy el~ c-ous regIons 11'. the Se:-.s c r Si. aCeS S \ 1) , S (~) . Le': ....; ( . ) be- thest.:
The input to B is the set of all candidate repres entations f o r al l sensor types in S(U and S(2) , rctri 0ved in domains Z. determined by the exoge neous parameters z. (~)~
Bl
'<=>
!:~ thi~
J
Three types of
S (2)
Str~ctur~l
1
Examl--10 3 : If Ni 15 et l,)n~uaSt: describing th e context 3tld features about a sp~cific ~r ob lem , then N. (X) ~ill contai:l in the first sublist th~ ~redickte rules which use at least one ke )'word about the scene found in L. (X) . The attributes hereof would sa~" whet1 suchla rule may be enCQUllt ered .
J
(\.
if the global sc ene cont0xt is
s~ecifica!ly ,
o r uS(l) and v S (2) have no , o r too few, joint observables z . , or a differenc e representation e cannot be d3fined. However , the joint loca tion Z12 is non - empty . One may then try to define a cooccurrence matrix k , (m , n) , mES(l) , 1 n£S(2) , by sampling.z by succ~ssive subdivi 12 Slons , and reg l sterlng the cor r espondillg sa~p led representations derived from uS(l) ' v (2) ; S Z12(m , n) lS then equal to the nunber of sampl ed representati o~s ~hcre s en sor output s £S(l) and sensor output s £ S(2) remain with iW specified fixed ra~ ge s . n SENSOR FliSION OPERATIONS (P?OBLE;'l C)
The knowledge represe~tati on X = (X . , L . (X) ,N. (X), i E S) lS derlved f o r : 5
1
1
1
each senso r /information channel i , for wh ich the sensor and kno~ledge bases are available each observatio~ of the sce~e o r event There f o r e , t o be able to inte r prct , recognize or unde r stand the joi:Lt set of observations abou~ t he scene , obta i ned from all s enso r s , one m~st be ab l e to combine the above kno.'lec!ge provided fr om each channel . The basic combinations se r ve again dif f e r ent purposes . The mair. combir'..at i ons are : i . fusio~ operatio~s , used in know l edge query ar.d inference ii . ma t ching ope r at i ons used in r ecognition and unde r standing goals iii . p r ojection operat i ons , used to display jo i nt l y the outcomes o f the dif f e r en t sen so r s Fusion ope r ations : Le v e l 1: Fu s ion ope r ation s o n vec t o r s pace s E c an on l y be ca rri e d out f o r space s of s ame d i men -
L
:-~~f l
~,'.. ;, ,:'.
~~
~ "/":L' ~ '
i-:::;: 11·.; :nc': L1r':.:t:: st di m€·:lsi:.; r:,
I. ':.'.::'
: :ii:", ~-:,~·.si ",:· : ·; ~1 1
11"'.
1':111
. Ch r umQt ici ty t riang le (quot i ent ope r ati o ::s)
!:. hat
a re E.'xter.ded ·.· .:ith dummy
S:'.d C'~·S
~_:""01
~ : I:: th e £'..:scd ~ r uI.h v, prc j ect i\...'!1S art.:: idc :·,ti . . : ..ll t\.. t ilt..: se lt:: cti Oj: c f ,.;r .J.:;.h c ;.;.t s ets , o r
". !.'. , :.:-: r:: ~ : r",:;·.
:;'"
~;
.. - ..:.. .. - .
• . : ~.L
....
cf
r, ITtt..:..t !":' :;:: 3 ,
!:; ,~::!,_ .
_ .. _
::;....:~i l~
s
~\ : '.
·..,;nl:.:h
e::'C-:-t
i: ·..::i".,:Ce::.;:-:i5
c ~'21~s ,
0~t~
~hich
0th~r
node s ar0
m~ll ed.
I:: ::hL: SL':!i.:.:r.tic c..: l l:" S0:' ~1 after i l,:si0:: .J. :. L:·· ~ ·..:.:.:ti0 : : :""..J rr0sl··c :·.ds :. C} a f i l:0 r L'-!-,J.::i e d by ~ r C' i J.-::~l:' l:-.• : :h1...~ S J.~L' ::let ..:..'f co!: s:.r~'ii: ·.t r:...:.l es ~tr L~0h ~ll r ul ~s i:. ~.
L,_· ': t::l
j. :
R0 mJrk :
The must
o bvi (· ~ s
f~.rm
f or Level 1
1' [ 0 -
- t.:C t 1,. ! ~ ::;· : l.~ .... ~
10'"
_I r ... ..:':-.:r·. ·
:i.:; ~:-:
::: . .
_ ; e 1" :. "': :. ." ;. s :
\'l Sl b1 0 IR
;' .
· :l:::i ~) l·
L ':2rl ~~'
r:rJl:i sil: ';:' :Ll1 : .. . ~ '''::' ~ i
:.v
. .:.:
~tl.'._ :
ti·.,' i rl'1 :·.i :::. i.ll
: R _\
L
in ~~ ~
~~~
Th is
: ~"re
di s ; L:'.:
. '·\· ....·rL'.y
:)r
s rarh l ~
~r \.i ~~~ ti 0::S
of :h·~·
'r:1,'.:
l_~·.:
': (- r
·...::-.l~ S
..'-' ·...·~:i :; r-:
':l:l' l .J. ':
r :i...:! , ll"
'1,: it!1 .::),]r:h.:
() \'~r 1 3y
~ r~
sl"\· l.:r,~ l
s·': :·.c is
c3r ri0d Ot!t
~:idt.<
i:""i ·..: t
b~'
usi ~~:
,"~h ':'1I:; :(: ls ,
.::.~.:.. li t!d .
" .. ~ .• :. i : .:-5 L.:. l l::-; r·.. ! , s .:::;'. ' !." 1 ,-: 1
!.. .•:
1·..:· s
th(· f.1(: r Sed t i.:cltures(X) ;;~i\ 'L ;:' 1..~L·;~ d(· fi:: t:c fv!" 5(1), S(~) t h\:· 1. r . . ,blc~ l.S ::c''.·,: tu mv di f y Lh~ l·L· ln:sc·r.t.at :i..u;,s Xs (1) J.:-Id Xs (2) i: , crd l2 r
:\ss :Jmi:~,-!
m a tchi ',~ · l·,~-: ·,:,.s
,~r·_· :
:; 11 r.l.:':,L'h i r;,·;
rr un !.a t:.ern r ,: co,; 1"! i. . '- '- .. , :-,,;~ . ...1 . ': : .... i.~:lb ·) l" :-t (,(..Jd r",· L.::' i·. :-: $ ':·}·" s : ·:!.' ir . .; 1. r ucl.:;durt.,s .-),:. !.' ': '.: r:'.i:.i miza. :io:': l_r0Ct:d ures "r ',h ' ljLq .h ic ;.roj cctio ns: , r., ~l:::ch I u i::t s i1'. d= 3 (3 t r a nsL.1tions c1 ::d :3 U · t~ tl ·:~: ~ '.s) .md tch 5l.:tS c·f 2 i-,OHl ts (2 tra r.s latiur.s , i·r·,::"c l:-d~:n= s
1 cctntio l1, 1 sca ling, 2 l e ng th warping s, i l;;~.: t, h t·.,:::: r.Hn ~:· t r i za t i e ns) .mat (.:h i i J.( · sL'qmcnts in d:::.2
1
r ~t~ t i01 ~ ,
LL"v t.;' l
(2 tran slations ,
w~r~ i t\g)
l" , tr.e fus ed gra ph G, all match ing op esubg ra~}l matc hes; they requir e the ~ U r.lbi:~.-~torial s e .:. r ch for the syrrunetr ic di f ference :.:.. ..... tt,..' ...... \ s ;_:~) -j L'i · !lS . . ~ f S.1m(; deqr ·~e . 8 ne sFec id 1 case i s t.h\:~ r\utch i :'.' ;.; u f d! )Cnt,r o r refcreri.ct.; i"iodes . " ~'.
r ~lti~)r lS
~r 0
t u ,.:l~ ; ·!t;L3. tt..: ~ l i1i~ht 'r l \""·:~ '·l :C(·rr esl<.ta ti;..:.'rJ iX } ,+ 1 ir~ th e hicrarch i c ~l Clils sif i ca t io!~ p r oS CL·S:::> .
TI:1 S f i~ld 15 ~l V0 r~' c ornr l (· x Ol1 ~, beCfi US ~ :1 0body k:-_'J'''''S i: ~ L3C' I"'.t..::r a 1 110\-.' t o gene rat e highe r l Evel rCI ~ r QSc~tat i cns
wit h out dec r easing the
f i~~ l elas pcssi bl e tuols , althouqh :~ct v~ lidatcd ur: rnul t isens0r d~t ~ , ar~ : ge ::cra li sed s ymboli c re presenta t ions, ge ne ralis ed s hape d escri!:t ions, quad trcc s hape re pr e sen tati ons , S effi2.::tic r.etworks. The: agg r ~gat io n F roc(~ ss wi l l a l most eve r ywhc r ~ use s~ m an tic d ll d CO I~tcxtual informat ion conta i ned in S , h owever , in a diffc rel:t way . Th e mos t eff icie nt teCh tlique s ar e t h e
r~ c""'y :~i Li~:ri
r cl tl· .
SOr.1 ~
f Dll ow·i r.g : (a)
C0vccu rr er:cc cO!ldit ions on represer,tation
(XS(lII .. , (X S (2)) ;' su ffic ient n umber of l ower level rep resent atio ns ~re jO iJltly p resen t , with c ondit i o ns o n the ir s equ~ncing i~i the z doma in.
se mar.tic grammar N, the .3n::· : srr-...;c t. I,!ra. l ma tch l l;"';- () f stri ngs st r ·...;ct:.Jr ~·:l m. .i.tch i r.S o f strinS 5 w'ith ur, k no ..·.;n \"·L..:m t; :. ts . :. : ': :'. tt:x t d·_·!.. t.:: ::dcr. :: '/ ! ',-::.~~i:·. :.: " r i:u-:..ch·..:s 1::: :-.'_Hnbcr s o f Juir.t (,CC ;J -
(b) Semantic grammar o f rCI)rcs entatior:s : ;\ ·: ;ram mar is de fi~ ed wit h th e repr esen tation s (X 1 . . . S (1) > I. X S (2)) t. a~ I,O!l- termln a l e lem(::r. ts, a nd r(:i . . rE::S ~nt at1 0 : ~S (X ). i :-. thf.: r. . r o d u cti tJr, r ule:
!"(:' ". c·:s
cer ived
~~v~1
Tr: t:. h~ fus ed
.3 :.
m<.1t.ch i1.o.j
'1" _:r~i t i 0 t;s
st r i ~g s.
LI..:\··:.: l
l:
Th ~} '
~ r0
·:,,·ctr. r i.
t ~o
surt s , or igi~ati Jlg in ·:·r ir, a subsF a..c e o f the
a rc ~ cs sibl e f r o m the '..;. si.;, ,=:; t.he J oi nt norm ~c;~- li::ia r ~r oject ior.s or oFe rati o~s a re :._ ··_ss ib l t: aMlJ!".g st:'csr.aces of t.h e fused vc:c -_ C!:" :):'.ac 2: :i~ ~~r J'.l:, ~
ii.
of
S r .· 0.C 0 ,
i . rcJ ~ cti ~~ s
s [, a,:: ,::,
. Ch ange Gf scal e : do~ ai~
s!.a tia l
to spati al
time domai n,
fre q~e~cy
tine to time fr equ e~cy !-'hasl2 dc..ma i:-l to t-has e freq:..;.er;cy domain d0m a i~,
. D..:: ri':a.t i o;. : t i :n ~
':0 sF Eec
co1 0J r t.o colour diff erefi ces si ~~a l
le~ e l
t o diff e r anti al l evel value to s pec tr al derivative . ~ o ~ -l i::ear ~ r ojec t ions th r ough a st r uct·. .:. r ir.9 e le :TIE;. t: s~ectra l
cl ose d are a t o skelet on r Egio~ to bou~dary
. Quatc r tl io n ope r a ti ons f o r time de pe ndent en t ities
S (+1
is ·. . . h en this gr,d:unar is {} tree sr arrunar , bec a 'J se it i s thEn equi va l er:. t to a sc Da~t ic ~~ tw o r k, that i s a direct ed g raph in which th~ ~ o des ilr e: t he 10we r level r ep r ese n t ations , a~d th8 a r cs the r r ed i catcs . t\ s1.ccia l
ca s~
La bEl aggrE:ga tiGI1 o r relax at i o r.: In f <:: r e !: ce est imat i on must b e used to derive ag gregation r~ l~s f~r i ~divid~a l lo~er - leve l labe l s. These r ules rn~s t mini mi se a d i stancE meaSJre o r loss (c)
a~d
f'..: r. ctio:, bet.,.:een (X ' 2.+ 1 S lo~ e r-level
reF rE sentatior.s ar,d
r epre sentatl on s
~.
J.s!l
Gr af-h r educt i on : I f th e repr e se!; t ati on s (XS(l));. a nd (:
Senso r fu si o n i s the scene unders tand i ng appr oach wh i c h use s da ta or info rmat i on o f mo r e tha n o ne
:\~7
type . The
types
informatior~
ma~'
diffe r th r ough
[llJ Koho ne:l , T . Ass oc iative Memory , Springer ,
the :
!'le'.' York ,
physical sensing ~ril1ciple sensor l oc.:;. tior, sensor desig~ o r settl:: g se~SGr
[12J
[13J Sesl , P . , ~a i::
sai=-: '. . · a\ · el(: : ~St.h e r si_(:ctr..:11 the sc:",s o r data rate
dir, .:.: , d~t:.,;cti\·ity
;:~ r0fil'2
~:~d
or [l~
~ile~' / RSP ,
R., Ra~g0 image ~::d~rsta!l
~3t: ~ ~: :
Co::f .
cvffii~'Jtc r
19 85 ,
r0~ c ~::iti0~ ,
_ 1\
ki:: cO~SlS
f~r
ri: ,ci £. ll;;'s t:'L:msel\'c:s ::'t: l ..:.:',';i'
i . sig~als la~alo~ , di0ital , l o~ic) ii . 2- D o r ~ - D 3rrays s~ch as Im3~es o r r ~d i3 tic:, fields iii . proc0dur31 i:~forn~t.i u !~ , s~ch ~s te Xt., s~eech , soft~a r e , bch a vi o ral r~10s In any eve~t , s~!:sor di\' crsl:~' 3.~d 1.ardl1elism are int ri nsic t o sensor f~sic:~ . EJch scnsor ,' illf o r mat ion type is used ir ~ ~arall~l f u r fe atu r e e xtraction from sa i d senso r sl-'ccific in f o rmati on . Different features are thus extract0d ir~ pa ral101 . The knowl e dge based u!lde r standil1Y lj r oces s then takes place afte rwards by ~cc2ssiJ1g features of the different sources , by a hierarchical retrie val and i nference/recuynition i r ocess . This approach is especially useful when no sensor alone gives good f ea tures, o r orl 1~' does so at th~ expense of heavy processing alld use o f conte xt information . Knowledge repr e sellta ti on techniques indispc~sible
for
these serlso r fusi on tasks. REFERENCES
[15J
\'iSl O,: 4 30
~ .
20r.1i::i:·,i:.,; stereoEsls a"d S~dC~
~e r c~ : : ti c !: ,
1 St 'IEE:::: Cv :,: . ~1rtificial a~E ' lic~~ic~s , 1984 , ~ . 15 6
l i..;hti:',:: mect.?
and kn o wledge processing are
rec ocn iti or ,
I:~r:E
P!'\,.'C.
J ' lit c hi " , ,', .,
soft~3re
~
a:~d
1985
e ~vir o ::~ e ~t
The i:Jhysi c.J l sl...::~si: ·.S to thr ee c3ta0ories :
1978
L . F . , ~~ ll:tro ductioJl to inf r ared image
acqui sltl on
bac,d"'idth
p r epr0c~ssing
Pa~ ,
Proc .
ir:telli~er,ce
~il ,
E., .' !itchi~ , ;\ . , ;\G 0~~ n"'dl, ..- . K. , ~ rir.lL':,ts i :: cUmClr,ins r.3.;'0 e a :',d ir:t0!:sity ~,:d'-i~ ::-,a ; 5 , C"" ::-'': ' ...; r o.j. h . J.r.laqe Pr e e ., \',,1 ~l , 1083 , 1- . 3:,.15
[l6J ~,,: ·.da , '1, :a':,,!:·.""th.o:; , ,.
{1::85) ,
D0dh i.o",·" la , R .,
I..J :. tir:i,:l cuo l·e ruti cw v f k r.o\·. . l~d~t: S v'J !' C05 , T . F... ruciL0 :\. 1 . Cen tC'r , ~'-'·..:i :: (! C\""'ffi£ ',:tl...:r St...:'l.'\'12€:S . .~' :,
t~ctical
~ ~rf,Jr0 ,
~I~A
,:"m0 ri c.::. ;, I:.stit'.,:t·,2 of S:.';.1CL' . [18]
Forr.ul~ ,
P~~er
No 83 - 2398 ,
;\ c r G~', a~tics
3.1~d
P . J ., Pe mbc-r ton , l~ . G. , of tactic~l SCI... . ::~'~ , 1:: " r\I)pl lC.J.t lUI1S of Artificial l!lt~llig e :~ ce ", Pruc . SPIE , Vol . 110 485 , Rov.:l.J. n c. ,
,t\. \' . ,
ll~84) ,
~~Il:~ X tU3 1
~::al~'sis
i . 18 9 -
[19J
T.D ., L o wranc~, J . D. , fischler , M. A. , (198 1) , i,!1 ir:fcrencc tcchr:ique or int e grating knov,'ledgt: fr om disparate sources , Pro c . 1981 lilt . J . cOIlference on arti -
Garv~~ ',
ficial
intelli~ ence ,
319 -
[20J Lambird , B . A. , Lavine , D., Kanal , L .N., (19B4) , Distributed architectu r e and pa -
rallel non - directional search for k now[l J
[2J
Kan ode , T . Recovery o f 3- D Sh ape of an Ob ject from a Sigle Vie w. T. R . CMU - CS - 79 - 153, Carr.egie Me110n Univ 1979 Mil gram , D. L. Region Extracti o n Gsing Con -
ve r gent evidence , Computer Graphics and image Processing , 1979 , Vol 2 , 1- 12 [3J
Davis , L . S . and Rozenfeld , A. Coo pe r ating
Processes f or Low Level Vis i on : A Sur ~ey.
T . R. T . R .- 851 , Univ . "laryland , Jan . 1980 .
[4J
Fu , K.S. Syntactic Patterr. Recogn iti on
Academic Pr ess , 1974 [5J
[6J
Samet , H. and Rozenfeld , h . Qu adtree Struc tures f o r Region Processing. AD- A- 077568 Nov . 1979 Winston, P . H. Learning Structural Descriptions fr om Example s , The Psychology of
Computer Vision . McGraw Hill , 1975 , Ch. [7 J
[8J
5.
Mitchie, A., Aggarwal , J .K. , Multipl e senso r int eg rati on/fusion through image proces s i ng . J . O~tica l engineering , Vol 25 , no 3 , march 1 986 , 380 - 386 Keown, D. M. Knowledge Str uctu ring in Task Ori ented Image Databases , Proc . IEEE
Wo r kshop on Picture Data Processing , Description and Management , Aug . 1980 , 145 - 151 [9J
Fu, K. S . and Yu , T . S . S tat is tical Patte r n Cl assi f ication Using Contextual Informa tion, Wiley , New York , Dec . 1 980
[lOJ Wri ght , F .L . Fusion o f Multisensor data , Si gnal , Oct . 1980 , 39 - 4 3
ledge based cartograph i c feature extra~
tion , Int . J. of Man-Machin e studies , Vol . 20 , 1 07 [21J Nii , P . H. , FeigcllbauQ , E. A. , Ant on , J . J ., (1982)
SiYJO a l - t o - synbo l
transformation ,
HASP/SlAP case study , the AI Ma gazine , SFring 1982 , 23 [22J Pa u , L . ,
(1981) , Fusion of multisensor data
rec ogn ition , in : Pattern re coynition theo r y and applications , J . i~
~att0rn
Kittler , K. S. Fu , L. Pau (Eds), D . Reide l Publ ., Do rdr ec ht, in the NATO ASI series
[23J Smith , R.G. , (1985 , Ref"o rt o n the 1984 Dis tributed artificial intelligence work shoj- , The [24J Pau , L .,
1,1
~lagazine ,
Fall 1985 , 234 -
(1986) , Multisensor f us i on for
visior: using arti f icial intelligence, ir. G . Oll us
(Ld) , " Digital ima ge proces -
sing in ind ~ stria l applications ", IFAC Pr oc ., Pe r gamo~ Press , L o nd o~ , 1987 (ISBN : 03 43465 - H) [25J Lajeu~esse , T. : . , Sensor fusion , Defence
Science &
eng i~ ee ri r:g ,
Sept . 1986 , 21 -
©
11-' .\(
10t h I rit't1l1i.tI \ \ '()ll d ( :tllI .L:lt""'"
\ llIll ilh, FK(; , I'IK,
KNOWLEDGE REPRESENTATION APPROACHES IN SENSOR FUSION L. Pau '/'1'111111('((1
('II !i'I'I'lty
"I
li"llIlIlnl"
li lrlg, ;.,8,
li}\ - ]Sll!i
J..\IIglir.
/) 1'111111111,
crease s , the f eatu r e number l\F will incrcC!.se and the r ecogn it ion systcn i :l terms o f data fl o \o.'s , and classification tir.les . T he key r cquire Qe~t is th u s t o improve the ove r a ll f e ature ext r ac tion and :...;clecti cl: , so that NF .:1!ld NT i nc r e as e at a slo~e r rate than ~S. ove r~h e lm
This pape r r e\"icv,;S some k:--:'o ',·,.'lt.?dye r eF res02 i( tati o J1 ap? r c~ches d e vot e d t o the senso r f us i on i)r oblem , as e rlc ou~te r e d wh en~ve r images , sig~als , t e xt must be combined tc l?r0vide the in!:-'ut to a cont r o ller o r to a!1 Inferen ce proced~re . The basic steps in volved i~ the derivati on of the kno ~ ledge repr e sentati o n scheme, are : loc~te a re p resent at i o n based on e x ogencous A. corl t e xt informati on B. compare two repr e s elltations to filld out if they refer t o the same object/en tity C. merging s e ns o r based f eatures fr o m th e va ri ous r ep res entations o f th e same o bj ec t int o a new set of feat u r es o r attributes D. aggregati!lg the r ev r csen tati on s into a joint fused r ep r e sentati on , usually mo re abstract than each o f the sensor r e l a t e d represent~ ti on s The importallce o f sensor fusi o n steDs first from the fact that it is gene rally correct to assune that improvements in cont r ol law simplicit! and robustness , as well as better classi fication r e sults , can be achieved by combining div e rse informa tion sources . The second e l e ment , is that e.g . spatially distributed s en sing , o r ot herwise di verse sensing , does inde e d re qu ire fusion as well .
Mo re o v e r , b~ca~sc most r ecognit i ol! proceSSeS a r ~ hiera r c hical , they carlIlot proceed to the ne x t 10v e l o f !-lattern representati o n until old l1.ultis en sor data have bCe!l fus ed t oget h e r with flew sens o r d ata , Ar:1ong the difficulties involved , the critic.} l o n e s a r e the :ollowi~g : 1.
senso r div e rsity (nature , location , access , acqu isiti on delays , speed) : Exampl e 1 . 1 : r usion of imay e ry , si gna ls , bina r y data and ve rbal r epo rts ; Examp l e 1 , 2 : Distribu ted sensor netwo rks; Example 1 . 3 : Corrununications to o n -l ine vs o ff-lin e sens o rs; Example 2 .4: Diff e renc es between real - tiDe and s l o w-r esponse senso r s .
2,
diversity i n s enso r r.ata r cp r e sentu.tions made j o intly availabl~ f o r f ea ture e xtraction: Example 1 . 5 : Graphic , syntactic , graph theor etic o r statistica l r eprese~ tati ons .
KEYWORDS Artificial intelli gence , s en s o r fusi on , visi on , kn o wledge r epresentation , r eg istrati o n , sensors , signal processing .
3.
conf lict between ove r a ll re cogn ition tine and feature complexi ty , as detcruined by the NF feature extracto r s ;
4.
diffe r ences in handling o f dYlla~ ic inf o r ma tion , especially of sensor outputs , changing feature s o~ patter:l re p r esentat ions .
INTRODUCTI ON
This pape r emphasizes k no wledg e representation for sensor fusion , and not specific applications . Said app lications are r,los tly in tu r get classifica ti on [10 ,1 2,22J , F.W [17J , industrial visi on [7 , 13 ,24 J , and napping [2 0 , 25J ,
In this paper will be p res ented a number of "p p roaches "nd techniqu e s by which multisensor data can be fUged to impro ve the feature selecti on and the o v e rall pe rf o rman ce of a classification or interpretation system [7 , 22 , 2·1, LSJ . ~t is :;er.0rally assumed that the latter is made ou t by :
THE SENSOR DATA FCSIGr; PROCE SS
NS separate sensing devic e s o r k~ o wledg e s ou rces i E 5, i = 1, ... , NS p r ov idi ~g pat tc r~ r epresentations
In o rd e r to :
NF classificati on f ea t u r e s, derived fr o@ the NS sensors
(a)
fuse multisens o r nata
(b)
gene rate an updat e d r epresentat i on (X~)i+1 fr o l!l ( X ). for an unk n o wn patte rn o r entity w , wi tff' tepresentation rut. r based on the sen sors in sensing set S , a ~umber of basic steps must be accomplished in the ove rall fUSlon an d represe n tation p ~ ocess :
NT processing and classification stages in a multilevel r ecogn ition p r oce dure , using a t most NF feat u r es , The ir:lpor tance of i nro ri .. dtion fusion stens first fr o m the fact that it is generally co rr ect to as sune that i mprovements in terms of classification error probability , rejection rate , and interp r eta t i on robustness, can on ly be achieved ut tl.e ex pense of addit i onal independent features delivered by ~ore separate sensors . On the other hand , as the nunber NS of sensors in-
323
A.
~ocate a re p resent a ti on x . based on the set of exogcneous va l ues z (x') characterising -r:.he acquisi ti.un o r thle 6utput o f sen sor j E S; i n othe r words , i!lcor.d.ng sensor o r knowledge data must be chp.cked as to re ferring to the same ~ [ l, 3J
B.
Associ'·' ::i ng Tho representati on s Us ( 1) and Vs (2)
:\~
I
L to find out if they refer to the same entity (u ) ~ (v); this assumes the location protlem A. tu be solved, and the i~tersectlo11 of (Z,(Cl);JES(l) and z~(v);J -S(2)} not to be ~ amp t y [ 7 , 13 , 24, 25· J ~
c.
P
featun:· .. t.-:-:xiliUL,_· ':·....'c..:Lr~: E (1); dun (:";,) =....: structural 0"r~l.rh, ',.,·It.h L(,dcs 1,.;hlCrr clr-t f03t~rt's f.; th0 arc 13bels ~rL list uf f~at~rc !:,dcs il1 th0 vectur X~, ~ith the
X
vl L
(X) 1
~:~~~~i~j-~ra~.::,
Merging features from two re~rese~tations x S(1 ) a~d xS(~) of the sane ~atter~ into a
set x , \..;nere S=S (l)u S (2); thlS assumes the c0m~a~iso~ problem 9. to be solved. [9,21,22,24J
__ ti;:.n here-c·r ',·;it:-: 1.·_:1,. :5
;~o\\,·
D.
two representatio~s (Xs(llil 3!:d (X s (2)i) l~to a new higher level ~e~res e~ta=lGn (X ),+] , where S~S(l) ~ S(2); this
:.;
1 is:.
(X) 1
~,~
solved f~~r
S
feat~re
merging prublem C. to
~bcv~
st~~s
r_: 1:..:s ..'::' ::i
are also
E.
association process attribute data fusion
r
KI~u~led0~ tyt;;<~
: ']st
call~d:
L~s~d
~~'5tc~s
tee. -:I:c 5'_':.5 .. r
,-
,1
chi
C~;:-,SiS"':..L·:;". :'
tvct~:1."'''';s,
·.,·itr: i..!.:.s:riL:..i:...·...;G I t . ·,.~··.. :::::).i.:
5iI.l~ L·· ,. :~.:.. r, :t:·h ....·s . .;1...., tI1ct·vfl..'Yl..- I r~ . ·s,. :.t ::lTI~.ll·.: ~~T:....:"; !_YGC(.·SS tv _. . . '--;.(.lrL·ss ti ',: r:.:. ··.·:1· ~:.;l . rv-
~1d("'1---.:.:.;tC.
(Prc,blcr.l K) used in the data bast;'. This dilt,], b3S0 is assumed to be divided into files or objects cG!:sistirlq of the required informatio~ arld retrievable with the proper entry keys as specified iri the above problem statenents.
~,n'::5 ·,,:'r.t.J.ti(j;-~ issu!~ th!.-~:'l: rh, -..;.t Llll '",·ht:>rL· :l:n:l: rl·.: . ri...-s0:·lt.J.ti ·,11 ll:\'l:ls t)'le uf ser:suY Jd:~.
t1L,:-.
ISSUES
In specific implementations, the issues are the following: 1. The main goal is to require better discriminating features, e.g. features giving the highest correct recogIlition rates with lowest false ali3.rms and reJections [9J. 2. The next goal is to avoid having to use high
performance and high cost (single) sensors and computers
[10,17J.
3. The third goal is to achieve sensor diversity and thus integrity, to cope with sensor failures or data link failures and destruction [17,25J 4. The fourth goal 1S to achieve higher recogni-
tion, or segmentation, or understanding speeds, through parallel sensor data pre-processing [2,4,12,14,15J. 5. Sensor diversity offers segmentation, charac-
terization, and feature extraction clues when retrieving a~d processing features fro~ other sensors. This is the cross-fertilization effect at the query and understanding level [12,15J
6.
Sensor fusion allows to avoid stereo sensors a~d ~rocessing, if one sensor or several sellsors of different types in combination provide range [1,7,13,14J.
In terms of hierarchical processing it turns out that different sensors provide each higher level ico~s which are quite sinilar in terms of their fornal re~resentation. This allows us to ex~ect that a proger knowledge representation framework 1,o;ill make it possible to combine such icons, even if they originate in different inforwation sources. At the saDe tiwe, this weans that the selection and structuring of the knowledge representa-
is highly critical for the performance of
any sensor fusion system.
hi<2r.:,1.rcti·-'::~11
Notation: object/ process / pattern to be fied and classified 1
i E
identi-
dimension of a vector space E. for the lileasurement vector Z., i E S 1 sensor/information c~annel index
vector space, with its
nor~
:'";...;si'~.!.
St'::_::>,
Thrce
rc:.:~'rcsc:;talic'n ll'\'·~'ls ':'l.re cU:·lsid~'rl~d: 1: The lu~ost 10\'~1 f the rCl"Yi..,s,_::tatiG:: is () vecter spa.ce .. (i) uf dimc1;siun d , f
contcxt/cnvirol:me~t3l
r~orcnetcr.
SX.:1ml..lle 1: I:". a tYl- iC.:J.I d=3 diLll::.si . Jl"dl S('·'...:II'_·, observed through a l~scr se:~sor, ass0ciJt~d tu a 2-D CCD array: Xl: 1st cuordir·;ate of l_c-,i:,t (I>Ci.ssivc·) x : 2:":d c()ordin.i3.te of i_vi:-.t Ci..lassi\'e) 2 x3: 3rd cocrdirlate (ra:-.c:L') ((lctive) x : o.nplitudc ef Si,:!"L.].} (LJrt..:y 1(:\-'e1) (j~~ssivc) 4 x5: freque~cy of las~r sot:rce (activ~) x6: wavelength of DeaSl:rLQe~t x (p~ssjve) 4 x : time (rassi\'c) 7 x8: estimated reflectivity (active) Xg: irrodi.:1tion field (active) Level 2: If X is the feature \'(:ctor 11. :.:: (i) :.:-rovided by L(~.(;el 1, the: L~':t:l 2 r~·l. rf:s'--::~·.+'::'.J.ti, 1S the list L.
(X)
dof1~cd
.s fallows:
10t G(i) ~e the: feature grai~h, ~hich sr~ci fies all feature ty~0S ~hich can be e!lCGU~t ered i~ £(i), and ostablishes the rolatio::s between those which are feasible/allGwable the: first sllblist ot L. (X) is the list L)f fcat~res ~resent i~ X.l the secol:d sublist coAtai~s all the labl:ls attac~ed to arcs c~c ~Gde ef ~hich ~t least is in the first s~blist.
ExanFle 2: If a F-ict~re is Cc.:·;.sid~red all c:lenents of which are nade ef ~riDitive shafEs, then G(i) is the gra~h reFrese~tatic~ of the correspondi~g picture descri~tio~ la~g~age f~~ objects
KNOWLEDGE REPRESENTATION (PROBLEM K)
'\\'t:
,~Y"-
th'...: ':-'D()\'t:
four problem, one of the most iml-)crt3::t cunsiderati0ns is the knowledge repr0sc~til
I~lPLEMENTATION
, .(!
tc bl:
c:.::alysis d3.ta fusion
=:1 ,'1.11 th(:sc:
d.
.:-;-
process and pusitional data fusion
c. u.
tio~
1:
b~
[1,6,14,16,19,21,22,23J.
alisnme~t
data
,~::-::. l"""-.':'~::;::..:l
L::r!":l.
~ggr~gatlng
assumes the T~~
!:="rE:dic~1t(:
allo~ed/expected
in the
sce~e.
The primitive shapes are: straight seSMe~~s, curved segme~ts, regio~s of specific share. The labels of graph G. are: length of these rrinitives, as observea in the sce~e and registered through coordinates in X., as well as other feature attributes (e.g. cu}vature, area, orientation, etc.). The values of these labels are 0Jk. Level 3:
The third level is a set of rules re-
presenting the context in which the scene is;
KIl()\dt'd gt' Rt'prt'st' IlI ;t1ioll .-'l.ppro ;lcht·s ill Sell sor Fusioll if M is the predicate form rep r esentation of the context graffiJ'lar , the" the result N. (X) o f the le vel 3 knu~ l~dge represen t a ti o n ~i l t be the list of predicate rules f~ om ~1 co~t3in i ng feat~res e n-
ca l match cannot be perfect , and ~ ill require two isomorphic distance measures d and d (2) b~ Slll S tween sensor ou tputs in S(l) and 5(2) : r
,. .',\1.::Itere d ir, L . (X) , \..:ith the cc r respor.ding attri -
b~t~ l~bels '~ flthe rcl e s . Mo re
..,
n~'E reS 0: .t t::C t.:~r ou .:;h tnL: St[;1a.r.tic sr alTU':1a r ~1 . , ~1 is s~iiused re~r0S0;1t ~ d b)' ~ set af logical ~r~di c~teS
'.1
~hich
c 0 rr es~o~d
t~ ,
~h~
r~writi~g
r~lcs
i::
.
. 'i ' Rt.:l~ r 1 : ',,' \ , ' d th 1.1:0,,1 "1 v..' , ': are C0ncate:!2:ted lIsts o f tt:?rmlr:.al a:"'.c ::0;: t~r l.\ ir.al sY:-:1bc.'1 1s fr,,', l:l tht! la:".guQge ~ ger:er.Jt es .
The tt!rr.1ir'.J.l symbcls u t this LlI.-:;:U.3.Ljt i nc lude all ft:ac.:.res o f . .;( j ) , 'JL: l_I.~1t;: me ;1te d b .... cthL'r symbols re -
t o th ~ CO~t0 x t . Thn list N ( X) ~ill be defi~ed by : ~ fi r sE s~blist o f.rules rl ~ s~ch that at 10J.st . . i::e fl..:J.ture ln L . ( X) 15 er.countt2 red in V ; this r:1>.:Clr:S '::1at L ~·ill be u:~ified ....·ith the ~redicJtc rcpres er~ ti~g r . cl secor.Q s"..lbllst , made ot the labels b of s~ch r~l~s ~hich are u~ificd ~ith L . (:;t. l~ted
Ou tput of problem K: The represelltation of the fused sens o r data is Xs = (Xi ' Li (X) , Ni (X) ; i E S) , covering ,Ill sen so r s In S . LOC ,\TI ON OF A REPRESENTATION (PROBLHl A) This process is equivalent to scanning the exo gcneous IJa r ameter space E of z. to locate by search a domaln Z contal~lng z~( w ) ,and to gene rate all assoclatJd rep r esentat10rls x . . This re quires a distance measure 6 between the vector z. ( :d and the set Z. ; il conpleted location will bJ characterised by 6(z. (w) , Z.) = O. In nost ca ses , the distance measute 6 iise l f may incorporate contextual info r mation , although this should be avo i d ed . On ly representations with e xogeneous va lues z. in Z will be considered hereafter . ASS()CIl,TION OF TlvO REPRESENTATIONS
(PR O ElLE~1
gr~~ . h
J
cODpariso~s
are possible :
statistical ma t ch o f sensor outputs z. , i mply ing and isomo r phisr:: betv.'een f eatu r es fk st r uctura l ma t ch fuzzy match
St atist i cal match (8 1 ) uS( l ) and v (2) re f e r t o t h e same en ti ty S ~(u) = w (v) if there e xi sts an isomorphism I be twe en t he sen s or outputs in S (l) and S (2) . Th is isomorph i sm i s an i nve r tib l e one - to - one co rres pondence . As ment i o ned i n t he above e x a mp les, t h e s tatisti -
r0~rt.:s t.: : ~~2ti0 ::S ;
req~ir0
~
G i ~ S (l))
lar I
nat ct lns
th~
cU ~i~rls o~
dist ~~ce
d_
b0t~~e!1
~lll
the
t~c:" grCl~ ' hs
."cd :: ( \. S (2) ) :
-( uS(l)) = -(\'S(2))<=>d c lC(t.: S (1)) ' C ('·S(2))) ~ .:::'5 m.) tchi LS d is t.:mccs d:=; ,
the
~os t
t YF iCJ.l o n es
ar c : (a)
Syrrunc t r ic differc!lc0 bet'. .;eL':', sub..;r.:l.i-hs t..... f ar:d l;(\"S(:2) ) , arid .:l cl)mbil~ato ri.J.l
G(u S(l))
search am0r:~ Jll ~ ossib l ~ subgr.J.lhs t o f i !ld , if a~y , d - close SL:.b~rd~~ls . It should !:ut be ~ x f~cted that the fl:ll g ra!Jhs ~ ill ~atch . (b) Lir.€a r combir:atio:L o f the nt.:.mbe:r o f fa CeS , cut sets , o ricntati ons , degrees a f the nod es , in the graphs G(u S(l)) ' G( v S (2)) ' Fuzzy ~Iatch (B3) This is typlcally the cas~ ~hc~ the sensor measu re merlts o r r 0I) res e ~tati o rls S(l) a~d S(2) are of such divers e natures that !)either a statistical nor a structural match are l"'o ssiblc . A fuzzy r c la ti o :lshiF R may the!) helF ill some instances : (a) either uS( l ) and v (2' have jOlnt observables S z. , and a olfferellce tcpresentation S ca~ be dJfined , such that a fuzzy r e lationship may be applied to i t . (b)
B)
This assumes first that these candidate represen tation from two different sensor famil i es have a non - enpty Joint location defined as Z12 = {Z ; j£S(l)} {Z. j£S(2)}. I f no t , the loca tlons arJ different , a ~d the rep r esentations u (I) ' v (2) cannot refer to the saDe entity S S to.! (ul = u.., (v) .
B2 B3
I
~atch (E:) case, th0 re~resc~t~ti o :.s uS(I)' \"S(2)3re terms uf strL:.ct-..:al ..jrurhs 0: h omoy el~ c-ous regIons 11'. the Se:-.s c r Si. aCeS S \ 1) , S (~) . Le': ....; ( . ) be- thest.:
The input to B is the set of all candidate repres entations f o r al l sensor types in S(U and S(2) , rctri 0ved in domains Z. determined by the exoge neous parameters z. (~)~
Bl
'<=>
!:~ thi~
J
Three types of
S (2)
Str~ctur~l
1
Examl--10 3 : If Ni 15 et l,)n~uaSt: describing th e context 3tld features about a sp~cific ~r ob lem , then N. (X) ~ill contai:l in the first sublist th~ ~redickte rules which use at least one ke )'word about the scene found in L. (X) . The attributes hereof would sa~" whet1 suchla rule may be enCQUllt ered .
J
(\.
if the global sc ene cont0xt is
s~ecifica!ly ,
o r uS(l) and v S (2) have no , o r too few, joint observables z . , or a differenc e representation e cannot be d3fined. However , the joint loca tion Z12 is non - empty . One may then try to define a cooccurrence matrix k , (m , n) , mES(l) , 1 n£S(2) , by sampling.z by succ~ssive subdivi 12 Slons , and reg l sterlng the cor r espondillg sa~p led representations derived from uS(l) ' v (2) ; S Z12(m , n) lS then equal to the nunber of sampl ed representati o~s ~hcre s en sor output s £S(l) and sensor output s £ S(2) remain with iW specified fixed ra~ ge s . n SENSOR FliSION OPERATIONS (P?OBLE;'l C)
The knowledge represe~tati on X = (X . , L . (X) ,N. (X), i E S) lS derlved f o r : 5
1
1
1
each senso r /information channel i , for wh ich the sensor and kno~ledge bases are available each observatio~ of the sce~e o r event There f o r e , t o be able to inte r prct , recognize or unde r stand the joi:Lt set of observations abou~ t he scene , obta i ned from all s enso r s , one m~st be ab l e to combine the above kno.'lec!ge provided fr om each channel . The basic combinations se r ve again dif f e r ent purposes . The mair. combir'..at i ons are : i . fusio~ operatio~s , used in know l edge query ar.d inference ii . ma t ching ope r at i ons used in r ecognition and unde r standing goals iii . p r ojection operat i ons , used to display jo i nt l y the outcomes o f the dif f e r en t sen so r s Fusion ope r ations : Le v e l 1: Fu s ion ope r ation s o n vec t o r s pace s E c an on l y be ca rri e d out f o r space s of s ame d i men -
L
:-~~f l
~,'.. ;, ,:'.
~~
~ "/":L' ~ '
i-:::;: 11·.; :nc': L1r':.:t:: st di m€·:lsi:.; r:,
I. ':.'.::'
: :ii:", ~-:,~·.si ",:· : ·; ~1 1
11"'.
1':111
. Ch r umQt ici ty t riang le (quot i ent ope r ati o ::s)
!:. hat
a re E.'xter.ded ·.· .:ith dummy
S:'.d C'~·S
~_:""01
~ : I:: th e £'..:scd ~ r uI.h v, prc j ect i\...'!1S art.:: idc :·,ti . . : ..ll t\.. t ilt..: se lt:: cti Oj: c f ,.;r .J.:;.h c ;.;.t s ets , o r
". !.'. , :.:-: r:: ~ : r",:;·.
:;'"
~;
.. - ..:.. .. - .
• . : ~.L
....
cf
r, ITtt..:..t !":' :;:: 3 ,
!:; ,~::!,_ .
_ .. _
::;....:~i l~
s
~\ : '.
·..,;nl:.:h
e::'C-:-t
i: ·..::i".,:Ce::.;:-:i5
c ~'21~s ,
0~t~
~hich
0th~r
node s ar0
m~ll ed.
I:: ::hL: SL':!i.:.:r.tic c..: l l:" S0:' ~1 after i l,:si0:: .J. :. L:·· ~ ·..:.:.:ti0 : : :""..J rr0sl··c :·.ds :. C} a f i l:0 r L'-!-,J.::i e d by ~ r C' i J.-::~l:' l:-.• : :h1...~ S J.~L' ::let ..:..'f co!: s:.r~'ii: ·.t r:...:.l es ~tr L~0h ~ll r ul ~s i:. ~.
L,_· ': t::l
j. :
R0 mJrk :
The must
o bvi (· ~ s
f~.rm
f or Level 1
1' [ 0 -
- t.:C t 1,. ! ~ ::;· : l.~ .... ~
10'"
_I r ... ..:':-.:r·. ·
:i.:; ~:-:
::: . .
_ ; e 1" :. "': :. ." ;. s :
\'l Sl b1 0 IR
;' .
· :l:::i ~) l·
L ':2rl ~~'
r:rJl:i sil: ';:' :Ll1 : .. . ~ '''::' ~ i
:.v
. .:.:
~tl.'._ :
ti·.,' i rl'1 :·.i :::. i.ll
: R _\
L
in ~~ ~
~~~
Th is
: ~"re
di s ; L:'.:
. '·\· ....·rL'.y
:)r
s rarh l ~
~r \.i ~~~ ti 0::S
of :h·~·
'r:1,'.:
l_~·.:
': (- r
·...::-.l~ S
..'-' ·...·~:i :; r-:
':l:l' l .J. ':
r :i...:! , ll"
'1,: it!1 .::),]r:h.:
() \'~r 1 3y
~ r~
sl"\· l.:r,~ l
s·': :·.c is
c3r ri0d Ot!t
~:idt.<
i:""i ·..: t
b~'
usi ~~:
,"~h ':'1I:; :(: ls ,
.::.~.:.. li t!d .
" .. ~ .• :. i : .:-5 L.:. l l::-; r·.. ! , s .:::;'. ' !." 1 ,-: 1
!.. .•:
1·..:· s
th(· f.1(: r Sed t i.:cltures(X) ;;~i\ 'L ;:' 1..~L·;~ d(· fi:: t:c fv!" 5(1), S(~) t h\:· 1. r . . ,blc~ l.S ::c''.·,: tu mv di f y Lh~ l·L· ln:sc·r.t.at :i..u;,s Xs (1) J.:-Id Xs (2) i: , crd l2 r
:\ss :Jmi:~,-!
m a tchi ',~ · l·,~-: ·,:,.s
,~r·_· :
:; 11 r.l.:':,L'h i r;,·;
rr un !.a t:.ern r ,: co,; 1"! i. . '- '- .. , :-,,;~ . ...1 . ': : .... i.~:lb ·) l" :-t (,(..Jd r",· L.::' i·. :-: $ ':·}·" s : ·:!.' ir . .; 1. r ucl.:;durt.,s .-),:. !.' ': '.: r:'.i:.i miza. :io:': l_r0Ct:d ures "r ',h ' ljLq .h ic ;.roj cctio ns: , r., ~l:::ch I u i::t s i1'. d= 3 (3 t r a nsL.1tions c1 ::d :3 U · t~ tl ·:~: ~ '.s) .md tch 5l.:tS c·f 2 i-,OHl ts (2 tra r.s latiur.s , i·r·,::"c l:-d~:n= s
1 cctntio l1, 1 sca ling, 2 l e ng th warping s, i l;;~.: t, h t·.,:::: r.Hn ~:· t r i za t i e ns) .mat (.:h i i J.( · sL'qmcnts in d:::.2
1
r ~t~ t i01 ~ ,
LL"v t.;' l
(2 tran slations ,
w~r~ i t\g)
l" , tr.e fus ed gra ph G, all match ing op esubg ra~}l matc hes; they requir e the ~ U r.lbi:~.-~torial s e .:. r ch for the syrrunetr ic di f ference :.:.. ..... tt,..' ...... \ s ;_:~) -j L'i · !lS . . ~ f S.1m(; deqr ·~e . 8 ne sFec id 1 case i s t.h\:~ r\utch i :'.' ;.; u f d! )Cnt,r o r refcreri.ct.; i"iodes . " ~'.
r ~lti~)r lS
~r 0
t u ,.:l~ ; ·!t;L3. tt..: ~ l i1i~ht 'r l \""·:~ '·l :C(·rr esl<.ta ti;..:.'rJ iX } ,+ 1 ir~ th e hicrarch i c ~l Clils sif i ca t io!~ p r oS CL·S:::> .
TI:1 S f i~ld 15 ~l V0 r~' c ornr l (· x Ol1 ~, beCfi US ~ :1 0body k:-_'J'''''S i: ~ L3C' I"'.t..::r a 1 110\-.' t o gene rat e highe r l Evel rCI ~ r QSc~tat i cns
wit h out dec r easing the
f i~~ l elas pcssi bl e tuols , althouqh :~ct v~ lidatcd ur: rnul t isens0r d~t ~ , ar~ : ge ::cra li sed s ymboli c re presenta t ions, ge ne ralis ed s hape d escri!:t ions, quad trcc s hape re pr e sen tati ons , S effi2.::tic r.etworks. The: agg r ~gat io n F roc(~ ss wi l l a l most eve r ywhc r ~ use s~ m an tic d ll d CO I~tcxtual informat ion conta i ned in S , h owever , in a diffc rel:t way . Th e mos t eff icie nt teCh tlique s ar e t h e
r~ c""'y :~i Li~:ri
r cl tl· .
SOr.1 ~
f Dll ow·i r.g : (a)
C0vccu rr er:cc cO!ldit ions on represer,tation
(XS(lII .. , (X S (2)) ;' su ffic ient n umber of l ower level rep resent atio ns ~re jO iJltly p resen t , with c ondit i o ns o n the ir s equ~ncing i~i the z doma in.
se mar.tic grammar N, the .3n::· : srr-...;c t. I,!ra. l ma tch l l;"';- () f stri ngs st r ·...;ct:.Jr ~·:l m. .i.tch i r.S o f strinS 5 w'ith ur, k no ..·.;n \"·L..:m t; :. ts . :. : ': :'. tt:x t d·_·!.. t.:: ::dcr. :: '/ ! ',-::.~~i:·. :.: " r i:u-:..ch·..:s 1::: :-.'_Hnbcr s o f Juir.t (,CC ;J -
(b) Semantic grammar o f rCI)rcs entatior:s : ;\ ·: ;ram mar is de fi~ ed wit h th e repr esen tation s (X 1 . . . S (1) > I. X S (2)) t. a~ I,O!l- termln a l e lem(::r. ts, a nd r(:i . . rE::S ~nt at1 0 : ~S (X ). i :-. thf.: r. . r o d u cti tJr, r ule:
!"(:' ". c·:s
cer ived
~~v~1
Tr: t:. h~ fus ed
.3 :.
m<.1t.ch i1.o.j
'1" _:r~i t i 0 t;s
st r i ~g s.
LI..:\··:.: l
l:
Th ~} '
~ r0
·:,,·ctr. r i.
t ~o
surt s , or igi~ati Jlg in ·:·r ir, a subsF a..c e o f the
a rc ~ cs sibl e f r o m the '..;. si.;, ,=:; t.he J oi nt norm ~c;~- li::ia r ~r oject ior.s or oFe rati o~s a re :._ ··_ss ib l t: aMlJ!".g st:'csr.aces of t.h e fused vc:c -_ C!:" :):'.ac 2: :i~ ~~r J'.l:, ~
ii.
of
S r .· 0.C 0 ,
i . rcJ ~ cti ~~ s
s [, a,:: ,::,
. Ch ange Gf scal e : do~ ai~
s!.a tia l
to spati al
time domai n,
fre q~e~cy
tine to time fr equ e~cy !-'hasl2 dc..ma i:-l to t-has e freq:..;.er;cy domain d0m a i~,
. D..:: ri':a.t i o;. : t i :n ~
':0 sF Eec
co1 0J r t.o colour diff erefi ces si ~~a l
le~ e l
t o diff e r anti al l evel value to s pec tr al derivative . ~ o ~ -l i::ear ~ r ojec t ions th r ough a st r uct·. .:. r ir.9 e le :TIE;. t: s~ectra l
cl ose d are a t o skelet on r Egio~ to bou~dary
. Quatc r tl io n ope r a ti ons f o r time de pe ndent en t ities
S (+1
is ·. . . h en this gr,d:unar is {} tree sr arrunar , bec a 'J se it i s thEn equi va l er:. t to a sc Da~t ic ~~ tw o r k, that i s a direct ed g raph in which th~ ~ o des ilr e: t he 10we r level r ep r ese n t ations , a~d th8 a r cs the r r ed i catcs . t\ s1.ccia l
ca s~
La bEl aggrE:ga tiGI1 o r relax at i o r.: In f <:: r e !: ce est imat i on must b e used to derive ag gregation r~ l~s f~r i ~divid~a l lo~er - leve l labe l s. These r ules rn~s t mini mi se a d i stancE meaSJre o r loss (c)
a~d
f'..: r. ctio:, bet.,.:een (X ' 2.+ 1 S lo~ e r-level
reF rE sentatior.s ar,d
r epre sentatl on s
~.
J.s!l
Gr af-h r educt i on : I f th e repr e se!; t ati on s (XS(l));. a nd (:
Senso r fu si o n i s the scene unders tand i ng appr oach wh i c h use s da ta or info rmat i on o f mo r e tha n o ne
:\~7
type . The
types
informatior~
ma~'
diffe r th r ough
[llJ Koho ne:l , T . Ass oc iative Memory , Springer ,
the :
!'le'.' York ,
physical sensing ~ril1ciple sensor l oc.:;. tior, sensor desig~ o r settl:: g se~SGr
[12J
[13J Sesl , P . , ~a i::
sai=-: '. . · a\ · el(: : ~St.h e r si_(:ctr..:11 the sc:",s o r data rate
dir, .:.: , d~t:.,;cti\·ity
;:~ r0fil'2
~:~d
or [l~
~ile~' / RSP ,
R., Ra~g0 image ~::d~rsta!l
~3t: ~ ~: :
Co::f .
cvffii~'Jtc r
19 85 ,
r0~ c ~::iti0~ ,
_ 1\
ki:: cO~SlS
f~r
ri: ,ci £. ll;;'s t:'L:msel\'c:s ::'t: l ..:.:',';i'
i . sig~als la~alo~ , di0ital , l o~ic) ii . 2- D o r ~ - D 3rrays s~ch as Im3~es o r r ~d i3 tic:, fields iii . proc0dur31 i:~forn~t.i u !~ , s~ch ~s te Xt., s~eech , soft~a r e , bch a vi o ral r~10s In any eve~t , s~!:sor di\' crsl:~' 3.~d 1.ardl1elism are int ri nsic t o sensor f~sic:~ . EJch scnsor ,' illf o r mat ion type is used ir ~ ~arall~l f u r fe atu r e e xtraction from sa i d senso r sl-'ccific in f o rmati on . Different features are thus extract0d ir~ pa ral101 . The knowl e dge based u!lde r standil1Y lj r oces s then takes place afte rwards by ~cc2ssiJ1g features of the different sources , by a hierarchical retrie val and i nference/recuynition i r ocess . This approach is especially useful when no sensor alone gives good f ea tures, o r orl 1~' does so at th~ expense of heavy processing alld use o f conte xt information . Knowledge repr e sellta ti on techniques indispc~sible
for
these serlso r fusi on tasks. REFERENCES
[15J
\'iSl O,: 4 30
~ .
20r.1i::i:·,i:.,; stereoEsls a"d S~dC~
~e r c~ : : ti c !: ,
1 St 'IEE:::: Cv :,: . ~1rtificial a~E ' lic~~ic~s , 1984 , ~ . 15 6
l i..;hti:',:: mect.?
and kn o wledge processing are
rec ocn iti or ,
I:~r:E
P!'\,.'C.
J ' lit c hi " , ,', .,
soft~3re
~
a:~d
1985
e ~vir o ::~ e ~t
The i:Jhysi c.J l sl...::~si: ·.S to thr ee c3ta0ories :
1978
L . F . , ~~ ll:tro ductioJl to inf r ared image
acqui sltl on
bac,d"'idth
p r epr0c~ssing
Pa~ ,
Proc .
ir:telli~er,ce
~il ,
E., .' !itchi~ , ;\ . , ;\G 0~~ n"'dl, ..- . K. , ~ rir.lL':,ts i :: cUmClr,ins r.3.;'0 e a :',d ir:t0!:sity ~,:d'-i~ ::-,a ; 5 , C"" ::-'': ' ...; r o.j. h . J.r.laqe Pr e e ., \',,1 ~l , 1083 , 1- . 3:,.15
[l6J ~,,: ·.da , '1, :a':,,!:·.""th.o:; , ,.
{1::85) ,
D0dh i.o",·" la , R .,
I..J :. tir:i,:l cuo l·e ruti cw v f k r.o\·. . l~d~t: S v'J !' C05 , T . F... ruciL0 :\. 1 . Cen tC'r , ~'-'·..:i :: (! C\""'ffi£ ',:tl...:r St...:'l.'\'12€:S . .~' :,
t~ctical
~ ~rf,Jr0 ,
~I~A
,:"m0 ri c.::. ;, I:.stit'.,:t·,2 of S:.';.1CL' . [18]
Forr.ul~ ,
P~~er
No 83 - 2398 ,
;\ c r G~', a~tics
3.1~d
P . J ., Pe mbc-r ton , l~ . G. , of tactic~l SCI... . ::~'~ , 1:: " r\I)pl lC.J.t lUI1S of Artificial l!lt~llig e :~ ce ", Pruc . SPIE , Vol . 110 485 , Rov.:l.J. n c. ,
,t\. \' . ,
ll~84) ,
~~Il:~ X tU3 1
~::al~'sis
i . 18 9 -
[19J
T.D ., L o wranc~, J . D. , fischler , M. A. , (198 1) , i,!1 ir:fcrencc tcchr:ique or int e grating knov,'ledgt: fr om disparate sources , Pro c . 1981 lilt . J . cOIlference on arti -
Garv~~ ',
ficial
intelli~ ence ,
319 -
[20J Lambird , B . A. , Lavine , D., Kanal , L .N., (19B4) , Distributed architectu r e and pa -
rallel non - directional search for k now[l J
[2J
Kan ode , T . Recovery o f 3- D Sh ape of an Ob ject from a Sigle Vie w. T. R . CMU - CS - 79 - 153, Carr.egie Me110n Univ 1979 Mil gram , D. L. Region Extracti o n Gsing Con -
ve r gent evidence , Computer Graphics and image Processing , 1979 , Vol 2 , 1- 12 [3J
Davis , L . S . and Rozenfeld , A. Coo pe r ating
Processes f or Low Level Vis i on : A Sur ~ey.
T . R. T . R .- 851 , Univ . "laryland , Jan . 1980 .
[4J
Fu , K.S. Syntactic Patterr. Recogn iti on
Academic Pr ess , 1974 [5J
[6J
Samet , H. and Rozenfeld , h . Qu adtree Struc tures f o r Region Processing. AD- A- 077568 Nov . 1979 Winston, P . H. Learning Structural Descriptions fr om Example s , The Psychology of
Computer Vision . McGraw Hill , 1975 , Ch. [7 J
[8J
5.
Mitchie, A., Aggarwal , J .K. , Multipl e senso r int eg rati on/fusion through image proces s i ng . J . O~tica l engineering , Vol 25 , no 3 , march 1 986 , 380 - 386 Keown, D. M. Knowledge Str uctu ring in Task Ori ented Image Databases , Proc . IEEE
Wo r kshop on Picture Data Processing , Description and Management , Aug . 1980 , 145 - 151 [9J
Fu, K. S . and Yu , T . S . S tat is tical Patte r n Cl assi f ication Using Contextual Informa tion, Wiley , New York , Dec . 1 980
[lOJ Wri ght , F .L . Fusion o f Multisensor data , Si gnal , Oct . 1980 , 39 - 4 3
ledge based cartograph i c feature extra~
tion , Int . J. of Man-Machin e studies , Vol . 20 , 1 07 [21J Nii , P . H. , FeigcllbauQ , E. A. , Ant on , J . J ., (1982)
SiYJO a l - t o - synbo l
transformation ,
HASP/SlAP case study , the AI Ma gazine , SFring 1982 , 23 [22J Pa u , L . ,
(1981) , Fusion of multisensor data
rec ogn ition , in : Pattern re coynition theo r y and applications , J . i~
~att0rn
Kittler , K. S. Fu , L. Pau (Eds), D . Reide l Publ ., Do rdr ec ht, in the NATO ASI series
[23J Smith , R.G. , (1985 , Ref"o rt o n the 1984 Dis tributed artificial intelligence work shoj- , The [24J Pau , L .,
1,1
~lagazine ,
Fall 1985 , 234 -
(1986) , Multisensor f us i on for
visior: using arti f icial intelligence, ir. G . Oll us
(Ld) , " Digital ima ge proces -
sing in ind ~ stria l applications ", IFAC Pr oc ., Pe r gamo~ Press , L o nd o~ , 1987 (ISBN : 03 43465 - H) [25J Lajeu~esse , T. : . , Sensor fusion , Defence
Science &
eng i~ ee ri r:g ,
Sept . 1986 , 21 -