The Mathematical Notion of Complexity

\'\ .\I.YSIS .\'\1> I>F(O\II'OSIIIO'\ ()F « )\II'U.'\ .S YSIT\IS

i

{.n p\l t\.!.lt t II- _\{ '1rh I r rcfltl t.d \\·,.11.1 ("II~tl · "" Hu ll ,tpc"!. Il u ll\.!.< tl\ 1'.1.'" I

THE MATHEMATICAL NOTION OF COMPLEXITY L. Lovasz

In an incr eas ingly compl e x society . s~rrounde~ oy a n increasin~ly comclex teeh~lology , th e ~eed for gene ral guid ~l i~es for the ~ o delling o f s u et comrlex syste~s is obvious . One of the most importa~t guii e lines is taking into accou~t this comple xi t y i tsel f , i . e . to introduce varicus ~easures of com ple xi ty . In this ~aper t~ o ma th em atically e xact mea s u r e s of compl e x ity . one based on infromat i on cont ent and the ot h e r bas e d on compu tat ional aspects , are surveyed .

O. Introducti on . Suppose that somebody shows you the followi ng deck of cards : A. 2 •...• K of clubs ; A. 2 ..... K o f diamonds ; A. 2 ..... K of hearts ; a nd A. 2 ••.•• K o f spades . You will agree that the deck is not ~ e ll shuf fl ed . But how do y o u kn ow ? The first answ sr that eve ryborly gives is that the p r obability that any sh uffling produces this orde r of the cards is so small that it i s virtually impo~sible that this event occurs at random . B~t the same sentence woul d b e v al i d for any particular ordering ! An d y e t , by shuffling the dec k we do obtain som e ord e r . Ther e must b e so me way (which we in tui tiv el y f ee ll to distin g~ i sh the above ordering from a " truly random" one . What i s this? The k ey tc the arswer is the o bs e rvati on that this ordsr of the ca r ds is very easil y described . ~or e ~recisely , ~ e can desc rib ? it e asier th an listin~ all cards (in this cas e , we usei th~ sOffi~what VAgu e ~ot.at i0n 11 " : blJ_ t or!e cO~_ l l ~ ea s i l :l f _i ne: f.lor f> prec is e descriptio~s of ~h o j~ck . Th i s o bs e r vatio~ ~otivates t. ~ e i~ea t o say th a t a seq lJe nC e is ~ljndom if i~· ca~not b p i ~ scrib ej short.e r tha~ listi!~g RIl o f its e lemen ts . A~j Dorp ?e~era1l~ , this this l e ajs to t h e ~ Gt. i0~ r ~ c o mB[e~ it y n ~ ~ s e q t..:er:c
RS

th ~

l~ ~~ t ~

o~

~~S

shn r t.es~

successful an d was abandoned by p r obabil ists after Kolnogorov bas ed probabil i ty theory on measure theo ry . I n that th e ory . how e v e r . it is im possi ble to distinguish betwpen two i n fi n i te 0 - 1 sequences . This ne ~sr theory . found ~ lso by"Ko l mogo r ov ~nd . independently . by ChAitin is abl e to tell wh ethe r o r not a singl e given sequence is ra ndom (in fact , it can tell how rAndom i t is) . In this s ens~ it sur pass e s th~ probability theory based on measures . But th e i de a of defining the compl e xity of a sequence (or of any oth e r structu r e) goes fA r b e yond th e fo undAtions o f pr obabil ity theory . Th e r elevance of this not i on to information proc e ssing . storage and re trieval is obvious : if a sequence can b e de scrib ed in a small space then this mea ns that the in f or~atio~ co~t e nt o f the sequence can be " squeezed " i~to this small space . At this point . a new rrobl e m occurs . If we try to compress a sequ~nce (a structure . a set o f data) into smRlle r Beaee . an3 t h e n ever.t ';al1y recover it . we have to b e ab l e to c arry C:lt t.h e ;l p- c ess a~y COi..pl~tr3tior.s ar.,l i~. fAC t. i i', " r ~a s o ~-, e'c17 11 ~ 1;:-. 7 . f... r.;a . ~. 0 :- ii ff i~~~ty ~i t h t~e ~ otio ~ 0 f i~fcr~ati o~ cc~ ­ ~1 e xit. ; sk ~ t. c~~d ~bov~ is t~a~ i t jn~s ~ot sa y a~ y th i~g abo'jt ho~ long joes it ta k e to c0m~~te th code froe the s e ~ ~=~ce o r :~ e seq~~~ ~c A fr o ~ i~s coje . I~E sit~ati o ~ ~s -?ve ~ ~ o r se : ~ e shall s ee ~~at the n ; ti~ ~~ coi~ o f a seq~~~c~ is ~ ot alg o rit~~ically comp:~ ta t le at al l . T~ i s fact f c rc es "13 to c8nsijer a iiff e r e ~~ ki~j c ~ ~ o ~~:exity Q

po S s i Q 1 os " j e s cri;:;-.i (I :, ! ! . n '3 r '3 t. :--: 7- \.; 0 r .-: II jescri;:tiC":," a;:,?p. r s tn Co !"' Ather va;; ;;e bu t it is fair ly eas y : ~ ~ akQ i~ rrecIse : say . it is t~ ~ le ~;t. ~ ~ f ~~ ~ s~ortQst 3AS IC ;r og rac '~~i c~ ~ ~ ~ ~ ra ~~2 :~~ ~i~2~ s~ q · ; ~ ~c ~ .

i~ ':'~1is

iiea o f

:: -? :"'i:.i~'.s

of a

r a ~.j::~.:,ess

se:::~ ~ ;-

~~~~ 5ra:~ ~~~~~~~~~~~b~1~ ~~~~ Cl ~ C f~~~:~ e ~ es~ _ ir.g ~o

y et

:0 t~ e

r e ~ar~

:i~ ~2

~~a:

~~ ~ r.

~~:

~~

f:r-

a tto ~~t

",:". ~

3clv ~

~c

V~ ~

~as

j~-?

reo:,:: Cr:".:. P ss:) ~"" .s f

~, ~

r

~

~ ' Q

i~:

f

t:

2~~~ Q

~~ ~~a t~~~:y

~ ~C2

~~e0 ~y

~a~~

~ ~s

r.~:

~- v~~~:c~s . : r. -? ~a:o~ ~~i2 2· ;~~~a~ji:.g ~r 0 ~~ e~

.·. ~~es : ?-

~ ..

~~

~r~ e j

t ~

i:. S ~ e :. ,. :'""' ~~ ~_ -; -:.::. ~

~rc~ ~f~~ ~ s 1 ; s~;er~~;~~~~f ~all ; the tte sar.:?

c::e ~">1~y

if eve ry s e C0 is s~l~: ~ :.e ~ .

, ~ ~ e r; +~ i2 S~:: ~ ~ 2~

s~~'::"

~

.; -? r 2 F

~te

~ e ~qi~.

~

v~~l

a i

e~p ~ p ~ i~

~0:

:r ~~2

?r o t:~~s a~ o ·;~ t

t~e

i~f o r~a~i o~

a~i

o~

r es~l~s

ti ~ e

cc~~lex i t y ,

ccmp[e~iiy,

c o ~c s r~i~g

sps2e

a~i

the

~ eq~ ir e i

task

e

:!s ~ ~~c~

~~e

fr ~

1/2 ,

~i ~h

s o - cBllei ccmpuiaiicna£

~i~i~~~

:0

c~r~ai~ a l ~ o ri:~~i2 a~~ ~ij e r ~~ c ~~ ~ta~ :~ ~ ~ ~ ~i o ~ c ~

acco~; ­

;:- o tatl y i~~ c :-~ ­

at~o~ c~~;l exi ty ~ e ~ti c ~ ~ j a t cv~ . ~ tes e ~;es:i o ~ 2 er e o ~~ e ~ e x ~ r e ~ e~ y ji ffi c~l:

~ ~ f~ ~ ~

s e-

~ta"':".

.

r ela Li0~

~~~

~ h eo ry

; o ~~lari:y j

t~i l ~ a~

ar o · ~ ~::

~ E a8:

~te ~

a ~ o ~g

E~ :CyS

a~j

gr e at

:~ eo ~ et icsl

20~ ­

ii ea s a:so ~ ~ e :~aL e ~ l ar~ e r a~j :arger ;art o f c~assic l ~att e ­ ~8 t ic2 : o ;~i~izati o~ . ~ · ;~ e ri ca~ ~ ~a y s is . "j ~t ~ r the~ r y , gr o~ ~ ~~ en ry , ~ r ~pt t eo r y

; · ;t ~ r

1105

sci~~:is:~ .

~t ese

1. Lovasz

I 106

-n

I!:i ~~~

2S_

~a:~ e ~a:ic2

~~

S~0 ~ S

:~

ca~

~

~~

~ ost

~ p

s~)~:i

class -

t~~

r s -~ ~~ t -

te

~i t ~:

~v

~;: ~S .

t:i

! ~~ ~

obta ~~lC~ g

e~

X

ass

~g

:~~2

; a ; ~ r ~ ~ i~ 2 :~ SS :~ ~ ~ : ~~ ) g~~ o v ~;.::: o3 '~:' - _ .. ;;:: :-:-. -:: :) f .i ~ .'3 a;:; ~ i '2

:'::"~ ~: 2

a~~

;:: ~ !::ial

~~~:j:'::"~~ ,

a;~:~:ati:~s

;r : t~~'::":i~. y

~~ ~~ ~:.

~~ 8S~

e x; s~~~j

~~ ~

Se

2~~pl ~xi : ~l· .

o t"':.ai~

i~

:t

2o~=lex'::":-; .

:~ecfe:ica~

is

Jf

r easo~abls

e~coji~g

encoji~g

tte

p~rposes

sc'::"e~c2

s~ch

of

~ ot

will

a

(tte act~61 details ~at~er for th e

for

a

of

Koimog( )10V code

~h~

t~,-?-

Si~c~

~~

sha:~

x.

'~C:7:;:

=- ~ :.: i

~.

o~ tt ~ ~i2~t ~a~j si~ ~ :. ...: ~-:, ~q>·.g ~~aiZ -::E= b:,t, ;;o~,

.'l ~

o~ ~o

cons~a~ t

;) ~-: :;. ~; : ~.

~ ~i s

cf

~ .:; ~-:;

a~~~~i ~ ~

?

b~

~! ~ ~ ~ ~ ~S 1. ~~

t:-:. :'. e;- s ,

~~ff~a~

Y

El :. ~

~n~2~a~~,

the

o~

;rogra~

coj~ ,

X

iefi~i~g

the~ wi~h

+ co~st a~i so

~:--.:'

~ i:~

i~~

:~ ~

s "':. ~ ~-..~ ;;

~~ ~

k

we expec -

n

1' S LEXICOGRAPHICALLY

A:iJ ??IIIT Tii:::

!!lTE STRliiG"

Even if writt en out prop e r l y in BASIC , this oroe;rarr: will have leng th at most 10g2n + log2 k + log2m + const . Here m ~ (~) and k ~ a n . Hence the l e ngth of this description of X is at most 10g2n + 10g2an + 10g2(~) + const ~

X.

If we take a~ o t~ e r di Rl~c~ of BASI C or any ot~ e r progrllrrl lar,e;' Jag e i"s tea ~ of 3ASIC , th e n we arriv e at a di//e~(nt valu e c ' (X) for the information complexity . This ambi guity , how~vsr t should ~ot both e r us too much . For, W2 can imi t at e one program language in th ~ other by writing an ap~ropriat ~ corn~iler , which takes onl y bo: ~ :.ri ~ :1 s;-.acs . T~is i~!=,· l i 2s tha~

s. ~ != -2 ~·.js

~~e

se~~e~ce

aii a

" ORDER ALL ST'tINGS eF LENGTH

~!TH

K(X) o f ths string X is defi~ed as th~ mini~lJ~ le~€th of Rny stri~g d0fi~ing x. i ~ shal l call A sl]ortpst ~rogrA~ rie fi ~i~~

~h ~ r ~

~o

stri~2

~e

If

But there is a ~ uc h si~Fler way to see that such seque~ces ca~ be coopresseci. More gen erally , if a string X contains abo~t alXI 1' s , 0 < a < 1 , then the string has inform atio r . corr.plexity at ::;ost H(a)IXI , where H(a) = - alog 2 a - (1 - a)10g2(1 - a) . To see t~is , consider the following program :

X

He ass :.:m o?: ths t \.J ~ !'la '/ ';- a CO~r::1 t, ,-~ :- '",ho s o? me mory arj outptlt t.ape ar e infinit e (or at least larg e E:: nough for rllr.Li:.g Y) , arld that ~ e fix som e dialect of BASIC . Th p in/n~ma ­ lien cempleKitu (or Ko'mnqn~nv compfe
X a

tte

2 o ~;r sss~s

deccii~g

s~rvey) .

t e a s~rifig . ~ E de~ot,e by lxl ~~ ~ length of X. ~ e say ths~ a stri~g v deline~ X if Y i s H BASIC ~rogra~ wtich , ~t ~ ~ run , r s s ;llts i~ ~ri~t, i!1€ o u t X. 2 e r e So let

~oj~

by at c~~ 8% .

ra:~er

coc;~:er

tc consiier sI: jata a~j a::"=. ;~:)g:,a~s e~coiei i~ the for~ of a G- 1 sec u e~ce (a ~i~ing for brevity) . ~o~ever , w~ shall s;ea~ of gra;hs or nu=bers or BASIC progra~s as stri~€s : i~ t~is case ~e ~ay thi~k Jf a~j

t~is

ave~ag e

~ E j l e ~g~h< . 92Txi

I~for~a:iJ~

cos:~~a~y

~s

:~ ~o~ ;

:: ~ a:€~ri:~~s . ~ ~ a:s ~ ~ ~ ~ti:~ s o ~~ ~~~:::a= ­ ~ ~~al ~o:ic~s ) ~ ~o~;~:s:'::" ~ ~a::" ~::~;:exi~y a~i ~~ ei r : c ~~ ec :i o ;5 : ) i~~~~=a:io~al

1.

: e ~g:~

e. -

iata

i~

a~j

v~s a ~ '.,; ::-. ~r8caci=-

o f length coin" , ~j 1/3 and

~g

tt p~obabil "'./ 2/3 . ih~:: "':.h e r:·-.,;ff ~a~ c o i ~ yi ~ljs a~ ~ !:c oji~g as a~otter stri~g

:; :: ~ ~ ::.. -=- :.:.::.. -: .

a

be a str a "tias e

- (alog a + (1 - a)10g2(1 - a»n . 2

So we have seen that if the frequency of 1' s in a string is substantially different from 1/2 , then the string can be compressed by a factor strictly less than 1 . Similarly we could prov e , say . that if the frequency of 1' s i~ the even positions is substantial ly different from 1i2 th2n th e string can b e compr e ssed .

Bl;t maybe ull stri~gs ca~ be comrressed? Mayh e all st. ri~gs ~av e so~ ~ (hiiden) regul arit:1 ~h ich ca n give ri se to a short de sc ri~ti o~ (cf. Sv~jk ' s lectu r e t o Sergeant ~aseklo)? It is not difficul~ to se e that : ' ~ s~, ~ !: c. O}: != 0 sit. e i s t, r 11 e : u { (m 0 ,) t a f. f. ",t1inq .; hone in/o1moi.ion comp£e~ilu e", .;{!n tio[l!l equal te thri., (pnc;th; . 3 0 alJr.ost e~ ll

ar e i?!ccr.,;;- e ss .i bl e . To see

se:;·.~-:-~.c -= ::;

~~l~:,. ~~~ = ~~ <; "-~~~t t.n " i~;:~e~o~~ p~R~rams so :~ e ~·;~t~T of s:ri~~s ~i:~ i~form n s:i ~ ~ C07~:~ ~ lu j at ~ 0S~ ~ ~ a~ ~ost 2 (ef ~ c ~rs~ , a~o~:g ~t e stri~€s ~i :~ le~gth a~j

~

:~ e r e

~AS IC <::: ... .:: -

- ,. _

:~e

c :""

a~e

sa~e

5::,ri~-.g s

i::

tiOL

~

~f

~ti~t

a~i

o~: ~:;~) .

c :.'"

:>::; :":". ;: le :·: :.. . . .<1·

if

~a~y

;r o~ ~a~s

~

a~e

t~e

50

:~ss

~ e a~i~gf~l

~hich

frac~i o ~

E.~

::: CS ·~

:. - i::

a~ti:~a~~l-

2-~~i! · ~2

of

~~a ~

r:-, C;S~

::..s a:

slo~l '

le~~"':.~

.. ( SE.y)

t:-. -2 :rac -

~c

·..;!1i~:--. are

2/~. ·...;.5

::-. -2 ~

~ ~: ~ C ~2~ : ~

c~!: ~a~i;~ o f

:: r

~ 2-::

t [; ~

_ ~SS~~2S~

~ .. :. . ":' !':-:-. e

-: c :-.

~~ : ~~~-:

_.

2 ': ~'.

*

-;' ". -:

·r ~ ~' ~··

: .• :'"

2- ':- ::... .;. 7- ~. -:'

~~ 2 ~i~~2

:;~

'7

iefi~e

of

ler:~~ :-_

2C:::;~~2212_e :c :s~g:t

is

~a~

~a~y

also

cc ~:~ e x

:r o bat~

ca~

!

te

:'0

give

~-. o:icr.

~s e i

as the

:heo~:l . :a11 an 7.undom i:.. .

The Mathematical Not ion of Complexity (n

-+

=)

where X denotes the string formed by the first nn elements of X. (For techr.ical reas ons . one defines rando mness by the more complicate d but s t r on ~er rroperty that K(X ) > n - f(n) if n> r.o(f) for every n function f such that ~2 -f (n) < But the simpler definition given above will suffice to demonstrate the idea . )

=.

We can see right away that altmo6t all int i nite 0 - 7 6equence6 a~e ~andom (i . e . if we generate an infinite 0 -1 sequence at random - each digit being 0 or 1 with n r o bability 1/2 . independently of each other then it will satisfiy the criterion given abo~e with probability 1) . Furthermore . va~~OU6 taw6 o{ ta~ge numge~6 a~e valid to~ all ~andom 0 - 7 6equence6 . For example . we have proved that if the f r equency of 1 ' s in a string is different from 1/2 then the information complexity of the string is less then its length ; hence . in any infinite 0 -1 sequence which is ra ndom in the sense defined above . the frequency of 1 ' s must be 1 /2 . We shall not go into more deta i ls of these f oundations of probability theory ; rather . I would like to discuss the significance of these ideas for computer science . To this end . however. we have to have a quick glance at another kind of complexity notion .

2 . Com p utational complexit y. An algo ri thm (p r og r am) . when run on a given input X. draws on various resources of the comput ation facility : the most commonly considered r esou r ces are time and space (memory) . For sake of simplicity . we shall restrict our selves to considering time here ; the theory can be extended to space considerations . One usually compares running time with the length Ixl of the input . and so the time complexity of an algorithm is defined as the function f . where f(n) is the maximum running time of the algorithm over all in puts of size n . (To be precise . this is the WO~6t ca6e complexity. Perhaps more important to practical applications is the ave~a g e complexity . but this notion leads to se r ious theoretical and technical diffic ulties , because o~e has to post l~late a~ a prio ri probability d is t r ibution on the input data . which is seldor.J natural. The r e fore we shall or.ly consider the worst case complexity here . )

No w suppose that we have a problem to solve . We shall ass~rne ~ hat the ans~e~ to the problem is always Ilyes'! or lI no" . ~anv orcb Isms have, of course , more cc~clicat~d·out ­ puts (n~mbers t vectors etc . ) , b;lt this siDplification does ~ot li~it ~ er.erality too , serio~sly , a~ least for t~~ purpose~ of sucn a survey . Tt~s a p~oCtem ca~ be defi~ed formally as a set of stri~gs (t~ose strings which , consijerej as i~puts to the proble~ , ~et a~ ar:s-...:er ":,res " ) . ~sual~y

the ~ ~.a r e

lor tne

SOl~~lO~

:a~j poss~b12

algor i th~s

01 ou ~ ~=081e~ ~ e ca~ O ~ c:~ r s? , ~e wo:;ld ii~e ~o

choose fr o~ . choose a~ efficie~t o~e . T~ere is ~o guar antee that there is a si~gle he~ algorit~~ ; so we ca~not iefi~e ~he co~~lex y of a ~roblem as the s~allest ~o~~lex ~ of algor lthms solvi~g i t. 3ut ~ e ca!: as qJ2stions IWC3 -8

I 107

like : is there a linear tir.Je algorithm or a polynomial time algori thm to solve the problem? This leads to the isfinition of complexily cta66e6 of problems . Let us sur vey some of the ~ost i~portant co~plexity classes . . . Starting from the top . so~e ~roblems may not have any algo ri thm at all to solve them . such probler.Js have bee~ fo und before comput ers came to existence . C~e of the first of these was the "Entscheidungsproblemll : given a mathematical assertion , is it provable? This q~estion was proved algorithmically unQeciQable by Church(1936) . Perhaps more familiar to a computer scientist is the halt ing p~ogeem : does a given BASIC program s t op in a finite time . We shall meet another a l gorithm ic ally undecidable probler.J in the next section . The next important class of probler.Js are those solvable in exponential t im e . A typical exponential time algorithm is one which enumerates all subsets of a set or searches through all cont i nuations of a game to find the winning move . Perhaps the most popular time complexity class is potwnomiat t i me (p) . One r easo n f?r ~his is that this is the magnitude of dlf fl culty wh ere most practically important probl ems lie or at least may lie . This complexity class has also proved very fruit ful i n theoretical studies . Polynomial - time algori t hms often coincide with those f e lt " good " or "non - trivial " and they are also r e lat ed to important notions in formal l ogic . One more nice distinguishing feature of polynomial - time algorithms is that they are precisely those algorithms which can' be programmed in BAS!C without usin~ th e G?TO statement and always using theOinput Slze as upper bounds in FOR statements . This gives a syntactical criterion for recognizing polynomial - time algorithms . A further class of snecial interest is tinea~ lime . Solvabiiity in linear time is requir ed in on - line situations . Let us hasten to point out that it is in general very difficult to decide if a given problem is solvable in polynomial (linear . e xponential •.•. ) time . For example . we do not k now i f it can be decided in polynomial time whether or not a giv'?n positive ' integer is a prime ! But there are other ~easures of complexity of a prob lec -.. : hich ofte~ give at least good hints as to how diffic~lt the algorithmic sol~tion of the r=oble~ will be .

A rath er tricky , but fun6a~e~tal ~otio~ here is non - de i e~m ini ~m . The idea is that a ~ood guess may ma k e it easier (o r even trivIal) to solve the probleffi . For exa ~~le , if we want to de cid e ~heth er or ~ot ~ give~ ~ ­ . digit ~umber A is ;rice , ~ ~e~ 2~essi~~ somehow a jivisor 3 of A, ~he ~~obleffi rs settled . It is a t~ivial ~ask ~c check if A/3 is ir:.je e:i ay: ir.teger . (:\0:'2 :.~a: ",,; e :iid ~ot si~ply guess t~e a~s~er : ~ e g~essej a divisor , so tr.at t~e correctr.ess of the guess~ng ~as easy to eheck ! ) 80ntinato=ics , or~i ~izatio n :a~ be f or~aliz~d se :'ha: gi ve~ a~ i~~ut X, ~ e are ~oo~i~~ f e r a~ ad6itio~al stri~g Y s~ch :'hat~ X aY:c Y "fit" ; aY:d it is s as·; ~c chec~ t~at X a~a Y "f i:' '' (s·jch a strlr:~ Y is so~e~i!:.es called a k'itne~~ for X) . ~ere "easy " ~ay mea~ any of the conplexity class2s ~e~tio~ed ~a~y

problens

i~

a~d y:~~ber theo ry

L. Lova s z

11 08

above ; and for different choices of these classes , we get interesting new complexity classes of problems . The first is class ical: If checking wheth er or not X and Y "fit" is algorithmically solvable (recur sive) then the set of strings X for which a Y exists is ~ecu~~iuely enume~alle . If checking wh ethe r or not X and Y "fit" takes only polynomial time in ! X! , then the problem of deciding wh ethe r or not such a Y exists for a given X belongs to non - dete~mini~tic ~olynomial time (NP) . Similarly one defines non - dete~mini~tic ex~onential time and non - dete~mini~tic einea~ time etc .

able mathematical theorems is recursively enumerable (for a theorem X, its proof Y is a witness for the fact that it is provable) . Since this set is not recursiv e (there is no algorithm to decide if a mathematical statement is provable) , here the larger strength o f non ~ determinism is well - known . On the othe r end of the scale, a very recent and technically difficult result of Paul, Pippinger , Szemer~di and Trott e r (1983) shows that non - deterministic linear time is a wid e r class of problems than deterministic linear t ime . But the problem wheth e r

P It is perhaps the class NP which is the richest in important examples . Virtually all of the basic problems in graph theory (colorability, planarity , connectivity etc . ) and optimization (travelling sales man , chinese postman , assignment etc . ) belong to this class or can be transformed into problems belonging to this class . We have mentioned already prime testing . Let us formulate one more example of a problem in NP which is very important in computer science. Let X encode a Boolean polynomial (i.e . a function of the form X

1

A( (x

2

v x

3

) A (x VX ) )" ... 3 1

whose variabl es range over {O , l} . Then the SATISFIABILITY problem asks whether one can give values to the variables so that the value of the function is 1 . Here a witness for X is simply a system of the variables for which X assumes the value 1 . Since it is trivial to evaluate the Boolean polynomial in polynomial time , the SATISFIABILITY problem is in NP . Let us remark that not only is SATISFI ABILITY in NP , but it is in a sense one of the hard es t problems in NP . Cook (1971) proved that every other problem in NP can be reduced to SATISFIABILITY in polynomial time in the following sense : Take any problem in NP , say the question whether a given number A is composite . Then we can associate with A a Boolean polynomial B such that A is composite if and only if B is satisfiable . Moreover , this construction can be carried out in poly nomial time . So if we had a "good" algor ithm to solv e th e SATISFIABILITY problem then it would also yield a "good"' algorithm to check for primality and in fact a "good" algorithm to solve any problem in NP . In the meanwhile it has turned out that this is not a special property of SATIS FIABILITY , but a large number of quite dosn - to - earth problem~ like graph coloring , travelling salesman or int e g e r linear programming , ar e also " hardest" problems in NP in this sense . Such problems are called NP - com~eete . For a precise intro duction to this subject , a~d for a catalog of NP - complete problems , see Garey and Johnson (1979) . It is intuitively clear that to allow guessing should enhance the power of alg orithms . In many ex amples , the problems become even trivial , like in the case of primality testing or SATISFIABILITY . How ever , to prove that non - dete r ministic complexity classes are really wider that the corresponding deterministic classes seems to be extremely d ifficult iR some cases . The "top" is relatively easy : it is quite clear that the set of all prov -

=

NP?

is unsettled in spite of its great import ance , appeal and the consid e rable effort that has gone into its solution attempts . The corresponding problem for exponent ial time is also unsett le d . For more about computational complexity theory, see also Aho , Hopcroft and Ullman (1974) and Papadimitriou and Steiglitz (1982) .

3 . The computational compl exit y of inform ation complexity . If we have a string X (which may encode a variety of structures , graphs , numbers etc.), we may want to determine its Kolmogorov complexity K(X) . Now here com es the bad news : this number is not algorithmically computable ! It is interesting to note that this fact also goes back in a sense to pre - comput e r times . The reader who is somewhat familiar with the history of mathematical paradoxes will recall the famous "Typ ewrite r Paradox'; Consider a typewriter (this only serves to fix an alphabet) and begin to define num bers by English sentences and symbols on the typewriter . So at the beginning , you can just write " 1 ", "2" etc ., but after a while phrases like" iexp exp exp Jl" or "the 35th Mersenn e prime" may be more compact descriptions . Since there are in finitely many natural numbers , there will be a first which needs more than 100 sym bols to define . But consider now the defin ition "the ti~~t natu~al numle~ not detinalle ly at mo~t 100 ~ymtol~"

.

This definition contains only 6 1 symbols (including spaces) , so this natural number after all i6 detinalle by at most 100 symbols - contradicting its own definition ! This contradiction can be resolved if we make it more precise using the notion of Kolmogorov complexity . It says that the definition "the first natural number with Kolmogorov complexity at most n " cannot be converted into a program which prints out this number . The reaso n why this is not possible can only be that the into~m ­ a t ion com~lexity ot a 6t~ing cannot le dete~mined

aego~ithmicaely .

At a first glance this result itself seems paradoxical . While it is tru e t hat there is no appearant etticient way to compute the Kolmogorov complexity , one may think that the following procedure works at least in theory : Let X be any string . Take 2!X ! +100 computers and fe ed into them the

The Mathematica l Notion of Comp l exity

21XI+100 strings of length Ixl+100 as programs . Most of them will of course stop with an e rr o r message . but some of them will work and print out something . Consider only those which have printed out X. and pick the one with the shortest input . The length of this i nput is of course K(X) . What is wrong? Well . some of the computers will run on infinitely . and at any point in time we shall not be able to tell whether a computer still running will eve r stop . So the above negative res ult also implies the f ollowing: No uppe~ Kound on ihe ~unninq which compuie~ X t~om code can Ke computed alqo~ ­ iihmically t~om X .

ii me ot ihe ii ~

alqo~ it hm

Kolmogo~ov

So the notion of info rm ational complexity is from an algorithmic point of view as bad as it can possibly be . It is quite natural to try to r epair this by r equ iri ng that the encoding and/or decoding algorithms have bounded time complexity . Th e theory loos es some of its elegance by this . but it gains in aprlicability . This extension of Kolmo go r ov s theory was worked out r ecently by Sipser ( 19 83) . Ko ( 1983) and Hartmanis (1983) . The p r ec ise definition is the following . Given two integer s a and b . we say that a st ri ng has time - limited i nformation complex ity (a . b) if there ex ists a BASIC p r ogram Y such that Y a . and a computer computes X fr om Y i n at most b steps . Of cou r se . a given string X does not have a un i que pair (a . b) associa ted with it : rather . there i s a trade - off between a and b: if more time is a llowed then the sequence can be compressed mor e efficiently . Howeve r. the exact nature of this t ra de - off is not yet worked out. Hartmanis ( 1983) discovered interesting connections b et ween the time - bounded i nform ation comple xi ty and the computational comp lexity discussed in Section 2 . To illustrate his results. let us consider the SATISFIABI LITY p r o blem mentioned before . It is often the case that the Boolean f unction involved in the problem is calculated from a much smaller st r uctu r e . One may hope that in this case the p r oblem is easier to solve . Suppose . f or examp le . that the Boolean functio n of l en~th n is computed fr om a ~t ring of ~e n g th 10g2n in time n log n (i . e . it has ti me - li~ited information complexity (10g2 n • n log n) ) . Hartmanis proved that if this problem can be solved in polynomial tim e then every problem in non - dete rmi nistic exponent ial time can be solved in exponen tial time . (So even very low info r mation complexity does not help much if the hypothesis that non - dete rmi nism extends exponential time is true . ) This surprising con n ection to one of the most f und amental unsolved probl ems in comp ute r science suggests furt her important applications of the not io n of information complexity .

4 . Examples and r emarks . In this last section we give some examples and r emarks which show some open problems concerning th ese notions and i ndicate their applicab ility. Suppose that we want to stor e trees on n points . There are two ess e ntially different questi ons here: th e trees i n question may

1109

be eaKeeled or uneageeeed . i. e . isomorphic trees are either dist inguish ed or not . First consider the case of unlabelled trees . It is c1 a r that the adjacency matrix . which takes n bits to store for a tree on n points is very r edundant . Listing all the edges is more efficient : it takes 2(n - 1) . . 1 0g2n bits (since strings of length 10g2n are needed to lab el the points) . It is well k nown that there are precisely nn - 2 trees on n points . so in eny encoding scheme . they must ~ave codes of length (n - 2)10g2n on the average . The inte r esting thing is that this is also enoug h! The so - called Prufer code encode s all t r ees on n lab elled points as sequences of length n - 2 . consisting of numbe r s 1 ••.•• n . In binary notation such a sequence will have l ength (n - 2)10g n . Together with a program of bounded fength which decodes the Prufer code . this shows that the Kolmogorov complex ity of any labell ed t r ee is at most (n - 2)10g2n, + const .

2

But of course the significance of the Prufer code is not that it takes only (n - 2) 1 0g2n digits ; such a code could eas ily be obtained by enumerating all trees on n tRoints and then letting the code of the k tree in this o r der be the number k. The point i s that the t r ee can be r econst ructed fr om i ts Pr ufer code in polynomial time . So the Kolm ogo rov complexity of any labelled tree on n points is at most (n - 2)10g2n + const even with polynomial t i me bound . Consid er now unlabelled trees . It is known that the number of such tr ees on n points roughly tn . wh ere t z 2 . 96 is a constant . So by the counting argument . the Kolmogorov complexity of an unlabelled tree is at least nlog t on the ave r age . and by the enumerat 2 i on argument . it is at most nlog 2 t + const . But I do not know any decent (say. polynom ial - time) algorithm to ach i eve this bound . The best code I k no w of which can be de coded i n polynomial time yields code strings of le ngth 2n . A very similar situation occured ea rli er in this paper when the Huffman code was dis cuss ed . We gave an enume rati on argument to show that the informat i on complexity implied by the Huffman code can be achieved much easier . But of course this argument did not make the information theoretical cod ing procedures obsolete : among other adventages . they are much more time - efficient . As a further possible application of these concepts . consider the following problem . Let p(x 1 ••••• x k ) be a polynomial with 2 integral coeffi~ients . like (x 1 - x ) + (x 2 +x ) 2 3 - 5x1(x4-x5)3 ••• IS this polynomial i dentical ly O? We all learn at school that by expand i ng the brackets . this question can be abswered : the polynomial is identically 0 iff all terms cancel . But . unfo r tunate ly. it may happen that temporarily exponent ially many te r ms arise. and so this algorithm has very poor time and space compl e xity . The f ollowing is better : subs ti tute randomly chosen values f or x 1 •...• x • If the value of th e polynomial is not O ~ k then the poly nomial is certainly not identically O. If the value is O. then the polynomial mayor