- Email: [email protected]
Public policy evaluation: Vol. II, Sage yearbooks in politics and public policy
BOOK REVIEWS
t i o n , social welfare, d e v e l o p m e n t a l disabilities, p u b l i c administration, nutrition, gerontology, rehabilitation, poverty, delinquency, criminal justice, and many others" (p. 18). In six c h a p t e r s a n d an epilogue, t h e a u t h o r s proc e e d to deal step b y step w i t h m o s t o f t h e m a j o r aspects of p r o g r a m e v a l u a t i o n . T h e y usually give b o t h sides o f t h e c o n t r o v e r s i a l areas w h i c h t h e y deal w i t h a n d are clear in discussing the issues. T h e y i n c l u d e a n u m b e r o f b r i e f descriptions as e x a m p l e s b u t a l m o s t all e x a m p l e s are f r o m t h e field o f m e n t a l h e a l t h a n d closely r e l a t e d areas (e.g., e v a l u a t i o n of criminal j u s t i c e or a n t i p o v e r t y programs.) T h e i r references also heavily e m p h a s i z e m e n t a l h e a l t h a n d t h e y o m i t such m a j o r p u b l i c h e a l t h / h e a l t h care a u t h o r i t i e s as Avedis D o n a b e d i a n a n d Osler P e t e r s o n . T h e style of t h e b o o k is c o n v e r s a t i o n a l w i t h occasional asides a n d it can easily b e read in a few h o u r s . T h e m o s t significant q u e s t i o n I have a b o u t b o t h b o o k s , is exactly w h a t a u d i e n c e will find t h e m useful? While discussion in b o t h b o o k s is a t a fairly simple level, b o t h sets of a u t h o r s still seem to p r e s u m e a c e r t a i n a m o u n t of k n o w l e d g e a b o u t scientific t e c h n i q u e s a n d a b o u t p r o g r a m e v a l u a t i o n o n t h e p a r t s o f t h e readers. N e i t h e r b o o k starts f r o m square one. F o r e x a m p l e , in F r a n k l i n a n d T h r a s h e r , social a u d i t s are i n t r o d u c e d "as a special case o f process e v a l u a t i o n a n d o f q u a l i t y c o n t r o l " (p. 35). A t t h a t p o i n t in t h e b o o k , n e i t h e r process e v a l u a t i o n n o r q u a l i t y c o n t r o l have b e e n p r o p e r l y e x p l a i n e d , n o r does t h e discussion o f soaial audits p r o v i d e an a d e q u a t e p i c t u r e o f t h a t for t h e n e o p h y t e . E p s t e i n a n d T r i p o d i , to take a n o t h e r e x a m p l e , cover "site a n d m e d i u m o f o b s e r v a t i o n " u n d e r t h e heading " o b s e r v a t i o n a l research p r i n c i p l e s . " T h e y state " o b s e r vational research requires t h a t t h e site for t h e o b s e r v a t i o n s b e s t a n d a r d i z e d . In o b s e r v i n g p a t i e n t b e h a v i o r in a m e n t a l hospital, o n e w o u l d observe t h e m at specified times a n d places in t h e i r day. In o b s e r v i n g g r o u p w o r k e r s at w o r k , o n e m i g h t c o n f i n e o n e s e l f to o b s e r v a t i o n s o f i n t e r a c t i o n s w i t h g r o u p s t h a t have j u s t b e g u n or groups t h a t have b e e n m e e t i n g for m o r e t h a n t w o m o n t h s . This k i n d of s t a n d a r d -
251
i z a t i o n is necessary for valid c o m p a r i s o n s o f t h e subjects o f o b s e r v a t i o n s " (p. 46). T h a t seems to b e all t h e y have to say o n the m a t t e r . If t h e b o o k s are for b e g i n n e r s , m u c h in t h e m seems to m e to b e b e y o n d q u i c k u n d e r s t a n d i n g a n d possibly bey o n d a n y u n d e r s t a n d i n g at all w i t h o u t a h e a v y a d d i t i o n a l dose o f l e c t u r i n g or o u t s i d e reading. O n t h e o t h e r h a n d , t h e b o o k s seem to b e at far t o o simple a level for use b y g r a d u a t e s t u d e n t s in a m a j o r a c a d e m i c field, such as psyc h o l o g y or sociology. F o r t h a t , t h e references are t o o few a n d t o o general a n d t h e e x a m p l e s a n d discussions are t o o u n c h a l l e n g i n g . If the b o o k s are w r i t t e n for a n a u d i e n c e w h o are n o t really b e g i n n e r s a n d yet, n o t g r a d u a t e stud e n t s (or graduates) in t h e behavioral sciences, w h o a m o n g t h o s e w h o w a n t to k n o w s o m e t h i n g a b o u t p r o g r a m e v a l u a t i o n or research t e c h n i q u e s -- are left? Most a d m i n istrators w h o have b e e n o u t o f s c h o o l for a while, fall i n t o t h e b e g i n n e r class as do m o s t p s y c h i a t r y residents, social w o r k s t u d e n t s a n d graduates, or politicians. Perhaps, political a n d policy science types c o u l d use the b o o k , b u t these days, m o s t of t h e m seem fairly s o p h i s t i c a t e d in t h e "hard science" approaches and therefore, would want m o r e in a b o o k t h a n these p r o v i d e . N e i t h e r b o o k is substantial e n o u g h to p u r c h a s e as a m a j o r r e f e r e n c e for a personal library or to c o n s u l t w h e n a difficult t e c h n i c a l quest i o n arises. T h e b o o k s do serve to i n d i c a t e w h a t k i n d s o f issues, studies, a n d t e c h n i c a l q u e s t i o n s e x p e r i e n c e d evaluators are c o n c e r n e d w i t h and, t h e r e f o r e , help to define t h e field for t h o s e w h o are novices. B o t h b o o k s , t h e r e f o r e , can b e r e c o m m e n d e d for t h e i r clarity o f p r e s e n t a t i o n a n d for t h e i r p a c k a g i n g a good deal into a very l i m i t e d a m o u n t o f space. Yet, I ' m afraid t h a t it will n o t b e a p p r o p r i a t e for large n u m b e r s o f b e g i n n e r s b e c a u s e t h e y are n o t q u i t e e l e m e n t a r y e n o u g h , n o r for a d v a n c e d s t u d e n t s , b e c a u s e t h e y are t o o e l e m e n t a r y . F o r t h e i n - b e t w e e n group t h e y have some i n t e r e s t as a step t o w a r d d e f i n i n g the e v a l u a t i o n a n d p l a n n i n g fields.
Public Policy Evaluation: Vol. II, Sage Yearbooks in Politics and Public Policy, e d i t e d b y K e n n e t h M. D o l b e a r e . Beverly
Hills: Sage Publications, 1975, 288 pp., $1 7.50 (cloth); $7.50 (paper). Reviewer: Beryce W. M a c L e n n a n Public Policy E v a l u a t i o n is the s e c o n d in a series o f yearb o o k s o n politics a n d p u b l i c policy. T h e v o l u m e is divided i n t o t h r e e parts; c o n c e p t u a l i z a t i o n , i m p l e m e n t a t i o n a n d a p p l i c a t i o n s to crime c o n t r o l . Papers are selected to illust r a t e t r e n d s in p o l i c y m e t h o d o l o g y a n d to describe some o f t h e tools g e n e r a t e d to realise these t r e n d s . T h e t h r e e p a p e r s in p a r t 1 c o n t r a s t w i t h each o t h e r . Bert a n d C o n n o l l y s t a t e t h a t t h e social sciences m u s t form u l a t e strategies to a t t a c k t h e s t r u c t u r a l sources o f p u b lic p r o b l e m s b e f o r e t h e y r e a c h crisis p r o p o r t i o n s . As illust r a t i o n , t h e y a n a l y z e t h e essential c o n f l i c t (to t h e m ) bet w e e n t h e r e s t o r a t i o n o f e n v i r o n m e n t a l integrity a n d t h e a u t o n o m y of t h e m a r k e t e c o n o m y . J o h n s t o n m a i n t a i n s t h a t t h e c o n n e c t i o n b e t w e e n r e s e a r c h findings a n d p o l i c y decisions m u s t b e m a d e b y social scientists. He develops a f r a m e w o r k for p o l i c y - o r i e n t e d research w h i c h includes
d e s c r i p t i o n , forecast, e x p l a n a t i o n a n d criticism a n d discusses the i d e n t i f i c a t i o n a n d v a l i d a t i o n of o u t c o m e indicators as m e a s u r e m e n t p r o b l e m s . C o l e m a n views p u b l i c policy e v a l u a t i o n as a guide to a c t i o n in policy decisions. He c o n t r a s t s such d e c i s i o n - o r i e n t e d r e s e a r c h w i t h academic research a n d states t h a t t h e f o r m e r m u s t b e timely a n d t h a t a l t h o u g h it m a y b e r o u g h a n d r e a d y it m u s t have a high p r o b a b i l i t y of a c c u r a c y in o r d e r to b e credible to decision m a k e r s . He m a i n t a i n s t h a t social audits are essential in research o n r e s o u r c e u t i l i z a t i o n a n d illustrates his thesis f r o m his w o r k in e d u c a t i o n . In p a r t 2, C o o k a n d Scioli c o m p a r e e x p e r i m e n t a l a n d q u a s i - e x p e r i m e n t a l designs to m e a s u r e t h e i m p a c t of p u b lic policy. T h e y e m p h a s i z e t h a t t h e p r o g r a m is t h e basic u n i t of analysis a n d t h a t p r o c e d u r a l a n d o u t c o m e objectives m u s t b e studied. T h e y e x a m i n e i n t e r n a l a n d e x t e r n a l
252
BOOK REVIEWS
threats to validity and discuss h o w to o v e r c o m e them with different research designs. Nagel advocates the application of existing statistical m e t h o d s (correlations, linear programming, m a t r i x decision approaches and cost benefit analyses) to public policy controversies and discusses their alternative uses. Caputo and Cole report the results of a survey c o n d u c t e d to study the initial policy implications of N i x o n ' s general revenue sharing program for m e t r o p o l itan America and discuss some of the problems which are c o m m o n to policy analyses and specific to studies o f revenue sharing. Gardiner introduces the third part by p o i n t i n g out the limited role played by evaluation in the m a n a g e m e n t of criminal justice programs. He relates this to the preoccupation of managers with day to day crises rather than the a c h i e v e m e n t of p r o g r a m m a t i c objectives. He identifies three p r o b l e m s for evaluators of criminal justice policy: the vagueness of goals, u n c e r t a i n t y regarding indicators of success or failure, and lack of clear relationships between criminal justice activities and goal attainment. These are illustrated by the three following papers. Parks utilizes official statistics and citizen surveys to evaluate police p e r f o r m a n c e in 45 m e t r o p o l i t a n neighborhoods. Kobrin and Lubrick argue that the scope of crime control is d e l i m i t e d by social ecology, that a normative level o f crime control is delimited by social ecology, that a normative level o f crime should be established for each envir o n m e n t under study and that sustained effort in crime
r e d u c t i o n should be directed to the amelioration of social conditions. Kelly examines the p r o b l e m s inherent in the evaluation of pretrial release programs and identifies the weakness of statistical data as guides to lower c o u r t activities. The three parts of this v o l u m e do not clearly relate to each o t h e r and the v o l u m e lacks a critical and integrating s u m m a t i o n so that it does not quite achieve the editor's primary goal " t o critically assess the state of the art of public policy evaluative research at mid-passage p o i n t in the social sciences' efforts to develop policy interpretative capabilities." However, the papers do examine questions which are fundamental to all policy evaluation: w h e t h e r it is appropriate to accept the basic organization of society in evaluating efforts to improve conditions; what dimensions must be e x a m i n e d and in what detail in order to obtain i n f o r m a t i o n essential to policy decision making; what is the state of the art in solving problems of design, data gathering and analysis. The problems of sustaining quality programs and permission to evaluate over time are not addressed and perhaps insufficient emphasis is placed on the d y n a m i c interaction b e t w e e n policies, programs and the changing environment. Neverthelesg, the papers are interesting and thought-provoking, and the volume has general relevance for public planners and evaluators across the social and political sciences and as a b o o k of readings for students of policy evaluation.
t i o n , social welfare, d e v e l o p m e n t a l disabilities, p u b l i c administration, nutrition, gerontology, rehabilitation, poverty, delinquency, criminal justice, and many others" (p. 18). In six c h a p t e r s a n d an epilogue, t h e a u t h o r s proc e e d to deal step b y step w i t h m o s t o f t h e m a j o r aspects of p r o g r a m e v a l u a t i o n . T h e y usually give b o t h sides o f t h e c o n t r o v e r s i a l areas w h i c h t h e y deal w i t h a n d are clear in discussing the issues. T h e y i n c l u d e a n u m b e r o f b r i e f descriptions as e x a m p l e s b u t a l m o s t all e x a m p l e s are f r o m t h e field o f m e n t a l h e a l t h a n d closely r e l a t e d areas (e.g., e v a l u a t i o n of criminal j u s t i c e or a n t i p o v e r t y programs.) T h e i r references also heavily e m p h a s i z e m e n t a l h e a l t h a n d t h e y o m i t such m a j o r p u b l i c h e a l t h / h e a l t h care a u t h o r i t i e s as Avedis D o n a b e d i a n a n d Osler P e t e r s o n . T h e style of t h e b o o k is c o n v e r s a t i o n a l w i t h occasional asides a n d it can easily b e read in a few h o u r s . T h e m o s t significant q u e s t i o n I have a b o u t b o t h b o o k s , is exactly w h a t a u d i e n c e will find t h e m useful? While discussion in b o t h b o o k s is a t a fairly simple level, b o t h sets of a u t h o r s still seem to p r e s u m e a c e r t a i n a m o u n t of k n o w l e d g e a b o u t scientific t e c h n i q u e s a n d a b o u t p r o g r a m e v a l u a t i o n o n t h e p a r t s o f t h e readers. N e i t h e r b o o k starts f r o m square one. F o r e x a m p l e , in F r a n k l i n a n d T h r a s h e r , social a u d i t s are i n t r o d u c e d "as a special case o f process e v a l u a t i o n a n d o f q u a l i t y c o n t r o l " (p. 35). A t t h a t p o i n t in t h e b o o k , n e i t h e r process e v a l u a t i o n n o r q u a l i t y c o n t r o l have b e e n p r o p e r l y e x p l a i n e d , n o r does t h e discussion o f soaial audits p r o v i d e an a d e q u a t e p i c t u r e o f t h a t for t h e n e o p h y t e . E p s t e i n a n d T r i p o d i , to take a n o t h e r e x a m p l e , cover "site a n d m e d i u m o f o b s e r v a t i o n " u n d e r t h e heading " o b s e r v a t i o n a l research p r i n c i p l e s . " T h e y state " o b s e r vational research requires t h a t t h e site for t h e o b s e r v a t i o n s b e s t a n d a r d i z e d . In o b s e r v i n g p a t i e n t b e h a v i o r in a m e n t a l hospital, o n e w o u l d observe t h e m at specified times a n d places in t h e i r day. In o b s e r v i n g g r o u p w o r k e r s at w o r k , o n e m i g h t c o n f i n e o n e s e l f to o b s e r v a t i o n s o f i n t e r a c t i o n s w i t h g r o u p s t h a t have j u s t b e g u n or groups t h a t have b e e n m e e t i n g for m o r e t h a n t w o m o n t h s . This k i n d of s t a n d a r d -
251
i z a t i o n is necessary for valid c o m p a r i s o n s o f t h e subjects o f o b s e r v a t i o n s " (p. 46). T h a t seems to b e all t h e y have to say o n the m a t t e r . If t h e b o o k s are for b e g i n n e r s , m u c h in t h e m seems to m e to b e b e y o n d q u i c k u n d e r s t a n d i n g a n d possibly bey o n d a n y u n d e r s t a n d i n g at all w i t h o u t a h e a v y a d d i t i o n a l dose o f l e c t u r i n g or o u t s i d e reading. O n t h e o t h e r h a n d , t h e b o o k s seem to b e at far t o o simple a level for use b y g r a d u a t e s t u d e n t s in a m a j o r a c a d e m i c field, such as psyc h o l o g y or sociology. F o r t h a t , t h e references are t o o few a n d t o o general a n d t h e e x a m p l e s a n d discussions are t o o u n c h a l l e n g i n g . If the b o o k s are w r i t t e n for a n a u d i e n c e w h o are n o t really b e g i n n e r s a n d yet, n o t g r a d u a t e stud e n t s (or graduates) in t h e behavioral sciences, w h o a m o n g t h o s e w h o w a n t to k n o w s o m e t h i n g a b o u t p r o g r a m e v a l u a t i o n or research t e c h n i q u e s -- are left? Most a d m i n istrators w h o have b e e n o u t o f s c h o o l for a while, fall i n t o t h e b e g i n n e r class as do m o s t p s y c h i a t r y residents, social w o r k s t u d e n t s a n d graduates, or politicians. Perhaps, political a n d policy science types c o u l d use the b o o k , b u t these days, m o s t of t h e m seem fairly s o p h i s t i c a t e d in t h e "hard science" approaches and therefore, would want m o r e in a b o o k t h a n these p r o v i d e . N e i t h e r b o o k is substantial e n o u g h to p u r c h a s e as a m a j o r r e f e r e n c e for a personal library or to c o n s u l t w h e n a difficult t e c h n i c a l quest i o n arises. T h e b o o k s do serve to i n d i c a t e w h a t k i n d s o f issues, studies, a n d t e c h n i c a l q u e s t i o n s e x p e r i e n c e d evaluators are c o n c e r n e d w i t h and, t h e r e f o r e , help to define t h e field for t h o s e w h o are novices. B o t h b o o k s , t h e r e f o r e , can b e r e c o m m e n d e d for t h e i r clarity o f p r e s e n t a t i o n a n d for t h e i r p a c k a g i n g a good deal into a very l i m i t e d a m o u n t o f space. Yet, I ' m afraid t h a t it will n o t b e a p p r o p r i a t e for large n u m b e r s o f b e g i n n e r s b e c a u s e t h e y are n o t q u i t e e l e m e n t a r y e n o u g h , n o r for a d v a n c e d s t u d e n t s , b e c a u s e t h e y are t o o e l e m e n t a r y . F o r t h e i n - b e t w e e n group t h e y have some i n t e r e s t as a step t o w a r d d e f i n i n g the e v a l u a t i o n a n d p l a n n i n g fields.
Public Policy Evaluation: Vol. II, Sage Yearbooks in Politics and Public Policy, e d i t e d b y K e n n e t h M. D o l b e a r e . Beverly
Hills: Sage Publications, 1975, 288 pp., $1 7.50 (cloth); $7.50 (paper). Reviewer: Beryce W. M a c L e n n a n Public Policy E v a l u a t i o n is the s e c o n d in a series o f yearb o o k s o n politics a n d p u b l i c policy. T h e v o l u m e is divided i n t o t h r e e parts; c o n c e p t u a l i z a t i o n , i m p l e m e n t a t i o n a n d a p p l i c a t i o n s to crime c o n t r o l . Papers are selected to illust r a t e t r e n d s in p o l i c y m e t h o d o l o g y a n d to describe some o f t h e tools g e n e r a t e d to realise these t r e n d s . T h e t h r e e p a p e r s in p a r t 1 c o n t r a s t w i t h each o t h e r . Bert a n d C o n n o l l y s t a t e t h a t t h e social sciences m u s t form u l a t e strategies to a t t a c k t h e s t r u c t u r a l sources o f p u b lic p r o b l e m s b e f o r e t h e y r e a c h crisis p r o p o r t i o n s . As illust r a t i o n , t h e y a n a l y z e t h e essential c o n f l i c t (to t h e m ) bet w e e n t h e r e s t o r a t i o n o f e n v i r o n m e n t a l integrity a n d t h e a u t o n o m y of t h e m a r k e t e c o n o m y . J o h n s t o n m a i n t a i n s t h a t t h e c o n n e c t i o n b e t w e e n r e s e a r c h findings a n d p o l i c y decisions m u s t b e m a d e b y social scientists. He develops a f r a m e w o r k for p o l i c y - o r i e n t e d research w h i c h includes
d e s c r i p t i o n , forecast, e x p l a n a t i o n a n d criticism a n d discusses the i d e n t i f i c a t i o n a n d v a l i d a t i o n of o u t c o m e indicators as m e a s u r e m e n t p r o b l e m s . C o l e m a n views p u b l i c policy e v a l u a t i o n as a guide to a c t i o n in policy decisions. He c o n t r a s t s such d e c i s i o n - o r i e n t e d r e s e a r c h w i t h academic research a n d states t h a t t h e f o r m e r m u s t b e timely a n d t h a t a l t h o u g h it m a y b e r o u g h a n d r e a d y it m u s t have a high p r o b a b i l i t y of a c c u r a c y in o r d e r to b e credible to decision m a k e r s . He m a i n t a i n s t h a t social audits are essential in research o n r e s o u r c e u t i l i z a t i o n a n d illustrates his thesis f r o m his w o r k in e d u c a t i o n . In p a r t 2, C o o k a n d Scioli c o m p a r e e x p e r i m e n t a l a n d q u a s i - e x p e r i m e n t a l designs to m e a s u r e t h e i m p a c t of p u b lic policy. T h e y e m p h a s i z e t h a t t h e p r o g r a m is t h e basic u n i t of analysis a n d t h a t p r o c e d u r a l a n d o u t c o m e objectives m u s t b e studied. T h e y e x a m i n e i n t e r n a l a n d e x t e r n a l
252
BOOK REVIEWS
threats to validity and discuss h o w to o v e r c o m e them with different research designs. Nagel advocates the application of existing statistical m e t h o d s (correlations, linear programming, m a t r i x decision approaches and cost benefit analyses) to public policy controversies and discusses their alternative uses. Caputo and Cole report the results of a survey c o n d u c t e d to study the initial policy implications of N i x o n ' s general revenue sharing program for m e t r o p o l itan America and discuss some of the problems which are c o m m o n to policy analyses and specific to studies o f revenue sharing. Gardiner introduces the third part by p o i n t i n g out the limited role played by evaluation in the m a n a g e m e n t of criminal justice programs. He relates this to the preoccupation of managers with day to day crises rather than the a c h i e v e m e n t of p r o g r a m m a t i c objectives. He identifies three p r o b l e m s for evaluators of criminal justice policy: the vagueness of goals, u n c e r t a i n t y regarding indicators of success or failure, and lack of clear relationships between criminal justice activities and goal attainment. These are illustrated by the three following papers. Parks utilizes official statistics and citizen surveys to evaluate police p e r f o r m a n c e in 45 m e t r o p o l i t a n neighborhoods. Kobrin and Lubrick argue that the scope of crime control is d e l i m i t e d by social ecology, that a normative level o f crime control is delimited by social ecology, that a normative level o f crime should be established for each envir o n m e n t under study and that sustained effort in crime
r e d u c t i o n should be directed to the amelioration of social conditions. Kelly examines the p r o b l e m s inherent in the evaluation of pretrial release programs and identifies the weakness of statistical data as guides to lower c o u r t activities. The three parts of this v o l u m e do not clearly relate to each o t h e r and the v o l u m e lacks a critical and integrating s u m m a t i o n so that it does not quite achieve the editor's primary goal " t o critically assess the state of the art of public policy evaluative research at mid-passage p o i n t in the social sciences' efforts to develop policy interpretative capabilities." However, the papers do examine questions which are fundamental to all policy evaluation: w h e t h e r it is appropriate to accept the basic organization of society in evaluating efforts to improve conditions; what dimensions must be e x a m i n e d and in what detail in order to obtain i n f o r m a t i o n essential to policy decision making; what is the state of the art in solving problems of design, data gathering and analysis. The problems of sustaining quality programs and permission to evaluate over time are not addressed and perhaps insufficient emphasis is placed on the d y n a m i c interaction b e t w e e n policies, programs and the changing environment. Neverthelesg, the papers are interesting and thought-provoking, and the volume has general relevance for public planners and evaluators across the social and political sciences and as a b o o k of readings for students of policy evaluation.