- Email: [email protected]
Educational quality indicators in the United States: Latest developments
Studies in EducationalEvaluation. Vol. 14, pp. 75-89, 1988
0191-491X/88 $0.00 + .50 Copyright © 1988 Pergamon Press plc
Printed in Great Britain. Atl rights reserved.
EDUCATIONAL QUALITY INDICATORS IN THE UNITED STATES: LATEST DEVELOPMENTS* Leigh Burstein University of Ca//fom/a, l o s Angeles
The i n t r o d u c t o r y c o m m e n t a r y on national quality indicators portrays a portion of the scenario of efforts in the United States. B u t a n y discussion of quality indicators in American e d u c a t i o n becomes i m m e d i a t e l y d a t e d once entered on the printed page; the activity level a n d the state of practice could best be characterized as "every which way at the same time". Agencies a n d organizations at both federal a n d state levels have been actively engaged in indicator developments; moreover, t h e s e developments have not b e e n restricted to agencies with a strictly e d u c a t i o n a l focus. A Selective sampling of indicator efforts would include at least the following: Since 1983, the U.S. D e p a r t m e n t of E d u c a t i o n h a s p u b l i s h e d State E d u c a t i o n S t a t i s t i c s c h a r t s (better k n o w n as "the Wall Chart") which r a n k s t a t e s on average scores form widely u s e d college a d m i s s i o n s tests [the Scholastic Aptitude Test (SAT) a n d American College Testing (ACT)] and other educational variables. This p u b l i c a t i o n h a s received considerable a t t e n t i o n from the press, the public, and various policy m a k e r s and educational college admissions test results for t h i s purpose. *Work on this paper was partially supported by a grant from the Office of E d u c a t i o n a l Research a n d I m p r o v e m e n t / D e p a r t m e n t of E d u c a t i o n (OERI/ED). However, t h e o p i n i o n s e x p r e s s e d h e r e i n do n o t n e c e s s a r i l y reflect the position or policy of OERI/ED a n d no official e n d o r s e m e n t by the OERI/ED should be inferred. 75
76
L. Burstein
D e s p i t e c o g e n t s u b s t a n t i v e a n d t e c h n i c a l c r i t i c i s m s of t hi s p r actice (e.g., Wainer, Holland, Swinton, a n d Wang, 1985), s t r o n g r e a c t i o n from state e d u c a t i o n officials, and the tacit a c k n o w l e d g e m e n t b y f e d e r a l officials t h a t a d m i s s i o n t e s t i n g r e s u l t s reflect only a limited p a r t of the p e r f o r m a n c e picture, t he Wall c h a r t still r e t a i n s s t a t e r a n k i n g s on t h e s e d a t a as a key component. In N o v e m b e r 1984, t he c ounc i l of Chief S t a t e School Officers (CCSSO) i s s u e d a p o s i t i o n p a p e r in s u p p o r t of s t a t e - b y - s t a t e c o m p a r i s o n s of e d u c a t i o n a l a c h i e v e m e n t a n d c o m m i t t i n g t h e i r o r g a n i z a t i o n to w or k with federal a n d s t a t e agenci es to improve t h e i r c a p a c i t y to g a t h e r , anal yze, a n d r e p o r t on a v a r i e t y of a s s e s s m e n t m a t t e r s . To c a r r y o u t t h e i r c o m m i t t m e n t , CCSSO e s t a b l i s h e d an e d u c a t i o n a s s e s s s m e n t c e n t e r to m a i n t a i n a n d d i s s e m i n a t e i n f o r m a t i o n on a s s e s s m e n t p r a c t i c e s of t he s t a t e s a n d s u p p o r t cl os e r a l i g n m e n t of s t a t e t e s t i n g p r o g r a m m e s a n d c o o r d i n a t i o n a m o n g international, national, a n d state a s s e s s m e n t a c t i v i t i e s . T h i s C C S S O c e n t e r is c u r r e n t l y p r o c e e d i n g to i m p l e m e n t a c o m p r e h e n s i v e pl a n t h a t would i n c l u d e c r o s s - s t a t e i n d i c a t o r s of e d u c a t i o n a l o u t c o m e s , e d u c a t i o n a l c o n t e x t , a n d e d u c a t i o n a l policies and pr a c t i c e s (CCSSO, 1984; Seldon, 1986a, 1986b). Th e C e n t e r for St a t i s t i cs (CS; f o r m e r l y the National C e n t e r for E d u c a t i o n Statistics) in the U.S. D e p a r t m e n t of E d u c a t i o n h a s b e e n actively involved in i n d i c a t o r d e v e l o p m e n t s in a n u m b e r of ways. According to the p r e s e n t l e a d e r s h i p at CS (Elliott and Hall, 1985), th e D e p a r t m e n t of E d u c a t i o n is a t t e m p t i n g to define its a g e n d a for f u t u r e statistical collections and a n a l y s e s a n d "has b e c o m e a g o a d for d e v e l o p m e n t of b e t t e r m e a s u r e s w h e r e available o n e s are i n a d e q u a t e .... of t i m e - s e r i e s (studies) w h e r e only one-time r e s e a r c h studies are available ..... a n d for theoretical a n d empirical s t u d i e s n e e d e d to develop s y n t h e s e s or indices of i n f o r m a t i o n or to relate one set of d a t a to anot her" (p.8). To c a r r y o u t its role in this effort, CS h a s developed p l a n s for the redesign of its e l e m e n t a r y a n d s e c o n d a r y d a t a collection p r o g r a m (Center for S t a t i s t i c s , 1986a; Hall, J a e g e r , K e a r n e y a n d Wiley, 1985); f u n d e d a s t u d y by t h e C e n t e r for t h e S t u d y of E v a l u a t i o n ( B u r s tein, B a k e r , A s c h b a c h e r , a n d Keesling, 1985) to explore
Quality in U.S.A.
77
issues s u r r o u n d i n g the development of n a t i o n a l indicators b a s e d on i n f o r m a t i o n from c u r r e n t state testing programs; s u p p o r t e d the CCSSO to develop state-by-state indicators; a n d h a s been at the forefront of new federal c o m m i t m e n t s to the g a t h e r i n g a n d r e p o r t i n g of i n t e r n a t i o n a l e d u c a t i o n s t a t i s t i c s (Center for Statistics, 1985, 1986B). O u t s i d e t h e m a i n s t r e a m e d u c a t i o n a l a r e n a , t h e National Science F o u n d a t i o n (NSF) h a s s u p p o r t e d v a r i o u s r e p o r t s a n d s t u d i e s (e.g., The N a t i o n a l S c i e n c e B o a r d C o m m i s s i o n on P r e c o l l e g i a t e E d u c a t i o n in M a t h e m a t i c s , Science, a n d Technology, 1983; Raizen a n d J o n e s , 1985; R o m b e r g a n d S m i t h , 1985; S h a v e l s o n et al., 1986) t h a t could lead to p o t e n t i a l l y i m p o r t a n t n e w s y s t e m s to m o n i t o r progress in m a t h e m a t i c s , science, a n d technology education.
The Impetus for Indicator Activity
The veritable a v a l a n c h e of quality indicators activity in the U.S. is a direct consequence of the p u s h for educational reforms nationally. Since the election of Ronald Reagan in 1980, there have been s u b s t a n t i a l c h a n g e s in federal e d u c a t i o n a l policy [Clark a n d Astuto (1986) provide a n insightful analysis of this policy shift]. The changes have generally been to reduce the federal role in direct s u p p o r t of e d u c a t i o n a l p r o g r a m s , especially those t a r g e t e d to specific c o n s t i t u e n c i e s . Both guidelines a n d financial s u p p o r t have been reduced, c o n s i s t e n t with the a d m i n i s t r a t i o n ' s philosophy of state a n d local control and financing of education, which coincidenfly justifies cuts in federal s u p p o r t for education. To m a i n t a i n their presence in the national e d u c a t i o n a l a r e n a at the s a m e time t h a t federal m a n d a t e s a n d costs are r e d u c e d , t h e R e a g a n a d m i n i s t r a t i o n h a s c h o s e n to h i g h l i g h t its responsibilities for g a t h e r i n g a n d reporting i n f o r m a t i o n on the s t a t u s of e d u c a t i o n a n d e d u c a t i o n a l reform. The release of W h a t Works: Research About T e a c h i n g a n d Learning (1986), a selective c o m p e n d i u m of forty one r e s e a r c h findings a n d related references; a n d First Lessons: A Report on Elementary_ E d u c a t i o n in America (Bennett, 1986) are indicative of the m a n n e r in which the D e p a r t m e n t of E d u c a t i o n i n t e n d s to exert leadership. S u c h is the b a c k d r o p a g a i n s t which quality indicators activities in the
78
L. Burstein
U.S. m u s t be i n t e r p r e t e d . Strong p r e s s u r e for i m p r o v e m e n t , a need for m e a n s to a s s e s s the impact of educational reforms, a n d c o n s e q u e n t political m a n e u v e r i n g to d e t e r m i n e who s e t s the s t a n d a r d s a n d who m e a s u r e s progress toward them. The major new actor is "the public" and its elected representatives. In the p a s t e d u c a t i o n a l e s t a b l i s h m e n t s at the federal, n a t i o n a l (e.g., the n a t i o n a l t e a c h e r s and a d m i n i s t r a t o r s organizations) and state (chief state school officers, state boards of education) retained m u c h of t h e f u n c t i o n a l c o n t r o l over e d u c a t i o n a l policies a n d p r a c t i c e s . Now g o v e r n o r s a n d s t a t e legislators are i n c r e a s i n g l y b e c o m i n g involved in e s t a b l i s h i n g e d u c a t i o n a l policies a n d s t a n d a r d s a n d the m e a n s to m e a s u r e a c h i e v e m e n t a n d progress. Types of Educational Indicators V a r i o u s s c h e m e s for categorizing e d u c a t i o n a l i n d i c a t o r s have been s u g g e s t e d (e.g., C o m m i t t e e on C o o r d i n a t i n g E d u c a t i o n a l I n f o r m a t i o n a n d Research; J a e g e r 1978, National Association of S e c o n d a r y School Principals, 1985; U.S. D e p a r t m e n t of E d u y c a t i o n , 1984). In the c u r r e n t l y prevailing climate, the two major d i m e n s i o n s along which an indicator can be classified have to do with the specific a s p e c t s of the e d u c a t i o n a l s y s t e m p r e s u m e d to be r e f l e c t e d by the indicator a n d the level of the s y s t e m to w h i c h it is targeted. The first dimension, n a m e l y the a s p e c t s reflected, classifies indicators into one of t h r e e t y p e s a c c o r d i n g to w h e t h e r t h e y refer to a condition affecting, a c h a r a c t e r i s t i c of, or a c o n s e q u e n c e of the e d u c a t i o n a l system. While t h e s e three a s p e c t s a p p e a r u n d e r a variety of n a m e s , "educational context" c a p t u r e s the c i r c u m s t a n c e s largely b e y o n d its control with which the e d u c a t i o n a l s y s t e m m u s t contend. Indicators of the d e m o g r a p h y of the s t u d e n t p o p u l a t i o n (e.g., e t h n i c diversity, E n g l i s h l a n g u a g e facility, socieconomic circumstances) of a n e d u c a t i o n a l j u r i s d i c t i o n (school, district, state, region) a n d of its economic resources (e.g., n a t u r a l resources, degree of i n d u s t r i a l i z a t i o n , f i n a n c i n g a u t h o r i t y ) fall w i t h i n t h i s c a t e g o r y , " E d u c a t i o n a l policies a n d practices" appropriately characterizes the second a s p e c t w i t h t h e t e r m s " o u t c o m e s " , " a c h i e v e m e n t s " or " p e r f o r m a n c e " d e n o t i n g the p r e s u m e d c o n s e q u e n c e s . M u c h of the c u r r e n t debate, a n d h e n c e activity, focuses on indicators of the quality of eductional outcomes. The r e m a i n d e r of the paper will, similarly, c o n c e n t r a t e on this aspect of quality indicators.
Quafity in U.S.A. 79
The second d i m e n s i o n of classification, the levels targeted, is defined by the u n i t s of observation, or p e r h a p s more precisely, the level at which i n d i c a t o r s are reported. Virtually a n y level a t w h i c h the activities and c o n s e q u e n c e s of the e d u c a t i o n a l s y s t e m can be e x a m i n e d can be u s e d to generate qualtiy indicators. Right now c i r c u m s t a n c e s for the n a t i o n as a whole a n d for t h e individual s t a t e s receive the m o s t public a t t e n t i o n . However, as s t a t e s begin to i m p l e m e n t their own versions of e d u c a t i o n a l reforms, i n d i c a t o r s at the level of the local e d u c a t i o n a l a u t h o r i t y (either district or school) are a p p e a r i n g more f r e q u e n t l y (e.g., California State D e p a r t m e n t of E d u c a t i o n , 1986; Fetler, 1986; Florida D e p a r t m e n t of Education, 1985). For t h e r e m a i n d e r of t h e p a p e r , we r e s t r i c t o u r a t t e n t i o n to d e v e l o p m e n t s involving n a t i o n w i d e a n d s t a t e - b y - s t a t e i n d i c a t o r s of the quality of educational outcomes. The c i r c u m s t a n c e s a n d issues with regard to each type of indicator will be discussed briefly. An examination of the the role of the National A s s e s s m e n t of E d u c a t i o n a l Progress (NAEP) will t h e n serve to illustrate the practical implications a n d complexities of attempting to satisfy the often competing p u r p o s e s of m e a s u r i n g n a t i o n a l educational process u s i n g cross-state comparisons of educational performance. Nationwide Indicators of Educational Quality As with almost every other d o m a i n of American society, both public and p r i v a t e s e c t o r s h a v e c o n t r i b u t e d to t h e d o c u m e n t a t i o n of n a t i o n a l e d u c a t i o n a l quality, especially with r e s p e c t to m e a s u r e m e n t of s t u d e n t performance. W h e n policy m a k e r s a n d r e s e a r c h e r s have raised questions a b o u t educational progress in the p a s t (e.g., Harnischfeger a n d Wiley, 1975;; Natinal I n s t i t u t e of Education, 1976; Wirtz et al., 1977), the performance d a t a cited have been d r a w n from a variety of sources: college a d m i s s i o n t e s t i n g p r o g r a m s , from t h e c o m m e r c i a l t e s t p u b l i s h e r s (e.g., t h e Iowa T e s t i n g program); a n d f e d e r a l l y - f u n d e d r e s e a r c h projects (e.g., n a t i o n a l l o n g i t u d i n a l s t u d i e s , c o m p e n s a t o r y e d u c a t i o n s t u d i e s a n d NAEP). Occasionally, the results from the cross-national s t u d i e s c o n d u c t e d by the International Association for Educational Achievement (IEA; e.g., Comber and Keeves, 1973; H u s e n , 1967, National Science Board 1986) enliven the challenges to existing educational policies a n d practices. E a c h of the cited d a t a b a s e s exhibits certain flaws which limit its utility in r e p r e s e n t i n g the "total" performance picture. Most suffer from a lack of
80
L, Burstein
r e p r e s e n t a t i v e n e s s to t h e n a t i o n as a whole, e i t h e r d u e to t h e selective p o p u l a t i o n s t a r g e t e d b y the t e s t s (e.g., only c o l l e g e - b o u n d or e c o n o m i c a l l y d i s a d v a n t a g e d s t ude nt s ) , or p r o b l e m s in the c o o p e r a t i o n r a t e s or r e a s o n s for participation. For example, lEA s t u d e n t s in the U.S. and the d e v e l o p m e n t of n o r m s for c o m m e r c i a l l y p u b l i s h e d s t a n d a r d i z e d t e s t s are p l a g u e d by the latter problem. C o n c e r n s can also be raised a b o u t t he c o n t e n t r e p r e s e n t a t i o n of e a c h of t h e s e p e r f o r m a n c e m e a s u r e s , the standardization of a d m i n i s t r a t i o n c o n d i t i o n s , a n d t h e f r e q u e n c y w i t h w h i c h t h e t e s t s are administered. B e c a u s e it w a s e x p l i c i t l y d e s i g n e d to m e a s u r e t h e a c h i e v e m e n t s of n a t i o n a l l y r e p r e s e n t a t i v e s a m p l e s , NAEP s t a n d s o u t theoretically as t he b e s t s o u r c e of d a t a on progress over time a l t h o u g h some h a v e q u e s t i o n e d its c o n t e n t s e l e c t i o n a n d i n f r e q u e n c y (this is deal t with m o r e fully below). To a great extent, t he c o m b i n a t i o n of public a n d private efforts and dat a s o u r c e s r e m a i n at the h e a r t of t he p r e s e n t efforts to a s s e s s c u r r e n t t r e n d s in e d u c a t i o n a l a c h i e v e m e n t (e.g., C ongr e ssi onal B u d g e t Office, 1986; National S c i e n c e B o ar d , 1986). B u t t he p r e s s u r e s to b r i n g t he d a t a b a s e s into line with th e perceived n e e d s of policy m a k e r s are g r e a t e r t h a n in earlier times as o u r d i s c u s s i o n of NAEP will highlight. Similarly, t h e r e n e w e d i n t e r e s t in i n t e r n a t i o n a l e d u c a t i o n a l st at i st i cs a n d c o m m i t m e n t to n a t i o n a l p a r t i c i p a t i o n in c r o s s - n a t i o n a l a s s e s s m e n t s of e d u c a t i o n a l a c h i e v e m e n t (Center for S t a t i s t i c s , 1986b) reflects a m a r k e d climatic c h a n g e . D u r i n g t he mid 1970's to early 1980's, a few individuals at the National I n st i t ut e of E d u c a t i o n , t he National Science F o u n d a t i o n and the t h e n National C e n t e r for E d u c a t i o n S t a t i s t i c s (com bi ned with t he S p e n c e r F o u n d a t i o n ) k e p t the S e c o n d I n t e r n a t i o n a l M a t h e m a t i c s S t u d y a n d o t h e r lEA s t u d i e s alive d e s pi t e t he a m b i v a l e n c e of t hei r agencies toward t he endeavor. C u r r e n t l y s t r o n g i n t e r e s t in the private a n d political s e c t o r in i n t e r n a t i o n a l e c o n o m i c rivalry a n d its perceived c o n n e c t i o n to t he q u a l i t y of a c o u n t r y ' s e d u c a t i o n a l s y s t e m g e n e r a t e s u p p o r t for collecting n a t i o n w i d e d a t a t h a t would be p a r t of a s y s t e m of i n t e r n a t i o n a l e d u c a t i o n a l i ndi cat ors. T h u s the i m m e d i a t e p r o s p e c t for i n t e r n a t i o n a l s t u d i e s in t he U.S. is bright, at least u n t i l s h o r t - r a n g e n a t i o n a l p r o v i n c i a l i s m agai n o v e r t a k e s l o n g - r a n g e good sense.
Quafity in U.S.A.
81
State-By-State Indicators of E d u c a t i o n a l Quality
While t h e p e r t i n e n c e of nat i onw i de i n d i c a t o r s to t he i s s u e of i n t e r e s t h e r e is self-evident, the role of s t a t e - b y - s t a t e c o m p a r i s o n s w a r r a n t s f u r t h e r explanation. An essential el em e nt of e d u c a t i o n a l reform in t he U.S. is state a c t i v i t y s in ce s t a t e s m a i n t a i n m u c h of t he p r o g r a m m a t i c a u t h o r i t y for e d u c a t i o n a n d in m o s t s t a t e s , a c c o u n t for t h e m a j o r p r o p o r t i o n of the f u n d i n g of local e d u c a t i o n a l p r o g r a m s . Given t h e A m e r i c a n p e n c h a n t for c omp etitio n a n d t he increasingly active role of the private b u s i n e s s sector in p r e s s i n g for e d u c a t i o n a l i m p r o v e m e n t s , it is not s u r p r i s i n g t h a t competition a m o n g t h e s t a t e s h a s b e e n s e e n as a significant e l e m e n t in t he p r e s e n t quality indicator efforts. The national as pe ct s of state c o m p a r i s o n s arise from t h e m e a n s to bring a b o u t state-level c o m p a r i s o n s of e d u c a t i o n a l quality. The k ey q u e s t i o n is who s h o u l d d e t e r m i n e the s t a n d a r d s a g a i n s t w h i c h different s t a t e s ar e to be c o m p a r e d given t h e n a t i o n a l i n t e r e s t in e d u c a t i o n a l i m p r o v e m e n t and the b l u r r e d lines of d e m a r c a t i o n b e t w e e n federal and state a u t h o r i t i e s (acting collectively) in the definition of nat i onal s t a n d a r d s ? While efforts in the a r e a of nationwide m e a s u r e s of e d u c a t i o n a l progress h av e a longer history, t he a g e n d a in s t a t e c o m p a r i s o n s is c u r r e n t l y m ore active an d controversial. As pointed out above, t he states, as reflected in the e n d o r s m e n t b y t he C ounci l of Chief S t a t e School Officers (CCSSO), have c o m m i t t e d to gathering comparative data. B u t st at es have diverse educational goals as reflected in c u r r i c u l u m guidelines, g r a d u a t i o n r e q u i r e m e n t s and o rien tatio n of their state testing p r o g r a m s (e.g., Goertz, 1986; B u r s t e i n et al., 1985). Moreover, s t a t e s v a r y c o n s i d e r a b l y in t h e d e m o g r a p h y of t h e i r s t u d e n t p o p u l a t i o n s a n d in the r e s o u r c e s t h e y c a n a n d do c o m m i t to s u p p o r t e d u c a t i o n . U n d e r s u c h c o n d i t i o n s , t h e sel ect i on of t he c o n t e n t s of t e s t s designed to m e a s u r e e d u c a t i o n a l a c h i e v e m e n t across the s t a t e s will probabl y be an a r d u o u s task. At p r e s e n t , b o t h the federal g o v e r n m e n t (through its p r o p o s e d c h a n g e s in th e C e n t e r for Statistics d a t a collection activities) a n d t he s t a t e s (through t h e efforts of t h e CCSSO a s s e s s m e n t cent er) are w o r k i n g to develop m e a s u r e s t h a t could be u s e d to c o m p a r e p e r f o r m a n c e across the states. This d ual effort c a n be either a blessing or a curse, d e p e n d i n g on w h e t h e r t he two g o v e r n m e n t a l levels c o o p e r a t e on s u c h m a t t e r s as t e s t c o n t e n t s , p r o g r a m a d m i n i s t r a t i o n , a n d f undi ng. While t he d i r e c t i o n s s u g g e s t e d b y c u r r e n t l y circulating idea p a p e r s (e.g., C e nt e r for Statistics, 1986a; Seldon, 1986b) are
82
L. Burstein
encouraging, agencies at the federal a nd st at e levels have yet to forge a joint c o m m i t m e n t to w o r k i n g toward a c o m m o n g r o u n d for resolving conflicts in the d o m a i n of c r o s s - s t a t e c o m p a r i s o n s of e d u c a t i o n a l quality.
The Role of NAEP
R e c e n t d e v e l o p m e n t s r e g a r d i n g the National A s s e s s m e n t of E d u c t i o n a l P ro g r es s ap tly i l l us t r a t e t he c h a n g e s b r o u g h t a b o u t by t he c u r r e n t wave of i n t e r e s t in q u a l i t y i n d i c a t o r s a n d t h e p o t e n t i a l for c o n t i n u i n g conflicts s u r r o u n d i n g its role as a n a t i o n a l m e a s u r e of e d u c a t i o n a l p r o g r e s s . Historically, NAEP c a n be t r a c e d to the desire by t he U.S. C o m m i s s i o n e r of E d u c a t i o n in the early 1960's to fulfill the original legislative m a n d a t e of the U.S. Office of E d u c a t i o n (the f o r e r u n n e r of t he D e p a r t m e n t of E ducat i on) to collect a n d d i s s e m i n a t e i n f o r m a t i o n on t h e c o n d i t i o n a n d p r o g r e s s of e d u c a t i o n . F r o m 1969 t h r o u g h 1983, a s s e s s m e n t s of t h e p e r f o r m a n c e of s e v e n y e a r - o l d s , eleven y e a r - o l d s a n d s e v e n t e e n y e a r - o l d s were collected from n a t i o n a l l y a n d regionally r e p r e s e n t a t i v e s a m p l e s in the a r e a s of writing, reading, literature, citizenship, social s t udi es, m a t h e m a t i c s , m usi c, art, and c a r e e r a n d o c c u p a t i o n a l d e v e l o p m e n t . M e a s u r e m e n t s in e a c h s u b j e c t were t a k e n at r o u g h l y four-or-five y e a r i n t e r v a l s with s u f f i c i e n t p r e c i s i o n to reliably identify na t i onw i de c h a n g e s in p e r f o r m a n c e . For its first dozen years, NAEP enjoyed a period of relative tranquility u n d e r th e g o v e r n a n c e of the E d u c a t i o n a l C o m m i s s i o n of t he St at es (ECS), an o r g a n i z a t i o n of the g o v e r n o r s a nd chief school officials from t he fifty states. D u r i n g t h i s ti m e it e s t a b l i s h e d a r e p u t a t i o n as a credible, low-profile, e ss en tially a political y a r d s t i c k of the n a t i o n ' s e d u c a t i o n a l progress. Among its m a j o r s t r e n g t h s were its c o n t r i b u t i o n s to t he t e c h n o l o g y of c u r r i c u l u m a s s e s s m e n t an d to t he d e v e l o p m e n t of t he capaci t y within s t a t e s to carry out t h e i r own a s s e s s m e n t s . O p e r a t i n g u n d e r t he a u s p i c e s of ECS legitimated NAEP's activ iti es w i t h s t a t e a n d local e d u c a t i o n a l a g e n c i e s , l e a d i n g to exceptionally high cooperation and hence nationally and regionally r e p r e s e n t a t i v e s a m p l e s with u n u s u a l l y high statistical integrity b y e d u c a t i o n s t an d ar d s . Th e decision w a s m a d e early on n o t to a t t e m p t s t a t e - b y - s t a t e reporting of NAEP results. T he cost of t he s a m pl i ng s u c h r e p o r t i n g would n e c c e s s i t a t e
Quality in U.S.A.
83
w a s c o n s i d e r e d to be prohibitive. Besides, s u c h c o m p a r i s o n s were perceived to be invidious by t h o s e e duc t i ona l agencies w h o s e c o o p e r a t i o n was seen as essential to th e p r o g r a m ' s success. In th e b r o a d e r s c h e m e of things, however, political i n n o c u o u s n e s s is n o t n e c e s s a r i l y a v i r t u e for a federally f u n d e d d a t a collection effort. T he failure to e s t a b l i s h a h i g h e r publ i c profile, b o t h w i t h i n a n d o u t s i d e t he e d u c a t i o n a l e s t a b l i s h m e n t , p r o v i d e d NAEP w i t h limited l everage in t he r e c u r r i n g f u n d i n g decisions of the federal g o v e r n m e n t . Over time the NAEP p r o g r a m s h r a n k due to rising costs a n d c u t s in its appropri at i ons, leading to less f r e q u e n t a s s e s s m e n t t h a n originally i nt ended. Yet the interval bet w een a s s e s s m e n t s w oul d ha ve h a d to d e c r e a s e m a r k e d l y or s o m e o t h e r m e a n s f o u n d to a t t r a c t a t t e n t i o n away from the a n n u a l reporting of t r e n d s in college a d m i s s i o n tes t results. T h e d e s i r e to h a v e NAEP s e r ve a l a r g e r role in t h e f o r m a t i o n of e d u c a t i o n a l policies a n d t h e m e a s u r e m e n t of t h e ,impact of e d u c a t i o n a l p r a c t i c e s was p e r h a p s t he m a j o r r e a s o n w h y t he c o n t r a c t for NAEP was shifted to the E d u c a t i o n a l Testing Service (ETS) 1 in 1983. ETS's proposed p l a n (Messick, B e a t o n a n d Lord, 1983), e c h o i n g a n a s s e s s m e n t of NAEP c o n d u c t e d b y Wirtz a n d LaPointe (1982), called for an e x p a n d e d role for NAEP as a "National r epor t card". While promising to m a i n t a i n t he high call calibre of th e i t em d e v e l o p m e n t w o r k c o n d u c t e d by ECS, ETS p r o p o s e d o t h e r m o d i f i c a t i o n s t h a t it believed w o u l d e n h a n c e NAEP's role in t he e s t a b l i s h m e n t of e d u c a t i o n a l s t a n d a r d s a n d in policy d e v e l o p m e n t s . The p r o p o s e d c h a n g e s w e r e p r i m a r i l y t e c h n i c a l : a s h i f t f r o m age-level to grade-level testing, m o r e f r e q u e n t t e s t i n g in core s u b j e c t s , a r e d e s i g n of t e s t s t h a t wo u l d e n c o u r a g e the e x a m i n a t i o n of r e l a t i o n s h i p s a m o n g t a s k s a n d b e t w e e n p e r f o r m a n c e a n d o t h e r s t u d e n t c h a r a c t e r i s t i c s , a n d a move toward scale-score reporting rather than the item-level performance statistics provided in ECS reports. T hes e c h a n g e s were seen as a significant shift to war d s tying p e r f o r m a n c e to the f u nct i oni ng of the e d u c a t i o n a l system, 1 T h r o u g h o u t its existence, NAEP has b e e n carried o u t u n d e r a c o n t r a c t from t h e F e d e r a l g o v e r n m e n t w i t h a p r i v a t e o r g a n i z a t i o n r a t h e r t h a n bei ng c o n d u c t e d directly by a specific federal agency. ECS was t h e c o n t r a c t o r from 1969 t h r o u g h 1983 at w h i c h time ETS w on a c o m p e t i t i o n for a five-year c o n t r a c t to c o n t i n u e t he a s s e s s m e n t . T he federal g o v e r n m e n t is c u r r e n t l y developing th e criteria a n d pr opos a l r e q u e s t for the n e x t c o m p e t i t i o n for the NAEP c o n t r a c t .
84
L. Burstein
t h u s p o t e n t i a l l y m a k i n g a s s e s s m e n t r e s u l t s m o r e u s e f u l for e d u c a t i o n a l d ecis io n m a k e r s while at t he s a m e time placing the c o n s e q u e n c e s of their a c t i o n s m o r e c l e a r l y in t h e p u b l i c view. Moreover, ETS q u i c k l y moved t o war d active e n c o u r a g e m e n t of state-level use of NAEP p e r f o r m a n c e d a t a by establishing a st a t e a s s e s s m e n t u n i t to facilitate its work in this area. As th e p l a n s for i n d i c a t o r s of s t a t e - b y - s t a t e e d u c a t i o n a l p e r f o r m a n c e h a v e crystallized, d i s c u s s i o n s a b o u t the s o u r c e of t h e t est i ng d a t a for state c o m p a r i s o n s have increasingly n a r r o w e d toward various alternatives t h a t rely on NAEP in s o m e way. In the fall of 1985, S e c r e t a r y of E d u c a t i o n B e n n e t t called for a s u b s t a n t i a l i n c r e a s e in t he NAEP t est i ng p r o g r a m while at t he s a m e time e n c o u r a g i n g s t at es to participate m ore fully in NAEP. The Center for Statistics' p l a n s for the redesign of their E l e m e n t a r y a n d S e c o n d a r y data collection p r o g r a m (Center for Statistics, 1986a) i n c o r p o r a t e a revised NAEP in th e p r o p o s e d i n t e g r a t e d d a t a s y s t e m a n d a n t i c i p a t e t h a t t he new syst em "will be c a p a b l e of e x p a n d i n g to S t a t e - R e p r e s e n t a t i v e s a m p l e s " . T he p r o p o s a l s guiding the i m p l e m e n t a a t i o n of t he CCSSO c o m m i t m e n t to obtain c o m p a r a b l e s t a t e - b y - s t a t e i n f o r m a t i o n on s t u d e n t a c h i e v e m e n t (CCSSO, 1984; Seldon, 1986b) call for the s t a t e s to " a t t e m p t to d r a w on t he NAEP item pool as a m e a n s of c o n s t r u c t i n g m e a s u r e s of s t u d e n t achi evem ent " and for i n c r e a s i n g the States' role in NAEP in order to i n s u r e t h a t NAEP c o n t e n t m o r e c o m p r e h e n s i v e l y reflects s t a t e p e r s p e c t i v e s . I n d e p e n d e n t of CCSSO actions, ETS h a s a l r e a d y t a k e n s t e ps to e x p a n d t he link b e t w e e n NAEP and individual s t a t e s t h r o u g h a j o i n t t es t i ng project with t he S o u t h e r n Regional E d u c a t i o n B o a r d in s e ve r al S o u t h e r n s t a t e s (SREB, 1985) a n d t h r o u g h i n c r e a s i n g th e n u m b e r of s t a t e s carrying out c o n c u r r e n t a s s e s s m e n t s . T h e r e are p l a n s for possible r e s t r u c t u r i n g t h r o u g h a n e w competetive bid for f u n d i n g . T hi s m i g h t lead to f u r t h e r d e c i s i o n s a b o u t h o w actively f u t u r e NAEP c o n t r a c t o r s will be in s t a t e - l e v e l r e p o r t i n g of e d u c a t i o n a l achievement. 2 2 T h e r e is a c e r t a i n i r o n y to t he i n c r e a s i n g a t t e n t i o n given to state-level r e p o r t i n g in the evolving role of NAEP. Yet NAEP's own s u c c e s s early on at e s t a b l i s h i n g itself as a credible, e s s e n t i a l l y a political, y a r d s t i c k a n d its c o n t r i b u t i o n s to t he d e v e l o p m e n t of t he c a p a c i t y w i t hi n s t a t e s to c a r r y out their own a s s e s s e m e n t s m a k e it an obvious t arget of o p p o r t u n i t y duri ng this p e r i o d w i t h its m o r e p r o m i n e n t role for e d u c a t i o n a l a s s e s s m e n t in e d u c a t i o n a l policy.
Quality in U.S.A. 85
Th e i n c r e a s e d b u r d e n s h e a p e d u p o n NAEP to serve multiple political, social, a n d e d u c a t i o n a l p u r p o s e s u n d e r t he s a m e u n d e r s u p p o r t e d u m b r e l l a are th e s o u r c e of "the potential for c o n t i n u i n g conflicts" to w hi ch we alluded earlier on. It is possible to foresee c i r c u m s t a n c e s t h a t could easily erode its s t a n d i n g a s a valid b a r o m e t e r of n a t i o n a l e d u c a t i o n a l p e r f o r m a n c e . Hopefully, t h e v a r i o u s i n t e r e s t s involved in t he deci si ons a b o u t NAEP can arrive at a f o r m u l a t h a t will allow its us e in the s t a t e - b y - s t a t e reporting while p r e s e r v i n g its i n t e g r i t y as a n a t i o n w i d e i n d i c a t o r , T h e r e are c e r t a i n l y t e c h n i c a l a l t e r n a t i v e s for a c c o m p l i s h i n g t h e s e dual goals b u t w h e t h e r t hey are practically a n d politically feasible r e m a i n s to be seen.
Speculation a b o u t F u t u r e Developments
Elliott a n d Hall (1985), a d m i n i s t r a t o r s of the C e n t e r for Statistics in the Office of E d u c a t i o n a l R e s e a r c h a n d I m p r o v e m e n t , c h a l l e n g e d e d u c a t o r s to p a r t i c i p a t e in t he d e v e l o p m e n t of m e a s u r e s for a s s e s s i n g h o w well, or how poorly, th e r e c e n t wave of e d u c a t i o n a l reforms a n d s u p p o r t is working. To this en d Elliott a n d Hall pos e d a set of q u e s t i o n s t h e y believed s h o u l d be a d d r e s s e d a b o u t w h e t h e r t he m e a s u r e s u s e d as e d u c a t i o n i n d i c a t o r s at the state a n d local level are reasonable. These q u e s t i o n s dealt with the c o n t e n t validity of existing a c h i e v e m e n t indicators as m e a s u r e s of w h a t is t a u g h t or s h o u l d be tau ght ; t h e ~.trength of t he r e l a t i o n s h i p s b e t w e e n t he i ndi cat ors a n d s t u d e n t p e r f o r m a n c e ; the alterabilitv of the conditions reflected by the indicators; the u n d e r s t a n d i n g of indicators; the u s e f u l n e s s of c u r r e n t m e a n s of r e D o r t i n ~ for v a r i o u s s t a k e h o l d e r s ; t he ability of c u r r e n t i n d i c a t o r s to r ef lect v a r i a t i o n in o p p o r t u n i t i e s a n d p e r f o r m a n c e a m o n g i m p o r t a n t s u b g r o u p s ; the c o m p a r a b i l i t y of m e a s u r e s a c r o s s locales; and finally their s u s c e p t a b i l i t y to distortion. Also highlighted was the nagging c o n c e r n a b o u t w h e t h e r th e m e a d i a a n d public officials could be e d u c a t e d to u s e i ndi cat or d a t a appropriately. Elliott a n d Hall ha ve p r e s e n t e d a t h o u g h t f u l a g e n d a for t he n a t i o n a l effort to develop e d u c a t i o n a l quality indicators. It is also an e n c o u r a g i n g one coming from federal officials who are w e l l -connect ed to the i m p l e m e n t a t i o n of f ed er al policy in t hi s area. B u t h i s t o r y h a s n o t b e e n k i n d to wishful t hin k in g a b o u t the role t h a t evaluative information actually plays in the highly p o l i t i c i z e d a t m o s p h e r e c r e a t e d b y t h e c o m p e t i n g i n t e r e s t involved in
86
L. Burstein
e d u c a t i o n a l reforms. We can expect the debate a r o u n d the a p p r o p r i a t e n e s s of various m e a s u r e s of educational outcomes to continue. Moreover, there are still virtually no available national d a t a t h a t capture the proximal qualities of e d u c a t i o n a l c o n t e x t s , r e s o u r c e s , a n d p r o c e s e s (i.e. w h a t e d u c a t o r s are actually faced with a n d w h a t t h e y do on a daily basis and w h a t s t u d e n t s are actually learning) as they impact on teaching and learning in schools and on the organization and m a n a g e m e n t of local educational systems. Yet, without s u s t a i n e d efforts, not to m e n t i o n s u b s t a n t i a l l y i n c r e a s e d financing, to improve t h e q u a l i t y a n d p r o g r a m fidelity of m e a s u r e s of e d u c a t i o n a l o u t c o m e s a n d to develop better m e a n s to contextualize t h e m by improving the c h a r a c t e r i z a t i o n s of the s e t t i n g s from w h i c h t h e y are g a t h e r e d , the n a t i o n a l m o v e m e n t toward the development of e d u c a t i o n a l quality indicators will r e m a i n largely a goad r a t h e r t h a n a guide to s u c c e s s f u l e d u c a t i o n a l reform.
References Bennett, W.J, (1986) First lessons: A report on elementary_ e d u c a t i o n in America. Washington, D.C. : U.S. Department of Education. Burstein, L., Baker, E.L., Aschbacher, P., and Keesling, J.W. (1985) Using state test d a t a for national indicators of education quality: A feasibility s t u d y . Los Angeles: Center for the S t u d y of Evaluation, G r a d u a t e School of Education, UCLA. Clark D.L. a n d Astuto T.A. (1986) The significance a n d p e r m a n e n c e of c h a n g e s in federal e d u c a t i o n a l policy 1980-1988. Occasional Paper, Bloomington, Indiana: Policy Studies Center of the University Council for Educational Administration, Indiana University, J a n u a r y . California State D e p a r t m e n t of Education. schools, Sacramento, CA, 1986.
Performance report for California
Center For Statistics. Plan for the redesign of the e l e m e n t a r y a n d secondary d a t a collection program. Working Paper, W a s h i n g t o n , D.C. : Office of E d u c a t i o n a l Research a n d Improvement, U.S. D e p a r t m e n t of Education, March 27, 1986 (a). C e n t e r For Staistics. "Proposal for a U.S. c o n s o r t i u m on c r o s s - n a t i o n a l e d u c a t i o n a l studies", Draft, W a s h i n g t o n D.C. : Office of E d u c a t i o n a l R e s e a r c h a n d Imporvement, U.S. D e p a r t m e n t of E d u c a t i o n , April 9, 1986 (b). Committee On Coordinating E d u c a t i o n a l Information a n d Research CCSSO Center on A s s e s s m e n t and Evaluation. Draft Report, Washington, D.C., November 15, 1985.
Quality in U.S.A.
87
C o m b e r L.C. a n d Keeves J. P. (1973) Science e d u c a t i o n in n i n e t e e n countries. New York: J o h n Wiley and Sons. C o n g r e s s i o n a l B u d g e t Offfice T r e n d s in E d u c a t i o n a l Washington D.C. Congress of the United States, 1986.
Achievement,
Council Of Chief S t a t e School Officers. E d u c a t i o n a l a s s e s s m e n t a n d evaluation in the United States. Position Paper, Washington, D.C, 1984. E11iott E.J. and Hall R. (1985) Indicators of performance: Measuring the educators, E d u c a t i o n a l M e a s u r e m e n t : I s s u e s a n d Practice, 4 (2), pp 6-8. Fetler, M. (1986) Accountability in California public schools. E d u c a t i o n a l Evaluation and Policy Analysis, 8_ (1), 31-44. Florida D e p a r t m e n t of E d u c a t i o n . Florida's progress toward excellence in e d u c a t i o n in t h e public schools. A Report to the State Board of E d u c a t i o n , T a l l a h a s s e e , FL: A s s e s s m e n t , Testing, a n d E v a l u a t i o n Section, B u r e a u of Program Support Services, December 1985. Goertz M.E. (1986) State educational standards: A 50-state survey, Research Report, Princeton, NJ: Educational Testing Services. Hall G., Jaeger, R.M. K e a m e y C.P. a n d Wiley D.E. (1985) Alternatives for a national d a t a s y s t e m on elementarv a n d s e c o n d a r y education, A Report p r e p a r e d for the Office of E d u c a t i o n a l R e s e a r c h a n d Improvement, December 20. Harnischfeger A. a n d Wiley D.E. (1975) Achievement test score decline: Do we need to worry?., Chicago: ML-Group for Policy Studies in Education. Husten. T. (ed.) (1967) International s t u d y of achievement in m a t h e m a t i c s : A c o m p a r i s o n of twelve countries, Stockholm a n d New York: Almquist a n d Vgiksell a n d J o h n Wiley and Sons. J a e g e r , R.M. (1978) A b o u t e d u c a t i o n a l i n d i c a t o r s : S t a t i s t i c s on the conditions a n d t r e n d s in education. In L.S. S h u l m a n (ed.). Review of Research in Education, 6. PP. 276-315. Katzenmeyer C.G. The f e d e r a l role in e n c o u r a g i n g s t a t e - b y - s t a t e a c h i e v e m e n t comparisons. Paper presented at the A n n u a l Meeting of the National Council on M e a s u r e m e n t in Education, San Francisco, April 1986. Messick, S., Beaton, A. a n d Lord. F. (1983) N a t i o n a l A ~ s e s s m e n t of E d u c a t i o n a l Progress reconsidered: A new design for a new era, NAEP R e p o r t 83-1~ Princeton NJ: National A s s e s s m e n t of E d u c a t i o n a l Progress. National Association Of Secondary School Principals. E d u c a t i o n indicators: A national report card. Reston VA: NASSP, 1985.
88
L. Burstein
National C e n t e r For E d u c a t i o n Statistics. E d u c a t i o n i ndi cat ors conference on international education statistics: Summary_ of d i s c u s s i o n s , Chevy Chase, Maryland: U.S. D e p a r t m e n t of E ducat i on, April 1985. National I n s t i t u t e of E d u c a t i o n . Declining t est scores: A c o n f e r e n c e report, Washington, D.C. : National Institute of Education, 1976. N a t i o n a l S c i e n c e B o a r d C o m m i s s i o n O n P r e c o l l e g e E d u c a t i o n In M a t h e m a t i c s , S c i e n c e a n d Technlogy. E d u c a t i n g A m e r i c a n s for t h e 2 1 s t cen tur y. Washington D.C. : National Science Foundat i on, 1983. National S cien c e B o a r d S c i e n c e i n d i c a t o r s : T he 1985 report, W a s h i n g t o n , D.C. : National Science Founda t i on, 1986. Raizen, S.A. a n d J o n e s L.V. (1985) I n d i c a t o r s of Precolle~e e d u c a t i o n in science and m a t h e m a t i c s . Washington~ D.C. : National A c a d e m y Press. R a n d C o r p o r a t i o n . Monitoring n a t i o n a l p r o g r e s s in m a t h e m a t i c s , science, an d t e c h n o l o g y education. Proposal to the National Science Foundat i on, S a n t a Monica, CA, 1985. R o m b e r g T. a n d S m i t h M.S. (1985) m a t h e m a t i c s monitoring center. E d u c a t i o n a l Research.
T h e e s t a b l i s h m e n t of a s c h o o l Madison, WI: W i s c o n s i n C e n t e r for
S e l d e n , R.W. S o m e c l a s s i c a l m e a s u r e m e n t i s s u e s c o n f r o n t i n g t he d e v e l o p m e n t of s t a t e - b y - s t a t e a s s e s s m e n t of s t u d e n t a c h i e v e m e n t . P a p e r p r e s e n t e d at t h e A n n u a l Meeting of t h e N at i onal Counci l on M e a s u r e m e n t in E d u c a t i o n , S a n Francisco, April 1986 (a). Selden, R.W. White P a p e r : Strategies and i s s u e s in t he d e v e l o p m e n t of c o m p a r a b l e i ndi cat or s for m e a s u r i n g s t u d e n t a c h i e v e m e n t . Washington, D.C. : S ta t e E d u c a t i o n A s s e s s m e n t Center, Council of Chief State School Officers, April 30, 1986 (b). S o u t h e r n R eg i ona l E d u c a t i o n B o a r d . M e a s u r i n g s t u d e n t a c h i e v e m e n t : C o m p a r a b l e t es t r e s u l t s for participating s o u t h e r n states, the South, and the Nation, Atlanta: S o u t h e r n Regional E d u c a t i o n Board, 1985. The National C om m i s s i on On Excellence In E d u c a t i o n . A n a t i o n at risk: The imr)erative for educat i onal reform. Washington, D.C. : U.S. G o v e r n m e n t Printing Office, 1983. U.S.
D e p a r t m e n t of E d u c a t i o n . S t a t e e d u c a t i o n s t a t i s t i c s : State p e r f o r m a n c e s , r e s o u r c e i m p u t s , a n d p o p u l a t i o n c h a r a c t e r i s t i c s 1972 an d 1982., 1984.
Wainer, H., Holland P.W., Swinton, S. a n d Wang M.H. (1985) On st at e e d u c a t i o n statistics. J o u r n a l of E d u c a t i o n a l Statistics, 10 (4), 293-326. U.S. D e p a r t m e n t of E d u c a t i o n . W h a t works: R e s e a r c h a b o u t t e a c h i n g and learning. Washington D.C. Wirtz, W. a n d Lapointe, A (1982) M e a s u r i n g t he qual i t y of e d u c a t i o n : A r e p o r t on a s s e s s i n g e d u c a t i o n a l progress. Washington, D.C.: Wirtz and Lapointe.
Quality in U.S.A.
89
Wirtz, W, et. al (1972) On f u r t h e r e x a m i n a t i o n : R e p o r t of t h e Advisory_ p a n e l o n t he S c h o l a s t i c A pt i t ude T e s t score decline. New York: The College Board.
The A ut hor LEIGH BURSTEIN is a P r o f e s s o r in t h e Social R e s e a r c h M e t h o d o l o g y Division of th e G r a d u a t e School of E d u c a t i o n at t he University of California, Los Angles. His w o r k on q u a l i t y i n d i c a t o r s i n c l u d e s a s t u d y on u s i n g i n f o r m a t i o n f r o m s t a t e t e s t i n g p r o g r a m s to c r e a t e c o m p a r a b l e c r o s s - s t a t e q u a l i t y i n d i c a t o r s a n d he s e r v e s as a n a d v i s o r y p a n e l i s t for t h e Rand C o r p o r a t i o n feasibility s t u d y on a c o m p r e h e n s i v e i n d i c a t o r s y s t e m for m o n i t o r i n g m a t h e m a t i c s , science, a n d t e c h n o l o g y e d u c a t i o n in t he United States.
0191-491X/88 $0.00 + .50 Copyright © 1988 Pergamon Press plc
Printed in Great Britain. Atl rights reserved.
EDUCATIONAL QUALITY INDICATORS IN THE UNITED STATES: LATEST DEVELOPMENTS* Leigh Burstein University of Ca//fom/a, l o s Angeles
The i n t r o d u c t o r y c o m m e n t a r y on national quality indicators portrays a portion of the scenario of efforts in the United States. B u t a n y discussion of quality indicators in American e d u c a t i o n becomes i m m e d i a t e l y d a t e d once entered on the printed page; the activity level a n d the state of practice could best be characterized as "every which way at the same time". Agencies a n d organizations at both federal a n d state levels have been actively engaged in indicator developments; moreover, t h e s e developments have not b e e n restricted to agencies with a strictly e d u c a t i o n a l focus. A Selective sampling of indicator efforts would include at least the following: Since 1983, the U.S. D e p a r t m e n t of E d u c a t i o n h a s p u b l i s h e d State E d u c a t i o n S t a t i s t i c s c h a r t s (better k n o w n as "the Wall Chart") which r a n k s t a t e s on average scores form widely u s e d college a d m i s s i o n s tests [the Scholastic Aptitude Test (SAT) a n d American College Testing (ACT)] and other educational variables. This p u b l i c a t i o n h a s received considerable a t t e n t i o n from the press, the public, and various policy m a k e r s and educational college admissions test results for t h i s purpose. *Work on this paper was partially supported by a grant from the Office of E d u c a t i o n a l Research a n d I m p r o v e m e n t / D e p a r t m e n t of E d u c a t i o n (OERI/ED). However, t h e o p i n i o n s e x p r e s s e d h e r e i n do n o t n e c e s s a r i l y reflect the position or policy of OERI/ED a n d no official e n d o r s e m e n t by the OERI/ED should be inferred. 75
76
L. Burstein
D e s p i t e c o g e n t s u b s t a n t i v e a n d t e c h n i c a l c r i t i c i s m s of t hi s p r actice (e.g., Wainer, Holland, Swinton, a n d Wang, 1985), s t r o n g r e a c t i o n from state e d u c a t i o n officials, and the tacit a c k n o w l e d g e m e n t b y f e d e r a l officials t h a t a d m i s s i o n t e s t i n g r e s u l t s reflect only a limited p a r t of the p e r f o r m a n c e picture, t he Wall c h a r t still r e t a i n s s t a t e r a n k i n g s on t h e s e d a t a as a key component. In N o v e m b e r 1984, t he c ounc i l of Chief S t a t e School Officers (CCSSO) i s s u e d a p o s i t i o n p a p e r in s u p p o r t of s t a t e - b y - s t a t e c o m p a r i s o n s of e d u c a t i o n a l a c h i e v e m e n t a n d c o m m i t t i n g t h e i r o r g a n i z a t i o n to w or k with federal a n d s t a t e agenci es to improve t h e i r c a p a c i t y to g a t h e r , anal yze, a n d r e p o r t on a v a r i e t y of a s s e s s m e n t m a t t e r s . To c a r r y o u t t h e i r c o m m i t t m e n t , CCSSO e s t a b l i s h e d an e d u c a t i o n a s s e s s s m e n t c e n t e r to m a i n t a i n a n d d i s s e m i n a t e i n f o r m a t i o n on a s s e s s m e n t p r a c t i c e s of t he s t a t e s a n d s u p p o r t cl os e r a l i g n m e n t of s t a t e t e s t i n g p r o g r a m m e s a n d c o o r d i n a t i o n a m o n g international, national, a n d state a s s e s s m e n t a c t i v i t i e s . T h i s C C S S O c e n t e r is c u r r e n t l y p r o c e e d i n g to i m p l e m e n t a c o m p r e h e n s i v e pl a n t h a t would i n c l u d e c r o s s - s t a t e i n d i c a t o r s of e d u c a t i o n a l o u t c o m e s , e d u c a t i o n a l c o n t e x t , a n d e d u c a t i o n a l policies and pr a c t i c e s (CCSSO, 1984; Seldon, 1986a, 1986b). Th e C e n t e r for St a t i s t i cs (CS; f o r m e r l y the National C e n t e r for E d u c a t i o n Statistics) in the U.S. D e p a r t m e n t of E d u c a t i o n h a s b e e n actively involved in i n d i c a t o r d e v e l o p m e n t s in a n u m b e r of ways. According to the p r e s e n t l e a d e r s h i p at CS (Elliott and Hall, 1985), th e D e p a r t m e n t of E d u c a t i o n is a t t e m p t i n g to define its a g e n d a for f u t u r e statistical collections and a n a l y s e s a n d "has b e c o m e a g o a d for d e v e l o p m e n t of b e t t e r m e a s u r e s w h e r e available o n e s are i n a d e q u a t e .... of t i m e - s e r i e s (studies) w h e r e only one-time r e s e a r c h studies are available ..... a n d for theoretical a n d empirical s t u d i e s n e e d e d to develop s y n t h e s e s or indices of i n f o r m a t i o n or to relate one set of d a t a to anot her" (p.8). To c a r r y o u t its role in this effort, CS h a s developed p l a n s for the redesign of its e l e m e n t a r y a n d s e c o n d a r y d a t a collection p r o g r a m (Center for S t a t i s t i c s , 1986a; Hall, J a e g e r , K e a r n e y a n d Wiley, 1985); f u n d e d a s t u d y by t h e C e n t e r for t h e S t u d y of E v a l u a t i o n ( B u r s tein, B a k e r , A s c h b a c h e r , a n d Keesling, 1985) to explore
Quality in U.S.A.
77
issues s u r r o u n d i n g the development of n a t i o n a l indicators b a s e d on i n f o r m a t i o n from c u r r e n t state testing programs; s u p p o r t e d the CCSSO to develop state-by-state indicators; a n d h a s been at the forefront of new federal c o m m i t m e n t s to the g a t h e r i n g a n d r e p o r t i n g of i n t e r n a t i o n a l e d u c a t i o n s t a t i s t i c s (Center for Statistics, 1985, 1986B). O u t s i d e t h e m a i n s t r e a m e d u c a t i o n a l a r e n a , t h e National Science F o u n d a t i o n (NSF) h a s s u p p o r t e d v a r i o u s r e p o r t s a n d s t u d i e s (e.g., The N a t i o n a l S c i e n c e B o a r d C o m m i s s i o n on P r e c o l l e g i a t e E d u c a t i o n in M a t h e m a t i c s , Science, a n d Technology, 1983; Raizen a n d J o n e s , 1985; R o m b e r g a n d S m i t h , 1985; S h a v e l s o n et al., 1986) t h a t could lead to p o t e n t i a l l y i m p o r t a n t n e w s y s t e m s to m o n i t o r progress in m a t h e m a t i c s , science, a n d technology education.
The Impetus for Indicator Activity
The veritable a v a l a n c h e of quality indicators activity in the U.S. is a direct consequence of the p u s h for educational reforms nationally. Since the election of Ronald Reagan in 1980, there have been s u b s t a n t i a l c h a n g e s in federal e d u c a t i o n a l policy [Clark a n d Astuto (1986) provide a n insightful analysis of this policy shift]. The changes have generally been to reduce the federal role in direct s u p p o r t of e d u c a t i o n a l p r o g r a m s , especially those t a r g e t e d to specific c o n s t i t u e n c i e s . Both guidelines a n d financial s u p p o r t have been reduced, c o n s i s t e n t with the a d m i n i s t r a t i o n ' s philosophy of state a n d local control and financing of education, which coincidenfly justifies cuts in federal s u p p o r t for education. To m a i n t a i n their presence in the national e d u c a t i o n a l a r e n a at the s a m e time t h a t federal m a n d a t e s a n d costs are r e d u c e d , t h e R e a g a n a d m i n i s t r a t i o n h a s c h o s e n to h i g h l i g h t its responsibilities for g a t h e r i n g a n d reporting i n f o r m a t i o n on the s t a t u s of e d u c a t i o n a n d e d u c a t i o n a l reform. The release of W h a t Works: Research About T e a c h i n g a n d Learning (1986), a selective c o m p e n d i u m of forty one r e s e a r c h findings a n d related references; a n d First Lessons: A Report on Elementary_ E d u c a t i o n in America (Bennett, 1986) are indicative of the m a n n e r in which the D e p a r t m e n t of E d u c a t i o n i n t e n d s to exert leadership. S u c h is the b a c k d r o p a g a i n s t which quality indicators activities in the
78
L. Burstein
U.S. m u s t be i n t e r p r e t e d . Strong p r e s s u r e for i m p r o v e m e n t , a need for m e a n s to a s s e s s the impact of educational reforms, a n d c o n s e q u e n t political m a n e u v e r i n g to d e t e r m i n e who s e t s the s t a n d a r d s a n d who m e a s u r e s progress toward them. The major new actor is "the public" and its elected representatives. In the p a s t e d u c a t i o n a l e s t a b l i s h m e n t s at the federal, n a t i o n a l (e.g., the n a t i o n a l t e a c h e r s and a d m i n i s t r a t o r s organizations) and state (chief state school officers, state boards of education) retained m u c h of t h e f u n c t i o n a l c o n t r o l over e d u c a t i o n a l policies a n d p r a c t i c e s . Now g o v e r n o r s a n d s t a t e legislators are i n c r e a s i n g l y b e c o m i n g involved in e s t a b l i s h i n g e d u c a t i o n a l policies a n d s t a n d a r d s a n d the m e a n s to m e a s u r e a c h i e v e m e n t a n d progress. Types of Educational Indicators V a r i o u s s c h e m e s for categorizing e d u c a t i o n a l i n d i c a t o r s have been s u g g e s t e d (e.g., C o m m i t t e e on C o o r d i n a t i n g E d u c a t i o n a l I n f o r m a t i o n a n d Research; J a e g e r 1978, National Association of S e c o n d a r y School Principals, 1985; U.S. D e p a r t m e n t of E d u y c a t i o n , 1984). In the c u r r e n t l y prevailing climate, the two major d i m e n s i o n s along which an indicator can be classified have to do with the specific a s p e c t s of the e d u c a t i o n a l s y s t e m p r e s u m e d to be r e f l e c t e d by the indicator a n d the level of the s y s t e m to w h i c h it is targeted. The first dimension, n a m e l y the a s p e c t s reflected, classifies indicators into one of t h r e e t y p e s a c c o r d i n g to w h e t h e r t h e y refer to a condition affecting, a c h a r a c t e r i s t i c of, or a c o n s e q u e n c e of the e d u c a t i o n a l system. While t h e s e three a s p e c t s a p p e a r u n d e r a variety of n a m e s , "educational context" c a p t u r e s the c i r c u m s t a n c e s largely b e y o n d its control with which the e d u c a t i o n a l s y s t e m m u s t contend. Indicators of the d e m o g r a p h y of the s t u d e n t p o p u l a t i o n (e.g., e t h n i c diversity, E n g l i s h l a n g u a g e facility, socieconomic circumstances) of a n e d u c a t i o n a l j u r i s d i c t i o n (school, district, state, region) a n d of its economic resources (e.g., n a t u r a l resources, degree of i n d u s t r i a l i z a t i o n , f i n a n c i n g a u t h o r i t y ) fall w i t h i n t h i s c a t e g o r y , " E d u c a t i o n a l policies a n d practices" appropriately characterizes the second a s p e c t w i t h t h e t e r m s " o u t c o m e s " , " a c h i e v e m e n t s " or " p e r f o r m a n c e " d e n o t i n g the p r e s u m e d c o n s e q u e n c e s . M u c h of the c u r r e n t debate, a n d h e n c e activity, focuses on indicators of the quality of eductional outcomes. The r e m a i n d e r of the paper will, similarly, c o n c e n t r a t e on this aspect of quality indicators.
Quafity in U.S.A. 79
The second d i m e n s i o n of classification, the levels targeted, is defined by the u n i t s of observation, or p e r h a p s more precisely, the level at which i n d i c a t o r s are reported. Virtually a n y level a t w h i c h the activities and c o n s e q u e n c e s of the e d u c a t i o n a l s y s t e m can be e x a m i n e d can be u s e d to generate qualtiy indicators. Right now c i r c u m s t a n c e s for the n a t i o n as a whole a n d for t h e individual s t a t e s receive the m o s t public a t t e n t i o n . However, as s t a t e s begin to i m p l e m e n t their own versions of e d u c a t i o n a l reforms, i n d i c a t o r s at the level of the local e d u c a t i o n a l a u t h o r i t y (either district or school) are a p p e a r i n g more f r e q u e n t l y (e.g., California State D e p a r t m e n t of E d u c a t i o n , 1986; Fetler, 1986; Florida D e p a r t m e n t of Education, 1985). For t h e r e m a i n d e r of t h e p a p e r , we r e s t r i c t o u r a t t e n t i o n to d e v e l o p m e n t s involving n a t i o n w i d e a n d s t a t e - b y - s t a t e i n d i c a t o r s of the quality of educational outcomes. The c i r c u m s t a n c e s a n d issues with regard to each type of indicator will be discussed briefly. An examination of the the role of the National A s s e s s m e n t of E d u c a t i o n a l Progress (NAEP) will t h e n serve to illustrate the practical implications a n d complexities of attempting to satisfy the often competing p u r p o s e s of m e a s u r i n g n a t i o n a l educational process u s i n g cross-state comparisons of educational performance. Nationwide Indicators of Educational Quality As with almost every other d o m a i n of American society, both public and p r i v a t e s e c t o r s h a v e c o n t r i b u t e d to t h e d o c u m e n t a t i o n of n a t i o n a l e d u c a t i o n a l quality, especially with r e s p e c t to m e a s u r e m e n t of s t u d e n t performance. W h e n policy m a k e r s a n d r e s e a r c h e r s have raised questions a b o u t educational progress in the p a s t (e.g., Harnischfeger a n d Wiley, 1975;; Natinal I n s t i t u t e of Education, 1976; Wirtz et al., 1977), the performance d a t a cited have been d r a w n from a variety of sources: college a d m i s s i o n t e s t i n g p r o g r a m s , from t h e c o m m e r c i a l t e s t p u b l i s h e r s (e.g., t h e Iowa T e s t i n g program); a n d f e d e r a l l y - f u n d e d r e s e a r c h projects (e.g., n a t i o n a l l o n g i t u d i n a l s t u d i e s , c o m p e n s a t o r y e d u c a t i o n s t u d i e s a n d NAEP). Occasionally, the results from the cross-national s t u d i e s c o n d u c t e d by the International Association for Educational Achievement (IEA; e.g., Comber and Keeves, 1973; H u s e n , 1967, National Science Board 1986) enliven the challenges to existing educational policies a n d practices. E a c h of the cited d a t a b a s e s exhibits certain flaws which limit its utility in r e p r e s e n t i n g the "total" performance picture. Most suffer from a lack of
80
L, Burstein
r e p r e s e n t a t i v e n e s s to t h e n a t i o n as a whole, e i t h e r d u e to t h e selective p o p u l a t i o n s t a r g e t e d b y the t e s t s (e.g., only c o l l e g e - b o u n d or e c o n o m i c a l l y d i s a d v a n t a g e d s t ude nt s ) , or p r o b l e m s in the c o o p e r a t i o n r a t e s or r e a s o n s for participation. For example, lEA s t u d e n t s in the U.S. and the d e v e l o p m e n t of n o r m s for c o m m e r c i a l l y p u b l i s h e d s t a n d a r d i z e d t e s t s are p l a g u e d by the latter problem. C o n c e r n s can also be raised a b o u t t he c o n t e n t r e p r e s e n t a t i o n of e a c h of t h e s e p e r f o r m a n c e m e a s u r e s , the standardization of a d m i n i s t r a t i o n c o n d i t i o n s , a n d t h e f r e q u e n c y w i t h w h i c h t h e t e s t s are administered. B e c a u s e it w a s e x p l i c i t l y d e s i g n e d to m e a s u r e t h e a c h i e v e m e n t s of n a t i o n a l l y r e p r e s e n t a t i v e s a m p l e s , NAEP s t a n d s o u t theoretically as t he b e s t s o u r c e of d a t a on progress over time a l t h o u g h some h a v e q u e s t i o n e d its c o n t e n t s e l e c t i o n a n d i n f r e q u e n c y (this is deal t with m o r e fully below). To a great extent, t he c o m b i n a t i o n of public a n d private efforts and dat a s o u r c e s r e m a i n at the h e a r t of t he p r e s e n t efforts to a s s e s s c u r r e n t t r e n d s in e d u c a t i o n a l a c h i e v e m e n t (e.g., C ongr e ssi onal B u d g e t Office, 1986; National S c i e n c e B o ar d , 1986). B u t t he p r e s s u r e s to b r i n g t he d a t a b a s e s into line with th e perceived n e e d s of policy m a k e r s are g r e a t e r t h a n in earlier times as o u r d i s c u s s i o n of NAEP will highlight. Similarly, t h e r e n e w e d i n t e r e s t in i n t e r n a t i o n a l e d u c a t i o n a l st at i st i cs a n d c o m m i t m e n t to n a t i o n a l p a r t i c i p a t i o n in c r o s s - n a t i o n a l a s s e s s m e n t s of e d u c a t i o n a l a c h i e v e m e n t (Center for S t a t i s t i c s , 1986b) reflects a m a r k e d climatic c h a n g e . D u r i n g t he mid 1970's to early 1980's, a few individuals at the National I n st i t ut e of E d u c a t i o n , t he National Science F o u n d a t i o n and the t h e n National C e n t e r for E d u c a t i o n S t a t i s t i c s (com bi ned with t he S p e n c e r F o u n d a t i o n ) k e p t the S e c o n d I n t e r n a t i o n a l M a t h e m a t i c s S t u d y a n d o t h e r lEA s t u d i e s alive d e s pi t e t he a m b i v a l e n c e of t hei r agencies toward t he endeavor. C u r r e n t l y s t r o n g i n t e r e s t in the private a n d political s e c t o r in i n t e r n a t i o n a l e c o n o m i c rivalry a n d its perceived c o n n e c t i o n to t he q u a l i t y of a c o u n t r y ' s e d u c a t i o n a l s y s t e m g e n e r a t e s u p p o r t for collecting n a t i o n w i d e d a t a t h a t would be p a r t of a s y s t e m of i n t e r n a t i o n a l e d u c a t i o n a l i ndi cat ors. T h u s the i m m e d i a t e p r o s p e c t for i n t e r n a t i o n a l s t u d i e s in t he U.S. is bright, at least u n t i l s h o r t - r a n g e n a t i o n a l p r o v i n c i a l i s m agai n o v e r t a k e s l o n g - r a n g e good sense.
Quafity in U.S.A.
81
State-By-State Indicators of E d u c a t i o n a l Quality
While t h e p e r t i n e n c e of nat i onw i de i n d i c a t o r s to t he i s s u e of i n t e r e s t h e r e is self-evident, the role of s t a t e - b y - s t a t e c o m p a r i s o n s w a r r a n t s f u r t h e r explanation. An essential el em e nt of e d u c a t i o n a l reform in t he U.S. is state a c t i v i t y s in ce s t a t e s m a i n t a i n m u c h of t he p r o g r a m m a t i c a u t h o r i t y for e d u c a t i o n a n d in m o s t s t a t e s , a c c o u n t for t h e m a j o r p r o p o r t i o n of the f u n d i n g of local e d u c a t i o n a l p r o g r a m s . Given t h e A m e r i c a n p e n c h a n t for c omp etitio n a n d t he increasingly active role of the private b u s i n e s s sector in p r e s s i n g for e d u c a t i o n a l i m p r o v e m e n t s , it is not s u r p r i s i n g t h a t competition a m o n g t h e s t a t e s h a s b e e n s e e n as a significant e l e m e n t in t he p r e s e n t quality indicator efforts. The national as pe ct s of state c o m p a r i s o n s arise from t h e m e a n s to bring a b o u t state-level c o m p a r i s o n s of e d u c a t i o n a l quality. The k ey q u e s t i o n is who s h o u l d d e t e r m i n e the s t a n d a r d s a g a i n s t w h i c h different s t a t e s ar e to be c o m p a r e d given t h e n a t i o n a l i n t e r e s t in e d u c a t i o n a l i m p r o v e m e n t and the b l u r r e d lines of d e m a r c a t i o n b e t w e e n federal and state a u t h o r i t i e s (acting collectively) in the definition of nat i onal s t a n d a r d s ? While efforts in the a r e a of nationwide m e a s u r e s of e d u c a t i o n a l progress h av e a longer history, t he a g e n d a in s t a t e c o m p a r i s o n s is c u r r e n t l y m ore active an d controversial. As pointed out above, t he states, as reflected in the e n d o r s m e n t b y t he C ounci l of Chief S t a t e School Officers (CCSSO), have c o m m i t t e d to gathering comparative data. B u t st at es have diverse educational goals as reflected in c u r r i c u l u m guidelines, g r a d u a t i o n r e q u i r e m e n t s and o rien tatio n of their state testing p r o g r a m s (e.g., Goertz, 1986; B u r s t e i n et al., 1985). Moreover, s t a t e s v a r y c o n s i d e r a b l y in t h e d e m o g r a p h y of t h e i r s t u d e n t p o p u l a t i o n s a n d in the r e s o u r c e s t h e y c a n a n d do c o m m i t to s u p p o r t e d u c a t i o n . U n d e r s u c h c o n d i t i o n s , t h e sel ect i on of t he c o n t e n t s of t e s t s designed to m e a s u r e e d u c a t i o n a l a c h i e v e m e n t across the s t a t e s will probabl y be an a r d u o u s task. At p r e s e n t , b o t h the federal g o v e r n m e n t (through its p r o p o s e d c h a n g e s in th e C e n t e r for Statistics d a t a collection activities) a n d t he s t a t e s (through t h e efforts of t h e CCSSO a s s e s s m e n t cent er) are w o r k i n g to develop m e a s u r e s t h a t could be u s e d to c o m p a r e p e r f o r m a n c e across the states. This d ual effort c a n be either a blessing or a curse, d e p e n d i n g on w h e t h e r t he two g o v e r n m e n t a l levels c o o p e r a t e on s u c h m a t t e r s as t e s t c o n t e n t s , p r o g r a m a d m i n i s t r a t i o n , a n d f undi ng. While t he d i r e c t i o n s s u g g e s t e d b y c u r r e n t l y circulating idea p a p e r s (e.g., C e nt e r for Statistics, 1986a; Seldon, 1986b) are
82
L. Burstein
encouraging, agencies at the federal a nd st at e levels have yet to forge a joint c o m m i t m e n t to w o r k i n g toward a c o m m o n g r o u n d for resolving conflicts in the d o m a i n of c r o s s - s t a t e c o m p a r i s o n s of e d u c a t i o n a l quality.
The Role of NAEP
R e c e n t d e v e l o p m e n t s r e g a r d i n g the National A s s e s s m e n t of E d u c t i o n a l P ro g r es s ap tly i l l us t r a t e t he c h a n g e s b r o u g h t a b o u t by t he c u r r e n t wave of i n t e r e s t in q u a l i t y i n d i c a t o r s a n d t h e p o t e n t i a l for c o n t i n u i n g conflicts s u r r o u n d i n g its role as a n a t i o n a l m e a s u r e of e d u c a t i o n a l p r o g r e s s . Historically, NAEP c a n be t r a c e d to the desire by t he U.S. C o m m i s s i o n e r of E d u c a t i o n in the early 1960's to fulfill the original legislative m a n d a t e of the U.S. Office of E d u c a t i o n (the f o r e r u n n e r of t he D e p a r t m e n t of E ducat i on) to collect a n d d i s s e m i n a t e i n f o r m a t i o n on t h e c o n d i t i o n a n d p r o g r e s s of e d u c a t i o n . F r o m 1969 t h r o u g h 1983, a s s e s s m e n t s of t h e p e r f o r m a n c e of s e v e n y e a r - o l d s , eleven y e a r - o l d s a n d s e v e n t e e n y e a r - o l d s were collected from n a t i o n a l l y a n d regionally r e p r e s e n t a t i v e s a m p l e s in the a r e a s of writing, reading, literature, citizenship, social s t udi es, m a t h e m a t i c s , m usi c, art, and c a r e e r a n d o c c u p a t i o n a l d e v e l o p m e n t . M e a s u r e m e n t s in e a c h s u b j e c t were t a k e n at r o u g h l y four-or-five y e a r i n t e r v a l s with s u f f i c i e n t p r e c i s i o n to reliably identify na t i onw i de c h a n g e s in p e r f o r m a n c e . For its first dozen years, NAEP enjoyed a period of relative tranquility u n d e r th e g o v e r n a n c e of the E d u c a t i o n a l C o m m i s s i o n of t he St at es (ECS), an o r g a n i z a t i o n of the g o v e r n o r s a nd chief school officials from t he fifty states. D u r i n g t h i s ti m e it e s t a b l i s h e d a r e p u t a t i o n as a credible, low-profile, e ss en tially a political y a r d s t i c k of the n a t i o n ' s e d u c a t i o n a l progress. Among its m a j o r s t r e n g t h s were its c o n t r i b u t i o n s to t he t e c h n o l o g y of c u r r i c u l u m a s s e s s m e n t an d to t he d e v e l o p m e n t of t he capaci t y within s t a t e s to carry out t h e i r own a s s e s s m e n t s . O p e r a t i n g u n d e r t he a u s p i c e s of ECS legitimated NAEP's activ iti es w i t h s t a t e a n d local e d u c a t i o n a l a g e n c i e s , l e a d i n g to exceptionally high cooperation and hence nationally and regionally r e p r e s e n t a t i v e s a m p l e s with u n u s u a l l y high statistical integrity b y e d u c a t i o n s t an d ar d s . Th e decision w a s m a d e early on n o t to a t t e m p t s t a t e - b y - s t a t e reporting of NAEP results. T he cost of t he s a m pl i ng s u c h r e p o r t i n g would n e c c e s s i t a t e
Quality in U.S.A.
83
w a s c o n s i d e r e d to be prohibitive. Besides, s u c h c o m p a r i s o n s were perceived to be invidious by t h o s e e duc t i ona l agencies w h o s e c o o p e r a t i o n was seen as essential to th e p r o g r a m ' s success. In th e b r o a d e r s c h e m e of things, however, political i n n o c u o u s n e s s is n o t n e c e s s a r i l y a v i r t u e for a federally f u n d e d d a t a collection effort. T he failure to e s t a b l i s h a h i g h e r publ i c profile, b o t h w i t h i n a n d o u t s i d e t he e d u c a t i o n a l e s t a b l i s h m e n t , p r o v i d e d NAEP w i t h limited l everage in t he r e c u r r i n g f u n d i n g decisions of the federal g o v e r n m e n t . Over time the NAEP p r o g r a m s h r a n k due to rising costs a n d c u t s in its appropri at i ons, leading to less f r e q u e n t a s s e s s m e n t t h a n originally i nt ended. Yet the interval bet w een a s s e s s m e n t s w oul d ha ve h a d to d e c r e a s e m a r k e d l y or s o m e o t h e r m e a n s f o u n d to a t t r a c t a t t e n t i o n away from the a n n u a l reporting of t r e n d s in college a d m i s s i o n tes t results. T h e d e s i r e to h a v e NAEP s e r ve a l a r g e r role in t h e f o r m a t i o n of e d u c a t i o n a l policies a n d t h e m e a s u r e m e n t of t h e ,impact of e d u c a t i o n a l p r a c t i c e s was p e r h a p s t he m a j o r r e a s o n w h y t he c o n t r a c t for NAEP was shifted to the E d u c a t i o n a l Testing Service (ETS) 1 in 1983. ETS's proposed p l a n (Messick, B e a t o n a n d Lord, 1983), e c h o i n g a n a s s e s s m e n t of NAEP c o n d u c t e d b y Wirtz a n d LaPointe (1982), called for an e x p a n d e d role for NAEP as a "National r epor t card". While promising to m a i n t a i n t he high call calibre of th e i t em d e v e l o p m e n t w o r k c o n d u c t e d by ECS, ETS p r o p o s e d o t h e r m o d i f i c a t i o n s t h a t it believed w o u l d e n h a n c e NAEP's role in t he e s t a b l i s h m e n t of e d u c a t i o n a l s t a n d a r d s a n d in policy d e v e l o p m e n t s . The p r o p o s e d c h a n g e s w e r e p r i m a r i l y t e c h n i c a l : a s h i f t f r o m age-level to grade-level testing, m o r e f r e q u e n t t e s t i n g in core s u b j e c t s , a r e d e s i g n of t e s t s t h a t wo u l d e n c o u r a g e the e x a m i n a t i o n of r e l a t i o n s h i p s a m o n g t a s k s a n d b e t w e e n p e r f o r m a n c e a n d o t h e r s t u d e n t c h a r a c t e r i s t i c s , a n d a move toward scale-score reporting rather than the item-level performance statistics provided in ECS reports. T hes e c h a n g e s were seen as a significant shift to war d s tying p e r f o r m a n c e to the f u nct i oni ng of the e d u c a t i o n a l system, 1 T h r o u g h o u t its existence, NAEP has b e e n carried o u t u n d e r a c o n t r a c t from t h e F e d e r a l g o v e r n m e n t w i t h a p r i v a t e o r g a n i z a t i o n r a t h e r t h a n bei ng c o n d u c t e d directly by a specific federal agency. ECS was t h e c o n t r a c t o r from 1969 t h r o u g h 1983 at w h i c h time ETS w on a c o m p e t i t i o n for a five-year c o n t r a c t to c o n t i n u e t he a s s e s s m e n t . T he federal g o v e r n m e n t is c u r r e n t l y developing th e criteria a n d pr opos a l r e q u e s t for the n e x t c o m p e t i t i o n for the NAEP c o n t r a c t .
84
L. Burstein
t h u s p o t e n t i a l l y m a k i n g a s s e s s m e n t r e s u l t s m o r e u s e f u l for e d u c a t i o n a l d ecis io n m a k e r s while at t he s a m e time placing the c o n s e q u e n c e s of their a c t i o n s m o r e c l e a r l y in t h e p u b l i c view. Moreover, ETS q u i c k l y moved t o war d active e n c o u r a g e m e n t of state-level use of NAEP p e r f o r m a n c e d a t a by establishing a st a t e a s s e s s m e n t u n i t to facilitate its work in this area. As th e p l a n s for i n d i c a t o r s of s t a t e - b y - s t a t e e d u c a t i o n a l p e r f o r m a n c e h a v e crystallized, d i s c u s s i o n s a b o u t the s o u r c e of t h e t est i ng d a t a for state c o m p a r i s o n s have increasingly n a r r o w e d toward various alternatives t h a t rely on NAEP in s o m e way. In the fall of 1985, S e c r e t a r y of E d u c a t i o n B e n n e t t called for a s u b s t a n t i a l i n c r e a s e in t he NAEP t est i ng p r o g r a m while at t he s a m e time e n c o u r a g i n g s t at es to participate m ore fully in NAEP. The Center for Statistics' p l a n s for the redesign of their E l e m e n t a r y a n d S e c o n d a r y data collection p r o g r a m (Center for Statistics, 1986a) i n c o r p o r a t e a revised NAEP in th e p r o p o s e d i n t e g r a t e d d a t a s y s t e m a n d a n t i c i p a t e t h a t t he new syst em "will be c a p a b l e of e x p a n d i n g to S t a t e - R e p r e s e n t a t i v e s a m p l e s " . T he p r o p o s a l s guiding the i m p l e m e n t a a t i o n of t he CCSSO c o m m i t m e n t to obtain c o m p a r a b l e s t a t e - b y - s t a t e i n f o r m a t i o n on s t u d e n t a c h i e v e m e n t (CCSSO, 1984; Seldon, 1986b) call for the s t a t e s to " a t t e m p t to d r a w on t he NAEP item pool as a m e a n s of c o n s t r u c t i n g m e a s u r e s of s t u d e n t achi evem ent " and for i n c r e a s i n g the States' role in NAEP in order to i n s u r e t h a t NAEP c o n t e n t m o r e c o m p r e h e n s i v e l y reflects s t a t e p e r s p e c t i v e s . I n d e p e n d e n t of CCSSO actions, ETS h a s a l r e a d y t a k e n s t e ps to e x p a n d t he link b e t w e e n NAEP and individual s t a t e s t h r o u g h a j o i n t t es t i ng project with t he S o u t h e r n Regional E d u c a t i o n B o a r d in s e ve r al S o u t h e r n s t a t e s (SREB, 1985) a n d t h r o u g h i n c r e a s i n g th e n u m b e r of s t a t e s carrying out c o n c u r r e n t a s s e s s m e n t s . T h e r e are p l a n s for possible r e s t r u c t u r i n g t h r o u g h a n e w competetive bid for f u n d i n g . T hi s m i g h t lead to f u r t h e r d e c i s i o n s a b o u t h o w actively f u t u r e NAEP c o n t r a c t o r s will be in s t a t e - l e v e l r e p o r t i n g of e d u c a t i o n a l achievement. 2 2 T h e r e is a c e r t a i n i r o n y to t he i n c r e a s i n g a t t e n t i o n given to state-level r e p o r t i n g in the evolving role of NAEP. Yet NAEP's own s u c c e s s early on at e s t a b l i s h i n g itself as a credible, e s s e n t i a l l y a political, y a r d s t i c k a n d its c o n t r i b u t i o n s to t he d e v e l o p m e n t of t he c a p a c i t y w i t hi n s t a t e s to c a r r y out their own a s s e s s e m e n t s m a k e it an obvious t arget of o p p o r t u n i t y duri ng this p e r i o d w i t h its m o r e p r o m i n e n t role for e d u c a t i o n a l a s s e s s m e n t in e d u c a t i o n a l policy.
Quality in U.S.A. 85
Th e i n c r e a s e d b u r d e n s h e a p e d u p o n NAEP to serve multiple political, social, a n d e d u c a t i o n a l p u r p o s e s u n d e r t he s a m e u n d e r s u p p o r t e d u m b r e l l a are th e s o u r c e of "the potential for c o n t i n u i n g conflicts" to w hi ch we alluded earlier on. It is possible to foresee c i r c u m s t a n c e s t h a t could easily erode its s t a n d i n g a s a valid b a r o m e t e r of n a t i o n a l e d u c a t i o n a l p e r f o r m a n c e . Hopefully, t h e v a r i o u s i n t e r e s t s involved in t he deci si ons a b o u t NAEP can arrive at a f o r m u l a t h a t will allow its us e in the s t a t e - b y - s t a t e reporting while p r e s e r v i n g its i n t e g r i t y as a n a t i o n w i d e i n d i c a t o r , T h e r e are c e r t a i n l y t e c h n i c a l a l t e r n a t i v e s for a c c o m p l i s h i n g t h e s e dual goals b u t w h e t h e r t hey are practically a n d politically feasible r e m a i n s to be seen.
Speculation a b o u t F u t u r e Developments
Elliott a n d Hall (1985), a d m i n i s t r a t o r s of the C e n t e r for Statistics in the Office of E d u c a t i o n a l R e s e a r c h a n d I m p r o v e m e n t , c h a l l e n g e d e d u c a t o r s to p a r t i c i p a t e in t he d e v e l o p m e n t of m e a s u r e s for a s s e s s i n g h o w well, or how poorly, th e r e c e n t wave of e d u c a t i o n a l reforms a n d s u p p o r t is working. To this en d Elliott a n d Hall pos e d a set of q u e s t i o n s t h e y believed s h o u l d be a d d r e s s e d a b o u t w h e t h e r t he m e a s u r e s u s e d as e d u c a t i o n i n d i c a t o r s at the state a n d local level are reasonable. These q u e s t i o n s dealt with the c o n t e n t validity of existing a c h i e v e m e n t indicators as m e a s u r e s of w h a t is t a u g h t or s h o u l d be tau ght ; t h e ~.trength of t he r e l a t i o n s h i p s b e t w e e n t he i ndi cat ors a n d s t u d e n t p e r f o r m a n c e ; the alterabilitv of the conditions reflected by the indicators; the u n d e r s t a n d i n g of indicators; the u s e f u l n e s s of c u r r e n t m e a n s of r e D o r t i n ~ for v a r i o u s s t a k e h o l d e r s ; t he ability of c u r r e n t i n d i c a t o r s to r ef lect v a r i a t i o n in o p p o r t u n i t i e s a n d p e r f o r m a n c e a m o n g i m p o r t a n t s u b g r o u p s ; the c o m p a r a b i l i t y of m e a s u r e s a c r o s s locales; and finally their s u s c e p t a b i l i t y to distortion. Also highlighted was the nagging c o n c e r n a b o u t w h e t h e r th e m e a d i a a n d public officials could be e d u c a t e d to u s e i ndi cat or d a t a appropriately. Elliott a n d Hall ha ve p r e s e n t e d a t h o u g h t f u l a g e n d a for t he n a t i o n a l effort to develop e d u c a t i o n a l quality indicators. It is also an e n c o u r a g i n g one coming from federal officials who are w e l l -connect ed to the i m p l e m e n t a t i o n of f ed er al policy in t hi s area. B u t h i s t o r y h a s n o t b e e n k i n d to wishful t hin k in g a b o u t the role t h a t evaluative information actually plays in the highly p o l i t i c i z e d a t m o s p h e r e c r e a t e d b y t h e c o m p e t i n g i n t e r e s t involved in
86
L. Burstein
e d u c a t i o n a l reforms. We can expect the debate a r o u n d the a p p r o p r i a t e n e s s of various m e a s u r e s of educational outcomes to continue. Moreover, there are still virtually no available national d a t a t h a t capture the proximal qualities of e d u c a t i o n a l c o n t e x t s , r e s o u r c e s , a n d p r o c e s e s (i.e. w h a t e d u c a t o r s are actually faced with a n d w h a t t h e y do on a daily basis and w h a t s t u d e n t s are actually learning) as they impact on teaching and learning in schools and on the organization and m a n a g e m e n t of local educational systems. Yet, without s u s t a i n e d efforts, not to m e n t i o n s u b s t a n t i a l l y i n c r e a s e d financing, to improve t h e q u a l i t y a n d p r o g r a m fidelity of m e a s u r e s of e d u c a t i o n a l o u t c o m e s a n d to develop better m e a n s to contextualize t h e m by improving the c h a r a c t e r i z a t i o n s of the s e t t i n g s from w h i c h t h e y are g a t h e r e d , the n a t i o n a l m o v e m e n t toward the development of e d u c a t i o n a l quality indicators will r e m a i n largely a goad r a t h e r t h a n a guide to s u c c e s s f u l e d u c a t i o n a l reform.
References Bennett, W.J, (1986) First lessons: A report on elementary_ e d u c a t i o n in America. Washington, D.C. : U.S. Department of Education. Burstein, L., Baker, E.L., Aschbacher, P., and Keesling, J.W. (1985) Using state test d a t a for national indicators of education quality: A feasibility s t u d y . Los Angeles: Center for the S t u d y of Evaluation, G r a d u a t e School of Education, UCLA. Clark D.L. a n d Astuto T.A. (1986) The significance a n d p e r m a n e n c e of c h a n g e s in federal e d u c a t i o n a l policy 1980-1988. Occasional Paper, Bloomington, Indiana: Policy Studies Center of the University Council for Educational Administration, Indiana University, J a n u a r y . California State D e p a r t m e n t of Education. schools, Sacramento, CA, 1986.
Performance report for California
Center For Statistics. Plan for the redesign of the e l e m e n t a r y a n d secondary d a t a collection program. Working Paper, W a s h i n g t o n , D.C. : Office of E d u c a t i o n a l Research a n d Improvement, U.S. D e p a r t m e n t of Education, March 27, 1986 (a). C e n t e r For Staistics. "Proposal for a U.S. c o n s o r t i u m on c r o s s - n a t i o n a l e d u c a t i o n a l studies", Draft, W a s h i n g t o n D.C. : Office of E d u c a t i o n a l R e s e a r c h a n d Imporvement, U.S. D e p a r t m e n t of E d u c a t i o n , April 9, 1986 (b). Committee On Coordinating E d u c a t i o n a l Information a n d Research CCSSO Center on A s s e s s m e n t and Evaluation. Draft Report, Washington, D.C., November 15, 1985.
Quality in U.S.A.
87
C o m b e r L.C. a n d Keeves J. P. (1973) Science e d u c a t i o n in n i n e t e e n countries. New York: J o h n Wiley and Sons. C o n g r e s s i o n a l B u d g e t Offfice T r e n d s in E d u c a t i o n a l Washington D.C. Congress of the United States, 1986.
Achievement,
Council Of Chief S t a t e School Officers. E d u c a t i o n a l a s s e s s m e n t a n d evaluation in the United States. Position Paper, Washington, D.C, 1984. E11iott E.J. and Hall R. (1985) Indicators of performance: Measuring the educators, E d u c a t i o n a l M e a s u r e m e n t : I s s u e s a n d Practice, 4 (2), pp 6-8. Fetler, M. (1986) Accountability in California public schools. E d u c a t i o n a l Evaluation and Policy Analysis, 8_ (1), 31-44. Florida D e p a r t m e n t of E d u c a t i o n . Florida's progress toward excellence in e d u c a t i o n in t h e public schools. A Report to the State Board of E d u c a t i o n , T a l l a h a s s e e , FL: A s s e s s m e n t , Testing, a n d E v a l u a t i o n Section, B u r e a u of Program Support Services, December 1985. Goertz M.E. (1986) State educational standards: A 50-state survey, Research Report, Princeton, NJ: Educational Testing Services. Hall G., Jaeger, R.M. K e a m e y C.P. a n d Wiley D.E. (1985) Alternatives for a national d a t a s y s t e m on elementarv a n d s e c o n d a r y education, A Report p r e p a r e d for the Office of E d u c a t i o n a l R e s e a r c h a n d Improvement, December 20. Harnischfeger A. a n d Wiley D.E. (1975) Achievement test score decline: Do we need to worry?., Chicago: ML-Group for Policy Studies in Education. Husten. T. (ed.) (1967) International s t u d y of achievement in m a t h e m a t i c s : A c o m p a r i s o n of twelve countries, Stockholm a n d New York: Almquist a n d Vgiksell a n d J o h n Wiley and Sons. J a e g e r , R.M. (1978) A b o u t e d u c a t i o n a l i n d i c a t o r s : S t a t i s t i c s on the conditions a n d t r e n d s in education. In L.S. S h u l m a n (ed.). Review of Research in Education, 6. PP. 276-315. Katzenmeyer C.G. The f e d e r a l role in e n c o u r a g i n g s t a t e - b y - s t a t e a c h i e v e m e n t comparisons. Paper presented at the A n n u a l Meeting of the National Council on M e a s u r e m e n t in Education, San Francisco, April 1986. Messick, S., Beaton, A. a n d Lord. F. (1983) N a t i o n a l A ~ s e s s m e n t of E d u c a t i o n a l Progress reconsidered: A new design for a new era, NAEP R e p o r t 83-1~ Princeton NJ: National A s s e s s m e n t of E d u c a t i o n a l Progress. National Association Of Secondary School Principals. E d u c a t i o n indicators: A national report card. Reston VA: NASSP, 1985.
88
L. Burstein
National C e n t e r For E d u c a t i o n Statistics. E d u c a t i o n i ndi cat ors conference on international education statistics: Summary_ of d i s c u s s i o n s , Chevy Chase, Maryland: U.S. D e p a r t m e n t of E ducat i on, April 1985. National I n s t i t u t e of E d u c a t i o n . Declining t est scores: A c o n f e r e n c e report, Washington, D.C. : National Institute of Education, 1976. N a t i o n a l S c i e n c e B o a r d C o m m i s s i o n O n P r e c o l l e g e E d u c a t i o n In M a t h e m a t i c s , S c i e n c e a n d Technlogy. E d u c a t i n g A m e r i c a n s for t h e 2 1 s t cen tur y. Washington D.C. : National Science Foundat i on, 1983. National S cien c e B o a r d S c i e n c e i n d i c a t o r s : T he 1985 report, W a s h i n g t o n , D.C. : National Science Founda t i on, 1986. Raizen, S.A. a n d J o n e s L.V. (1985) I n d i c a t o r s of Precolle~e e d u c a t i o n in science and m a t h e m a t i c s . Washington~ D.C. : National A c a d e m y Press. R a n d C o r p o r a t i o n . Monitoring n a t i o n a l p r o g r e s s in m a t h e m a t i c s , science, an d t e c h n o l o g y education. Proposal to the National Science Foundat i on, S a n t a Monica, CA, 1985. R o m b e r g T. a n d S m i t h M.S. (1985) m a t h e m a t i c s monitoring center. E d u c a t i o n a l Research.
T h e e s t a b l i s h m e n t of a s c h o o l Madison, WI: W i s c o n s i n C e n t e r for
S e l d e n , R.W. S o m e c l a s s i c a l m e a s u r e m e n t i s s u e s c o n f r o n t i n g t he d e v e l o p m e n t of s t a t e - b y - s t a t e a s s e s s m e n t of s t u d e n t a c h i e v e m e n t . P a p e r p r e s e n t e d at t h e A n n u a l Meeting of t h e N at i onal Counci l on M e a s u r e m e n t in E d u c a t i o n , S a n Francisco, April 1986 (a). Selden, R.W. White P a p e r : Strategies and i s s u e s in t he d e v e l o p m e n t of c o m p a r a b l e i ndi cat or s for m e a s u r i n g s t u d e n t a c h i e v e m e n t . Washington, D.C. : S ta t e E d u c a t i o n A s s e s s m e n t Center, Council of Chief State School Officers, April 30, 1986 (b). S o u t h e r n R eg i ona l E d u c a t i o n B o a r d . M e a s u r i n g s t u d e n t a c h i e v e m e n t : C o m p a r a b l e t es t r e s u l t s for participating s o u t h e r n states, the South, and the Nation, Atlanta: S o u t h e r n Regional E d u c a t i o n Board, 1985. The National C om m i s s i on On Excellence In E d u c a t i o n . A n a t i o n at risk: The imr)erative for educat i onal reform. Washington, D.C. : U.S. G o v e r n m e n t Printing Office, 1983. U.S.
D e p a r t m e n t of E d u c a t i o n . S t a t e e d u c a t i o n s t a t i s t i c s : State p e r f o r m a n c e s , r e s o u r c e i m p u t s , a n d p o p u l a t i o n c h a r a c t e r i s t i c s 1972 an d 1982., 1984.
Wainer, H., Holland P.W., Swinton, S. a n d Wang M.H. (1985) On st at e e d u c a t i o n statistics. J o u r n a l of E d u c a t i o n a l Statistics, 10 (4), 293-326. U.S. D e p a r t m e n t of E d u c a t i o n . W h a t works: R e s e a r c h a b o u t t e a c h i n g and learning. Washington D.C. Wirtz, W. a n d Lapointe, A (1982) M e a s u r i n g t he qual i t y of e d u c a t i o n : A r e p o r t on a s s e s s i n g e d u c a t i o n a l progress. Washington, D.C.: Wirtz and Lapointe.
Quality in U.S.A.
89
Wirtz, W, et. al (1972) On f u r t h e r e x a m i n a t i o n : R e p o r t of t h e Advisory_ p a n e l o n t he S c h o l a s t i c A pt i t ude T e s t score decline. New York: The College Board.
The A ut hor LEIGH BURSTEIN is a P r o f e s s o r in t h e Social R e s e a r c h M e t h o d o l o g y Division of th e G r a d u a t e School of E d u c a t i o n at t he University of California, Los Angles. His w o r k on q u a l i t y i n d i c a t o r s i n c l u d e s a s t u d y on u s i n g i n f o r m a t i o n f r o m s t a t e t e s t i n g p r o g r a m s to c r e a t e c o m p a r a b l e c r o s s - s t a t e q u a l i t y i n d i c a t o r s a n d he s e r v e s as a n a d v i s o r y p a n e l i s t for t h e Rand C o r p o r a t i o n feasibility s t u d y on a c o m p r e h e n s i v e i n d i c a t o r s y s t e m for m o n i t o r i n g m a t h e m a t i c s , science, a n d t e c h n o l o g y e d u c a t i o n in t he United States.