Indicators of education systems James N. Johnstone

people working in different situations are natural. It is not possible for the reviewer to agree more with the author in his conclusion that real improvement would emanate from the development and implementation of a good curriculum. But he would not like to undermine the role of evaluation as an effective means of educational transformation.

location. A few short-answer questions aim to explore values and attitudes, but the "expected answers' are too closed and condensed for a guess to be hazarded as to whether they are likely to succeed in this purpose. There is a suggestion of lack of discrimination in matching the most appropriate forms of question to particular objectives, one of the high-level skills in this area.

H. S. Srivastava

Sample Unit Tests in Geography, H. S. Srivastava and K. Seshan, (National Council of Educational Research and Training, New Delhi, 1980), 172pp.

Sample Unit Tests in Geography consists of a set of twenty tests designed for formative evaluation purposes, assessing the curriculum of India's Classes IX and X. The tests were devised by the Examination Reform Unit of the National Council of Educational Research and Training at New Delhi. The tests reflect an attenuated Bloomian scheme of objectives, ranging ,from 'knowledge' to ~application', each major category broken down into sub-categories. These sub-categories will cause some confusion to those brought up in the Bloom idiom. This is not to suggest it should be slavishly followed. But it is necessary to adopt a logical subsumptive process. This does not seem present, for example, in an 'application' category, in which the sub-category descriptors include 'selects relevant facts', 'analyses', 'predicts', 'judges adequacy', 'proposes a plan', 'infers', 'establishes relationship', and so on. In addition to the cognitive objectives, the tests aim to develop geographical skills and assist in the development of 'positive attitudes towards peoples and environments'. Each of the twenty sections contains a blueprint or specification, in the form of a matrix, in which the proportion of marks to be allotted to the subtopics in the content, to the different abilities to be tested, and the assessment instrument to be used (here essays (very few), short answer questions, very short answer questions, and objective questions), is pre-specified. Most of the objective items are of the multiple-choice simple completion type. Each test includes a scoring key for the multiple choice items and a marking scheme for the rest, together with expected outline answers (usually very bare), estimated times needed for answers, and expected difficulty levels. No evidence is provided, however, of statistical analysis, to elicit the difficulty and discrimination levels of items. Questions of different types, while separately clustered, are not explicitly signalled, going against normal British practice, certainly in objective tests. Unfortunately, too many of the multiple choice items lend credence to what seems to me a generally misguided view that such items best test low level abilities and traditional content. The actual questions are varied in quality. A number of the multiple-choice items are relatively sophisticated, genuinely assessing higher level abilities including, for example, simulations of industrial 202

A major problem is the nature of the content which is being tested, reflecting largely though not wholly a pretty traditional view of geography. The main content headings do not promise a lively approach: 'temperature'; 'wind systems'; 'rainfall': 'climatic and natural regions'; 'soils'; 'mineral wealth': 'fuels'; and 'international trade', to take just a selection. A range of maps, graphs and sketch diagrams, as well as verbal material, provides the basis of the questions, but there are no photographs. A positive feature of exercises of this type is that the planning is not only necessarily careful, but also is made explicit. The skills and the values embodied in the production, as well as its weaknesses, are exposed to the light of day, and laid open to evaluation. The Examination Reform Unit's work is clearly worthy of further development which, to bear full fruit, should be paralleled and linked with a Curriculum Reform Unit. A 'new curriculum' is mentioned in the Preface, but there appears little evidence of it here. Certainly the present unit's work seems in advance of the curriculum it is testing. W. E. Marsden Indicators of Education Systems James N. Johnstone. (1981) International Institute for Educational Planning. London: Kogan Page; Paris: UNESCO In this long (279 pages plus appendices) and comprehensive book, James Johnstone offers what must now rank as a definitive work for those who believe that 'proper planning and researching of education systems must be based on sound quantiative methods' (p. 279). The book offers, in the author's words, 'an outline of some of the main concepts for defining, developing and using indicators of education systems . . . which politicians, planners and researchers might consider using in their examination of education systems'. In an initial chapter on 'The Concept of an.Indicator of an Education System'. Johnstone suggests that indicators have four principal uses: the statement and development of policies in a more definite and coherent way; the monitoring of change in systems; representing education system characteristics in research studies; and the facilitation of the formation of a valid and reliable classification of education systems. Subsequent chapters deal at a more or less sophisticated statistical level with the nature of education systems data, the theoretical and empirical identification of indicators for national and regional education systems, and how to present and use indicators in educational planning. research and classification.

The exposition is clear, thorough and sensitive to the m a n y problems of collecting and using data of this sort. Those who will criticise this book will be, I suspect, those, like me, who cite its sins of omission rather than commission, i.e. those who have grave doubts about any attempt to assess 'school quality' or 'system effectiveness' in purely quantitative terms. Phrases such as 'indicators measuring the ways education systems operate - the ways they process inputs to provide outputs' (p. 149) are characteristic of the book, as is the use of multivariate statistical analysis. Both these features are clear testimony to a particular perspective. It is not surprising then that Johnstone is m u c h in sympathy with the I E A studies which 'proved it is possible to measure national achievement levels'. The l E A studies have been extensively criticised, because, despite their statistical sophistication, they offer almost no interpretive insights as to the sources of the variation they identify. This is equally the shortcoming of the 'indicator' approach. A p a r t from some obvious banalities such as the assertion 'that levels of economic development and the extensiveness of educational provision in a country are intimately and positively inter-related', (p. 256) there are a n u m b e r of more pleasing but equally dubious illustrations of the use of indicators, such as the representation of empirically defined education system characteristics by different facial features such as smile or size of ears. Thus in the representation in diagram 5-7 'openness of eye' reflects a high proportion of second level students in the general education stream. Provided the reader is sufficiently familiar with the key, such an approach at least has the benefit of novelty. The more fundamental question is whether those planners and administrators whose job it is to make decisions about educational resources are likely to be encouraged by a book like this into an increasingly corporate management approach. Johnstone specifically denies that indicators can have any role to play in passing judgement on the degree to which a particular goal is being achieved, their use in an evaluation such as degree of decentralization being confined to the location of a particular system on a centralization/decentralization scale. This seems to me to be an untenable distinction, in that any way of categorizing an education system is itself a selection and a value-judgement. In this case the selection is that of confining the analysis to quantifi:~ble variables, the value-judgement being that cducafional outcomes may be adequately represented in numerical terms. Although Johnstone is at pains to distinguish between l~sychological and educational assessments and the measurement of education systems (p. 50), he is nevertheless vulnerable to the same criticism that can be levelled at attempts to quantify numerically pupils' learning. A relatively trivial decision to allocate numerical quantities to the :~ssessment of students' written performance at an elite university in the nineteenth century has had enormous significance for the ways in which we

think about educational practice and pupils' learning ever since. It is to be hoped that educational planners will not fall victim to a similar seduction in which the convenient shorthand which numbers provide is taken to be a true measure of educational quality. For this latter we must look to the unscientific, often ungeneralizable, detailed, interpretive studies which offer the only genuine content validity in any evaluation of the workings of an education system. It is perhaps unfair to criticize Johnstone's book for not doing what it did not set out to do, but in that it reveals no awareness of the short-comings of the perspective offered, the criticism is justified. Patricia Broadfoot Universal Primary Education in Nigeria: A Study of Kano State, M a r k Bray, 1981, 1SBN No. 0 71~)9 0933X pp 167 & appxs., etc. £ 5 . 5 0 Paperback, London, Routledge and Kegan Paul The foreword to this book makes reference to ' . fact collecting on such an extensive scale' and in many ways this is just what Bray has done. 'Facts' are even harder to establish in Nigeria than in the West because of the unreliability of government statistical data. However, given this limitation, Bray has done a n admirable job in gathering together a great deal of information on the attempt to implement U P E in K a n o State. In so doing he gives a thorough account of the history of U P E schemes in Nigeria in general and in Kano State in particular. The federal government has pumped a considerable a m o u n t of money into education in n o r t h e r n Nigeria in an attempt to reduce employment disparities between the north and the south as rapidly as possible. Yet, like other states in the north, Kano has faced a n u m b e r of major problems in implementing U P E and the book considers each in turn. One problem has been resistance f r o m those sections of the Hausa Muslim population that perceive Western education as a threat to traditional morals and standards. Particularly interesting in this regard is the account of Islamiyya schools which offer one way forward by attempting to merge aspects of Koranic education with the main features of Western schooling. A second set of problems concern the costs and logistics of implementation. The rate of expansion has meant that the money has been spread very thinly and that there is a resulting shortage of all teaching resources from school buildings to textbooks. .Most serious of all, however, is the shortage of qualified primary school teachers as only a small minority in Kano State are properly trained and certificated. On top of this, staff turnover is very high and motivation and morale are low. These sorts of problems raise the issue of the quality of education being provided by the U P E scheme. The message from this book is that auality has suffered at the hands of quantity. Quality is 203