Evaluating assessment: A comment and a perspective

TECHNOLOGICAL

FORECASTING

Evaluating

AND SOCIAL CHANGE

Assessment:

14, 147- 152 (1979)

A Comment and a

Perspective BRUCE KOPPEL

ABSTRACT This paper takes issue with the Porter and Rossini proposal to apply the logic of experimental design to the evaluation of technology assessment. Experimental design is intraparadigmatic and thus not suited to clarifying alternative technology-assessment (TA) models. An alternative starting point for evaluating TA is suggested that rests on the intersection of two dimensions: the process of doing TA and the social accountability of TA. Three evaluation arenas are discussed: social learning, constructive social invention, and enhanced synaptic options.

Introduction In a recent paper, Porter and Rossini [12] apply the logic of experimental and quasi-experimental design to the evaluation of technology assessments and technology forecasts. As the number of technology assessments has increased, attention is rightly turning to the question of how to assess “assessment.” The approach suggested by Porter and Rossini is important because it utilizes a design framework well-discussed (if not widely applied) in the evaluation research literature and proposes that it is epistemologitally appropriate for both TA and technology forecasting (TF). The present article focuses on the viability of their effort in relation to technology assessment. It is argued that there are several serious fallacies in the Porter-Rossini proposal. An alternative starting point, not as well operationalized, but arguably more appropriate for thinking about assessing assessment, is outlined. Technology Assessment and Experimental Design: The Fallacies of Compatability Experimental and quasi-experimental design [31 are approaches to the following problem in causal inference: How can one be “certain” that plausible alternative hypotheses to that being proposed do not account for observed data relationships? Inadequate attention to this problem is equivalent to underestimating potential distortion influences on internal and external validity.’ Porter and Rossini’s advocacy of the logic of experimental design to determine the validity and utility of TA concentrates on possible distortions to internal validity. This is problematic for two broad reasons, each of which can be considered as fallacies in their proposal.

BRUCE KOPPEL is a Research Associate with the East-West Center Resource Systems Institute in Honolulu, Hawaii. He is director of the Institute’s Technology Assessing Project and in that capacity is spending one year in the Philippines at the Southeast Asian Regional Center for Graduate Study and Research in Agriculture (SEARCA) helping in the development of a technology-assessment program for agriculture. I Internal validity refers to the appropriateness of an hypothesis for a particular observed data set. External validity refers to the generalizabihty of that same hypothesis to data sets other than that used to establish internal validity. @ Elsevier North Holland,

Inc., 1979

0040-1625/79/06014706$02.25

148

BRUCE KOPPEL

1. For TA, external validity is the major issue, i.e., generalizability to situations and parameters not entirely anticipated. 2. Experimental design is intraparadigmatic. It cannot screen alternative models, but rather can only provide a basis for excluding alternative hypotheses within a model. This means that even the proposed focus on internal validity cannot be realized unless a high degree of consensus on the maintained hypotheses of all TAs can be justifiably assumed. To make that assumption at this time would be empirically unjustifiable [I, IO]. Porter and Rossini say as much when they conclude that “One benefit of a coordinated effort to evaluate TA would be a better sense of the objectives of TA and its performance in meeting those objectives. ” Their approach would avoid, however, the major issue requiring evaluation: the viability of alternative TA models, and 12~~the “truth” of particular TAs. The second fallacy is the most critical. Experimental design seeks to make inference more feasible through elaborate attention to empirical comparison. By what methodologies is the method of discriminating comparison realized? The answer is the structuring of multiple observations. However, the evaluation issue in TA is not a method or methodological problem; it is very definitely an epistemological issue, the issue of plural epistemologies. Data correspondence per se is not particularly critical for this question [8, 9, 141. The issue is not reducible to the spacing, timing, or style of observation, but rather to more basic assumptions about what can or ought to be observed. Economists have called this the problem of “identification” [6, 71. In Basmann’s words

PI: the testing of theoretical premises about. [a] parameter is logically prior to its estimation. This is particularly so in case theoretical restrictions, supposedly containing true parameters, have been imposed on their statistical estimates and this imposition is claimed to be effective in enhancing efficiency of estimation.

What types of data will be appropriate is a derivative of the structural relationships that constitute a model. This is the epistemological side of the problem. Determining the “truth” of a given set of structural relationships is not possible through inference from the empirical relationships mandated by the structural relationships. That is one of the evaluative binds faced by TA. A second evaluative bind is that determining the validity of some empirical relationship in the future is not possible by direct extrapolation from observed relationships; it is necessary to specify also that there are no changes in structural relationships. If there is a structural change between the observation period and the prediction period, the empirical relationships can be obtained only from the new structure. However, to do that requires knowledge of the old structural parameters and a priori knowledge about the change. Knowledge of empirical coefficients will not be sufficient because of the high autonomy of structural relationships and the relatively low autonomy of empirical parameters. This means that while most empirical relationships will be affected by single structural changes, the reverse will not be the case. Econometricians have approached their model evaluation problem principally through more complete specification of structural relationships with careful attention to explicit statements about errors. The plausibility of assumed randomness of errors or particular patterns of intercorrelation are major enabling steps in econometric model testing. Technology assessment cannot exclude itself from any of the difficulties being dis-

EVALUATING

ASSESSMENT

149

cussed. The uncomfortable probability is that it has a more, rather than less, difficult time. The inabilities of TA to identify broad invariances in the operations of systems reflects how little we generally know about quite fundamental issues. However, TA has the additional problem of explicitly recognizing that it is not a purely positive science endeavor-it is also very much a normative science enterprise. Technology assessment cannot stop at asking about errors in systems, a positive science inquiry. It must also ask about errors of systems. The latter question is not subject to resolution simply by more or better data; the issue is neither purely positivistic nor is the content of falsifiability well understood. The value questions in TA put it beyond a framework that would rank TAs on the basis of methodological performance as if there were a single epistemological umbrella and no alternative models. Even if TA were a purely positivistic exercise, the application of an intraparadigmatic evaluative framework would still be questionable. Why? Both the lack of consensus about and knowledge of the operation of systems-a necessary (though, as suggested earlier, not a sufficient) condition for the application of experimental design for evaluative purposes. The evaluator would confront alternative models, each characterized by a set of maintained hypotheses, not all of which would be known. Under these conditions, attributions of relative internal validities will be at best partial judgments. What is a more accurate description of the likely outcome is that however dazzling the methodologies, the judgments will reflect a social fact-the self-view of the TA profession. This, of course, is the most serious point about the enlarging TA evaluation discussion. The discussion itself is a positive indicator of “professionalization,” but because of the unique aspirations of TA, the discussion also signifies that TA is at a crossroads. The TA movement can define professionalism narrowly, settle further into methodological involution, and court esoteric judgments and a narrowing constituency. The movement can also seek more innovative and, in the sense of what TA is about, more compatible views of professional accountability. The challenge inherent in the latter course is that evaluative criteria will not be precise and will not necessarily move toward convergence. At best, the criteria will be agreed-on arenas for debate, but the critical difference is that they will be divergent rather than convergent arenas [4]. Experimental design rests on the adequacy of the analogy as an epistemological tool. That is the logic of comparison and derived generalizability. Technology assessment requires a different perspective to support its advancementa perspective in which TA stands both within and outside itself. In the next section, one approach to making that statement operational is proposed. DIVERGENT

EVALUATION:

ARENAS

OF STRATEGIC

CONCERN

Directions for evaluating TA should be derived from a broader view of what technology assessing is-not a methodological exercise but an epistemological endeavor; the social organization of very complex knowledge and, paradoxically, very complex ignorance. Moreover, directions for evaluating TA should proceed from an awareness that cross-culturally little is known about the content of technology assessment, so inevitable thrusts toward specific universal evaluation indicators should be considered very cautiously [5, lo]. What, then, is a good starting point. Emphasis might best be placed on generic objectives to which diverse and unanticipated TAs might reasonably be expected to aspire and to those themes that might constitute the social accountability of TA (see Fig. 1). In other words, the objectives that can serve as discriminators do not have to be common attributes of all TAs, but only expectations and aspirations applicable to TA itself. An

150

BRUCE KOPPEL

Fig. 1. Toward a Me&Framework

for TA evaluation.

additional criterion needs to be added to preclude self-validation and rigidity; the objectives need to be strategic concerns for society. The strategic concern dimension is directed at the interface between the processes of doing TA and the social accountability of TA. Three objectives-cum-arenas of strategic concern are proposed: (1) social learning, (2) constructive social invention, and (3) enhancement of synaptic options. SOCIAL LEARNING

Social learning can vary from more intense and wide-scoped discussion of an existing or recognized problem to perception of new problem areas. This is the broadest expectation that can be assigned to technology assessing. The actual scope and coherence of recognition may be quite limited, reflecting, among other factors, constraints derived from the particular operating institutional context. Recognition, for example, of complex horizontal problems will be very difficult for societies highly organized around vertical institutions [ 131. A number of rural development strategies, especially those described as have been operationalized in ways that mirror the functional division of “integrated,” labor among agencies. There are visions of more comprehensive approaches to rural development, but the visions are incongruent with most existing institutional arrangements. To the degree that TA strives to enhance the feasibility of more effective social control over the choice and application of technologies, then even just debate involving alternative perspectives on already recognized, vaguely perceived, and currently unacknowledged problem and opportunity areas are components of social learning. CONSTRUCTIVE

SOCIAL INVENTION

A more significant form of social learning is a reorganization in some manner of the social fabric within which learning proceeds. Social invention can vary from projects to ideologies; the acid test is the emergence of something that was not there with that precise structure and function before. If the social invention contributes to more effective social control of technology, ranging from improved abilities to anticipate to enlarged capacities to reconsider, then it can be considered constructive. The question then becomes: To what extent does TA stimulate and is itself stimulated by and characterized by constructive social invention? ENHANCEMENT

OF SYNAPTIC

OnIONS

According to Webster’s Third New International Dictionary, a synapse is “the locus at which the nervous impulse passes from the axon of one neuron to the dendrites of

EVALUATING

151

ASSESSMENT

between the another. ” A synaptic option is an option that facilitates communication problem-perception processes in adjacent domains. It is an option that contributes to the synergistic creation of a new vision at the boundaries of older visions. For example, energy has for a long time been viewed as a technological and scientific problem. In some quarters it has been viewed as a measure of economic development. A synaptic option currently being discussed involves viewing energy as a quality of life problem, not simply with technological and economic components, but also with other components including values, ideologies, international relations, and so on. Enhanced positioning of synaptic options in problem definition and opportunity recognition is the most difficult, but in many senses, the most crucial step in a society’s learning process-the process of relearning. Synaptic options are perspectives tbat represent an awareness of lapses in problem construction. Such options are not the incremental fine-tuning of existing choices; they are the signposts of an alternative vision. Sustaining synaptic options will yield altered extrusions of a society’s image of itself. Technology assessment can contribute to these ends not only through the content of specific outputs, but through its processes as well. The point is illustrated in Fig. 2, using the recombinant-DNA debate as a simple example. It is not unimportant to consider individual TAs. However, when viewing TA as a broader process, as a type of social invention that operates on a temporal knowledge frontier as well as between contemporary value enclaves, it is more important to view individual TAs in light of the larger TA process. In other words, TA evaluation requires answering two questions:

Social learning

Constructive social invention

Enhancement of synaptic options

Recombinant DNA exposes new possibilities and risks in biological research and c; research utilization

I, I I I How can research be best I I directed and how can misapplications I 8 The , 9 be avoided and risks minimized? 1 , outputs I , II of ’ I I TA II 1 Does the role of biological I I I and basic research in general I I I require redefinition in light III of new recognition of social I ’ w goals, values, and risks? : : II I i i ; ] 1 Arenas for , expression, i 1 I discussion I t ; Arenas for 1 I decision I making 1 i Arenas for “new” arenas

Fig. 2. A content-process

Content directed ----- Process directed

TA evaluation

framework.

152

BRUCE KOPPEL

1. What is the normative appropriateness of the TA process? 2. What is the positive adequacy of the specific TA’s within the process? Porter and Rossini share better suited to question instead on the evaluation issue of whether TAs are of whether TA represents

a similar view of TA as process, but their evaluative scheme is 2 only. They do not consider evaluation of process, but focus of individual components within process. For TA, the positive accurate is subordinate to and dependent on the normative issue an acceptable vision [ 111.

Conclusion Between and within the predominantly vertical institutions of societies, TA will function in contexts that vary widely (and without any necessary monotonic relationships to GNP) on existing capacities for social learning, capabilities for constructive social invention, and proficiency at sustaining consideration of synaptic choices. Evaluation of TA between varied social contexts (both within and between nations) will, therefore, require standardization of yardsticks to the social contexts in which TA functions and by which it is shaped. It follows that evaluation is, in a very fundamental sense, a dialectical endeavor that considers simultaneously the capability of societies to assure high self-determination through deliberate experimentation and the role of TA in stretching and reshaping that capability. The TA community would do well to emulate this more fundamental relationship, to apply it to the TA community itself. It should seek excellence in terms of the more significant attributes and goals of TA and avoid confusing refinement of means for clarification of ends. References 1. Arnstein, Sherry R. and Christakis, Alexander N., Eds., Perspectives on Technology Assessment, Crofton Publishing Co., Newton, Mass., 1975. 2. Basmann, R., On the Applicability of the Identificabihty Test Statistic in Predictive Testing of Explanatory Economic Models, Ind. Econ. J. 13, 387-388 (1965-1966). 3. Campbell, Donald and Stanley, Julian, Experimental and Quasi-Experimental Designs forResearch, Rand McNally, Chicago, 1966. in Perspective, in Amstein and 4. Berg, Mark R., Chen, Kan, and Zissis, George J., Methodologies Christakis [I]. 5. Chen, Kan and Zacher, Lech, Toward Effective International Technology Assessments, paper presented at Conference on Technology Assessing: The Quest for Coherence, East-West Center, Honolulu, Hawaii, 30 May-10 June 1977. 6. Christ, C., Econometric Models and Methods, Wiley, New York, 1966. 7. Fisher, F., The Identification Problem in Econometrics, McGraw-Hill, New York, 1966. 8. Haavelmo, T., The Probability Approach in Econometrics, Econometrica 12, supplement (1944). 9. Koopmans, T., Rubin, H., and Leipnik, R., Measuring the Equation Systems of Dynamic Economics, in T. Koopmans, Ed., Statistical Inference in Dynamic Economic Models, Wiley, New York, 1950. JO. Koppel, B., Technology Assessing: The Quest for Coherence, East-West Center, Honolulu, Hawaii, 1977. Il. Kuhn, T., The Structure of Scienrifc Revolutions, University of Chicago Press, 1962. 12. Porter, A. and Rossini, F., Evaluation Designs for Technology Assessments and Forecasts, Technol. Forecast. Sot. Change 10, 369-380 (1977). 13. Rose, D., New Laboratories for Old, Daedalus 103, 143-156 (1974). 14. Theil, H., Specification Errors and the Estimation of Economic Relationships, Revue de l’lnstitut De Sratistique 25, 149-155 (1957). Received 12 September

1977