- Email: [email protected]
Assessing a problem-oriented social technology
486
Assessing a Problem-oriented
Social Technology
ASSESSING
A PROBLEM-
ORIENTED
SOCIAL
TECHNOLOGY A general conduct Selwyn Enzer
In a commercial environment, TA is frequently seen as an extension of technological forecasting with the purpose to provide arguments for company policy. Following the contribution illustrating this approach, which was published in the last issue,l Futures is now introducing another concept of TA as a policyoriented activity in the area of “social technology”. This article discusses methodological lessons learnt from an attempt to assess the impacts of the introduction of no-fault automobile insurance on a nationwide basis. It is concerned with the development of systematic procedures for analysing two types of future conditions: (1) the unintended social impacts likely to result from a given innovation in a relatively stable external environment; and (2) the changes in intended impacts and additional unintended impacts likely to result from coupling the given innovation with other prospective changes in the external world. assessment is a term applied to a class of activities designed to identify and evaluate the full range of social impacts likely to result from the introduction of innovation in societal processes. These innovations may result from new technologies (social or physical), changing the nature or use of existing technologies, or from responding to current problems or opportunities. The central feature of technology assessment which distinguishes it from systems analysis is the emphasis it places on the social impacts likely to TECHNOLOGY
Mr Selwyn Enzer is with the Institute for the Future, Menlo Park, California 94025, USA. His article is based on a study “Some Impacts of No-Fault Automobile Insurance-A Technology Assessment” in two volumes (R-30, March 1974; R-31, May 1974) available from the Institute for the Future. The activity described in the study and this article was performed under a grant from the National Science Foundation partly to develop methods to improve the nascent technology assessment capabilities and partly in response to a real social issue in the USA.
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
487
result from the new environment, with explicit attention to the indirect and higher order consequences of change that are likely to be precipitated by the basic innovation. In other words, technology assessments generally assume that the basic purposes which motivated the change are (or will be) realised, and proceeds from that assumption to explore what else may happen and how this new situation may affect particular classes and groups, and society as a whole. Technology assessment is a policy-oriented activity in that it is concerned with society’s management of technology and not the development or elaboration of technological options. If it is to be an effective tool, its relationship with other decision-making functions must be understood. Technology assessment can be viewed as the middle of a three-link chain: 1. Selection, schedule, and timing of technologies for assessment. 2. Technology assessment. 3. Utilisation of the results in a decision-making context. After deciding which technologies should be assessed and when they should be assessed, or reassessed if appropriate, the technology assessment itself is concerned with producing the information necessary to promote further debate and to enable the decision makers to act with a broader and more far-sighted perspective. The third link in the chain is concerned with the use of the results of the assessments in the decision-making process. If the results of a technology assessment do not enter into or affect any decisions, the assessment need not have been made in the first place. To some extent, our ability to perform either of the first two steps affects the criteria imposed on the other. For example, if we can confidently identify a minimal set of sources of change which are important to assess, we will be able to devote a proportionately larger effort to each assessment than we could if the list of topics were great. Clearly, then, the smaller list would permit more stringent criteria for technology assessment. Conversely, if we had methods which could rapidly and accurately assess the potential impacts of many types of change, we would be able to conduct more assessments and would then be able to include what may appear to be marginally important issues. To date, the problem of how to conduct technological assessments has received the greatest attention and is indeed the focus of this article. However, the problem of deciding which technology to assess may prove to be the more important one since, without performing that task efficiently, the technology assessment function may either require more human resources than can be devoted to it or may be forced to omit areas of change that may contain truly important impacts which could have been appropriately modified if properly assessed. The third step in the chain has received very little attention; the coupling between the analytic results and decision making is a problem which has always presented great difficulty to the planning community. Contact with decision makers is a key element in assuring that the results of the assessment are at least considered by them. Most of the criticism of technology assessments has centred on our inability to do them effectively. The big problems here are the open-ended nature of the problem and our inability to anticipate future developments with a reasonable
FUTURES
December 1974
488
Assessing a Problem-oriented
Social Technology
degree of accuracy. However, looking ahead, examining our choices, and selecting from among them is a basic societal characteristic which is considered good by definition ; and the fact that we may lack certain skills which can increase our long-range vision is insufficient justification for not pursuing technology assessment. Some criticism has also been directed not so much at the assessments themselves as at the value of the contribution they can make in the absence of an effective set of social indicators with which to evaluate the impacts identified in the assessments. However, society has been evaluating alternatives and making decisions without such indicators throughout history, and it would be oversimplified to argue that society should not attempt to direct innovative processes and use techniques until their impacts have been effectively evaluated. Types, methods Technology l
l
and timing of assessments
assessment faces two distinct methodological
which technologies should be assessed, when they reassessed, and the future period to be considered; development of effective techniques.
problems : should
be assessed or
The research described here was concerned primarily with the second problem. When researchers develop a tool in an activity which is focused on a specific subject and at the same time attempt to make it applicable to other subjects, they typically try to select the most general approach at the outset; but they frequently find later that, in order to produce meaningful results for the specific case study, they must adopt a more specialised approach. This study followed that course. Indeed, it may have been more pronounced in the present effort because the purpose which the tool was to serve-technology assessment -was not, and is still not, singular. The 1969 National Academy of Engineering study, Technology: Processes of Assessment and Choice,2 pointed out that technology assessments which are initiated by problems differ in structure from those initiated by new technologies. This difference alone obviates the generality of any one assessment, since only with problem-oriented assessments is it possible to identify, and hence evaluate, intended effects. Furthermore, the nature of the source of change can also affect the design of the assessment procedures. In this regard, social changes generally introduce a greater range of options than is usually possible with technical changes. two Using these distinctions, four types of assessments can be identified: concerned with physical and two with social technologies. These can be illustrated with the following examples. A problem-oriented, physical technology assessment could be one concerned with the societal impacts resulting from the expanded use of nuclear fuel to meet the increasing demand for electricity. An innovation-oriented, physical technology assessment might focus on the societal impacts resulting from the development and use of liquidThe first case is concerned with the nitrogen temperature superconductors. application of an existing technology to specific problems, whereas the second is concerned with the what, where, and how of the use of an as yet non-existent
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
489
capability which may evolve from basic scientific research. The important methodological distinction here is primarily the existence of a specific problem, not the fact that the technology being assessed may already exist, because in the course of widespread application of any “existing” technology, developments and improvements are likely to occur which may have the effect of converting it into a virtually “new” technology. When faced with a specific problem, the assessor must be concerned with alternative methods of solving that particular problem and changes in the problem itself, in addition to impacts which may result. Without the specific problem, alternatives and the whole concept of intended impacts are non-existent. Assessments encounter different variants when the sources of change are of a social nature. Consider, for example, the case study used as the vehicle for performing our research, no-fault automobile insurance. This is a problemoriented assessment of a technology which, because it is man-made, can take on a tremendous variety of forms. Unlike the example of a physical technology whose variants are limited by scientific and technological feasibility, the variations of no-fault insurance plans are limited only by our imagination and our own socio-legal system. As a result, the number of alternatives that must be included in social-technology assessments is apt to be much larger than those encountered in physical-technology assessments. Similar distinctions can be made for innovation-oriented social technologies. This type of change is more difficult to identify, but one might regard the potential introduction of instant referenda into the mechanism of democratic government as one future prospect. The ranges and types of alternatives to be considered for these assessments make it reasonable to conclude that four different methodological approaches may be required for the menu of possible technology assessments: one for the problem-oriented, social-technology assessment; another for the problemoriented, physical-technology assessment; and another pair for the innovationoriented assessments. Timing is an important consideration in any technology assessment: when it should be made and the future period that should be considered. These should be determined so that the assessments are made sufficiently early to allow for action or reaction to any of the potential future impacts. In addition, the assessment should be conducted before public debate has completely polarised those affecting and affected by the technology. In our case study the latter consideration was violated. As a result the experts’ judgments, essential to certain aspects of the assessment, often appeared to be emotionally rather than rationally determined. This situation precluded the use of such judgmental techniques as Delphi and cross-impact analyses from being used in the case study. These techniques, which are appropriate for many of the key problems of technology assessment, depend upon logically thoughtout judgments. A technology assessment must recognise the potential existence of diverse values and goals even among experts and must therefore be prepared to cope with such sources of bias in judgmental inputs. However, if the assessment is conducted at a time when the public debate is near to or just past its peak, emotion often replaces logic in the judgmental processes, and adjustments must be made in the methods of assessment to accommodate this condition. Hence
FUTURES
December 1974
490
Assessing a Problem-oriented
Social Technology
the technique developed in conducting our study reflects an analytic approach that is intended to be responsive to what might be called “hotly-debated”, problem-oriented social technologies. Focusing the assessment The methodology developed and tested in the course of this research came about as a result of a series of trials, errors, and modifications. If the study were to be repeated, the logic used would be that presented graphically in Figure 1, and Iterate .r
_c
innovations
__________
-_-------_-
____
1
and
alternatives
Identification
Figure 1. Automobile
accident
losses and possible
of
means of control
descriptively in Table 1. Central to the procedure is the development and use of a quantitative model. Although many of the data used in the present study’s model were subjectively derived (a situation which will probably apply in most social-technology assessments), the model is considered a key element because it enables the assessment team to screen a large number of alternatives rapidly and systematically in order to identify a small set of potentially important combinations. In any problem-oriented assessment, the first step is defining the problem fully. This definition must be stated in broad terms and must include both the elements of the problem and possible means of controlling it. This step should produce : l l
a listing and description of the issues which the problem area encompasses; identification of the specific means by which the various innovations respond
FUTURES
December 1974
Assessing a Problem-oriented Social Technology
l l
491
to the problem elements (these are the descriptors that can be used to distinguish one innovation from another); the arguments which support or oppose these innovations;3 and a listing of all other identifiable means by which the elements of the problem can be affected or controlled.
This first step identifies which elements and alternatives should be looked at in detail; the second and third define specific changes and options which should be considered. TABLE
1. A GENERAL METHODOLOGY FOR ASSESSING ORIENTED SOCIAL TECHNOLOGY
A PROBLEM-
1. Define the problem in terms of causes and indicators, including both pro and con arguments. 2. Identify and describe the innovation (including alternatives) to be assessed in terms of its impact on the problem indicators. 3. Identify and describe possible external future changes (ie, other than the above class of innovations) which are related to the problem and may affect either the causes or indicators of the problem. These relations are key inputs to the analysis which follows. Review historical experience with similar situations for guidelines and potentially obscure impacts. Develop as specific a model of the problem as is possible, incorporating the causes, linkages, and indicators described in steps 2 and 3 above. Develop a series of baseline futures using the model(s) with and without the innovation(s) being assessed, assuming a virtually constant external environment. Conduct a series of sensitivity analyses with the model(s) by evaluating changes in outcomes produced by individually perturbing the variables most susceptible to change in the future. 8. Identify and describe possible future changes that may alter the basic structure of the model. 9. Evaluate the changes that may result in the baseline futures if the structure of the model were to be changed. 10. Use the results of steps 6, 7 and 9 above to identify possible important combinations of innovative alternatives, external changes, and variations in the structure of the model; evaluate these cases separately. 11. Describe the outcomes (intended and unintended) in terms of key variations in the innovative alternatives and key external changes; indicate areas of greatest possible influence over these outcomes.
Steps 2 and 3 can proceed simultaneously. In the second, a set of the major variations of the social innovation under assessment and key alternative means for responding to the problem elements are identified and described. In the no-fault insurance case study, this meant describing the various no-fault statutes and other proposed automotive reform schemes in such terms as who requires insurance, what type is required, who is eligible for benefits, what benefits are received, what exemptions are made, and what means of recovery are available. This aspect of the assessment proved to be higly useful in distinguishing subtle yet important differences among apparently similar options. The third step involves identifying and describing possible future changes, external to the proposed social innovations, which could affect the elements of the problem if they occurred. The no-fault insurance case used a relevance analysis to identify areas that should be explored for such changes. Figure 3 shows that losses from automobile accidents can be reduced in three ways. It should be noted that the impacts identified in the relevance analysis can work two ways, ie, the introduction of no-fault insurance may affect the item in the relevance analysis and, conversely, a change in an item may affect the import-
FUTURES
December
1911
Assessing a Problem-oriented
492
Social
R.?duction
Technology
of auto accident
losses
I
l Improve
highway
l improve
design
vehicle
occupant 0 Improve
vehicle
l Improve
drivers safer
traffic
limprove
laws
l Enforce
laws
strictly
l Restrict
auto
use
aImprove
0 Improve svstems 0 Reduce
demand
design
for
damage vehicle design safety
highway
(barriers,
transport traffic
vehicle
property
pedestrian
0 Alternative
for
design l Improve
0 Adopt
design
safety
design
travel
l Property damage l Medical expenses
obstacles)
.
Provide services
timely
emergency
.
Improve
.
Promote greater safety devices
control
for
for
medical
services use of
lLOSS
of
l Pain and suffering
.
. Death,
l Police
dismemberment and disfigurement
income
Loss of productivity
q Medical . Court
resources resources resources
. LOSS of future earnings . Loss of services
lFuneral
expenses
Figure
2.
Automobile
accident
losses and possible
means of control
ante of no-fault insurance. Representation of the relevance analysis makes it easier to identify impact areas; eg, it becomes apparent that external developments in health insurance and income maintenance should be considered in the assessment since they affect the economic losses which result from accidents. Together, these first three steps provide the focus for the assessment. If the listings are too detailed or include too many marginally relevant topics, the subsequent assessment can easily become unmanageable. Conversely, if the listings are too abridged, important areas of impact may be omitted entirely. Utilisation of the problem elements and causal linkages as the means of providing focus for the assessment is open to the criticism that it may be overly constraining, by possibly precluding the identification of impacts in remote social areas. However, the ideal approach involving the systematic consideration of every societal area is obviously unmanageable. In our case study, an initial effort was made to use the entire area of the automobile and society as the focus for a relevance analysis which started with the societal goals of mobility, safety, and equity in the event of a mishap, with minimal adverse impacts on the environment. When the analysis became extremely large and unmanageable it was concluded that it would probably be more productive to use the problem elements and causal linkages in the highly systematic phase of the assessment and use a less structured approach, such as brainstorming, to explore more remote impact possibilities. The fourth step, the historical study, may or may not be relevant in any given assessment. In the no-fault insurance case study, it was possible to do it,
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
493
but it proved to be of little value. In those cases where historical data would be of such relevance as to be significantly pertinent, a true technology assessment would probably not be necessary.
Investigationoffactors The fifth step is to develop a model (or set of models) which includes the problem elements, the suspected impact areas, and the factors which affect either of these. The objective behind developing a model of the areas being assessed is to permit rapid and accurate evaluation of many alternatives. These models can take many forms and may be qualitative or quantitative.4 The factors included in analytic models are: (1) key output variables whose trends are to be forecast (indicators of the more direct areas of impact) ; (2) the relevant factors which can be (or may be) changed and which are suspected to have an impact on the output variables; and (3) the more readily forecastable quantities for which statistical data either exist or can be collected and which appear to be logically related to the key output variables and the relevant factors. These are then incorporated in a logic diagram which appears to describe the causal relations faithfully. With this structure an attempt is made to derive a set of mathematical equations which compute each successive item from the former. Where the correlation is poor, the factors may have to be changed or the structure revised. This trial-and-error process is continued until the model correlates well with the statistics on which it is based. In a regression model, the quality of the correlation is judged exclusively by how accurately the model can be used to retrodict the historical data.5 If the analysis is successful, a model which is statistically verifiable is obtained. The model may then be used to forecast the nominal future course of the output variables by inputting forecasts of the basic factors (GNP in the case of the no-fault insurance model). This forecast reflects the impacts that could be expected, assuming that all the forces at work in the past continue to operate in the same manner in the future. In the no-fault insurance assessment, this forecast was used as a baseline against which alternatives were computed. Tying the model to historical data generally limits the range over which the impact of external changes can be accurately computed. However, the prime purpose of this model was to assess impacts likely to result from changing the forces that were at work in the past. In this regard, it is questionable whether a model which exhibits good statistical correlation but is of limited accuracy in evaluating extensive future changes is most appropriate for this purpose.” It may be more desirable in some assessments to build models whose conceptual logic is more satisfying but whose results may be less valid in a statistical sense. This compromise may provide much more information on obscure impacts of individual changes. In problem-oriented assessments many commonalities are expected to exist between the model containing the technology being assessed and the model which does not include this innovation. This suggests that modularising the
FUTURES
December 1974
494
Assessing a Problem-oriented
Social Technology
analysis may prove to be highly efficient in pursuing problem-oriented assessments. In the no-fault insurance case, the entire loss-forecasting portion of the model was common to all futures; the expected losses were considered independent of the reparation system in force. The sixth step consists of using the model to compute a set of so-called baseline futures. Actually, these can be baseline pasts as well as futures. In other words, they can be used to recompute what the past might have been had the innovations been in effect, as well as what the futures are likely to be, given the introduction of each alternative innovation. In the no-fault insurance case study, this was really a crude assessment of the direct impacts likely to result from the introduction of various no-fault insurance schemes. In this particular example (and as might be expected in similar assessments of this type), the “desirable direction” is not always obvious. 7 This is one of the reasons why no attempt was made to use the elements of the model as indicators of desirability or importance. This aspect of the assessment produces a display of what might be expected with introduction of a specific innovation, all other external factors being equal, and provides the essential reference with which the model can be used to explore areas of more obscure impacts which may result from a combination of occurrences. The important change variables are identified in the seventh step, which consists of a series of sensitivity analyses. These show the impact on the output elements of the model when one variable is perturbed. Information of this type for different variables allows a rapid determination of when external developments are likely to have reinforcing or counteracting impacts. These results, together with the specific forecast effects identified in step three, are then used to identify important combinations of external developments, important not in the sense that they are more likely to occur than the others, but that if they occur they are likely to have a much greater impact on the problem than are others. The results produced from models (which are invariably static) are not final, but merely a description of trends that could be expected if the interactions remain the same as those described in this model. The subsequent steps address the ways in which the model itself can change. However, these aspects are much less systematic or structured than those used in developing the model. Evaluation
of changes
In steps eight and nine, changes which are external to the model and which might affect its basic structure are evaluated. In the no-fault insurance case study, a series of brainstorming sessions was conducted to identify the most important possible changes whose occurrence would affect the basic model structure and alternative means for obtaining some insight into these changes. These sessions focused on six issues whose effects would be primarily a function of behavioural changes among the various people involved in the process of resolving automotive accident disputes; it was felt that the only source of information about these changes was the judgment of people highly familiar with the functioning of the present reparation system. It was decided to use a questionnaire to obtain these judgments.
FUTURES
December 1874
Assessing a Problem-oriented
Social Technology
495
In this questionnaire, these sources of change were evaluated to determine the general degree and direction of the impacts which would be realised if the basic pattern on which the model is based were to change. The no-fault insurance questionnaire was concerned with such factors as the attitudes of juries towards the adversaries in a court suit with and without the presence of no-fault insurance, and the propensity to sue, or to settle out-of-court, of both plaintiffs and insurers in those cases in which the so-called loss thresholds were exceeded, and the like. Because of the concern over the polarisation of viewpoints with regard to no-fault insurance the responses were collated according to the respondents’ background and affiliation. Hence it is possible to review the ways in which the various categories of respondents view the anticipated changes. Possibly important combinations of external changes are identified for analysis in step ten. These are evaluated in two ways for mutually enhancing or inhibitIf the change listed betow were to occur.
estricted
or prohibited
chicle travel
.
major
Restricted
or prohibited cities
er availability
of these changes subsequently occurring would.
.
numbe
cities
into major
transport
the probability
motor
in a significant
mf areas within
. . . then
entry
of im
between
I
mm Figure 3. Cross-impact
coupling
among external
J
events
ing effects. First, the external changes are evaluated to determine whether or not the occurrence of one development (which affects the model) is likely to enhance or inhibit the likelihood of occurrence of another (which also affects the model) (F’g 1 ure 3). The matrix aids in identifying strongly reinforcing developments. The second evaluation (Figure 4) is concerned with the direction and magnitude of the direct impacts likely to be incurred in the explicit elements of the model. Items noted with a “C” in this figure may have very strong impacts if they are accompanied by other developments. Both these evaluations use matrix formats and initially address the impacts between pairs; ie, one external event on another, or one external event on one element in the model. After considering the impacts between pairs, higher-order combinations are generally more easily identified. This information makes it easier to identify combinations of changes where
FUTURES
December
1974
496
Assessing a Problem-oriented
If the change listed below were to occur. . .
...
Social Technolog
then these elements in the model would be directly affected.
Total auto 1 Numberof expenditures vehicles
Miles travelled
.
Number of
Restricted or prohibited motor vehicle travel in a significant number of areas within major cities
I
Restricted or prohibited entry into major cities
Greater availability of improved mass transport between major cities c
+
Greater availability of improved mass transport within major cities
Figure 4. Direct effect of external
events on the model
mutual reinforcement among the causal linkages is strong. These situations may promote important or even potentially “explosive” situations. Each of these situations is evaluated separately and potential controls or modifications that can be used to capitalise on or ameliorate any of the situations are identified. Identifying
the outcomes
The above results are the basis for the eleventh step, consisting of describing outcomes which are likely to be of greatest concern and which should be considered most carefully in directing the technology in question. Any general recommendations and guidelines which the research has produced should be offered here. Identifying the most important possible outcomes poses a difficult methodological problem. A likely outcome is not necessarily an important one, although an outcome which almost surely will not occur is definitely not important. As a result, the steps described here tend to stress highly sensitive areas and combinations of developments which reinforce rather than offset each other. Since one of the primary considerations in a technology assessment is the identification and elaboration of possibly important but obscure and higher-order impacts, methodology must explicitly include low-probability developments. Thus, in identifying technological impacts, many of the assessors will frequently be called upon to discriminate important outcomes having virtually no chance of occurrence from those having a slight yet sufficient chance to be worthy of further deliberation. It is quite possible that the utility of these highly unlikely outcomes will depend
FUTURES
December 1974
Assessing a Problem-oriented Social Technology
497
upon the nature of the actions or guidelines that can be suggested to capitalise on the possible consequences. If the actions are relatively easy to implement and not disruptive with respect to the accommodation of other options, they are likely to be enacted. On the other hand, even when such actions are not implemented they should be used to establish monitoring procedures which, as a minimum, assure an early-warning posture. Summary The core of the methodological problem involved in technology assessment is the development of systematic procedures for the identification of important higher-order, indirect effects that may result from the introduction or redirection of a societal innovation. The approach developed in this study employs a combination of both highly organised and open-ended explorations and, generally speaking, follows what might be called good analytic practice, or the basic steps in the scientific method-not something new, but rather something old which has been modified to direct the attention of the research to more obscure factors. The approach begins with a detailed definition and understanding of the problem area to identify the scope and range of factors that should be included in the assessment. This open-ended phase of the assessment includes a review of the literature, consultation with experts, and many brainstorming sessions. Because the output of this activity bounds the scope of the assessment, it is important that it be not so narrow as to eliminate important, but non-obvious areas of investigation. On the other hand, without any delimitation, the scope of any issue worthy of an assessment always approaches the infinite. One of the most serious criticisms of this approach is that because it establishes the focus of the so-called manageable set of factors to be investigated from the problem area itself, it can easily overlook important impacts that may occur in more remote issues. The manageable set is then structured as an explicit model which can assess various combinations of eventualities and alternative actions rapidly and accurately. However, because quantitative models are invariably limited by our understanding of complex situations, available data, and computational capabilities, they tend to be static copies of dynamic situations. As a result, it is important to reopen the investigation periodically to the open-ended format to inquire whether the situations identified within the model may produce changes which upset the structure embodied in the model. The procedures developed in this study appear to be of assistance in the identification of important developments of this type, but do not contain any systematic procedure for explicitly identifying these among the infinite number of developments which may take place in the future. It is quite possible that a truly systematic approach to this aspect of technology assessment may never be fully realised. The study did develop and demonstrate an approach to assessing a certain class of issues in a way that encourages the search for areas which are suspected to contain higher-order impacts and to include these explicitly in the analytic investigation, but even this approach is only effective for second- or perhaps third-order impacts.
FUTURES
December 1974
498
Assessing a Problem-oriented
Social Technology
Notes and references 1. Richard C. Davis, “Applying Technology Assessment in the Multi-Product Company”, Futures, Vol. 6, No. 5, October 1974, pages 413-419 2. National Academy of Sciences, Technology: Processes of Assessment and Choice, Report to US Congress, House Committee on Science and Astronautics (July 1969) 3. In the case study, the results of this portion of the first step were presented in the report, The Automobile ~ns~~a~~eSystem : Current Statw and Some Proposed ~e~~s~orls, by Selwyn Enzer, Gordon J. Slotsky, Dennis L. Little, James E. Doggart, and David A. Long, Working Paper WP-18, Institute for the Future (February 1972) 4. In the automobile insurance case study, a considerable (yet invariably insufficient) amount of statistical data was available concerning the system that had been in existence in the USA: ie, the tort (or fault) system. Hence it was possible to develop a mathematical model of this system which was statistically relatable and verifiable over a mea~ngful historical period. It was also possible to use this model to describe the change that might result if the innovation proposed (or its alternatives) were introduced in the future. The no-fault insurance model was based on regression equations, ie, it contained quantitative relationships which were statistically verifiable, to the extent that statistics were available. Furthermore, the degree to which the model replicated the historical statistics was the only criterion for determining the quality of the modeifs performance. 5. In the no-fault insurance model, it was found that most of the essential variables could be accurately computed from Gross National Product (GNP). In other words, by starting with statistical data on GNP, the model was used to compute such items as expenditures for personal transport, number of accidents, and bodily injuries. Furthermore, when starting with the known values of GNP for previous years these computations produced results which correlate very closely with the actual statistics for these “dependent” factors. Several factors were included in the model for which there were few or no statistical data. In such cases, either assumptions based on expert judgment are included or the variable is evaluated parametrically, or both. Parametrically evaluating a variable consists of using the model to assess the impacts on the output variables for a wide range of change in the parameter being evaluated. Using a model in this manner is called a sens~t~~~t~ u~ulys~sand is discussed in the text. 6. As can be seen in Volume I of our study, the no-fault insurance model’s exclusive use of this criterion severely limited the range of change variables that could be perturbed. However, this approach generally results in higher-confidence forecasts in those areas where the forces at work are not considered likely to undergo drastic changes, which appears to be the case in that portion of the no-fault insurance model concerned with forecasting losses. Considerations in Making Technology Assessments 7. Selwyn Enzer, “Practical Useful”, an address delivered at a joint symposium of the Institute of Management Sciences, the World Future Society, and the George Washington University Program of Policy Studies in Science and Technology, “Technology Assessment: Washington, DC, The George Washington A Future-Oriented Policy Tool”, University (5 June 1972)
FUTURES
December 1974
Assessing a Problem-oriented
Social Technology
ASSESSING
A PROBLEM-
ORIENTED
SOCIAL
TECHNOLOGY A general conduct Selwyn Enzer
In a commercial environment, TA is frequently seen as an extension of technological forecasting with the purpose to provide arguments for company policy. Following the contribution illustrating this approach, which was published in the last issue,l Futures is now introducing another concept of TA as a policyoriented activity in the area of “social technology”. This article discusses methodological lessons learnt from an attempt to assess the impacts of the introduction of no-fault automobile insurance on a nationwide basis. It is concerned with the development of systematic procedures for analysing two types of future conditions: (1) the unintended social impacts likely to result from a given innovation in a relatively stable external environment; and (2) the changes in intended impacts and additional unintended impacts likely to result from coupling the given innovation with other prospective changes in the external world. assessment is a term applied to a class of activities designed to identify and evaluate the full range of social impacts likely to result from the introduction of innovation in societal processes. These innovations may result from new technologies (social or physical), changing the nature or use of existing technologies, or from responding to current problems or opportunities. The central feature of technology assessment which distinguishes it from systems analysis is the emphasis it places on the social impacts likely to TECHNOLOGY
Mr Selwyn Enzer is with the Institute for the Future, Menlo Park, California 94025, USA. His article is based on a study “Some Impacts of No-Fault Automobile Insurance-A Technology Assessment” in two volumes (R-30, March 1974; R-31, May 1974) available from the Institute for the Future. The activity described in the study and this article was performed under a grant from the National Science Foundation partly to develop methods to improve the nascent technology assessment capabilities and partly in response to a real social issue in the USA.
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
487
result from the new environment, with explicit attention to the indirect and higher order consequences of change that are likely to be precipitated by the basic innovation. In other words, technology assessments generally assume that the basic purposes which motivated the change are (or will be) realised, and proceeds from that assumption to explore what else may happen and how this new situation may affect particular classes and groups, and society as a whole. Technology assessment is a policy-oriented activity in that it is concerned with society’s management of technology and not the development or elaboration of technological options. If it is to be an effective tool, its relationship with other decision-making functions must be understood. Technology assessment can be viewed as the middle of a three-link chain: 1. Selection, schedule, and timing of technologies for assessment. 2. Technology assessment. 3. Utilisation of the results in a decision-making context. After deciding which technologies should be assessed and when they should be assessed, or reassessed if appropriate, the technology assessment itself is concerned with producing the information necessary to promote further debate and to enable the decision makers to act with a broader and more far-sighted perspective. The third link in the chain is concerned with the use of the results of the assessments in the decision-making process. If the results of a technology assessment do not enter into or affect any decisions, the assessment need not have been made in the first place. To some extent, our ability to perform either of the first two steps affects the criteria imposed on the other. For example, if we can confidently identify a minimal set of sources of change which are important to assess, we will be able to devote a proportionately larger effort to each assessment than we could if the list of topics were great. Clearly, then, the smaller list would permit more stringent criteria for technology assessment. Conversely, if we had methods which could rapidly and accurately assess the potential impacts of many types of change, we would be able to conduct more assessments and would then be able to include what may appear to be marginally important issues. To date, the problem of how to conduct technological assessments has received the greatest attention and is indeed the focus of this article. However, the problem of deciding which technology to assess may prove to be the more important one since, without performing that task efficiently, the technology assessment function may either require more human resources than can be devoted to it or may be forced to omit areas of change that may contain truly important impacts which could have been appropriately modified if properly assessed. The third step in the chain has received very little attention; the coupling between the analytic results and decision making is a problem which has always presented great difficulty to the planning community. Contact with decision makers is a key element in assuring that the results of the assessment are at least considered by them. Most of the criticism of technology assessments has centred on our inability to do them effectively. The big problems here are the open-ended nature of the problem and our inability to anticipate future developments with a reasonable
FUTURES
December 1974
488
Assessing a Problem-oriented
Social Technology
degree of accuracy. However, looking ahead, examining our choices, and selecting from among them is a basic societal characteristic which is considered good by definition ; and the fact that we may lack certain skills which can increase our long-range vision is insufficient justification for not pursuing technology assessment. Some criticism has also been directed not so much at the assessments themselves as at the value of the contribution they can make in the absence of an effective set of social indicators with which to evaluate the impacts identified in the assessments. However, society has been evaluating alternatives and making decisions without such indicators throughout history, and it would be oversimplified to argue that society should not attempt to direct innovative processes and use techniques until their impacts have been effectively evaluated. Types, methods Technology l
l
and timing of assessments
assessment faces two distinct methodological
which technologies should be assessed, when they reassessed, and the future period to be considered; development of effective techniques.
problems : should
be assessed or
The research described here was concerned primarily with the second problem. When researchers develop a tool in an activity which is focused on a specific subject and at the same time attempt to make it applicable to other subjects, they typically try to select the most general approach at the outset; but they frequently find later that, in order to produce meaningful results for the specific case study, they must adopt a more specialised approach. This study followed that course. Indeed, it may have been more pronounced in the present effort because the purpose which the tool was to serve-technology assessment -was not, and is still not, singular. The 1969 National Academy of Engineering study, Technology: Processes of Assessment and Choice,2 pointed out that technology assessments which are initiated by problems differ in structure from those initiated by new technologies. This difference alone obviates the generality of any one assessment, since only with problem-oriented assessments is it possible to identify, and hence evaluate, intended effects. Furthermore, the nature of the source of change can also affect the design of the assessment procedures. In this regard, social changes generally introduce a greater range of options than is usually possible with technical changes. two Using these distinctions, four types of assessments can be identified: concerned with physical and two with social technologies. These can be illustrated with the following examples. A problem-oriented, physical technology assessment could be one concerned with the societal impacts resulting from the expanded use of nuclear fuel to meet the increasing demand for electricity. An innovation-oriented, physical technology assessment might focus on the societal impacts resulting from the development and use of liquidThe first case is concerned with the nitrogen temperature superconductors. application of an existing technology to specific problems, whereas the second is concerned with the what, where, and how of the use of an as yet non-existent
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
489
capability which may evolve from basic scientific research. The important methodological distinction here is primarily the existence of a specific problem, not the fact that the technology being assessed may already exist, because in the course of widespread application of any “existing” technology, developments and improvements are likely to occur which may have the effect of converting it into a virtually “new” technology. When faced with a specific problem, the assessor must be concerned with alternative methods of solving that particular problem and changes in the problem itself, in addition to impacts which may result. Without the specific problem, alternatives and the whole concept of intended impacts are non-existent. Assessments encounter different variants when the sources of change are of a social nature. Consider, for example, the case study used as the vehicle for performing our research, no-fault automobile insurance. This is a problemoriented assessment of a technology which, because it is man-made, can take on a tremendous variety of forms. Unlike the example of a physical technology whose variants are limited by scientific and technological feasibility, the variations of no-fault insurance plans are limited only by our imagination and our own socio-legal system. As a result, the number of alternatives that must be included in social-technology assessments is apt to be much larger than those encountered in physical-technology assessments. Similar distinctions can be made for innovation-oriented social technologies. This type of change is more difficult to identify, but one might regard the potential introduction of instant referenda into the mechanism of democratic government as one future prospect. The ranges and types of alternatives to be considered for these assessments make it reasonable to conclude that four different methodological approaches may be required for the menu of possible technology assessments: one for the problem-oriented, social-technology assessment; another for the problemoriented, physical-technology assessment; and another pair for the innovationoriented assessments. Timing is an important consideration in any technology assessment: when it should be made and the future period that should be considered. These should be determined so that the assessments are made sufficiently early to allow for action or reaction to any of the potential future impacts. In addition, the assessment should be conducted before public debate has completely polarised those affecting and affected by the technology. In our case study the latter consideration was violated. As a result the experts’ judgments, essential to certain aspects of the assessment, often appeared to be emotionally rather than rationally determined. This situation precluded the use of such judgmental techniques as Delphi and cross-impact analyses from being used in the case study. These techniques, which are appropriate for many of the key problems of technology assessment, depend upon logically thoughtout judgments. A technology assessment must recognise the potential existence of diverse values and goals even among experts and must therefore be prepared to cope with such sources of bias in judgmental inputs. However, if the assessment is conducted at a time when the public debate is near to or just past its peak, emotion often replaces logic in the judgmental processes, and adjustments must be made in the methods of assessment to accommodate this condition. Hence
FUTURES
December 1974
490
Assessing a Problem-oriented
Social Technology
the technique developed in conducting our study reflects an analytic approach that is intended to be responsive to what might be called “hotly-debated”, problem-oriented social technologies. Focusing the assessment The methodology developed and tested in the course of this research came about as a result of a series of trials, errors, and modifications. If the study were to be repeated, the logic used would be that presented graphically in Figure 1, and Iterate .r
_c
innovations
__________
-_-------_-
____
1
and
alternatives
Identification
Figure 1. Automobile
accident
losses and possible
of
means of control
descriptively in Table 1. Central to the procedure is the development and use of a quantitative model. Although many of the data used in the present study’s model were subjectively derived (a situation which will probably apply in most social-technology assessments), the model is considered a key element because it enables the assessment team to screen a large number of alternatives rapidly and systematically in order to identify a small set of potentially important combinations. In any problem-oriented assessment, the first step is defining the problem fully. This definition must be stated in broad terms and must include both the elements of the problem and possible means of controlling it. This step should produce : l l
a listing and description of the issues which the problem area encompasses; identification of the specific means by which the various innovations respond
FUTURES
December 1974
Assessing a Problem-oriented Social Technology
l l
491
to the problem elements (these are the descriptors that can be used to distinguish one innovation from another); the arguments which support or oppose these innovations;3 and a listing of all other identifiable means by which the elements of the problem can be affected or controlled.
This first step identifies which elements and alternatives should be looked at in detail; the second and third define specific changes and options which should be considered. TABLE
1. A GENERAL METHODOLOGY FOR ASSESSING ORIENTED SOCIAL TECHNOLOGY
A PROBLEM-
1. Define the problem in terms of causes and indicators, including both pro and con arguments. 2. Identify and describe the innovation (including alternatives) to be assessed in terms of its impact on the problem indicators. 3. Identify and describe possible external future changes (ie, other than the above class of innovations) which are related to the problem and may affect either the causes or indicators of the problem. These relations are key inputs to the analysis which follows. Review historical experience with similar situations for guidelines and potentially obscure impacts. Develop as specific a model of the problem as is possible, incorporating the causes, linkages, and indicators described in steps 2 and 3 above. Develop a series of baseline futures using the model(s) with and without the innovation(s) being assessed, assuming a virtually constant external environment. Conduct a series of sensitivity analyses with the model(s) by evaluating changes in outcomes produced by individually perturbing the variables most susceptible to change in the future. 8. Identify and describe possible future changes that may alter the basic structure of the model. 9. Evaluate the changes that may result in the baseline futures if the structure of the model were to be changed. 10. Use the results of steps 6, 7 and 9 above to identify possible important combinations of innovative alternatives, external changes, and variations in the structure of the model; evaluate these cases separately. 11. Describe the outcomes (intended and unintended) in terms of key variations in the innovative alternatives and key external changes; indicate areas of greatest possible influence over these outcomes.
Steps 2 and 3 can proceed simultaneously. In the second, a set of the major variations of the social innovation under assessment and key alternative means for responding to the problem elements are identified and described. In the no-fault insurance case study, this meant describing the various no-fault statutes and other proposed automotive reform schemes in such terms as who requires insurance, what type is required, who is eligible for benefits, what benefits are received, what exemptions are made, and what means of recovery are available. This aspect of the assessment proved to be higly useful in distinguishing subtle yet important differences among apparently similar options. The third step involves identifying and describing possible future changes, external to the proposed social innovations, which could affect the elements of the problem if they occurred. The no-fault insurance case used a relevance analysis to identify areas that should be explored for such changes. Figure 3 shows that losses from automobile accidents can be reduced in three ways. It should be noted that the impacts identified in the relevance analysis can work two ways, ie, the introduction of no-fault insurance may affect the item in the relevance analysis and, conversely, a change in an item may affect the import-
FUTURES
December
1911
Assessing a Problem-oriented
492
Social
R.?duction
Technology
of auto accident
losses
I
l Improve
highway
l improve
design
vehicle
occupant 0 Improve
vehicle
l Improve
drivers safer
traffic
limprove
laws
l Enforce
laws
strictly
l Restrict
auto
use
aImprove
0 Improve svstems 0 Reduce
demand
design
for
damage vehicle design safety
highway
(barriers,
transport traffic
vehicle
property
pedestrian
0 Alternative
for
design l Improve
0 Adopt
design
safety
design
travel
l Property damage l Medical expenses
obstacles)
.
Provide services
timely
emergency
.
Improve
.
Promote greater safety devices
control
for
for
medical
services use of
lLOSS
of
l Pain and suffering
.
. Death,
l Police
dismemberment and disfigurement
income
Loss of productivity
q Medical . Court
resources resources resources
. LOSS of future earnings . Loss of services
lFuneral
expenses
Figure
2.
Automobile
accident
losses and possible
means of control
ante of no-fault insurance. Representation of the relevance analysis makes it easier to identify impact areas; eg, it becomes apparent that external developments in health insurance and income maintenance should be considered in the assessment since they affect the economic losses which result from accidents. Together, these first three steps provide the focus for the assessment. If the listings are too detailed or include too many marginally relevant topics, the subsequent assessment can easily become unmanageable. Conversely, if the listings are too abridged, important areas of impact may be omitted entirely. Utilisation of the problem elements and causal linkages as the means of providing focus for the assessment is open to the criticism that it may be overly constraining, by possibly precluding the identification of impacts in remote social areas. However, the ideal approach involving the systematic consideration of every societal area is obviously unmanageable. In our case study, an initial effort was made to use the entire area of the automobile and society as the focus for a relevance analysis which started with the societal goals of mobility, safety, and equity in the event of a mishap, with minimal adverse impacts on the environment. When the analysis became extremely large and unmanageable it was concluded that it would probably be more productive to use the problem elements and causal linkages in the highly systematic phase of the assessment and use a less structured approach, such as brainstorming, to explore more remote impact possibilities. The fourth step, the historical study, may or may not be relevant in any given assessment. In the no-fault insurance case study, it was possible to do it,
FUTURES
December
1974
Assessing a Problem-oriented
Social Technology
493
but it proved to be of little value. In those cases where historical data would be of such relevance as to be significantly pertinent, a true technology assessment would probably not be necessary.
Investigationoffactors The fifth step is to develop a model (or set of models) which includes the problem elements, the suspected impact areas, and the factors which affect either of these. The objective behind developing a model of the areas being assessed is to permit rapid and accurate evaluation of many alternatives. These models can take many forms and may be qualitative or quantitative.4 The factors included in analytic models are: (1) key output variables whose trends are to be forecast (indicators of the more direct areas of impact) ; (2) the relevant factors which can be (or may be) changed and which are suspected to have an impact on the output variables; and (3) the more readily forecastable quantities for which statistical data either exist or can be collected and which appear to be logically related to the key output variables and the relevant factors. These are then incorporated in a logic diagram which appears to describe the causal relations faithfully. With this structure an attempt is made to derive a set of mathematical equations which compute each successive item from the former. Where the correlation is poor, the factors may have to be changed or the structure revised. This trial-and-error process is continued until the model correlates well with the statistics on which it is based. In a regression model, the quality of the correlation is judged exclusively by how accurately the model can be used to retrodict the historical data.5 If the analysis is successful, a model which is statistically verifiable is obtained. The model may then be used to forecast the nominal future course of the output variables by inputting forecasts of the basic factors (GNP in the case of the no-fault insurance model). This forecast reflects the impacts that could be expected, assuming that all the forces at work in the past continue to operate in the same manner in the future. In the no-fault insurance assessment, this forecast was used as a baseline against which alternatives were computed. Tying the model to historical data generally limits the range over which the impact of external changes can be accurately computed. However, the prime purpose of this model was to assess impacts likely to result from changing the forces that were at work in the past. In this regard, it is questionable whether a model which exhibits good statistical correlation but is of limited accuracy in evaluating extensive future changes is most appropriate for this purpose.” It may be more desirable in some assessments to build models whose conceptual logic is more satisfying but whose results may be less valid in a statistical sense. This compromise may provide much more information on obscure impacts of individual changes. In problem-oriented assessments many commonalities are expected to exist between the model containing the technology being assessed and the model which does not include this innovation. This suggests that modularising the
FUTURES
December 1974
494
Assessing a Problem-oriented
Social Technology
analysis may prove to be highly efficient in pursuing problem-oriented assessments. In the no-fault insurance case, the entire loss-forecasting portion of the model was common to all futures; the expected losses were considered independent of the reparation system in force. The sixth step consists of using the model to compute a set of so-called baseline futures. Actually, these can be baseline pasts as well as futures. In other words, they can be used to recompute what the past might have been had the innovations been in effect, as well as what the futures are likely to be, given the introduction of each alternative innovation. In the no-fault insurance case study, this was really a crude assessment of the direct impacts likely to result from the introduction of various no-fault insurance schemes. In this particular example (and as might be expected in similar assessments of this type), the “desirable direction” is not always obvious. 7 This is one of the reasons why no attempt was made to use the elements of the model as indicators of desirability or importance. This aspect of the assessment produces a display of what might be expected with introduction of a specific innovation, all other external factors being equal, and provides the essential reference with which the model can be used to explore areas of more obscure impacts which may result from a combination of occurrences. The important change variables are identified in the seventh step, which consists of a series of sensitivity analyses. These show the impact on the output elements of the model when one variable is perturbed. Information of this type for different variables allows a rapid determination of when external developments are likely to have reinforcing or counteracting impacts. These results, together with the specific forecast effects identified in step three, are then used to identify important combinations of external developments, important not in the sense that they are more likely to occur than the others, but that if they occur they are likely to have a much greater impact on the problem than are others. The results produced from models (which are invariably static) are not final, but merely a description of trends that could be expected if the interactions remain the same as those described in this model. The subsequent steps address the ways in which the model itself can change. However, these aspects are much less systematic or structured than those used in developing the model. Evaluation
of changes
In steps eight and nine, changes which are external to the model and which might affect its basic structure are evaluated. In the no-fault insurance case study, a series of brainstorming sessions was conducted to identify the most important possible changes whose occurrence would affect the basic model structure and alternative means for obtaining some insight into these changes. These sessions focused on six issues whose effects would be primarily a function of behavioural changes among the various people involved in the process of resolving automotive accident disputes; it was felt that the only source of information about these changes was the judgment of people highly familiar with the functioning of the present reparation system. It was decided to use a questionnaire to obtain these judgments.
FUTURES
December 1874
Assessing a Problem-oriented
Social Technology
495
In this questionnaire, these sources of change were evaluated to determine the general degree and direction of the impacts which would be realised if the basic pattern on which the model is based were to change. The no-fault insurance questionnaire was concerned with such factors as the attitudes of juries towards the adversaries in a court suit with and without the presence of no-fault insurance, and the propensity to sue, or to settle out-of-court, of both plaintiffs and insurers in those cases in which the so-called loss thresholds were exceeded, and the like. Because of the concern over the polarisation of viewpoints with regard to no-fault insurance the responses were collated according to the respondents’ background and affiliation. Hence it is possible to review the ways in which the various categories of respondents view the anticipated changes. Possibly important combinations of external changes are identified for analysis in step ten. These are evaluated in two ways for mutually enhancing or inhibitIf the change listed betow were to occur.
estricted
or prohibited
chicle travel
.
major
Restricted
or prohibited cities
er availability
of these changes subsequently occurring would.
.
numbe
cities
into major
transport
the probability
motor
in a significant
mf areas within
. . . then
entry
of im
between
I
mm Figure 3. Cross-impact
coupling
among external
J
events
ing effects. First, the external changes are evaluated to determine whether or not the occurrence of one development (which affects the model) is likely to enhance or inhibit the likelihood of occurrence of another (which also affects the model) (F’g 1 ure 3). The matrix aids in identifying strongly reinforcing developments. The second evaluation (Figure 4) is concerned with the direction and magnitude of the direct impacts likely to be incurred in the explicit elements of the model. Items noted with a “C” in this figure may have very strong impacts if they are accompanied by other developments. Both these evaluations use matrix formats and initially address the impacts between pairs; ie, one external event on another, or one external event on one element in the model. After considering the impacts between pairs, higher-order combinations are generally more easily identified. This information makes it easier to identify combinations of changes where
FUTURES
December
1974
496
Assessing a Problem-oriented
If the change listed below were to occur. . .
...
Social Technolog
then these elements in the model would be directly affected.
Total auto 1 Numberof expenditures vehicles
Miles travelled
.
Number of
Restricted or prohibited motor vehicle travel in a significant number of areas within major cities
I
Restricted or prohibited entry into major cities
Greater availability of improved mass transport between major cities c
+
Greater availability of improved mass transport within major cities
Figure 4. Direct effect of external
events on the model
mutual reinforcement among the causal linkages is strong. These situations may promote important or even potentially “explosive” situations. Each of these situations is evaluated separately and potential controls or modifications that can be used to capitalise on or ameliorate any of the situations are identified. Identifying
the outcomes
The above results are the basis for the eleventh step, consisting of describing outcomes which are likely to be of greatest concern and which should be considered most carefully in directing the technology in question. Any general recommendations and guidelines which the research has produced should be offered here. Identifying the most important possible outcomes poses a difficult methodological problem. A likely outcome is not necessarily an important one, although an outcome which almost surely will not occur is definitely not important. As a result, the steps described here tend to stress highly sensitive areas and combinations of developments which reinforce rather than offset each other. Since one of the primary considerations in a technology assessment is the identification and elaboration of possibly important but obscure and higher-order impacts, methodology must explicitly include low-probability developments. Thus, in identifying technological impacts, many of the assessors will frequently be called upon to discriminate important outcomes having virtually no chance of occurrence from those having a slight yet sufficient chance to be worthy of further deliberation. It is quite possible that the utility of these highly unlikely outcomes will depend
FUTURES
December 1974
Assessing a Problem-oriented Social Technology
497
upon the nature of the actions or guidelines that can be suggested to capitalise on the possible consequences. If the actions are relatively easy to implement and not disruptive with respect to the accommodation of other options, they are likely to be enacted. On the other hand, even when such actions are not implemented they should be used to establish monitoring procedures which, as a minimum, assure an early-warning posture. Summary The core of the methodological problem involved in technology assessment is the development of systematic procedures for the identification of important higher-order, indirect effects that may result from the introduction or redirection of a societal innovation. The approach developed in this study employs a combination of both highly organised and open-ended explorations and, generally speaking, follows what might be called good analytic practice, or the basic steps in the scientific method-not something new, but rather something old which has been modified to direct the attention of the research to more obscure factors. The approach begins with a detailed definition and understanding of the problem area to identify the scope and range of factors that should be included in the assessment. This open-ended phase of the assessment includes a review of the literature, consultation with experts, and many brainstorming sessions. Because the output of this activity bounds the scope of the assessment, it is important that it be not so narrow as to eliminate important, but non-obvious areas of investigation. On the other hand, without any delimitation, the scope of any issue worthy of an assessment always approaches the infinite. One of the most serious criticisms of this approach is that because it establishes the focus of the so-called manageable set of factors to be investigated from the problem area itself, it can easily overlook important impacts that may occur in more remote issues. The manageable set is then structured as an explicit model which can assess various combinations of eventualities and alternative actions rapidly and accurately. However, because quantitative models are invariably limited by our understanding of complex situations, available data, and computational capabilities, they tend to be static copies of dynamic situations. As a result, it is important to reopen the investigation periodically to the open-ended format to inquire whether the situations identified within the model may produce changes which upset the structure embodied in the model. The procedures developed in this study appear to be of assistance in the identification of important developments of this type, but do not contain any systematic procedure for explicitly identifying these among the infinite number of developments which may take place in the future. It is quite possible that a truly systematic approach to this aspect of technology assessment may never be fully realised. The study did develop and demonstrate an approach to assessing a certain class of issues in a way that encourages the search for areas which are suspected to contain higher-order impacts and to include these explicitly in the analytic investigation, but even this approach is only effective for second- or perhaps third-order impacts.
FUTURES
December 1974
498
Assessing a Problem-oriented
Social Technology
Notes and references 1. Richard C. Davis, “Applying Technology Assessment in the Multi-Product Company”, Futures, Vol. 6, No. 5, October 1974, pages 413-419 2. National Academy of Sciences, Technology: Processes of Assessment and Choice, Report to US Congress, House Committee on Science and Astronautics (July 1969) 3. In the case study, the results of this portion of the first step were presented in the report, The Automobile ~ns~~a~~eSystem : Current Statw and Some Proposed ~e~~s~orls, by Selwyn Enzer, Gordon J. Slotsky, Dennis L. Little, James E. Doggart, and David A. Long, Working Paper WP-18, Institute for the Future (February 1972) 4. In the automobile insurance case study, a considerable (yet invariably insufficient) amount of statistical data was available concerning the system that had been in existence in the USA: ie, the tort (or fault) system. Hence it was possible to develop a mathematical model of this system which was statistically relatable and verifiable over a mea~ngful historical period. It was also possible to use this model to describe the change that might result if the innovation proposed (or its alternatives) were introduced in the future. The no-fault insurance model was based on regression equations, ie, it contained quantitative relationships which were statistically verifiable, to the extent that statistics were available. Furthermore, the degree to which the model replicated the historical statistics was the only criterion for determining the quality of the modeifs performance. 5. In the no-fault insurance model, it was found that most of the essential variables could be accurately computed from Gross National Product (GNP). In other words, by starting with statistical data on GNP, the model was used to compute such items as expenditures for personal transport, number of accidents, and bodily injuries. Furthermore, when starting with the known values of GNP for previous years these computations produced results which correlate very closely with the actual statistics for these “dependent” factors. Several factors were included in the model for which there were few or no statistical data. In such cases, either assumptions based on expert judgment are included or the variable is evaluated parametrically, or both. Parametrically evaluating a variable consists of using the model to assess the impacts on the output variables for a wide range of change in the parameter being evaluated. Using a model in this manner is called a sens~t~~~t~ u~ulys~sand is discussed in the text. 6. As can be seen in Volume I of our study, the no-fault insurance model’s exclusive use of this criterion severely limited the range of change variables that could be perturbed. However, this approach generally results in higher-confidence forecasts in those areas where the forces at work are not considered likely to undergo drastic changes, which appears to be the case in that portion of the no-fault insurance model concerned with forecasting losses. Considerations in Making Technology Assessments 7. Selwyn Enzer, “Practical Useful”, an address delivered at a joint symposium of the Institute of Management Sciences, the World Future Society, and the George Washington University Program of Policy Studies in Science and Technology, “Technology Assessment: Washington, DC, The George Washington A Future-Oriented Policy Tool”, University (5 June 1972)
FUTURES
December 1974