- Email: [email protected]
Handbook of systems analysis: Craft issues and procedural choices
TECHNOLOGICAL
FORECASTING
AND SOCIAL CHANGE
38,207-209
(1990)
Book Review Hugh J. Miser and Edward Quade, eds., Handbook of Systems Analysis: Craft Issues and Procedural Choices, North-Holland, New York, 1988, 681 pages, $115.00. This is the second volume of a three-volume handbook. The editors intend the entire set as a handbook, that is, as a compendium of what is currently considered “good practice” in the field of systems analysis. By systems analysis, they mean the application of the scientific method of inquiry to existing or planned systems, where the systems are dominated by technology and have boundaries that can be fairly sharply defined. The editors make clear that people-dominated problems (“policy analysis”) are important, but are beyond the scope of this handbook. They set themselves two goals in preparing a handbook of systems analysis: first, to restrict attention to “the central core of the field,” and second, to offer “normative standards of quality and substance where experience warrants.” The editors are clearly among the best-qualified people in the United States to carry out this task, if it can be done at all. This second volume is on craft issues. It is aimed primarily at systems analysis professionals (p. 19). The editors describe craft knowledge as “less explicit than formal theoretical knowlege, but more objective than pure intuition” (p. 7). Quade explains further (p. 313) that in solving a systems analysis problem by calculating something, the important craft issue is not that the analyst was competent to make certain calculations, but that the analyst identified the right calculations to make and was able to interpret the results creatively. The editors point out that all scientific and professional endeavors require craftsmanship, that is, knowledge gained through experience, either personal or that of colleagues. They list a number of issues and choices that involve “craft,” in the sense that experience can help but there can be no formal rules for dealing with them. These include: How to approach a mess of problems associated with a situation that needs improvement and isolate one or more problems whose solutions will be relevant and useful to those with responsibilities for decision and action. How to increase the decision maker’s options by designing new and better alternatives. How to choose variables for investigation and how to quantify them in a useful way. How to choose models that represent the important variables in the problem, whiile balancing the competing factors of comprehensiveness and ease of calculation.
How well does this volume achieve its goal of providing “craft” information? Because there are no objective standards to judge “craft,” we need to use two criteria: What are we told about? and Who tells us? That is, are the things we are told merely platitudes, or do they really represent hard-won experience? And are the people who tell us really in a position to know? The bulk of the book, and its real meat, consists of Parts II, Selecting Analysis Approaches; III, Setting Boundaries for the Analysis; IV, Some Components of Analysis; and V, Professional Issues. Naturally there is some overlap among these sections, as well as frequent references in one part to material in another part. Nevertheless, the divisons seem fairly clear. 0 1990 by Elsevier Science Publishing
Co., Inc.
0040-1625/90/$3.50
BOOK REVIEW
208
Part II begins with a discussion of strategy. This chapter is written by the editors themselves, and their long experience in systems analysis shines brilliantly through in what they write. They make clear the need for a strategy in each analysis, stating that among other things a strategy l l
l
helps organize the body of information related to the problem furnishes a logical structure for the intellectual work of solving the problem provides the initial basis for the analysis plan, including a determination of the kinds of skills needed
They also make concrete the kinds of strategic choices available instance, they list possible strategic choices as l
l l
working within existing constraints or trying to relax them satisficing vs optimizing seeking a unitary measure of performance vs using multiple mance
to analysts.
measures
For
of perfor-
They note (p. 69) that the central tools of systems analysis are models. They provide a brief discussion of modeling and models. However, this volume of the handbook is not about modeling. For details, they refer the reader to Volume 1. Subsequent chapters in Part II cover use of expert judgment (by Olaf Helmer), use of operational gaming (by Ingolf Stahl), and social experimentation (by Rae W. Archibald and Joseph P. Newhouse). These chapters maintain the same high level as that by Quade and Miser. Part III, on setting boundaries, includes chapters on generating and screening alternatives, with emphasis on how to set limits while maintaining adequate coverage within those limits (Warren E. Walker), on coping with uncertainty (Yehezkel Dror), and on uses and limitations of quantitative methods (Quade). Part IV, on components of analysis, might have been combined with Part II, since it’s not really clear how to separate components from approaches. Nevertheless, the chapters are well written. Brita Schwarz discusses forecasting and scenarios; H. G. Massey discusses the importance of cost estimation in systems analysis and presents a range of methods, with discussion of the experience of analysts with each. Harry Hatry discusses program evaluation, including not only objectives of evaluation but how to improve its usefulness and when to use controlled experiments. Part V, on professional issues, gets even more deeply into “craft” issues. Miser discusses the underlying concepts for systems analysis, including the idea that systems analysis is intended to result in action and that it inherits many of its professional attitudes from the science-based professions. Miser and Quade devote a chapter to verification (assuring that the models analysts actually build are the models they intended to build) and validation (assuring that the models they intended to build are like the real system in all relevant aspects). Bruce Goeller contributes a chapter on how to evaluate success in systems analyis, illustrating his points with examples drawn from some of the outstanding analysis projects of the past decade. The final chapter, on quality control, presents Miser and Quade’s views on how to assure that the analysis is done right. This book has to be judged in its own terms. The neophyte reading it is not going to learn how to do systems analysis. It is intended for the person who already knows the academically teachable tools of the trade: mathematical modeling, statistical analysis,
BOOK REVIEW
209
design of experiments, linear programming, queuing theory, search theory, etc. It is a compendium of experience-based guideposts and warning-here arc some things that worked in some well-done analyses; here are some of the traps and pitfalls analysts have learned to beware through bitter experience. On its own terms, it succeeds well. The contributors are all people who have “seen the elephant” and who can speak with authority about what they have learned. The apprentice systems analyst can learn a great deal here. Even the experienced professional analyst can benefit from this book, because the range of experience of the editors and contributors is greater than what any one analyst can reasonably accumulate in a working lifetime. The editors are to be commended on a fine job. In addition, we can bc grateful to the International Institute for Applied Systems Analysis for initiating the work and to North-Holland Publishing Company for supporting its completion. Mechanically, the book is well made, and it seems to bc remarkably free of misprints. JOSEPH P. MARTIN0 Associate Editor
Received 19 March 1990
FORECASTING
AND SOCIAL CHANGE
38,207-209
(1990)
Book Review Hugh J. Miser and Edward Quade, eds., Handbook of Systems Analysis: Craft Issues and Procedural Choices, North-Holland, New York, 1988, 681 pages, $115.00. This is the second volume of a three-volume handbook. The editors intend the entire set as a handbook, that is, as a compendium of what is currently considered “good practice” in the field of systems analysis. By systems analysis, they mean the application of the scientific method of inquiry to existing or planned systems, where the systems are dominated by technology and have boundaries that can be fairly sharply defined. The editors make clear that people-dominated problems (“policy analysis”) are important, but are beyond the scope of this handbook. They set themselves two goals in preparing a handbook of systems analysis: first, to restrict attention to “the central core of the field,” and second, to offer “normative standards of quality and substance where experience warrants.” The editors are clearly among the best-qualified people in the United States to carry out this task, if it can be done at all. This second volume is on craft issues. It is aimed primarily at systems analysis professionals (p. 19). The editors describe craft knowledge as “less explicit than formal theoretical knowlege, but more objective than pure intuition” (p. 7). Quade explains further (p. 313) that in solving a systems analysis problem by calculating something, the important craft issue is not that the analyst was competent to make certain calculations, but that the analyst identified the right calculations to make and was able to interpret the results creatively. The editors point out that all scientific and professional endeavors require craftsmanship, that is, knowledge gained through experience, either personal or that of colleagues. They list a number of issues and choices that involve “craft,” in the sense that experience can help but there can be no formal rules for dealing with them. These include: How to approach a mess of problems associated with a situation that needs improvement and isolate one or more problems whose solutions will be relevant and useful to those with responsibilities for decision and action. How to increase the decision maker’s options by designing new and better alternatives. How to choose variables for investigation and how to quantify them in a useful way. How to choose models that represent the important variables in the problem, whiile balancing the competing factors of comprehensiveness and ease of calculation.
How well does this volume achieve its goal of providing “craft” information? Because there are no objective standards to judge “craft,” we need to use two criteria: What are we told about? and Who tells us? That is, are the things we are told merely platitudes, or do they really represent hard-won experience? And are the people who tell us really in a position to know? The bulk of the book, and its real meat, consists of Parts II, Selecting Analysis Approaches; III, Setting Boundaries for the Analysis; IV, Some Components of Analysis; and V, Professional Issues. Naturally there is some overlap among these sections, as well as frequent references in one part to material in another part. Nevertheless, the divisons seem fairly clear. 0 1990 by Elsevier Science Publishing
Co., Inc.
0040-1625/90/$3.50
BOOK REVIEW
208
Part II begins with a discussion of strategy. This chapter is written by the editors themselves, and their long experience in systems analysis shines brilliantly through in what they write. They make clear the need for a strategy in each analysis, stating that among other things a strategy l l
l
helps organize the body of information related to the problem furnishes a logical structure for the intellectual work of solving the problem provides the initial basis for the analysis plan, including a determination of the kinds of skills needed
They also make concrete the kinds of strategic choices available instance, they list possible strategic choices as l
l l
working within existing constraints or trying to relax them satisficing vs optimizing seeking a unitary measure of performance vs using multiple mance
to analysts.
measures
For
of perfor-
They note (p. 69) that the central tools of systems analysis are models. They provide a brief discussion of modeling and models. However, this volume of the handbook is not about modeling. For details, they refer the reader to Volume 1. Subsequent chapters in Part II cover use of expert judgment (by Olaf Helmer), use of operational gaming (by Ingolf Stahl), and social experimentation (by Rae W. Archibald and Joseph P. Newhouse). These chapters maintain the same high level as that by Quade and Miser. Part III, on setting boundaries, includes chapters on generating and screening alternatives, with emphasis on how to set limits while maintaining adequate coverage within those limits (Warren E. Walker), on coping with uncertainty (Yehezkel Dror), and on uses and limitations of quantitative methods (Quade). Part IV, on components of analysis, might have been combined with Part II, since it’s not really clear how to separate components from approaches. Nevertheless, the chapters are well written. Brita Schwarz discusses forecasting and scenarios; H. G. Massey discusses the importance of cost estimation in systems analysis and presents a range of methods, with discussion of the experience of analysts with each. Harry Hatry discusses program evaluation, including not only objectives of evaluation but how to improve its usefulness and when to use controlled experiments. Part V, on professional issues, gets even more deeply into “craft” issues. Miser discusses the underlying concepts for systems analysis, including the idea that systems analysis is intended to result in action and that it inherits many of its professional attitudes from the science-based professions. Miser and Quade devote a chapter to verification (assuring that the models analysts actually build are the models they intended to build) and validation (assuring that the models they intended to build are like the real system in all relevant aspects). Bruce Goeller contributes a chapter on how to evaluate success in systems analyis, illustrating his points with examples drawn from some of the outstanding analysis projects of the past decade. The final chapter, on quality control, presents Miser and Quade’s views on how to assure that the analysis is done right. This book has to be judged in its own terms. The neophyte reading it is not going to learn how to do systems analysis. It is intended for the person who already knows the academically teachable tools of the trade: mathematical modeling, statistical analysis,
BOOK REVIEW
209
design of experiments, linear programming, queuing theory, search theory, etc. It is a compendium of experience-based guideposts and warning-here arc some things that worked in some well-done analyses; here are some of the traps and pitfalls analysts have learned to beware through bitter experience. On its own terms, it succeeds well. The contributors are all people who have “seen the elephant” and who can speak with authority about what they have learned. The apprentice systems analyst can learn a great deal here. Even the experienced professional analyst can benefit from this book, because the range of experience of the editors and contributors is greater than what any one analyst can reasonably accumulate in a working lifetime. The editors are to be commended on a fine job. In addition, we can bc grateful to the International Institute for Applied Systems Analysis for initiating the work and to North-Holland Publishing Company for supporting its completion. Mechanically, the book is well made, and it seems to bc remarkably free of misprints. JOSEPH P. MARTIN0 Associate Editor
Received 19 March 1990