- Email: [email protected]
Software technology and integration engineering
Software Technology
and Integration
Engineering
Robert C. McHenry and Jerry A. Rand IBM, Guithcrshtrrg,
Mur:\~Iattd
Perhaps the least appreciated area of modern software technology is top-down development. It is far more than a programming technique suitable for application to an individual work assignment: It is a rich and powerful technique for project implementation and for system integration. The characteristics of the top-down process (executable as a system and self-integrating) suggest that the process may be a foundation of integration engineering.
INTRODUCTION A number of trends in data processing appear to justify establishing a discipline of integration engineering. These trends include aggregating larger systems, requiring higher availability, distributing systems, and developing systems concurrently. Mills has defined software as logical doctrine for the harmonious cooperation of people and machines [ 11. When a system is defined to be a coherent assemblage of people and hardware with specific capabilities, the system engineering and related operating rules are, in fact, implemented in the operational software. From this perspective, we speculate that system integration and integration engineering technology can be approached from software technology. Our speculation can be illustrated, and perhaps substantiated, by a single approach: top-down development. Top-down development is one of the least appreciated facets of software technology. Topdown development is more than an individual work assignment technique: it is a profound technical and managerial strategy as well [2]. The key characteristics of top-down development suggest that the evolving software is always executable as a system, and the process is self-integrating. An adaptation of the top-
down approach was conceived by O’Neill in 1972 and demonstrated in the overlapped development and integration of the TRIDENT Command and Control System [3]. The concept is that software, unlike hardware, can be implemented in a system sense to be always executable and that system integration can proceed from the software (i.e., the computer) to the nonprogrammable hardware. The authors generalized the top-down adaptation by defining a top for any system and by relating the system top to an integration strategy. The top of any system is the logic for transitioning the system from state to state. This definition is general since even the simplest system must be turned on and off. In more complex systems there are many states, including initialization, maintenance, development. training, reduced function, full function, and termination. A key integration engineering hypothesis is that transitioning logic is a candidate integration strategy. The authors, in an earlier report [4], constructed a hypothetical example system to illustrate the integration engineering approach described in this paper and provide a framework for supporting rationale.
INTEGRATION
PROBLEMS
This section identifies problems fects on system development. clude the following:
treatment of hardware tern components:
of Systems North
and Softwale
Holland.
Inc..
IYXO
I, 1X7-193
(1980)
and software
preoccupation
with full-up
late initiation
of integration
poorly designed
5‘: Elsevier
efin-
partitioning of the system into subsystems without regard to integration activities and life cycle factors:
delayed interface testing testing or schedule;
The Journal
that have adverse These problems
as separate
user-oriented
sys-
functions:
and test activities: at the expense
maintenance
and training
of functional states:
and
187 Olh4-I212;xo/o3 IX7-(17$02.?r
188
extended schedule for reduced function state, design, and test. In traditional development, subsystems are chosen early in the cycle. In fact, subsystem-oriented specialists frequently prepare the detailed requirements that are reflected in a system specification. The most obvious and painful manifestations of the shortcomings of the traditional approach occur during system integration where interface imperfection and imbalance become evident. It follows that development efficiency has suffered. Less obviously, but of vital significance, the effectiveness of the resulting system may be impaired. The most common instance of premature partitioning is arbitrary allocation of function to hardware or software components. Equally serious is preoccupation with operational function rather than a comprehensive analysis of all system states. This preoccupation leads to belated and hurried treatment of the balance of the system design and associated hasty project planning. Thus integration and testing activities make inadequate provision for maintenance, training, and reduced function states. A new approach is needed, one that determines the states of the system and deters arbitrary subsystem choices. These logical states provide guidance for subsequent development of design solutions without overly constraining the choice of solution. The authors believe that such an approach may be offered by integration engineering. The need for integration engineering technology is greater than the need for systems, hardware, or software engineering technology. Far more tools exist to support design and development. In fact, implementational technology has progressed beyond our ability to employ it effectively. The continuing improvements in data processing hardware have led to the introduction of programmable hardware into many traditional nonprogrammable subsystems. Although the resulting systems may be thought of as distributed, they often result from subsystem rather than system decisions.
OBJECTIVES
The objectives of integration engineering will be realized by an approach that places state transitioning at the top of the system. Component boundaries for integration are not drawn between hardware and software. The boundaries are drawn around analyses that define the system states and the state transitioning. Maintenance, training, and reduced function states thereby become an integral part of the design and de-
R. C. McHenry and J. A. Rand velopment process and appear as early milestones on the integration schedule. Early deliveries are aimed at integrating the lowest operating level (i.e., the maintenance state), that is, the configuration with the minimum available hardware. Each successive delivery (and integration) can be viewed as a transition from one operating state to the next higher one. After integration of the final deliveries (full-up state), the preceding deliveries are retained rather than discarded. Delivery of a maintenance state allows for integration of hardware/software from a component level . through the subsystem to the system level with continuous availability of system test elements. This allows for verification of performance at each level of integration without dependence on the correct performance of higher levels. The objectives served by this approach are as follows: minimum time spent on interface misunderstandings; elimination of individual state design and integration activity; states for maintenance and training becoming a natural part of the design process; early deployment
of a partial system; and
overall shorter schedule because of increased overlap of development and integration.
APPROACH
This section outlines an integration engineering approach. Although it may appear to belabor the obvious, the reader is reminded that the critical path of large system development always passes through the system integration phase. This reminder takes on significance when the constituent paths that converge to the final path are examined. These paths all too often converge simultaneously at system integration. System integration should not be confined to the final days of a multiyear development. The system manager typically has adequate time to phase system integration throughout development activities. To accomplish phased integration, the manager must invest in early planning and process engineering to schedule the development of interfacing components properly. A key concept is maximization of the overlap of development and integration. Explicit interface specification is required to govem the development or modification of interacting components. This definition must be accomplished early to ensure that interface implementation will occur prior to integration. The interface definitions
Software
Technology
and Integration
Engineering
and consequent development or modification actions require mutual agreement cognizant among developers. Interface implementation, generally, is accomplished by the component developer. Typically, the initial interface implementation for each component is performed without availability of interfacing components. Thus, at an implementing level, the components are decoupled and some simulation of the missing is required component for interface verification. Some underlying concepts of integration engineering are presented in the following subsections.
Top-Down
Implementation
189
reduced by resource casualty (e.g.. equipment failure) and describes the correspondingly reduced function the system provides. Degraded states is a narrow term because the system may be required to support operator training, equipment maintenance. initialization, termination, or other functions in addition to the operational functions that led to the system’s development. Preoccupation with the full operational function state can lead to an unsatisfactory system since the early work must also specify the procedures required to transition the system from state to state. The requirements determination process that precede5 system development must specify the functional availability (e.g.. tolerable outage) requirements to guide the eventual development of procedure configurations for each state. Each state has intrinsic procedures and exists in a broader procedural environment which enables transition from state to state. A system top is the transitioning procedure, and this perspective helps drive the entire process (function. development, test. operation).
The top-down approach is patterned after the natural approach to system design and requires that programming proceed from developing the control architecture (interface) statements and data definitions downward to developing and integrating the function units. Top-down programming is an ordering of system development that allows for continual integration of the parts as they are developed and provides for interfaces prior to the parts being developed. In top-down structured programming, the system is organized into segments. The top-level segment contains the highest level of control logic and decisions within the program. and either passes control to lower-level segments. or identifies lower-level segments for in-line inclusion. Since this will invoke or include lower-level segments, code must exist for the next lower-level segment. This code, called a program stub, may be empty, may output a message, or may provide minimal function. These stubs are later expanded in full functional segments, which in turn require lower-level segments. This process continues until all functions within a system are defined in executable code. This approach provides the ability to evolve the system in amannerthat maintains the characteristic of being always operable, extremely modular. and always available for testing.
Prototype systems are developed for concept validation, feasibility determination, or benchmark calibration purposes. The successful prototype generally represents an early step in a major system acquisition. Although the successful prototype overcomes the certainty of specific failures. the typical prototype does not guarantee full-scale success. Serious problems or disappointing performance may arise in scaling up to an operational system. Consideration must be given to scaling problems throughout the prototype planning. One approach to prototyping is to scale down from the system design to the prototype design. In an evolutionary development approach, the prototype is a subset of the intended system. The prototype may. in fact. be an operational state of the intended system.
Operating
Integration
Modes
Examining system operation reveals the heart of the question: Where is the top of the system? The whole procedural question of development, test, operation, and modification finds focus in the life cycle. Operating states or modes describe the various configurations of system resources (people, programs. and equipment) that operate the system. In many development projects, the narrow term “degraded states” describes configurations that are
Prototypes
Since software, unlike hardware. can be implemented in a system sense to be always executable, the system integration should proceed from software (i.e ., the computer) to the nonprogrammable hardware. Phased deliveries of software to software/hardware integration ensure intermediate evaluations where managers can determine progress and acceptability of concurrently developed components. Phased deliveries provide the means of integrating hardware and
190
R. C. McHenry and J. A. Rand
software incrementally rather than as one enormous task late in the project. The problems are found and corrected earlier, and more schedule is available for retesting. The test system should also be built incrementally by phasing its development to match the system test and integration phasing. Earlier, operating states were described and the state transitioning procedures were considered to be a top of any system. A process is postulated to exploit the transitioning procedures as a strategy for implementing and integrating the larger than software system. The process considerations are the following: The number of states and transitions required for integrating the system may exceed the number required for operating the system. The testing of reduced function states is progressive and is an integration process by-product rather than an afterthought to full function state testing. The more critical functions (i.e., those functions occurring in more states) receive more testing. Where development funding is spread over many years (or under design-to-cost limits) the evolving system can be executed as a reduced system for testing or limited operation.
will perform its specified functions in the simulated environment and to reduce the number of problems encountered during system test and evaluation. Simulation programs are designed to exercise programs in the absence of component equipments and the environment that stimulates the systems. The use of simulation programs during the development, integration, and verification of the software is extensive. Simulation software must be transferable between the sites. Software maintenance must be provided throughout the life cycle starting with initial software deliveries. The working mechanism for software maintenance is established in the top-down development approach where program modifications represent additional stages beyond those required during development. System Readiness
Functions
In complex systems, the various assessment and hardware maintenance support functions must be integrated into the operational system. These functions include: interface tests to assess hardware integration during operation and testing; operability tests to assess functional availability;
Incremental
Development
The top-down approach is applied in discrete stages (initial, intermediate, final) to improve manageability. Each stage is carried to readiness for system integration. The initial stage begins early in the schedule, after the design specifications are developed, to exercise and validate the interface between executive and functional software and to obtain an early validation of computer utilization allocations. The intermediate stages are developed to exercise and validate the functional software interfaces and selected critical system functions. Simulation software provides inputs to the functional software in lieu of actual hardware inputs. Intermediate stage software is used for interface validation and early system hardware integration. The scheduling of initial and intermediate stages must provide time for appraisal and redirection of development. The final stage is developed to provide all functions the software is required to perform. Software integration and verification is a controlled process by which intermediate and final deliveries are integrated in simulation environments that, at successive tests, more fully approximate actual use. The objectives are to verify that the integrated software
equipment tests to assess availability and to support corrective or preventive maintenance; component status determination guration; and data extraction analysis.
for
to support reconfi-
presentation
or
subsequent
In addition, the personnel readiness functions (training and exercise) can be integrated into the operational system. Applicability
to Subsystems
The analytical process by which a system is partitioned into states can be applied to subsystems and lower-level subdivisions. A result of the process is the transitioning procedure which employs system parameters in its conditional logic. Although we may consider any conditional expression as choosing among states, at some point we cease to distinguish states unless a software/hardware/people reconfiguration or major component state change is involved. At the system level we may wish to consider intrasubsystem states as substates. (Substates have the property of being nested within states.) Now if we consider testing and more particularly
Software
Technology
and Integration
191
Engineering
integration to be conditional in nature, we may consider test and integration as transitioning procedures. Similarly, incremental development may be considered in a transitional context.
TRANSITION
MANAGEMENT
A key hypothesis of integration engineering is that the state transition management is a natural part (i.e., the top) of the system and should not require schedule extensions for testing. Transition management is the implementation of the integration engineering approach stated in the section on the integration engineering approach. Some work remains to be done in order to generalize transition implementation. The issues include the degree to which: transition management system:
is dependent
the transition management a distributed system: the actual configuration/state plication functions; and system
readiness
functions
function
on the operating is distributed
is transparent
in
to the ap-
are implemented.
Distributed
is completely management function dependent on the system architecture. For example, in a distributed system the transition management function itself could be distributed. The issues raised here are the same as those raised in connection with a distributed data management system (e.g., master/ slave versus peer to peer, deadlock avoidance, functional integrity). The resolution of these issues leads directly to questions about global status data, information hiding, and the connectivity between nodes. One possible solution to some of these questions is to utilize functional and physical addressing in the data bus (or set of buses) connecting nodes. Computers peripherals, and user terminals are attached to the bus. Each device on the data bus is assigned a hardware (physical) bus address. The address is also a unique identifier for hardware reconfiguration. Upon initialization, status change, or user command each computer builds and broadcasts its version of the equipment status. The reconfiguration process is then summarized as follows:
The transition
I. Resolve 3 _.
Transition management, like integration engineering, exposes most system design issues from the basic system architecture to component reliability requirements. However, every system is in some way unique in the parameters that define transition management. For this reason the authors have been unable to generalize by resolving the issues raised.
Operating
System
management can be viewed as part of. a superset of, or a subset of the operating system. In practice. transition management is all three. The system loader (e.g., initialization) is a superset of the operating system since it loads and initializes the executive program. The decision logic that determines where application functions reside for interface routing is a subset of the operating system. The logic to respond to hardware and software faults is part of the operating system and designed to match the computer chosen. Therefore, under today’s operating system concepts. transition management is dependent on the chosen operating system and computer. If, however. the operating system interface to the functions required by transition management (e.g., system loader, fault recovery, interrupt processing) is well defined and documented, then the degree of dependence would be minimized.
Transition
Management
3. 4.
5. 6.
status conflicts between computers Select the software configuration. Load and initialize software functions. Assign functional addresses on the data bus for: a. processes, b. data bases, and c. user designations. Initiate user alerts and displaying system status. Accept user commands for system recontiguration or initialization.
In this manner tem allocation:
the process
causes
three levels of sys-
Allocate functions or lost functions to each available computer (i.e., establish the set of functional bus addresses for each computer). Allocate data files, primary and alternative, to mass memory units (i.e., establish data set functional bus addresses for each mass memory). Confirm and/or change terminal designations for alerts. controller modes, and other user-owned functions (i.e., functional bus addresses for each terminal). Each unique set of functional bus addresses for any computer is considered as a different software configuration. Each unique set of functional bus addresses for any mass memory unit is considered as a different data base configuration. Also. each unique set of functional bus addresses for any terminal is considered as a different terminal configuration.
192
R. C. McHenry and J. A. Rand
The last allocation process should be performed only under operator direction, but the first two allocations can be performed by using a finite state transition table. The table maps the old state (set of functional bus addresses) to a new state based on the equipment status that changed.
Application
Functions
Obviously, the application functions must be designed to be independent of the system configuration. This is, however, achievable in any system and made easier by the functional addressing proposed above.
System Readiness
The system readiness functions are the single most important element to transition management, and one of the critical tools for integration engineering. If the system has the flexibility and redundancy to recover totally (or partially) from a single failure, but cannot detect the failure when it occurs, then the system flexibility is wasted. The system should maintain status on each data processing component in one of three categories: (1) on line, (2) off line, or (3) idle. The on-line category is strictly reserved for equipment that is currently in use by the operational system. The off-line category is used for equipment that has failed or that was placed off line by the operator for test. The idle category is reserved for equipment that has been repaired but is not in use by the operational system. The equipment is in a standby mode and can be placed on line automatically during the next reconfiguration. The data processing system would then change equipment from on to off line based on test results from the system readiness functions. The system readiness functions must provide fully integrated, on-line system tests and equipment diagnostics that operate in real time. The function should continually assess system readiness to meet the defined mission while causing minimal degradation to the operational system. The readiness functions should include the following tests: Operability tests assess the capability of the entire system to perform as required. The test should address the system as a whole and verify that the various complete functions can be performed. Interface tests assess the capability of all hardware units to intercommunicate properly and to perform their electrical functions correctly. This test should address each individual hardware unit and all of its
signal interfaces. Detected hardware unit failures should be immediately displayed. Alignment tests assist in the assessment and/or system alignment parameters.
of sensor
Performance monitoring (PM) tests check each equipment unit and interface for proper operation. Fault location (FL) tests exercise the unit to make a detailed determination of the exact nature and location of the failure for repair. They require that the subject equipment be dedicated to testing and repair. These tests are required to operate “on line” in that they will run concurrently with the operational program without any system degradation, as opposed to an “off line” test which is conducted independently of the operational program and prohibits that program from meeting its designated operational requirements. Although the FL tests are not fully on line, they run under control of the operational executive and degrade the system no further than loss of the malfunctioning equipment. These system tests must be fully integrated with the operational program and utilize operational program modules to provide tests of all hardware and operational-software functions and interfaces throughout the system. This can be accomplished by use of operator selected test exercises and printouts that log the flow of selected messages throughout the system. Hardware operability can be determined by operation of PM tests to detect failures and FL programs to isolate failures for repair. The purpose of the operability test is to determine and display the operational readiness of system functions. This is accomplished by display of functions affected by equipment failures and by running “operability readiness exercises.” Equipment failures can also be correlated to major system functions and the result presented in a system status display. Operability readiness exercises provide a comprehensive check of total system performance by executing operational functions using simulated stimuli and a planned sequence of operator interventions. The expected results can be compared to the system reponse for verification of operability. Intermediate calculations and interface data printouts can be logged out on a printer unit and made available for detailed analysis. The purpose of the interface test is to determine and display the readiness of the data processing equipment. This is accomplished by periodic checks of the equipment utilizing the on line PM software and
Software
Technology
and Integration
193
Engineering
by display of all equipment status at the operator’s console. The PM software can operate periodically on line with the operational software and check all equipment units and interfaces for the ability to perform hardware functions and to transmit and receive data correctlq The purpose of the alignment test is to provide support to system alignment. This can be accomplished by execution of alignment procedures that correlate and display alignment parameters for operator assessment and calibration. The alignment parameters displayed should reflect performance of not only the equipment. but also the operational algorithm. Individual PM tests perform the equipment check function of the interface test. The PM software should: provide fault detection, defined as the performance of tests to ascertain that equipment and interfaces are petforming properly or improperly and to indicate its operatin@nonopel-ating status as “go/no-go” indication: allocate part of the scheduled operation to performance monitoring tests in a background mode run continuously under the system’s control: be nondisruptive to the operational system by executing concurrently with operational software on a time-shared basis without interruption in the availability and use of equipment and without damage to operational data: and report on the status of hardware operational functions. The system
FL software
failures
and affected
should:
provide fault isolation and identification of a failed component within the equipment indicated as nonfunctioning by PM on manual request: run concurrently
with the operational
program:
and
diagnosis equipment that is logically off line to the operational program, but perhaps still physically connected. The unit(s) would be dedicated to maintenance activities and the operational program would not have access to the equipment during this time. This system readiness concept supports scheduled preventative maintenance and corrective maintethe following This offer\ nance. approach advantages: System operational readiness is continually and status data available at all times.
assessed
System operators are familiar with the tests since they are an operational exercise which uses the actual equipments. programs. and procedures. Software interfaces. are tested.
as well as hardware
intelfaces.
Implementation and retrofit costs are greatly by use of the existing operational code. These tests can provide training requirements engineering.
reduced
a portion of the operational and tools for integration
REFERENCES H. 1). Mills. Software Engineering. .S(~i~,/xc~195 (42X3). I 199- 1205 (1977). C. L. McGowan and R. C. McHenry, Software Muagement. in Xo~c,rrrc./~ f~ir~c~c~iic~~~.\ i/r .So/f~~.cox, ‘~w/I/~~~/II,:‘?‘t E. Ditor. ed.). MIT Press, Cambridge. Massachu4etts. 1978. R. C. McHenry and J. A. Rand. Integration Engineering: An Approach to Rapid System Development. FSD 77. 0179. IBM Corporation. Gaithersburg. Maryland. 1977. R. C. McHenry and J. A. Rand. Software Technology and Integration Engineering. FSD 78-0034. IBM Corporation. Gaithersburg. Maryland. 1978.
and Integration
Engineering
Robert C. McHenry and Jerry A. Rand IBM, Guithcrshtrrg,
Mur:\~Iattd
Perhaps the least appreciated area of modern software technology is top-down development. It is far more than a programming technique suitable for application to an individual work assignment: It is a rich and powerful technique for project implementation and for system integration. The characteristics of the top-down process (executable as a system and self-integrating) suggest that the process may be a foundation of integration engineering.
INTRODUCTION A number of trends in data processing appear to justify establishing a discipline of integration engineering. These trends include aggregating larger systems, requiring higher availability, distributing systems, and developing systems concurrently. Mills has defined software as logical doctrine for the harmonious cooperation of people and machines [ 11. When a system is defined to be a coherent assemblage of people and hardware with specific capabilities, the system engineering and related operating rules are, in fact, implemented in the operational software. From this perspective, we speculate that system integration and integration engineering technology can be approached from software technology. Our speculation can be illustrated, and perhaps substantiated, by a single approach: top-down development. Top-down development is one of the least appreciated facets of software technology. Topdown development is more than an individual work assignment technique: it is a profound technical and managerial strategy as well [2]. The key characteristics of top-down development suggest that the evolving software is always executable as a system, and the process is self-integrating. An adaptation of the top-
down approach was conceived by O’Neill in 1972 and demonstrated in the overlapped development and integration of the TRIDENT Command and Control System [3]. The concept is that software, unlike hardware, can be implemented in a system sense to be always executable and that system integration can proceed from the software (i.e., the computer) to the nonprogrammable hardware. The authors generalized the top-down adaptation by defining a top for any system and by relating the system top to an integration strategy. The top of any system is the logic for transitioning the system from state to state. This definition is general since even the simplest system must be turned on and off. In more complex systems there are many states, including initialization, maintenance, development. training, reduced function, full function, and termination. A key integration engineering hypothesis is that transitioning logic is a candidate integration strategy. The authors, in an earlier report [4], constructed a hypothetical example system to illustrate the integration engineering approach described in this paper and provide a framework for supporting rationale.
INTEGRATION
PROBLEMS
This section identifies problems fects on system development. clude the following:
treatment of hardware tern components:
of Systems North
and Softwale
Holland.
Inc..
IYXO
I, 1X7-193
(1980)
and software
preoccupation
with full-up
late initiation
of integration
poorly designed
5‘: Elsevier
efin-
partitioning of the system into subsystems without regard to integration activities and life cycle factors:
delayed interface testing testing or schedule;
The Journal
that have adverse These problems
as separate
user-oriented
sys-
functions:
and test activities: at the expense
maintenance
and training
of functional states:
and
187 Olh4-I212;xo/o3 IX7-(17$02.?r
188
extended schedule for reduced function state, design, and test. In traditional development, subsystems are chosen early in the cycle. In fact, subsystem-oriented specialists frequently prepare the detailed requirements that are reflected in a system specification. The most obvious and painful manifestations of the shortcomings of the traditional approach occur during system integration where interface imperfection and imbalance become evident. It follows that development efficiency has suffered. Less obviously, but of vital significance, the effectiveness of the resulting system may be impaired. The most common instance of premature partitioning is arbitrary allocation of function to hardware or software components. Equally serious is preoccupation with operational function rather than a comprehensive analysis of all system states. This preoccupation leads to belated and hurried treatment of the balance of the system design and associated hasty project planning. Thus integration and testing activities make inadequate provision for maintenance, training, and reduced function states. A new approach is needed, one that determines the states of the system and deters arbitrary subsystem choices. These logical states provide guidance for subsequent development of design solutions without overly constraining the choice of solution. The authors believe that such an approach may be offered by integration engineering. The need for integration engineering technology is greater than the need for systems, hardware, or software engineering technology. Far more tools exist to support design and development. In fact, implementational technology has progressed beyond our ability to employ it effectively. The continuing improvements in data processing hardware have led to the introduction of programmable hardware into many traditional nonprogrammable subsystems. Although the resulting systems may be thought of as distributed, they often result from subsystem rather than system decisions.
OBJECTIVES
The objectives of integration engineering will be realized by an approach that places state transitioning at the top of the system. Component boundaries for integration are not drawn between hardware and software. The boundaries are drawn around analyses that define the system states and the state transitioning. Maintenance, training, and reduced function states thereby become an integral part of the design and de-
R. C. McHenry and J. A. Rand velopment process and appear as early milestones on the integration schedule. Early deliveries are aimed at integrating the lowest operating level (i.e., the maintenance state), that is, the configuration with the minimum available hardware. Each successive delivery (and integration) can be viewed as a transition from one operating state to the next higher one. After integration of the final deliveries (full-up state), the preceding deliveries are retained rather than discarded. Delivery of a maintenance state allows for integration of hardware/software from a component level . through the subsystem to the system level with continuous availability of system test elements. This allows for verification of performance at each level of integration without dependence on the correct performance of higher levels. The objectives served by this approach are as follows: minimum time spent on interface misunderstandings; elimination of individual state design and integration activity; states for maintenance and training becoming a natural part of the design process; early deployment
of a partial system; and
overall shorter schedule because of increased overlap of development and integration.
APPROACH
This section outlines an integration engineering approach. Although it may appear to belabor the obvious, the reader is reminded that the critical path of large system development always passes through the system integration phase. This reminder takes on significance when the constituent paths that converge to the final path are examined. These paths all too often converge simultaneously at system integration. System integration should not be confined to the final days of a multiyear development. The system manager typically has adequate time to phase system integration throughout development activities. To accomplish phased integration, the manager must invest in early planning and process engineering to schedule the development of interfacing components properly. A key concept is maximization of the overlap of development and integration. Explicit interface specification is required to govem the development or modification of interacting components. This definition must be accomplished early to ensure that interface implementation will occur prior to integration. The interface definitions
Software
Technology
and Integration
Engineering
and consequent development or modification actions require mutual agreement cognizant among developers. Interface implementation, generally, is accomplished by the component developer. Typically, the initial interface implementation for each component is performed without availability of interfacing components. Thus, at an implementing level, the components are decoupled and some simulation of the missing is required component for interface verification. Some underlying concepts of integration engineering are presented in the following subsections.
Top-Down
Implementation
189
reduced by resource casualty (e.g.. equipment failure) and describes the correspondingly reduced function the system provides. Degraded states is a narrow term because the system may be required to support operator training, equipment maintenance. initialization, termination, or other functions in addition to the operational functions that led to the system’s development. Preoccupation with the full operational function state can lead to an unsatisfactory system since the early work must also specify the procedures required to transition the system from state to state. The requirements determination process that precede5 system development must specify the functional availability (e.g.. tolerable outage) requirements to guide the eventual development of procedure configurations for each state. Each state has intrinsic procedures and exists in a broader procedural environment which enables transition from state to state. A system top is the transitioning procedure, and this perspective helps drive the entire process (function. development, test. operation).
The top-down approach is patterned after the natural approach to system design and requires that programming proceed from developing the control architecture (interface) statements and data definitions downward to developing and integrating the function units. Top-down programming is an ordering of system development that allows for continual integration of the parts as they are developed and provides for interfaces prior to the parts being developed. In top-down structured programming, the system is organized into segments. The top-level segment contains the highest level of control logic and decisions within the program. and either passes control to lower-level segments. or identifies lower-level segments for in-line inclusion. Since this will invoke or include lower-level segments, code must exist for the next lower-level segment. This code, called a program stub, may be empty, may output a message, or may provide minimal function. These stubs are later expanded in full functional segments, which in turn require lower-level segments. This process continues until all functions within a system are defined in executable code. This approach provides the ability to evolve the system in amannerthat maintains the characteristic of being always operable, extremely modular. and always available for testing.
Prototype systems are developed for concept validation, feasibility determination, or benchmark calibration purposes. The successful prototype generally represents an early step in a major system acquisition. Although the successful prototype overcomes the certainty of specific failures. the typical prototype does not guarantee full-scale success. Serious problems or disappointing performance may arise in scaling up to an operational system. Consideration must be given to scaling problems throughout the prototype planning. One approach to prototyping is to scale down from the system design to the prototype design. In an evolutionary development approach, the prototype is a subset of the intended system. The prototype may. in fact. be an operational state of the intended system.
Operating
Integration
Modes
Examining system operation reveals the heart of the question: Where is the top of the system? The whole procedural question of development, test, operation, and modification finds focus in the life cycle. Operating states or modes describe the various configurations of system resources (people, programs. and equipment) that operate the system. In many development projects, the narrow term “degraded states” describes configurations that are
Prototypes
Since software, unlike hardware. can be implemented in a system sense to be always executable, the system integration should proceed from software (i.e ., the computer) to the nonprogrammable hardware. Phased deliveries of software to software/hardware integration ensure intermediate evaluations where managers can determine progress and acceptability of concurrently developed components. Phased deliveries provide the means of integrating hardware and
190
R. C. McHenry and J. A. Rand
software incrementally rather than as one enormous task late in the project. The problems are found and corrected earlier, and more schedule is available for retesting. The test system should also be built incrementally by phasing its development to match the system test and integration phasing. Earlier, operating states were described and the state transitioning procedures were considered to be a top of any system. A process is postulated to exploit the transitioning procedures as a strategy for implementing and integrating the larger than software system. The process considerations are the following: The number of states and transitions required for integrating the system may exceed the number required for operating the system. The testing of reduced function states is progressive and is an integration process by-product rather than an afterthought to full function state testing. The more critical functions (i.e., those functions occurring in more states) receive more testing. Where development funding is spread over many years (or under design-to-cost limits) the evolving system can be executed as a reduced system for testing or limited operation.
will perform its specified functions in the simulated environment and to reduce the number of problems encountered during system test and evaluation. Simulation programs are designed to exercise programs in the absence of component equipments and the environment that stimulates the systems. The use of simulation programs during the development, integration, and verification of the software is extensive. Simulation software must be transferable between the sites. Software maintenance must be provided throughout the life cycle starting with initial software deliveries. The working mechanism for software maintenance is established in the top-down development approach where program modifications represent additional stages beyond those required during development. System Readiness
Functions
In complex systems, the various assessment and hardware maintenance support functions must be integrated into the operational system. These functions include: interface tests to assess hardware integration during operation and testing; operability tests to assess functional availability;
Incremental
Development
The top-down approach is applied in discrete stages (initial, intermediate, final) to improve manageability. Each stage is carried to readiness for system integration. The initial stage begins early in the schedule, after the design specifications are developed, to exercise and validate the interface between executive and functional software and to obtain an early validation of computer utilization allocations. The intermediate stages are developed to exercise and validate the functional software interfaces and selected critical system functions. Simulation software provides inputs to the functional software in lieu of actual hardware inputs. Intermediate stage software is used for interface validation and early system hardware integration. The scheduling of initial and intermediate stages must provide time for appraisal and redirection of development. The final stage is developed to provide all functions the software is required to perform. Software integration and verification is a controlled process by which intermediate and final deliveries are integrated in simulation environments that, at successive tests, more fully approximate actual use. The objectives are to verify that the integrated software
equipment tests to assess availability and to support corrective or preventive maintenance; component status determination guration; and data extraction analysis.
for
to support reconfi-
presentation
or
subsequent
In addition, the personnel readiness functions (training and exercise) can be integrated into the operational system. Applicability
to Subsystems
The analytical process by which a system is partitioned into states can be applied to subsystems and lower-level subdivisions. A result of the process is the transitioning procedure which employs system parameters in its conditional logic. Although we may consider any conditional expression as choosing among states, at some point we cease to distinguish states unless a software/hardware/people reconfiguration or major component state change is involved. At the system level we may wish to consider intrasubsystem states as substates. (Substates have the property of being nested within states.) Now if we consider testing and more particularly
Software
Technology
and Integration
191
Engineering
integration to be conditional in nature, we may consider test and integration as transitioning procedures. Similarly, incremental development may be considered in a transitional context.
TRANSITION
MANAGEMENT
A key hypothesis of integration engineering is that the state transition management is a natural part (i.e., the top) of the system and should not require schedule extensions for testing. Transition management is the implementation of the integration engineering approach stated in the section on the integration engineering approach. Some work remains to be done in order to generalize transition implementation. The issues include the degree to which: transition management system:
is dependent
the transition management a distributed system: the actual configuration/state plication functions; and system
readiness
functions
function
on the operating is distributed
is transparent
in
to the ap-
are implemented.
Distributed
is completely management function dependent on the system architecture. For example, in a distributed system the transition management function itself could be distributed. The issues raised here are the same as those raised in connection with a distributed data management system (e.g., master/ slave versus peer to peer, deadlock avoidance, functional integrity). The resolution of these issues leads directly to questions about global status data, information hiding, and the connectivity between nodes. One possible solution to some of these questions is to utilize functional and physical addressing in the data bus (or set of buses) connecting nodes. Computers peripherals, and user terminals are attached to the bus. Each device on the data bus is assigned a hardware (physical) bus address. The address is also a unique identifier for hardware reconfiguration. Upon initialization, status change, or user command each computer builds and broadcasts its version of the equipment status. The reconfiguration process is then summarized as follows:
The transition
I. Resolve 3 _.
Transition management, like integration engineering, exposes most system design issues from the basic system architecture to component reliability requirements. However, every system is in some way unique in the parameters that define transition management. For this reason the authors have been unable to generalize by resolving the issues raised.
Operating
System
management can be viewed as part of. a superset of, or a subset of the operating system. In practice. transition management is all three. The system loader (e.g., initialization) is a superset of the operating system since it loads and initializes the executive program. The decision logic that determines where application functions reside for interface routing is a subset of the operating system. The logic to respond to hardware and software faults is part of the operating system and designed to match the computer chosen. Therefore, under today’s operating system concepts. transition management is dependent on the chosen operating system and computer. If, however. the operating system interface to the functions required by transition management (e.g., system loader, fault recovery, interrupt processing) is well defined and documented, then the degree of dependence would be minimized.
Transition
Management
3. 4.
5. 6.
status conflicts between computers Select the software configuration. Load and initialize software functions. Assign functional addresses on the data bus for: a. processes, b. data bases, and c. user designations. Initiate user alerts and displaying system status. Accept user commands for system recontiguration or initialization.
In this manner tem allocation:
the process
causes
three levels of sys-
Allocate functions or lost functions to each available computer (i.e., establish the set of functional bus addresses for each computer). Allocate data files, primary and alternative, to mass memory units (i.e., establish data set functional bus addresses for each mass memory). Confirm and/or change terminal designations for alerts. controller modes, and other user-owned functions (i.e., functional bus addresses for each terminal). Each unique set of functional bus addresses for any computer is considered as a different software configuration. Each unique set of functional bus addresses for any mass memory unit is considered as a different data base configuration. Also. each unique set of functional bus addresses for any terminal is considered as a different terminal configuration.
192
R. C. McHenry and J. A. Rand
The last allocation process should be performed only under operator direction, but the first two allocations can be performed by using a finite state transition table. The table maps the old state (set of functional bus addresses) to a new state based on the equipment status that changed.
Application
Functions
Obviously, the application functions must be designed to be independent of the system configuration. This is, however, achievable in any system and made easier by the functional addressing proposed above.
System Readiness
The system readiness functions are the single most important element to transition management, and one of the critical tools for integration engineering. If the system has the flexibility and redundancy to recover totally (or partially) from a single failure, but cannot detect the failure when it occurs, then the system flexibility is wasted. The system should maintain status on each data processing component in one of three categories: (1) on line, (2) off line, or (3) idle. The on-line category is strictly reserved for equipment that is currently in use by the operational system. The off-line category is used for equipment that has failed or that was placed off line by the operator for test. The idle category is reserved for equipment that has been repaired but is not in use by the operational system. The equipment is in a standby mode and can be placed on line automatically during the next reconfiguration. The data processing system would then change equipment from on to off line based on test results from the system readiness functions. The system readiness functions must provide fully integrated, on-line system tests and equipment diagnostics that operate in real time. The function should continually assess system readiness to meet the defined mission while causing minimal degradation to the operational system. The readiness functions should include the following tests: Operability tests assess the capability of the entire system to perform as required. The test should address the system as a whole and verify that the various complete functions can be performed. Interface tests assess the capability of all hardware units to intercommunicate properly and to perform their electrical functions correctly. This test should address each individual hardware unit and all of its
signal interfaces. Detected hardware unit failures should be immediately displayed. Alignment tests assist in the assessment and/or system alignment parameters.
of sensor
Performance monitoring (PM) tests check each equipment unit and interface for proper operation. Fault location (FL) tests exercise the unit to make a detailed determination of the exact nature and location of the failure for repair. They require that the subject equipment be dedicated to testing and repair. These tests are required to operate “on line” in that they will run concurrently with the operational program without any system degradation, as opposed to an “off line” test which is conducted independently of the operational program and prohibits that program from meeting its designated operational requirements. Although the FL tests are not fully on line, they run under control of the operational executive and degrade the system no further than loss of the malfunctioning equipment. These system tests must be fully integrated with the operational program and utilize operational program modules to provide tests of all hardware and operational-software functions and interfaces throughout the system. This can be accomplished by use of operator selected test exercises and printouts that log the flow of selected messages throughout the system. Hardware operability can be determined by operation of PM tests to detect failures and FL programs to isolate failures for repair. The purpose of the operability test is to determine and display the operational readiness of system functions. This is accomplished by display of functions affected by equipment failures and by running “operability readiness exercises.” Equipment failures can also be correlated to major system functions and the result presented in a system status display. Operability readiness exercises provide a comprehensive check of total system performance by executing operational functions using simulated stimuli and a planned sequence of operator interventions. The expected results can be compared to the system reponse for verification of operability. Intermediate calculations and interface data printouts can be logged out on a printer unit and made available for detailed analysis. The purpose of the interface test is to determine and display the readiness of the data processing equipment. This is accomplished by periodic checks of the equipment utilizing the on line PM software and
Software
Technology
and Integration
193
Engineering
by display of all equipment status at the operator’s console. The PM software can operate periodically on line with the operational software and check all equipment units and interfaces for the ability to perform hardware functions and to transmit and receive data correctlq The purpose of the alignment test is to provide support to system alignment. This can be accomplished by execution of alignment procedures that correlate and display alignment parameters for operator assessment and calibration. The alignment parameters displayed should reflect performance of not only the equipment. but also the operational algorithm. Individual PM tests perform the equipment check function of the interface test. The PM software should: provide fault detection, defined as the performance of tests to ascertain that equipment and interfaces are petforming properly or improperly and to indicate its operatin@nonopel-ating status as “go/no-go” indication: allocate part of the scheduled operation to performance monitoring tests in a background mode run continuously under the system’s control: be nondisruptive to the operational system by executing concurrently with operational software on a time-shared basis without interruption in the availability and use of equipment and without damage to operational data: and report on the status of hardware operational functions. The system
FL software
failures
and affected
should:
provide fault isolation and identification of a failed component within the equipment indicated as nonfunctioning by PM on manual request: run concurrently
with the operational
program:
and
diagnosis equipment that is logically off line to the operational program, but perhaps still physically connected. The unit(s) would be dedicated to maintenance activities and the operational program would not have access to the equipment during this time. This system readiness concept supports scheduled preventative maintenance and corrective maintethe following This offer\ nance. approach advantages: System operational readiness is continually and status data available at all times.
assessed
System operators are familiar with the tests since they are an operational exercise which uses the actual equipments. programs. and procedures. Software interfaces. are tested.
as well as hardware
intelfaces.
Implementation and retrofit costs are greatly by use of the existing operational code. These tests can provide training requirements engineering.
reduced
a portion of the operational and tools for integration
REFERENCES H. 1). Mills. Software Engineering. .S(~i~,/xc~195 (42X3). I 199- 1205 (1977). C. L. McGowan and R. C. McHenry, Software Muagement. in Xo~c,rrrc./~ f~ir~c~c~iic~~~.\ i/r .So/f~~.cox, ‘~w/I/~~~/II,:‘?‘t E. Ditor. ed.). MIT Press, Cambridge. Massachu4etts. 1978. R. C. McHenry and J. A. Rand. Integration Engineering: An Approach to Rapid System Development. FSD 77. 0179. IBM Corporation. Gaithersburg. Maryland. 1977. R. C. McHenry and J. A. Rand. Software Technology and Integration Engineering. FSD 78-0034. IBM Corporation. Gaithersburg. Maryland. 1978.