- Email: [email protected]
An application of mathematical programming concepts to behavioural research design
Applied Ergonomics 1982, 13.4, 263-268
An application of mathematical programming concepts to behavioural research design K.R. Laughery, Jr. Calspan Corporation, USA
More and more frequently, human factors specialists are being asked to design behavioural research and evaluation techniques that will be applied many times to many different systems, and not always by behavioural scientists. One way to meet the repeatability requirement is to base evaluation technique design on the concepts of mathematical optimisation (eg, linear programming). This paper presents a general model for the application of mathematical programming concepts to behavioural research design, and an example of the use of this approach to design a simulator certification programme for the Strategic Air Command (SAC).
Keywords: Mathematical programming, simulation, behavioural research Because of the many cost overruns that have resulted from inattention to personnel related expenses, those who purchase systems have become increasingly aware of the importance of the human component to both training and human engineering design costs. Perhaps the most visible evidence of this new awareness in the US Federal government is Directive A-109 from the Office of Management and Budget (OMB). This directive requires that training and other personnel considerations be addressed during the earliest concept stages of systems acquisition and throughout the procurement process. If we are to supply accurate human factors costing information throughout the system acquisition process, we will have to develop standardised evaluation techniques (eg, experimental designs, performance measures) that are tied to specific phases of systems acquisition. It is imperative that systems designers be confident that the human will be able to perform as required at various stages of design development. Standardisation of human factors evaluation procedures is essential since it is impractical to assume that human factors personnel will always be available to assist in the design of every system or in the design of any one system at every stage of its development. Even if adequate personnel were available, it would not be cost-effective continually to re-invent analysis and evaluation techniques. In addition, the adoption of standardised approaches to evaluation would facilitate comparison of results and encourage the development of human factors/training guidelines on the basis of historical data. The US Department of Defense (DOD) has already recoguised the need to develop standardised techniques for analysing the behaviour of people using major training systems. The many DOD projects currently under way to achieve this goal include the Army's
Early Training Estimation System (ETES) and the Air Force's Simulator Certification programme. In the near future, one of the major jobs of the human factors professional will be to develop standardised procedures and techniques for evaluating the 'human side' of a system. These procedures will be applied over and over on different systems, frequently by personnel untrained in behavioural analysis and experimentation. The personnel using the human factors evaluation techniques may not be able to develop proper experimental procedures, controls, or data collection and analysis methods. Therefore, they will require specific steps to be performed in the preparation and execution of testing procedures. The development of these procedures will require that human factors professionals select from among the best available evaluative tools those that can be most effectively used by people having little or no experience in conducting human factors evaluations. The task of the human factors professional, therefore, will be to select and document the appropriate tools for any particular recurring evaluation problem. In selecting the appropriate evaluation procedures, the human factors professional must not simply select the ideal procedures. From a theoretical perspective, if he selects the ideal procedure, then the limitations of the actual testing environment may render that procedure infeasible. At that point, the test manager (who may have no human factors expertise) may be required to establish completely new evaluation procedures. Rather than selecting the ideal procedure, the human factors professional must select and document the best procedure given the constraints of the environment in which the human factors evaluation will be conducted. By considering these constraints, the evaluation procedures he selects and documents will be feasible and,
0003-6870/82/04 0263-06 $03.00O 1982Butterworth& Co (Publishers)Ltd
AppliedErgonomics December 1982
263
therefore, used in actual evaluations. It is the concept of 'selecting the best alternative within the real-world constraints' which leads us into the application of mathematical programming concepts to behavioural research design.
'=20 6C 5= . 5C
Background
4~
Mathematical programming techniques have been in use simce the 1940s. This paper will not discuss the details of the techniques. The interested reader should see Bazaraa and Jarvis (1977). However, to help the uninitiated reader understand how mathematical programming techniques can be applied to human factors evaluative study design, this paper discusses basic concepts of mathematical programming and gives one example of how these techniques can be used to solve a human factors problem. The key to solving a problem using mathematical programming is to formulate the problem correctly. The first step is to state the objectives quantitatively. This requires determining tradeoffs among attributes in the case of multi-attribute objectives. The second step is to identify resource constraints that might limit the extent to which the objectives can be satisfied. Both steps can be performed simultaneously. Once the problem is correctly stated, various techniques are available to identify the solution that maximises achievement of objectives while staying within the resource constraints.
Example: An aircrew training school has just completed preliminary testing of an aircrew training simulator, which they would like to incorporate into the training programme in a way that minimises the total cost of simulator and aircraft training. This is the objective. It has been determined that 1 h of aircraft training costs $2500 and 1 h of simulator training costs $500. We are, however, constrained because the student must receive at least the equivalent of 50 h of aircraft training; 2 h of simulator training are equivalent to 1 h of flight training (as determined during preliminary testing). Additionally, simulator time is limited by availability to no more than 20 h per student. Finally, because of concerns about the effect of an extreme reduction in flight training on crew member morale, a minimum of 30 h of flight training is required. What mix of simulator and flight training would minimise costs while staying within the above constraints? In this problem, we have two variables. Let us call them A and S, where A = Number of aircraft training hours, and S = Number of simulator training hours. Given these, we are trying to minimise the value of the following equation: 2500 A + 500 S = total cost of simulator and aircraft training subject to the following constraints: A < 30 (minimum flying hours constraint) S < 20 (maximum simulator availability constraint) A + 2S < 50 (minimum 'flight training equivalent' constraint) A > 0, S > 0 (we cannot have negative training hours). Rather than solving this problem through linear programming, let us explore it graphically in Fig. 1.
264
AppliedErgonomics
December 1982
}
4C 35
o
~iS
= 50 A =30
5C 25 20
Not enouqh aircraft traininq
15
,5
1
,5
2o
2'5
Simulator training hours ($)
Fig. 1
Graphical representationof simulator/aircraft training hours programme
Note that the only feasible solutions are those in the shaded area marked 'Feasible Region'. The lowest total cost of training within the feasible region is that associated with 2 h of simulator training and 40 h of aircraft training at a total cost of $110,000 per student. This can be ascertained by determining that any line parallel to the two solid lines marked P and Q represents a set of solutions with the same total cost. Through observation, we can see that the parallel line which passes through the point (S = 20, A = 40) represents the feasible solution having the least cost. This solution best satisfies the objectives while staying within the constraints.
Application of mathematical programming concepts to determining 'optimal' human factors evaluative procedures The example above shows how mathematical programming techniques may be used to solve specific human factors problems. This paper is not intended to describe how mathematical programming can be applied to all types of human factors problems, although quantitative approaches to optimisation (as in the above example) should be applied when appropriate. This paper focuses upon how these techniques can be applied to evaluative research methodology design. In many ways, the problem of selecting the best human factors evaluative techniques is analogous to the mathematical optimisation problem. First, the evaluative procedure should have clearly defined goals or objectives: ie, "What are we trying to find out about the system at this particular phase of design?" Second, there are many constraints affecting the choice of technique. For example, what experimental techniques are available, what types of resources (personnel and material) will be available to conduct these evaluations, how much time will be allowed to conduct the evaluation, and so on. When an evaluative technique is to be applied many times, inattention to these constraints can lead to selecting a technique that
is ineffective. Once objectives and constraints are identified, we can select the one technique that best satisfies the objectives from among the available techniques that appear feasible. Not all human factors problems can be formulated in quantitative terms. However, at a minimum, the concepts can be rigorously applied to help ensure that the product, an evaluative technique for human factors analysis, is the best one available. The following paragraphs define specific steps in the development of evaluative techniques based on the concepts of mathematical programming. 1. Identify objectives In many cases, the questions that the human factors analysis is intended to answer are ill-defined. For example, if we want to evaluate tasks performed by the human that are critical to mission success, should we consider normal system operation, emergency operation, or both? These are largely management decisions, although human factors personnel may better recognise what the study objectives might be. The selection of objectives for the evaluation will involve discussions with personnel involved in the systems acquisition process, as well as an examination of regulations requiring the evaluation and available publications explaining the requirements. Most evaluations involve several goals. Ideally, decision analysis techniques (eg, multiattribute utility-function development) should be applied. At a minimum, interviews with decision makers should allow analysts to determine which study objectives are important at the specific phase of systems acquisition in which the evaluative technique will be used. 2. Determine constraints At this point, two subquestions must be addressed: (1) what techniques are available, and (2) what resources will be available to conduct the evaluation? A determination of available techniques is, basically, a literature review. It may also involve discussing the requirements with personnel experienced in conducting studies with similar objectives. The total set of available techniques will define the 'problem solution space'. However, not all techniques will be feasible for a particular type of evaulation because of resource constraints. Some may require too much time, experienced personnel that are not available, etc. The technique selected must be practical for the environment in which it will be used (see Payne, 1978). Determining resource constraints requires several steps. First, we must determine what classes of resources may be required. This can be done by examining the techniques found in the literature and listing the classes of resources required for each technique (eg, types of equipment, performance measures, computational requirements, personnel expertise). Secondly, we must examine all potential sources of information to determine how these resources may be constrained. A visit to some of the agencies that will conduct the evaluations is essential. Discussions with individuals in these organisations will provide insight into what can and cannot be done. Another approach is to develop 'scenarios' of how the evaluation might be conducted. Each scenario would describe what the evaluation process would entail in terms of the resources and steps involved. The scenarios would be presented to appropriate individuals (eg, those who will conduct the evaluation, supervisors) for review and comment. Scenarios have the advantage of stating requirements within an appropriate context.
3. Select the best feasible alternative After defining what we wish to achieve and identifying our alternatives for achieving those goals that are within the resource constraints, we must select the technique that best satisfies those goals. In many cases, it is feasible to establish direct links between resources consumed and the extent to which the objectives are met. Resource classes and constraints can be translated into variable types with upper and lower bounds. The satisfaction of objectives can be linked to resources consumed by the various alternative evaluation procedures. In these instances, mathematical progamming may be employed. However, many of the alternative techniques may not lend themselves to such quantitative relationships with the objectives (eg, a fully factoral design v s a central composite design). Even in these circumstances, the technique that best satisfies the objectives can usually be selected through an examination of available literature. One should expect that there will be alternative courses of action that do not have clear implications with respect to the objectives. In these cases, expert judgement must be applied at this point. Remember that the human factors specialist's judgement applied at this point is probably better than the judgement of a technician attempting to apply this evaluative technique at a later time. 4. Validate remits The old computer adage 'garbage in, garbage out' applies to this approach to evaluative technique development. We are only as sure of our results as we are of our assumptions (ie, objectives and constraints). Consequently, if there is significant uncertainty about the assumptions that we have developed, a validation should be conducted. The best evaluation of the technique would be its application on a system. At the very least, the technique should be presented to the agencies who will apply it to ensure that they feel confident that it meets their needs and that they are capable of performing it.
Application of this approach to the development of a simulator certification (SIMCERT) methodology Background This approach was used to design a Simulator Certification (SIMCERT) programme for the US Strategic Air Command (SAC) of the US Air Force. The initial requirement for conducting SIMCERT was identified in Air Force Regulation AFR 50-11. This regulation mandates that all simulators delivered to the Air Force initially be certified during the operational test and evaluationphase of acquisition. AFR 50-11 defines SIMCERT as "The process of verifying the specific aircrew tasks that can be (a) effectively trained in an aircrew training device, and (b) credited toward training requirements as established in (appropriate) publications". In short, SIMCERT seeks to determine how each simulator facilitates the learning of specific skills and permits valid performance assessment of those skills. Each Air Force command was required to publish a supplement to AFR 50-11 setting up a SIMCERT process that meets its specific needs. In May 1979, Calspan Corporation and the 93rd Bombardment Wing began to develop a SIMCERT programme that would be used repeatedly on all SAC
Applied Ergonomics
December 1982
265
simulators. The task was to develop the evaluative procedures to be used by SAC personnel, not to conduct the studies per se. The mathematical programming approach used to develop these procedures is described below. 1. Identify objectives of SIMCERT While AFR 50-11 generally defined SIMCERT objectives, there was much room for interpretation. Therefore, it was considered essential to define concrete objectives clearly so that the SIMCERT methodology could be designed to address these objectives specifically. The approach used to determine the appropriate SAC SIMCERT objectives was to delineate all reasonable alternatives as shown in Table 1, and then present them to the appropriate management decision makers within SAC for review.
As a result of this process, it was decided that any objective addressing issues related to optimising the training system were not to be a part of SAC SIMCERT. Instead, SIMCERT would be the examination of the training device, or set of devices, as currently fixed within a given instructional system. The purpose of SAC SIMCERT is to ensure that an existing training device facilitates adequate training and adequate student performance evaluation within a given training system, not to aid in system development. The observation was by no means obvious at the outset of the study. It was decided that objectives 1, 7 and 9 (Table 1) were the questions that the SIMCERT process should answer. 2. Determine constraints on performing SI MCE R T
Next, two subquestions were posed: (1) what techniques were available for satisfying the objectives of the SIMCERT programme, and (2) to what extent would resources be available to perform SIMCERT. The available techniques were determined through a literature review in areas relevant Table 1: Candidate objectives of SIMCERT 1.
Does device permit training transfer of objectives for which it was designed?
to simulator test and evaluation (Ditzian and Laughery, 1979). Many techniques for evaluating simulator transfer of training, retention issues, and fidelity assessments were unearthed. However, available SAC resources limited the techniques that could be applied. The resources that would be limited and the extent of the limitations were determined in three steps. First, the available techniques were examined to determine categories of resources that might be required. The result of this analysis is displayed in Table 2. Second, three senarios of how SIMCERT could be conducted were developed and presented to appropriate members of the SAC training community for review and comment. Each of these scenarios described a potential approach to SIMCERT with emphasis on the resources that would be required to conduct SIMCERT. Each of the scenarios presented different resource requirements. Third, the comments made by SAC personnel were reviewed to determine where and how the various resources would be constrained. As previously stated, the advantage of using scenarios is that the requirements of the evaluation process are seen within an appropriate context. A statement of the degree to which the various resources were constrained is included in Table 2. For a more complete description, the reader is referred to Laughery and Ditzian (1980a).
3. Selecting the best approach to SIMCERT
It was now reasonably clear which techniques were feasible for SIMCERT. Those techniques which best satisfied the objectives, as determined through the literature review and subjective evaluations, were combined to form the methodology that was recommended to SAC. A six step methodology was selected. The first three steps were performed early in the simulator's life cycle, after all preliminary tests had been conducted. These three steps, termed 'initial certification', sought to ensure adequate transfer of training and appropriate use of the simulator as a performance measurement device (objectives 1 and 7 from Table 1). These three steps are as follows: 1. Document the simulator's physical fidelity.
2.
What modifications can be made on the device to permit successful transfer of learning of the tasks for which it was designed? (eg, added features, etc).
2. Document the way in which the simulator is used in the instructional system (eg, number of hours per training objective).
3.
What modifications (hardware/software) could be made to the device to reduce training time on the trainer?
3. Conduct a transfer of training and performance measurement validation study.
4.
What modifications in the course syllabus will increase the training effectiveness of the training device?
5.
What additional objectives can be taught in the device (besides those for which it was originally intended)?
6.
In what ways is the device 'over designed'?
7.
Can the student be evaluated entirely in a training device or is operational evaluation required (ie, aircraft evaluation)?
8.
How useful are the instructional features of the training device?
9.
How does performance degrade over time with simulator training vs aircraft only training?
10.
266
What will be the instructor's attitude towards the simulator?
Applied Ergonomics
December 1982
Since the SIMCERT process was not intended to be prescriptive regarding ways to improve transfer of training, explanations of why poor transfer had occurred did oot need to be included. SIMCERT was to be descriptive: the simulator provided transfer and valid performance assessment or it did not. Prescriptive data had to be gathered and utilised prior to the SIMCERT process. If during SIMCERT the simulator was found to provide positive transfer and/or was found to be an appropriate performance measurement tool for the training objectives studied, then it was certified; otherwise it was not. In establishing the objectives (Step 1 above), this was deemed appropriate. The structure of Step 3 (the transfer study) was such that the consumption of constrained resources (Table 2) was minimised. For further details, see Laughery and Ditzian (1980b). The documentation of fidelity and the way in which the simulator was used were not explicit objectives of SIMCERT.
2. Monitor the way in which the simulator is being used in the instructional system.
Table 2: Resources required for SIMCERT
Degree of constraint 1.
Additional instructor time required to conduct study.
H
2.
Technician time to set up simulator.
P
3.
ISD personnel time to generate performance measures of training proficiency.
P
4.
Scientific personnel time to establish test parameters (eg, number of subjects/group).
U
5.
6.
ISD personnel time to generate specific training objectives to be taught in the training device. Secretary/keypunch time support to encode data.
P U
7.
Secretary/typist time to type report.
U
8.
Test director time to ensure testing is conducted in a proper and timely manner.
P
Support personnel time to evaluate/encode data.
U
10.
Scientific personnel time to interpret analysis.
U
11.
Additional student time for training required by test.
P
Additional simulator training time required to conduct study.
U
Simulator time for set-up (eg, testing, inputting performance measures).
P
Additional aircraft training required to conduct study.
H
Aircraft availability for reconfiguration for the study.
H
16.
Computer to analyse data
P
17.
Travel costs for scientific support.
U
18.
Data storage medium (eg, computer cards, mag cards).
P
9.
12. 13. 14. 15.
H = Highly constrained P = Partially constrained U = Virtually unconstrained
Rather, these steps were intended to provide baseline data for the second phase of SIMCERT, recurrent evaluation. This was to be conducted throughout the simulator's lifespan to ensure the continuation of training transfer (Objective 1), valid performance measurement (Objective 7), and to discover any long-term degradation of performance attributable to the use of the simulator (Objective 9). The resource constraints analysis indicated that the most limited resources were those which were required to conduct transfer studies (eg, limited additional instructor time, limited aircraft study time, etc). Therefore, it would have been impossible to re-perform the transfer study on a regular basis to ensure continuing transfer and performance measurement validity. Rather, it was decided that a feasible approach to satisfying these objectives would be by the following three steps of recurrent evaluation. 1. Monitor simulator fidelity.
3. Monitor the performance of crew members who have been trained in the simulator. These steps would be performed on a continuing basis after initial certification. The underlying assumption was that if fidelity and the way in which the simulator was used did not change during recurrent evaluation from the baseline observed during initial certification, then we would reasonably assume that transfer remained at the level observed during initial certification. If either of these two factors changed, then either corrections were made to return to initial certification standards or the simulator was de-certified for training. However, since simulator usage practices and fidelity do not account for all variability in simulator training transfer, the final step of monitoring crew member performance (Step 3 of recurrent evaluation) was included as a cross-check, It was found during the resource constraints analysis that this was performed routinely and that the data could be incorporated into the SIMCERT process with almost no additional resource costs. A flowchart of the entire SIMCERT process is included in Fig. 2. The net result was a proposed methodology which best satisfied the objectives of SIMCERT while staying within real-world constraints. Better methods could have been proposed (eg, recurring transfer studies) which would have been useless. Good methods could have been proposed which were aimed at undesired study objectives. However, because of the mathematical programming conceptual structure of methodology design, the desired median was found. 4. Validation of proposed methodology The proposed methodology was reviewed by members of the SAC training community. Their initial comments were favourable with respect to its feasibility. However, since the SIMCERT methodology was to be used on many simulators, including some which already existed, it was possible to apply the methodology and empirically to determine its feasibility. It was applied to three simulators at Castle AFB, California. Some minor changes to the certification process were identified as the result of this validation process, but the fundamental approach to SIMCERT has remained the same. The development process had succeeded in producing a methodology that met the needs of the SIMCERT programme.
Discussion When mathematical programming concepts were applied to the design of a SIMCERT programme, the result was a markedly different programme than the author had originally conceived. First, the objectives of the SIMCERT process were initially perceived to be much broader in scope than those that were f'mally selected. Second, many of the constraints that were originally anticipated turned out to be non-existent or not as extreme as expected. Third, unanticipated constraints surfaced. If the SIMCERT programme had been developed without determining applicable objectives and constraints, the result would have been a SIMCERT programme that attempted to answer more questions than needed. The focusing of the effort resulted in a much more powerful simulator certification tool.
Applied Ergonomics
December 1982
267
I
w
I E
Has NO~,~performance,,,. "~. deteri~ated J" signi antly
I l
~
I t I
II
I Asses~
-
Jl
It';a%',*.',°' ~ of trainiFR I I m~remen, oralpertcemqnce
Hove ~'~ • ' _iF r ~.occ4lptoi01ef limits
1"7 ~i'~'~if'~;v' I -
', I fidehty L,.c,
I
;' ] I I I I l
.
II '1
Yes
IYe%e ~
--to°.a,
M o n i t o ¢ Oll~licmion simu~oto¢ troinl~. Il J .- - oecl~curt~ ~ " 7 in the il~structiollo I'--~"% m the meo~s ~ system "with which /
I .,.m,ne I and events
/ ~:~;;~
Document simulator
\
traini~ a~o,icotion
with.respect ~
Jforthesetosks
tarne . . l de=t~rmmea ] T_o_us_ __. /
Fig. 2
j I
I
1
• •" Assess s*n~otor transfer of troiflin(J for mese events
I I ~ I
~ ~ /C~ertify N / simuldtol \ [ !or.aPpropriate ) | toslL~ ~ -' - on-(J'event¢ / ~'-'"'"" /
Recurrent evoJuatian
Graphical representation of the SIMCERT process
Conclusion
When developing evaluative techniques, the task of the human factors specialist is frequently not to find the 'perfect' techniques. Instead, the task is to find the optimal techniques that (1) satisfy the objectives of the evaluation, and (2) can be performed by the personnel in the environment where they will be used. This is no easy task. The concepts of mathematical programming provide useful guidelines to the design of these evaluative techniques. When properly applied, they can be expected to result in a product that effectively serves the ultimate users of the evaluative techniques. Acknowledgements
This work was supported in part by the Strategic Air Command under Contract No F33657-78-C-0491. The author also acknowledges Dr Jan L. Ditzian and Major George M. Houtman for their assistance in the Simulator Certification programme development discussed herein. References
Air Force Regulation 1977 AFR 50-11 'Management and utilisation of training devices', Department of the Air Force, HQ USAF, Washington, DC, October.
268
I
Of utiliSation hove changed
I Initial certificotion
Monitor performance of traine~s who have bge.n tralqed I wlth the slmulatar
fqr which means
whiCh I°*ea' teOI
[_~inodequacies,,,..,=/ worth ~ I correcting ~ ~ ~,~ \ ~
o
I
Applied Ergonomics
December 1982
Bazaraa, M.S., and Jaxvis, J.J. 1977 'Linear programming and network flows'. John Wiley and Sons, New York, NY. Ditzian, J.L., and Laughery, K.R. 1979 SAC Simulator Certification - Literature Review (Calspan T.M. 79-14-2.1). Calspan Corporation, Buffalo, NY: Ca~pan Corporation, August. Laughery, K.R., and Ditzian, J.L. 1980a SAC Simulator Certification - Objectives and Constraints (Calspan T.M. 79-14- 2.2). Buffalo, NY: Calspan Corporation, February. Langhery, K.R., and Ditzian, J.U 1980b SAC Simulator Certification - Candidate Methodologies (Calspan T.M. 79-14.2.2). Buffalo, NY: Calspan Corporation, February. Laughery, K.R., and Houtman, G.M. 1980 'Development of a Simulator Certification Methodology for the Strategic Air Command', Proceedings of the Seventh Annual Psychology in the Department of Defense Symposium, USAF Academy, CO, April.
Payne, T.A. 1978 'A practical guide to conduct of studies of transfer of learning', Draft report prepared for the Air Force Human Resources Laboratory, Flying Training Division, Williams AFB, AZ, April.
An application of mathematical programming concepts to behavioural research design K.R. Laughery, Jr. Calspan Corporation, USA
More and more frequently, human factors specialists are being asked to design behavioural research and evaluation techniques that will be applied many times to many different systems, and not always by behavioural scientists. One way to meet the repeatability requirement is to base evaluation technique design on the concepts of mathematical optimisation (eg, linear programming). This paper presents a general model for the application of mathematical programming concepts to behavioural research design, and an example of the use of this approach to design a simulator certification programme for the Strategic Air Command (SAC).
Keywords: Mathematical programming, simulation, behavioural research Because of the many cost overruns that have resulted from inattention to personnel related expenses, those who purchase systems have become increasingly aware of the importance of the human component to both training and human engineering design costs. Perhaps the most visible evidence of this new awareness in the US Federal government is Directive A-109 from the Office of Management and Budget (OMB). This directive requires that training and other personnel considerations be addressed during the earliest concept stages of systems acquisition and throughout the procurement process. If we are to supply accurate human factors costing information throughout the system acquisition process, we will have to develop standardised evaluation techniques (eg, experimental designs, performance measures) that are tied to specific phases of systems acquisition. It is imperative that systems designers be confident that the human will be able to perform as required at various stages of design development. Standardisation of human factors evaluation procedures is essential since it is impractical to assume that human factors personnel will always be available to assist in the design of every system or in the design of any one system at every stage of its development. Even if adequate personnel were available, it would not be cost-effective continually to re-invent analysis and evaluation techniques. In addition, the adoption of standardised approaches to evaluation would facilitate comparison of results and encourage the development of human factors/training guidelines on the basis of historical data. The US Department of Defense (DOD) has already recoguised the need to develop standardised techniques for analysing the behaviour of people using major training systems. The many DOD projects currently under way to achieve this goal include the Army's
Early Training Estimation System (ETES) and the Air Force's Simulator Certification programme. In the near future, one of the major jobs of the human factors professional will be to develop standardised procedures and techniques for evaluating the 'human side' of a system. These procedures will be applied over and over on different systems, frequently by personnel untrained in behavioural analysis and experimentation. The personnel using the human factors evaluation techniques may not be able to develop proper experimental procedures, controls, or data collection and analysis methods. Therefore, they will require specific steps to be performed in the preparation and execution of testing procedures. The development of these procedures will require that human factors professionals select from among the best available evaluative tools those that can be most effectively used by people having little or no experience in conducting human factors evaluations. The task of the human factors professional, therefore, will be to select and document the appropriate tools for any particular recurring evaluation problem. In selecting the appropriate evaluation procedures, the human factors professional must not simply select the ideal procedures. From a theoretical perspective, if he selects the ideal procedure, then the limitations of the actual testing environment may render that procedure infeasible. At that point, the test manager (who may have no human factors expertise) may be required to establish completely new evaluation procedures. Rather than selecting the ideal procedure, the human factors professional must select and document the best procedure given the constraints of the environment in which the human factors evaluation will be conducted. By considering these constraints, the evaluation procedures he selects and documents will be feasible and,
0003-6870/82/04 0263-06 $03.00O 1982Butterworth& Co (Publishers)Ltd
AppliedErgonomics December 1982
263
therefore, used in actual evaluations. It is the concept of 'selecting the best alternative within the real-world constraints' which leads us into the application of mathematical programming concepts to behavioural research design.
'=20 6C 5= . 5C
Background
4~
Mathematical programming techniques have been in use simce the 1940s. This paper will not discuss the details of the techniques. The interested reader should see Bazaraa and Jarvis (1977). However, to help the uninitiated reader understand how mathematical programming techniques can be applied to human factors evaluative study design, this paper discusses basic concepts of mathematical programming and gives one example of how these techniques can be used to solve a human factors problem. The key to solving a problem using mathematical programming is to formulate the problem correctly. The first step is to state the objectives quantitatively. This requires determining tradeoffs among attributes in the case of multi-attribute objectives. The second step is to identify resource constraints that might limit the extent to which the objectives can be satisfied. Both steps can be performed simultaneously. Once the problem is correctly stated, various techniques are available to identify the solution that maximises achievement of objectives while staying within the resource constraints.
Example: An aircrew training school has just completed preliminary testing of an aircrew training simulator, which they would like to incorporate into the training programme in a way that minimises the total cost of simulator and aircraft training. This is the objective. It has been determined that 1 h of aircraft training costs $2500 and 1 h of simulator training costs $500. We are, however, constrained because the student must receive at least the equivalent of 50 h of aircraft training; 2 h of simulator training are equivalent to 1 h of flight training (as determined during preliminary testing). Additionally, simulator time is limited by availability to no more than 20 h per student. Finally, because of concerns about the effect of an extreme reduction in flight training on crew member morale, a minimum of 30 h of flight training is required. What mix of simulator and flight training would minimise costs while staying within the above constraints? In this problem, we have two variables. Let us call them A and S, where A = Number of aircraft training hours, and S = Number of simulator training hours. Given these, we are trying to minimise the value of the following equation: 2500 A + 500 S = total cost of simulator and aircraft training subject to the following constraints: A < 30 (minimum flying hours constraint) S < 20 (maximum simulator availability constraint) A + 2S < 50 (minimum 'flight training equivalent' constraint) A > 0, S > 0 (we cannot have negative training hours). Rather than solving this problem through linear programming, let us explore it graphically in Fig. 1.
264
AppliedErgonomics
December 1982
}
4C 35
o
~iS
= 50 A =30
5C 25 20
Not enouqh aircraft traininq
15
,5
1
,5
2o
2'5
Simulator training hours ($)
Fig. 1
Graphical representationof simulator/aircraft training hours programme
Note that the only feasible solutions are those in the shaded area marked 'Feasible Region'. The lowest total cost of training within the feasible region is that associated with 2 h of simulator training and 40 h of aircraft training at a total cost of $110,000 per student. This can be ascertained by determining that any line parallel to the two solid lines marked P and Q represents a set of solutions with the same total cost. Through observation, we can see that the parallel line which passes through the point (S = 20, A = 40) represents the feasible solution having the least cost. This solution best satisfies the objectives while staying within the constraints.
Application of mathematical programming concepts to determining 'optimal' human factors evaluative procedures The example above shows how mathematical programming techniques may be used to solve specific human factors problems. This paper is not intended to describe how mathematical programming can be applied to all types of human factors problems, although quantitative approaches to optimisation (as in the above example) should be applied when appropriate. This paper focuses upon how these techniques can be applied to evaluative research methodology design. In many ways, the problem of selecting the best human factors evaluative techniques is analogous to the mathematical optimisation problem. First, the evaluative procedure should have clearly defined goals or objectives: ie, "What are we trying to find out about the system at this particular phase of design?" Second, there are many constraints affecting the choice of technique. For example, what experimental techniques are available, what types of resources (personnel and material) will be available to conduct these evaluations, how much time will be allowed to conduct the evaluation, and so on. When an evaluative technique is to be applied many times, inattention to these constraints can lead to selecting a technique that
is ineffective. Once objectives and constraints are identified, we can select the one technique that best satisfies the objectives from among the available techniques that appear feasible. Not all human factors problems can be formulated in quantitative terms. However, at a minimum, the concepts can be rigorously applied to help ensure that the product, an evaluative technique for human factors analysis, is the best one available. The following paragraphs define specific steps in the development of evaluative techniques based on the concepts of mathematical programming. 1. Identify objectives In many cases, the questions that the human factors analysis is intended to answer are ill-defined. For example, if we want to evaluate tasks performed by the human that are critical to mission success, should we consider normal system operation, emergency operation, or both? These are largely management decisions, although human factors personnel may better recognise what the study objectives might be. The selection of objectives for the evaluation will involve discussions with personnel involved in the systems acquisition process, as well as an examination of regulations requiring the evaluation and available publications explaining the requirements. Most evaluations involve several goals. Ideally, decision analysis techniques (eg, multiattribute utility-function development) should be applied. At a minimum, interviews with decision makers should allow analysts to determine which study objectives are important at the specific phase of systems acquisition in which the evaluative technique will be used. 2. Determine constraints At this point, two subquestions must be addressed: (1) what techniques are available, and (2) what resources will be available to conduct the evaluation? A determination of available techniques is, basically, a literature review. It may also involve discussing the requirements with personnel experienced in conducting studies with similar objectives. The total set of available techniques will define the 'problem solution space'. However, not all techniques will be feasible for a particular type of evaulation because of resource constraints. Some may require too much time, experienced personnel that are not available, etc. The technique selected must be practical for the environment in which it will be used (see Payne, 1978). Determining resource constraints requires several steps. First, we must determine what classes of resources may be required. This can be done by examining the techniques found in the literature and listing the classes of resources required for each technique (eg, types of equipment, performance measures, computational requirements, personnel expertise). Secondly, we must examine all potential sources of information to determine how these resources may be constrained. A visit to some of the agencies that will conduct the evaluations is essential. Discussions with individuals in these organisations will provide insight into what can and cannot be done. Another approach is to develop 'scenarios' of how the evaluation might be conducted. Each scenario would describe what the evaluation process would entail in terms of the resources and steps involved. The scenarios would be presented to appropriate individuals (eg, those who will conduct the evaluation, supervisors) for review and comment. Scenarios have the advantage of stating requirements within an appropriate context.
3. Select the best feasible alternative After defining what we wish to achieve and identifying our alternatives for achieving those goals that are within the resource constraints, we must select the technique that best satisfies those goals. In many cases, it is feasible to establish direct links between resources consumed and the extent to which the objectives are met. Resource classes and constraints can be translated into variable types with upper and lower bounds. The satisfaction of objectives can be linked to resources consumed by the various alternative evaluation procedures. In these instances, mathematical progamming may be employed. However, many of the alternative techniques may not lend themselves to such quantitative relationships with the objectives (eg, a fully factoral design v s a central composite design). Even in these circumstances, the technique that best satisfies the objectives can usually be selected through an examination of available literature. One should expect that there will be alternative courses of action that do not have clear implications with respect to the objectives. In these cases, expert judgement must be applied at this point. Remember that the human factors specialist's judgement applied at this point is probably better than the judgement of a technician attempting to apply this evaluative technique at a later time. 4. Validate remits The old computer adage 'garbage in, garbage out' applies to this approach to evaluative technique development. We are only as sure of our results as we are of our assumptions (ie, objectives and constraints). Consequently, if there is significant uncertainty about the assumptions that we have developed, a validation should be conducted. The best evaluation of the technique would be its application on a system. At the very least, the technique should be presented to the agencies who will apply it to ensure that they feel confident that it meets their needs and that they are capable of performing it.
Application of this approach to the development of a simulator certification (SIMCERT) methodology Background This approach was used to design a Simulator Certification (SIMCERT) programme for the US Strategic Air Command (SAC) of the US Air Force. The initial requirement for conducting SIMCERT was identified in Air Force Regulation AFR 50-11. This regulation mandates that all simulators delivered to the Air Force initially be certified during the operational test and evaluationphase of acquisition. AFR 50-11 defines SIMCERT as "The process of verifying the specific aircrew tasks that can be (a) effectively trained in an aircrew training device, and (b) credited toward training requirements as established in (appropriate) publications". In short, SIMCERT seeks to determine how each simulator facilitates the learning of specific skills and permits valid performance assessment of those skills. Each Air Force command was required to publish a supplement to AFR 50-11 setting up a SIMCERT process that meets its specific needs. In May 1979, Calspan Corporation and the 93rd Bombardment Wing began to develop a SIMCERT programme that would be used repeatedly on all SAC
Applied Ergonomics
December 1982
265
simulators. The task was to develop the evaluative procedures to be used by SAC personnel, not to conduct the studies per se. The mathematical programming approach used to develop these procedures is described below. 1. Identify objectives of SIMCERT While AFR 50-11 generally defined SIMCERT objectives, there was much room for interpretation. Therefore, it was considered essential to define concrete objectives clearly so that the SIMCERT methodology could be designed to address these objectives specifically. The approach used to determine the appropriate SAC SIMCERT objectives was to delineate all reasonable alternatives as shown in Table 1, and then present them to the appropriate management decision makers within SAC for review.
As a result of this process, it was decided that any objective addressing issues related to optimising the training system were not to be a part of SAC SIMCERT. Instead, SIMCERT would be the examination of the training device, or set of devices, as currently fixed within a given instructional system. The purpose of SAC SIMCERT is to ensure that an existing training device facilitates adequate training and adequate student performance evaluation within a given training system, not to aid in system development. The observation was by no means obvious at the outset of the study. It was decided that objectives 1, 7 and 9 (Table 1) were the questions that the SIMCERT process should answer. 2. Determine constraints on performing SI MCE R T
Next, two subquestions were posed: (1) what techniques were available for satisfying the objectives of the SIMCERT programme, and (2) to what extent would resources be available to perform SIMCERT. The available techniques were determined through a literature review in areas relevant Table 1: Candidate objectives of SIMCERT 1.
Does device permit training transfer of objectives for which it was designed?
to simulator test and evaluation (Ditzian and Laughery, 1979). Many techniques for evaluating simulator transfer of training, retention issues, and fidelity assessments were unearthed. However, available SAC resources limited the techniques that could be applied. The resources that would be limited and the extent of the limitations were determined in three steps. First, the available techniques were examined to determine categories of resources that might be required. The result of this analysis is displayed in Table 2. Second, three senarios of how SIMCERT could be conducted were developed and presented to appropriate members of the SAC training community for review and comment. Each of these scenarios described a potential approach to SIMCERT with emphasis on the resources that would be required to conduct SIMCERT. Each of the scenarios presented different resource requirements. Third, the comments made by SAC personnel were reviewed to determine where and how the various resources would be constrained. As previously stated, the advantage of using scenarios is that the requirements of the evaluation process are seen within an appropriate context. A statement of the degree to which the various resources were constrained is included in Table 2. For a more complete description, the reader is referred to Laughery and Ditzian (1980a).
3. Selecting the best approach to SIMCERT
It was now reasonably clear which techniques were feasible for SIMCERT. Those techniques which best satisfied the objectives, as determined through the literature review and subjective evaluations, were combined to form the methodology that was recommended to SAC. A six step methodology was selected. The first three steps were performed early in the simulator's life cycle, after all preliminary tests had been conducted. These three steps, termed 'initial certification', sought to ensure adequate transfer of training and appropriate use of the simulator as a performance measurement device (objectives 1 and 7 from Table 1). These three steps are as follows: 1. Document the simulator's physical fidelity.
2.
What modifications can be made on the device to permit successful transfer of learning of the tasks for which it was designed? (eg, added features, etc).
2. Document the way in which the simulator is used in the instructional system (eg, number of hours per training objective).
3.
What modifications (hardware/software) could be made to the device to reduce training time on the trainer?
3. Conduct a transfer of training and performance measurement validation study.
4.
What modifications in the course syllabus will increase the training effectiveness of the training device?
5.
What additional objectives can be taught in the device (besides those for which it was originally intended)?
6.
In what ways is the device 'over designed'?
7.
Can the student be evaluated entirely in a training device or is operational evaluation required (ie, aircraft evaluation)?
8.
How useful are the instructional features of the training device?
9.
How does performance degrade over time with simulator training vs aircraft only training?
10.
266
What will be the instructor's attitude towards the simulator?
Applied Ergonomics
December 1982
Since the SIMCERT process was not intended to be prescriptive regarding ways to improve transfer of training, explanations of why poor transfer had occurred did oot need to be included. SIMCERT was to be descriptive: the simulator provided transfer and valid performance assessment or it did not. Prescriptive data had to be gathered and utilised prior to the SIMCERT process. If during SIMCERT the simulator was found to provide positive transfer and/or was found to be an appropriate performance measurement tool for the training objectives studied, then it was certified; otherwise it was not. In establishing the objectives (Step 1 above), this was deemed appropriate. The structure of Step 3 (the transfer study) was such that the consumption of constrained resources (Table 2) was minimised. For further details, see Laughery and Ditzian (1980b). The documentation of fidelity and the way in which the simulator was used were not explicit objectives of SIMCERT.
2. Monitor the way in which the simulator is being used in the instructional system.
Table 2: Resources required for SIMCERT
Degree of constraint 1.
Additional instructor time required to conduct study.
H
2.
Technician time to set up simulator.
P
3.
ISD personnel time to generate performance measures of training proficiency.
P
4.
Scientific personnel time to establish test parameters (eg, number of subjects/group).
U
5.
6.
ISD personnel time to generate specific training objectives to be taught in the training device. Secretary/keypunch time support to encode data.
P U
7.
Secretary/typist time to type report.
U
8.
Test director time to ensure testing is conducted in a proper and timely manner.
P
Support personnel time to evaluate/encode data.
U
10.
Scientific personnel time to interpret analysis.
U
11.
Additional student time for training required by test.
P
Additional simulator training time required to conduct study.
U
Simulator time for set-up (eg, testing, inputting performance measures).
P
Additional aircraft training required to conduct study.
H
Aircraft availability for reconfiguration for the study.
H
16.
Computer to analyse data
P
17.
Travel costs for scientific support.
U
18.
Data storage medium (eg, computer cards, mag cards).
P
9.
12. 13. 14. 15.
H = Highly constrained P = Partially constrained U = Virtually unconstrained
Rather, these steps were intended to provide baseline data for the second phase of SIMCERT, recurrent evaluation. This was to be conducted throughout the simulator's lifespan to ensure the continuation of training transfer (Objective 1), valid performance measurement (Objective 7), and to discover any long-term degradation of performance attributable to the use of the simulator (Objective 9). The resource constraints analysis indicated that the most limited resources were those which were required to conduct transfer studies (eg, limited additional instructor time, limited aircraft study time, etc). Therefore, it would have been impossible to re-perform the transfer study on a regular basis to ensure continuing transfer and performance measurement validity. Rather, it was decided that a feasible approach to satisfying these objectives would be by the following three steps of recurrent evaluation. 1. Monitor simulator fidelity.
3. Monitor the performance of crew members who have been trained in the simulator. These steps would be performed on a continuing basis after initial certification. The underlying assumption was that if fidelity and the way in which the simulator was used did not change during recurrent evaluation from the baseline observed during initial certification, then we would reasonably assume that transfer remained at the level observed during initial certification. If either of these two factors changed, then either corrections were made to return to initial certification standards or the simulator was de-certified for training. However, since simulator usage practices and fidelity do not account for all variability in simulator training transfer, the final step of monitoring crew member performance (Step 3 of recurrent evaluation) was included as a cross-check, It was found during the resource constraints analysis that this was performed routinely and that the data could be incorporated into the SIMCERT process with almost no additional resource costs. A flowchart of the entire SIMCERT process is included in Fig. 2. The net result was a proposed methodology which best satisfied the objectives of SIMCERT while staying within real-world constraints. Better methods could have been proposed (eg, recurring transfer studies) which would have been useless. Good methods could have been proposed which were aimed at undesired study objectives. However, because of the mathematical programming conceptual structure of methodology design, the desired median was found. 4. Validation of proposed methodology The proposed methodology was reviewed by members of the SAC training community. Their initial comments were favourable with respect to its feasibility. However, since the SIMCERT methodology was to be used on many simulators, including some which already existed, it was possible to apply the methodology and empirically to determine its feasibility. It was applied to three simulators at Castle AFB, California. Some minor changes to the certification process were identified as the result of this validation process, but the fundamental approach to SIMCERT has remained the same. The development process had succeeded in producing a methodology that met the needs of the SIMCERT programme.
Discussion When mathematical programming concepts were applied to the design of a SIMCERT programme, the result was a markedly different programme than the author had originally conceived. First, the objectives of the SIMCERT process were initially perceived to be much broader in scope than those that were f'mally selected. Second, many of the constraints that were originally anticipated turned out to be non-existent or not as extreme as expected. Third, unanticipated constraints surfaced. If the SIMCERT programme had been developed without determining applicable objectives and constraints, the result would have been a SIMCERT programme that attempted to answer more questions than needed. The focusing of the effort resulted in a much more powerful simulator certification tool.
Applied Ergonomics
December 1982
267
I
w
I E
Has NO~,~performance,,,. "~. deteri~ated J" signi antly
I l
~
I t I
II
I Asses~
-
Jl
It';a%',*.',°' ~ of trainiFR I I m~remen, oralpertcemqnce
Hove ~'~ • ' _iF r ~.occ4lptoi01ef limits
1"7 ~i'~'~if'~;v' I -
', I fidehty L,.c,
I
;' ] I I I I l
.
II '1
Yes
IYe%e ~
--to°.a,
M o n i t o ¢ Oll~licmion simu~oto¢ troinl~. Il J .- - oecl~curt~ ~ " 7 in the il~structiollo I'--~"% m the meo~s ~ system "with which /
I .,.m,ne I and events
/ ~:~;;~
Document simulator
\
traini~ a~o,icotion
with.respect ~
Jforthesetosks
tarne . . l de=t~rmmea ] T_o_us_ __. /
Fig. 2
j I
I
1
• •" Assess s*n~otor transfer of troiflin(J for mese events
I I ~ I
~ ~ /C~ertify N / simuldtol \ [ !or.aPpropriate ) | toslL~ ~ -' - on-(J'event¢ / ~'-'"'"" /
Recurrent evoJuatian
Graphical representation of the SIMCERT process
Conclusion
When developing evaluative techniques, the task of the human factors specialist is frequently not to find the 'perfect' techniques. Instead, the task is to find the optimal techniques that (1) satisfy the objectives of the evaluation, and (2) can be performed by the personnel in the environment where they will be used. This is no easy task. The concepts of mathematical programming provide useful guidelines to the design of these evaluative techniques. When properly applied, they can be expected to result in a product that effectively serves the ultimate users of the evaluative techniques. Acknowledgements
This work was supported in part by the Strategic Air Command under Contract No F33657-78-C-0491. The author also acknowledges Dr Jan L. Ditzian and Major George M. Houtman for their assistance in the Simulator Certification programme development discussed herein. References
Air Force Regulation 1977 AFR 50-11 'Management and utilisation of training devices', Department of the Air Force, HQ USAF, Washington, DC, October.
268
I
Of utiliSation hove changed
I Initial certificotion
Monitor performance of traine~s who have bge.n tralqed I wlth the slmulatar
fqr which means
whiCh I°*ea' teOI
[_~inodequacies,,,..,=/ worth ~ I correcting ~ ~ ~,~ \ ~
o
I
Applied Ergonomics
December 1982
Bazaraa, M.S., and Jaxvis, J.J. 1977 'Linear programming and network flows'. John Wiley and Sons, New York, NY. Ditzian, J.L., and Laughery, K.R. 1979 SAC Simulator Certification - Literature Review (Calspan T.M. 79-14-2.1). Calspan Corporation, Buffalo, NY: Ca~pan Corporation, August. Laughery, K.R., and Ditzian, J.L. 1980a SAC Simulator Certification - Objectives and Constraints (Calspan T.M. 79-14- 2.2). Buffalo, NY: Calspan Corporation, February. Langhery, K.R., and Ditzian, J.U 1980b SAC Simulator Certification - Candidate Methodologies (Calspan T.M. 79-14.2.2). Buffalo, NY: Calspan Corporation, February. Laughery, K.R., and Houtman, G.M. 1980 'Development of a Simulator Certification Methodology for the Strategic Air Command', Proceedings of the Seventh Annual Psychology in the Department of Defense Symposium, USAF Academy, CO, April.
Payne, T.A. 1978 'A practical guide to conduct of studies of transfer of learning', Draft report prepared for the Air Force Human Resources Laboratory, Flying Training Division, Williams AFB, AZ, April.