- Email: [email protected]
Statistical aspects in a simulation study
Comout.at Indua.Engn&Vol.5. No. 3. pp 195-204.1981 Printed in GreatBritain.
0~3521S11030195-10502,0010 PergamonPre~sLid
STATISTICAL ASPECTS IN A SIMULATION STUDY AMITAVA MITRA a n d JAMES F. Cox Department of Management, Auburn University, AL 36849, U.S.A.
and ABE KELLIZY Management Consultant, International Business Services, Washington, DC 20005, U.S.A. (Received for publication 30 March 1981)
Abstract--This article discusses some of the general statistical considerations that are relevant to undertaking a simulation study. The major phases of conducting a simulation study along with the corresponding statistical aspects are explored. The objective of this paper is to provide a tutorial to aid simulation users in identifying and using statistical techniques. The analysis should be helpful in developing models of real world systems. A simulation of a cafeteria is provided which illustrates the use of appropriate statistical considerations and statistical tools.
I. I N T R O D U C T I O N
The study and analysis of systems by means of simulation techniques has gained wide usage in recent times [!]. The availability of high-speed computers has facilitated this process. Industries have accepted simulation as a valuable tool in decision making because of its understandability and its ability to represent "real world" problems. However, in any simulation analysis, various statistical aspects should be considered starting from the system observation and data gathering phases to the system validation and analysis phases. These statistical aspects, though nothing new to a statistician, have to be considered in a simulation study, and the practitioner therefore needs to have a working knowledge of them. The purpose of this paper is to present some statistical issues and their resolution that may be encountered at various stages of a simulation analysis, and thereby to provide a framework of guidelines to users undertaking simulation. The various statistical techniques are applied to a simulation of a cafeteria to illustrate their use and interpretation. The paper is by no means an exhaustive list of the different statistical methods that may be used in a simulation analysis. Instead, for each step in the analysis the paper discusses the general statistical considerations and provides some methods for handling the issue. The motivation for this study evolved from the idea of providing a tutorial to aid simulation users who are not involved in the field of statistics but nevertheless need to identify and use statistical techniques. This paper attempts to provide the linkage between statistical methods and their usage in simulation rather than discuss available statistical tools separate from the actual design of a simulation model. Statistical techniques have received attention in the simulation literature[2-12]. Our paper explains the statistical aspects to be considered at each state in developing and implementing a simulation and also provides an example showing how these statistical considerations are resolved. It is not feasible, however, to discuss in depth the theoretical basis of statistical procedures mentioned in the paper, e.g. analysis of variance, regression analysis, spectral analysis, etc. Naylor et al.[8] discuss the experimental design aspect and also provide general descriptions of the _F-test, multiple comparison procedures, multiple ranking procedures, spectral analysis, and sequential sampling. Fishman and Kiviat[13] discuss statistical considerations related to verification, validation, and problem analysis. Kleijnen[14, 15] provides a good review of variance reduction techniques and other statistical techniques that are appropriate to simulation studies. Section 2 discusses the statistical considerations related to simulation modeling. An application of simulation to a cafeteria, illustrating this methodology may be found in Section 3. 195
196
AMITAV*~MITRAel al
2, M E T H O D O L O G Y
A general framework for simulation analysis is shown in Fig. 1. From the planning stage to the model validation stage feedback loops exist between the phases till an acceptable model is formulated, after which analysis of the system may be performed. (a) Planning Problem definition and the explicit statement of objectives are crucial to any study, simulation or otherwise. The simulation user has to identify the problem concisely and decide upon the objectives prior to initiating the study. (b) Data collection and analysis (1) Observation is the initial step in the investigation of the system. If, however, the user is undertaking a simulation study of a proposed system, judgement must be used in place of the data collection and analysis phases. (2) A thorough familiarization of the system is achieved through an analysis of the observed data. Methods and amount of data to be collected are normally governed by a balance of the costs involved and the time allocated for the model construction against the estimated usefulness or benefits of the completed model. The first statistical analysis is usually a macroanalysis of the system. Computations of aggregate statistics, namely, mean, standard deviation, range, and low and high values indicate general system characteristics. Plots of histograms (relative frequency and cumulative), are also helpful. Based on the results of the macroanalysis, a microanalysis may be necessary. The macroanalysis should also indicate the type and magnitude of additional data to be gathered. If the output is sensitive to the input factors, then more information on the input should be obtained. (3) Several techniques should be useful in the microanalysis of data. When it is desired to estimate the explicit relationship between the dependent variable and one or more independent variables, regression analysis may be beneficial. The model is specified as (1)
Yi = ~8o+//,x,i +//2x:i +' " " + ,8~xvi+ ei, i = 1,2 . . . . . n
where y is the dependent variable; x,, x2. . . . . xp are the independent variables;//o,//, . . . . . //p are the unknown parameters; and ~ is the traditional error component. In a search for the "best" set of independent variables, stepwise regression procedures can be employed. This search method essentially computes a sequence of regression equations where at each step an I
PLANNING 1. 2. 3.
L ~|
Problem definition Ob]ectives of study Allocation of resources
l I DATA COLLECTION AND ANALYSIS
ANALYSIS OF SYSTEM i. 2.
i. 2. 3.
Study system characteristics Effect of change in parameter values on response
T MODEL VALIDATION
Observation of system Macroanalysis of data Microanalysis of data
I
I MODEL BUILDING
I I-
Compare simulated response with actual response If discrepancies exist, make appropriate changes
2. 3.
Fig. I. Framework for simulation analysis.
Identification of significant factors affecting response Analysis of exogenous and endogenous variables Develop a model
Statistical aspects in a simulation study
197
independent variable is added or deleted, depending on whether it is a forward or a backward selection procedure. The Criteria for adding or deleting an independent variable is dependent on the error sum of squares reduction and usually measured by the -F-statistic. Often times in simulation analysis rather than identify an explicit relation between the dependent and the independent variables, the problem may be to find out whether certain variables affect the response. In such cases, the analysis of variance is more appropriate to test if a factor has an effect. Also, qualitative variables as well as quantitative variables can be handled effectively in analysis of variance. The basic assumptions of analysis of variance are that the experimental errors are normally and independently distributed with a constant variance. The procedure tests differences between means through a comparison of variances by using an -F-test. If the data to be analyzed is generated in the simulation, then the experimental errors can be made independent by using different sequences of random numbers. If the degrees of freedom is very large, then non-normality has little effect on the power of the _F-test[16]. Also, inequality of the variances has little effect on the power of the test in the case of equal number of observations per ceil. If however deviations from the assumptions are critical, nonparametric or distribution-free analysis of variance techniques should be used[17]. For a review of regression analysis and analysis of variance see [18]. Where historical data is available, testing to determine which factors statistically affect the response is often neglected. For example, where data has been gathered over time, analysis of variance procedures may be used to test if time, e.g. day of the week, time of day, etc., significantly influences the factors. If it does, then the simulation procedure should be designed to reflect this influence; namely, different distributions depending on the day and time of day should be used. The problem of analyzing and identifying empirical distributions is discussed below. Once the significant input factors (and their associated levels, if not fixed) have been identified, the distributions of these input variables must be analyzed. Historical data can be examined to identify the empirical distribution of the variables. Statistical tests may be performed to determine if the observed data are from a specified theoretical distribution. If the hypothesis, that the observations are from a specified distribution, is not rejected, then that distribution is used in the simulation. If the hypothesis is rejected, then the empirical distribution is used as the benchmark for generating random numbers for values of the variable. The chi-square goodness of fit test and the Kolmog'orov-Smirnov test are two methods for testing the above mentioned hypothesis. In the chi-square test, if the sample data is divided into k classes, with O;, i = i,2 . . . . . k representing the observed frequency in class i, and Ei, i = 1,2 . . . . . k representing the expected or theoretical frequency in cell i as determined from the specified distribution, then
k
x 2 = ~ (Oi- E32/Ei
(2)
has a chi-square distribution with k-p-i degrees of freedom where p represents the number of parameters of the specified distribution estimated from the sample statistics. This calculated ~2 is compared to the tabulated values of g 2 for a specified level of significance. The KolmogorovSmirnov test is as follows: Let F(x) denote the completely specified theoretical cumulative distribution function under the null hypothesis. Let S,(x) be the sample cumulative distribution function based on n observations. So for any x, S,(x)=k/n where k is the number of observations less than or equal to x. Let the maximum deviation between the empirical distribution function and the hypothesized theoretical distribution function be denoted by /9. = m a x l F ( x )
- S.(x)l.
(3)
x
The null hypothesis is that the empirical distribution is the same as the specified theoretical distribution. If the observed value of D, is greater than or equal to the critical value given in tables [19] for the chosen level of significance, the null hypothesis will be rejected. In the output
198
AMITAVAMITR.*,et al
of the KOLMO subroutine [20] the z statistic equals (v'n)D,, and the probability represents the probability of the statistic being greater than or equal to z. Small values of z, and correspondingly large values of probability, imply a good fit. The Kolmogorov-Smirnov test has at least two advantages over the chi-square test. First, it can be used with small sample sizes where the validity of the chi-square test would be questionable. Second, it often appears to be a more powerful test than the chi-square test for any sample size[21]. Statistical estimation methods (e.g. maximum likelihood) may be used to estimate the parameters of the specified distribution from the observed sample. In such cases the chi-square goodness of fit may still be used with an appropriate reduction in the number of degrees of freedom. (c) Model building (1) The next phase, the development of a model, is based on the results of the macroanalysis and microanalysis of data. In the previous stage significant factors affecting response were identified. Now, these factors or a subset of them may be included in the model depending on the cost and resource limitation. (2) The detailed analysis of the endogenous and exogenous variables as mentioned in the previous stage (for example, studying inter-arrival time distributions and their relationship to the time of day or day of the week, comparing empirical service time distributions to theoretical distributions, studying queue characteristics and their effect on arriving customers) should now provide the backbone for model building. Developing a model is not a one-shot approach. The steps of data analysis, formulating a model, and validating the model form a continuous process with feedback loops. (3) A typical simulation model is developed by using a flowchart describing the relationships among the various system components. The flowchart or block diagram should outline the logical sequence of events in generating the time paths of the model's variables. The advantage of a detailed block diagram is that it greatly simplifies coding of the computer program. (d) Model validation (I) The problem of validating simulation models is a difficult task. Practical, statistical, and even philosophical complexities are involved. For a more philosophical discussion (rationalism, empiricism, and positive economics in validation) see Naylor[22] and Naylor and Finger[9]. Fishman and Kiviat[13] describe verification as testing whether the model, as well as parts of the model, behaves as assumed whereas validation means testing if the model output conforms to the observed output of the real world system. This observed output or response should be different from the data that was used to build the simulation model. Several statistical measures and techniques can be used in the validation process. The chi-square goodness of fit test, described in the section on data analysis, may again be employed for testing the simulated output against observed output. The disadvantages of this test, namely, its relative sensitivity to non-normality and the problem of selecting categories, may be avoided by using the Kolmogorov-Smirnov two-sample test[19], (a minor variant of the Kolmogorov-Smirnov one-sample test). The difference is that now the test is based on the statistic 19,,., defined as D,,., = maxlFm(x)-
G.(x)i.
(4)
where F,,(x) is the sample distribution function of a sample of size m from one population and G,(x) is the sample distribution function of a sample of size n from the second population. The analysis of variance can also be utilized. Tests comparing the output characteristics of simulated responses and observed responses can be constructed to see if a significant difference exists between them. Expert opinion may also be used in such tests. Suppose the expert expects a particular factor to become significant at some specified levels. The results of a simulation model incorporating the levels mentioned by the expert can be compared to the observed response. Naylor and Finger[9] list several possible measures of goodness of fit of a simulation model. namely, number, timing and direction of turning points; average value of a simulated time path:
Statistical aspects in a simulation study
199
the whole time path itself; etc. Spectral analysis can be applied to the simulation output and the actual output to see if the spectra (time paths) are the same. For more details on spectral analysis see[4,22]. A simple regression analysis between the simulation and the actual output can also be performed, and one may test if the intercept is zero and the slope is one[22]. (2) If there is a significant difference between the simulated and the actual output, as may be found by the above mentioned procedures, then changes in the model are necessitated. Sometimes, further analysis of data is required to gain more information about the system which, in turn, will facilitate the model building procedure.
(e) Analysis of 'system (i) Once the computer simulation model has been validated, one can proceed with the study of system characteristics. For example, in the queueing framework we may study the behavior of the number of persons served in the system over a specified length of time, the maximum queue length, the average waiting time, etc. (2) Experimental design techniques can be used to determine the effects of parameters, variables and relationships. These factors are varied at several levels. For example, one may wish to consider the effect of a qualitative factor, namely, queue discipline, at three levels such as first in first out, random and shortest processing time. Procedures such as random designs, group-screening designs, factorial designs, fractional factorial designs, response surface designs, etc. may be employed [23]. The simulation user, depending on the particular problem of interest, should choose the appropriate procedure. 3. A P P L I C A T I O N : S I M U L A T I O N OF A C A F E T E R I A
The cafeteria management wanted to develop a simulation model of their newly established "mini dell" line to examine possible methods of improving service. They wished to construct a model of the present system and investigate changes within the simulation model prior to actually implementing changes to the existing system. This analysis is limited to studying the present system. The cold lunch line at a medium-sized university was simulated. Service is provided five days a week from 11:00 a.m. to 1:00 p.m. Arriving customers form a queue (the primary line) where they are processed on a first come first served basis. Once a person enters the service area, he chooses among a number of combinations of foods and makes several service stops while in the line. Figure 2 depicts schematically the lunch line area. Stations 1-4 are operated by two workers who hand out sandwiches and potato chips and soup and cheese respectively. The SERVICE PERSONNEL
D NNSm I <
Custo~re
in Service
<
Area
<
L..
)
)
P r i o r y Queue (Firsts)
TI
Secondary Queue (Seconds) Service Area: 1. 2. 3. 4.
Main Sandwiches Potato Chips Other Sandwiches Cheese & Soup
5. 6. 7. 8.
Desserts Napkins & Sugar Coffee, Hot Water, Glasses Cashier
Fig. 2. Schematic diagram of cafeteria lunch line area.
AMITAVA MITRA et
200
al.
remainder of the line consists of self-service stations until the cashier station. About 35% of the customers choose to come back for seconds (and thirds . . . . ). These seconds enter a separate (secondary) queue, which has priority over the primary queue. Seconds are served after the current customer in station 1. In the lunch line mode, familiarization with the system was achieved by interviewing the cafeteria manager and by direct observation of the system during various time periods. Some simplifying assumptions were made at this phase: (i) service time for primary customers begins when they enter the service facility and ends after paying the cashier; (ii) the term "seconds" as used in this study includes seconds, thirds, etc.; (iii) since seconds do not pay the cashier again, their service time begins when they give their plate to the service worker and ends upon receiving their plate from the server. In the macroanalysis phase, data was provided by management and included the number of primary customers served in half-hour intervals from 1!:00 a.m. to 1:00 p.m. 5 days/week over a 7-week period. Summary statistics, presented in Table 1, in conjunction with actual observation indicated a busy period in the first half-hour period of each of two hours studied. Analysis of variance was utilized in determining whether the number of primary customers served over the 7-week period differed by week (F value = 13.07, p = 0.0001), by day of week (F value = 2.26, p = 0.066), and/or by half-hour increment (F value = 11.76, p = 0.0001). Further analysis of the weekly data by plots and regression analysis (F = 4.14, p = 0.10) indicated a slight growth in the number of customers served over the time period. The regression model for the number of primary customers served in half-hour increments (Y) vs time in weeks (X) was found to be Y= 112.91 + 3.564X. The t statistic for the slope coefficient was significant (t = 2.036, p = 0.10) and the intercept was significant (t = 14.419, p = 0.0001). Management indicated that the number of customers served had increased over the 7 weeks and had now stabilized near or at present system capacity; that the customer population was identical Monday-Friday; and that the first half-hour increment of each hour seemed to be the busy period. Based on the interviews with management, direct observation of the system, and the macroanalysis of the 7-week data, five 1-hr studies were conducted. Detailed information on interarrival times, primary service time, and secondary service time was cbtained. The shape of the empirical distributions was analyzed using frequency charts and cumulative histograms. The first ten minutes of each hour comprised a busy period where the arrival pattern had a different distribution from the remaining 50rain. Based upon this analysis the interarrival time distribution for primary customers was divided into two distributions. In Table 2, various statistical measures are presented based on a sample of 50 for each of the two time periods Table I. Summary statistics of number of primary cuslomer~ served
~i~
period
Mean
Standard deviation
Minimum value
Maximum value
11:00-11:30
am
129.96
15.47
102.O0
185.60
11:30-12:00
m.
126.14
17.15
gl.OO
182.00
12:O0-12:30
pm
136.09
18.33
IO2.00
195.O0
12:30-1:00
pm
116.51
19.68
8a.O0
171.O0
Table 2. Macro statistics of primary interarrival times Statistical Measure
Time period 12:OO m . - 1 2 : l O
p.m.
12:10-1:O0
7.32
sec
1.Oq
sec
Range
~.00
sec
?0.0~ ~ s e c
I
Median
7.06
sec
17.1~
I
Mean Standard
deviation
I7.QQ 5.00
~e{:
Statistical aspects in a simulationstudy
201
12:00 noon(m.)-12:10 p.m. and 12:l0 p.m.-1:00 p.m. Likewise the same pattern was observed for the other hour. The cause of this busy period was associated with the ending of university classes on or near the hour. Next each sample distribution was compared to the normal, exponential, and uniform distribution (each with specified parameters). The Kolmogorov-Smirnov test (KOLMO subroutine from IBM Scientific Subroutine Package[20]) was utilized. A sample of 60 was selected from the 5 hr of study and compared to the theoretical distributions whose parameters were estimated from data provided by management. The results of the analysis of the primary customer interarrival time distribution for 12:l0 p.m.-1:00 p.m. are given in Table 3. The empirical distributions were not well-approximated by the specified theoretical distributions as indicated by the z statistic in Table 3. Therefore, these empirical distributions were used in the simulation for the generation of interarrival times for this time period. Similar analyses were conducted for interarrival times for the first 10-min period of the hour, and the primary and secondary service times for all time periods. Primary service time was independent of the time period, therefore one empirical distribution was used to generate these times in the simulation. The same was true for the secondary service times. Based on the preceding microanalysis, empirical distributions were the "best fit" for primary customer arrivals (both the 10-min and 50-rain periods), for secondary customer arrivals, for primary service time, and for secondary service time. A simulation model, programmed in General Purpose System Simulation (GPSS1360)[24, 25] was based on the results of the macroanalysis and microanalysis. A flowchart specifying the details of the simulation model is given in Fig. 3. The model was validated by collecting additional data over a second 5-day period and then testing the simulation output characteristics for five computer runs (using different random number seeds) against these observed system characteristics by using an analysis of variance procedure (Statistical Analysis System [26]). The response characteristics tested were the number of primary customers served, the maximum queue length, the average queue length, and the number of secondary customers served. In Table 4. the results of an analysis of variance model indicates no significant difference between the simulation output and the observed output for the number of primary customers served. Similar analysis of variance models were constructed for the maximum queue length (F = 1.15, p = 0.299); average queue length (F = 0.47, p = 0.502); and the number of secondary customers served (F = 0.07, p = 0.791); and the results indicated no significant difference between the simulated output and the actual system output. The Kolmogorov-Smirnov two-sample test was also used for validation by comparing the empirical distributions of the simulated and actual output. As an illustration, for a sample of 10 observations of the number of primary customers served, the maximum absolute difference between the empirical cumulative distributions, D,oao. was equal to 0.40. The critical values from tables[19] for the 5% level and the 1% level of significance are 0.6 and 0.7 respectively. Hence the hypothesis that the two empirical distributions are the same is not Table 3. Kolmogoro'.-Smirnov test for primary interarrival time distribution for the period 12:I0 p.m.-1:00 p.m. /
Distributions Normal
Standard
16.21 sec deviation
Mean
lb.2I
Mean
Exponential
Unifor-
:
-
Lower
limit
Upper
limit
~_. ,¢
,n.
L. n
! rohabi
se¢
I ity
.lb8
sec ;
6
sec
30
.,.~7
sec
" . (lOb
Table 4. Validationusingan analysisof varianceprocedure with simulationresults and actual systemresults as classes Dependent
vat=able:
Source
Le~rees
Model
Erlor
Number o~ 1
l~
freedom
Of
prlmar}
:us~:-el5
Sum of
squares !~.3
"3!.
5er,.e: Me,i~
~,:t~,:e
:..
.
?-,,,.~e .
='--7
202
AMITAV~Mrn~ et al.
~ENERAT~.
i
(I) Generals Ornvin~ customers according to empirical arrival distribution. ( 12:10- I OO ) (2) Test the clock time, if Qreofer than IO minutes (6OO,OOO
C 1 ~ 2
milliseconds) ¢onfir~us, else terminate T ~
R
3
(3) TrQnafsr oil entering customers to the primary queue. (4) Generate orrivlnQ customers according to empirical
GENERATE
arrival distribution (12:OO-12:10) (5) Test the clock time,if less than IOminutes (600,000 c Is-s ° ° ' / ~ ,
5
milliseconds)continue,elSe terminate. (6)AssiQn primary service time to parameter 2
(7) Join primary lunch queue.
7 (8) Seize first service location in primary service area
(9) Deport from primary bunch queue. R=peote~ (IO)Pnmory service time for orrivin~ customers at each in o Series od~nce block VIO=9895+FNSPSTO/8, of 8 (11} Seize next service location
1 RELEASEr ~ , ,2
),
(12)ReJeose present service area
(13) Transfer 35% of the customers coming ~rough the
TRANSFER
I~Wnory Service area to the seconds waitinQ line (14) Amgn seconds service time to parameter 3,
14
V9=11340+ FN$SSTD. (15) Customers retuminQ for Second ServinQs of sandv,nches hove priority ove~ primary customers (16)Jam seconds lunch queue.
(17) Seize the sandwich sorvice focil;ty
(18) Deport seconds queue
(19) Seconds terence time for sandwiches
(20) Release the service facility
T
~
21
(21) Torminote one transaction
Fig. 3, Flowchart for simulation model.
Statistical aspects in a simulation study
203
Table 5. Sample simulation results Average number o f customers in queue
Maximum queue Primary queue
length
Number served
no waiting
time
Average waiting
time
&l
20.03
1
~.q7
min
2
O.la
a
O.10
min
Secondary queue
Facility number
Number of persons who experience
1
2
3
330
23g
238
99.8
71.9
71.9
&
5
6
7
8
237
236
236
235
234
72.5
72.1
71.2
~.~
64.7
Percentage utilization
rejected at these levels. Additionally, the results of the simulation (X) and the actual response characteristic (Y) for primary number of customers served were regressed. The resulting regression model was significant (F = 74.10, p = 0.0001) and the equation is Y = - 35.50 + I. 156X. The slope coefficient was not significantly different from one (t = 1.165. p =0.30) and the intercept was not significantly different from zero (t = - 1.106, p = 0.40). After the model validation phase, the system was studied and analyzed by simulation. Sample results of queue statistics and facility statistics for a simulation of a I-hr period are shown in Table 5. The average waiting time for persons in the primary queue is about 5 rain. The stations have a high utilization with the first station being busy most of the time. Further analysis may be done by refining the model by determining ways to reduce bottlenecks in the system by possible inclusion of multiple facilities in parallel for station 1. by using multiple servers, etc. On the whole, the model was a good representation of reality. 4. C O N C L U S I O N S
The statistical aspects involved in a simulation study have been considered. Several techniques for handling the statistical issues have been mentioned and a fe~ of these have been illustrated by using an example of a simulation model. Many statistical issues in simulation models are problem dependent. This article will acquaint the user with the general statistical considerations for each stage of simulation modeling, and will assist him in selecting the particular statistical methods that conform to his particular problem.
REFERENCES I. J. F. Cox. W. N Ledbetter 2,, J. M. Smith, Simulation as an aid to corporate decision making SimMation 29. 117-120 (1977). 2. D. S. Burdick & T. H. Naylor, Design of computer simulation experiments for industrial system~ Commun. ACM 9(5), 329-339 (1966). 3. G. S. Fishman, Correlated simulation experiments, Simulation 23, 177-180 (1974). 4. G. S. Fishman & P J. Kiviat, The analysis of simulation-generated time series Management Sci. 13(7), 525-557
(1967) 5. J. P. C. Kleilnen. Design and analysis of simulations: Practical statistical techniques. Simuhm,,n 28.81-90 (1977). 6. G. A. Mihram, Some practical aspects of the verification and validation of simulation models Ops Res. Quart. 23(I), 17-29 0972). 7. G. A. Mihram, The scientific method and statistics in simulation. Simulation 24. 14-16 (19"5~ 8. T. H. Naylor. D S. Burdick & W. E. Sasser, Computer simulation experiments with economic s~,stems: The problem of experimental design. J. Am. Statis. Assoc. 62, 1315-1337 (1967). 9. T. H. Naylor & J. Mr Finger, Verification of computer simulation models. Manaeeme,~t St + 14, 2 ~. B92-B 101 11967)
204
AMITAVAMITRAet al.
10. T. H. Naylor, K. Wertz & T. H. Wonnacott, Methods of analyzing data from computer simulation experiments. Commun. ACM 10( 11), 703-710 (1967). II. W. E. Sasser & T. H. Naylor, Computer simulation of economic systems: An example model. Simulation 8, 21-32 (1967). 12. R L. Van Horn, Validation of simulation results. Management Sci. 17(5), 247-258 (1971). 13. G. S. Fishman & P. J. Kiviat, Digital computer simulation: Statistical considerations. Rand Rep. RM-5387-PR (Nov. 1967). 14. J. P. C. Kleijnen, Statistical Techniques in Simulation Part I. Marcel Dekker, New York (1974). 15. J. P. C. Kleijnen, Statistical Techniques in Simulation Part II. Marcel Dekker, New York (1975). 16. H. Scheffe, The Analysis o[ Variance. Wiley, New York (1964). 17. M. L. Purl & P. K. Sen, Nonparametric Methods in Multirariate Analysis. Wiley, New York (1971). 18. J. Neter & W. Wasserman, Applied Linear Statistical Models. irwin, Homewood, Illinois 0974). 19. B. W. Lindgren. Statistical Theory, 2nd Edn. Macmillan, New York (1968). 20. IBM application Program: System1360. Scientil~c subroutine package, version III. Programmer's Manual, Program Number 360-A-CM-03X (Aug. 1970). 21. H. W. Lilliefors, On the Kolmogorov-Smirnov lest for normality with mean and variance unknown. J. Am. Starts. Assoc. 62, 399-..402 0967). 22. T. H. Naylor, Computer Simulation Experiments with Models o/Economic Systms. Wiley, New York (1971). 23. O. L. Davies, The Design and Analysis o,f Industrial Experiments. Hafner, New York (1971). 24. H. Maisel & G. Onugnoli, Simulation o[ Discrete Stochastic Systems. Science Research Associates, Chicago (1972). 25. T. J. Schriber, Simulation Using GPSS. Wiley, New York (1974). 26. A.J. Barr & J. H. Ooodnight, A User's Guide to the Statistical Analysis System. Institute of Statistics, Raleigh Division (Aug. 1972). 27. F..I. Massey, Jr., The Kolmogorov-Smirnov test for goodness of fit. J. Am. Starts. Assoc. 46, 68-78 (1951).
0~3521S11030195-10502,0010 PergamonPre~sLid
STATISTICAL ASPECTS IN A SIMULATION STUDY AMITAVA MITRA a n d JAMES F. Cox Department of Management, Auburn University, AL 36849, U.S.A.
and ABE KELLIZY Management Consultant, International Business Services, Washington, DC 20005, U.S.A. (Received for publication 30 March 1981)
Abstract--This article discusses some of the general statistical considerations that are relevant to undertaking a simulation study. The major phases of conducting a simulation study along with the corresponding statistical aspects are explored. The objective of this paper is to provide a tutorial to aid simulation users in identifying and using statistical techniques. The analysis should be helpful in developing models of real world systems. A simulation of a cafeteria is provided which illustrates the use of appropriate statistical considerations and statistical tools.
I. I N T R O D U C T I O N
The study and analysis of systems by means of simulation techniques has gained wide usage in recent times [!]. The availability of high-speed computers has facilitated this process. Industries have accepted simulation as a valuable tool in decision making because of its understandability and its ability to represent "real world" problems. However, in any simulation analysis, various statistical aspects should be considered starting from the system observation and data gathering phases to the system validation and analysis phases. These statistical aspects, though nothing new to a statistician, have to be considered in a simulation study, and the practitioner therefore needs to have a working knowledge of them. The purpose of this paper is to present some statistical issues and their resolution that may be encountered at various stages of a simulation analysis, and thereby to provide a framework of guidelines to users undertaking simulation. The various statistical techniques are applied to a simulation of a cafeteria to illustrate their use and interpretation. The paper is by no means an exhaustive list of the different statistical methods that may be used in a simulation analysis. Instead, for each step in the analysis the paper discusses the general statistical considerations and provides some methods for handling the issue. The motivation for this study evolved from the idea of providing a tutorial to aid simulation users who are not involved in the field of statistics but nevertheless need to identify and use statistical techniques. This paper attempts to provide the linkage between statistical methods and their usage in simulation rather than discuss available statistical tools separate from the actual design of a simulation model. Statistical techniques have received attention in the simulation literature[2-12]. Our paper explains the statistical aspects to be considered at each state in developing and implementing a simulation and also provides an example showing how these statistical considerations are resolved. It is not feasible, however, to discuss in depth the theoretical basis of statistical procedures mentioned in the paper, e.g. analysis of variance, regression analysis, spectral analysis, etc. Naylor et al.[8] discuss the experimental design aspect and also provide general descriptions of the _F-test, multiple comparison procedures, multiple ranking procedures, spectral analysis, and sequential sampling. Fishman and Kiviat[13] discuss statistical considerations related to verification, validation, and problem analysis. Kleijnen[14, 15] provides a good review of variance reduction techniques and other statistical techniques that are appropriate to simulation studies. Section 2 discusses the statistical considerations related to simulation modeling. An application of simulation to a cafeteria, illustrating this methodology may be found in Section 3. 195
196
AMITAV*~MITRAel al
2, M E T H O D O L O G Y
A general framework for simulation analysis is shown in Fig. 1. From the planning stage to the model validation stage feedback loops exist between the phases till an acceptable model is formulated, after which analysis of the system may be performed. (a) Planning Problem definition and the explicit statement of objectives are crucial to any study, simulation or otherwise. The simulation user has to identify the problem concisely and decide upon the objectives prior to initiating the study. (b) Data collection and analysis (1) Observation is the initial step in the investigation of the system. If, however, the user is undertaking a simulation study of a proposed system, judgement must be used in place of the data collection and analysis phases. (2) A thorough familiarization of the system is achieved through an analysis of the observed data. Methods and amount of data to be collected are normally governed by a balance of the costs involved and the time allocated for the model construction against the estimated usefulness or benefits of the completed model. The first statistical analysis is usually a macroanalysis of the system. Computations of aggregate statistics, namely, mean, standard deviation, range, and low and high values indicate general system characteristics. Plots of histograms (relative frequency and cumulative), are also helpful. Based on the results of the macroanalysis, a microanalysis may be necessary. The macroanalysis should also indicate the type and magnitude of additional data to be gathered. If the output is sensitive to the input factors, then more information on the input should be obtained. (3) Several techniques should be useful in the microanalysis of data. When it is desired to estimate the explicit relationship between the dependent variable and one or more independent variables, regression analysis may be beneficial. The model is specified as (1)
Yi = ~8o+//,x,i +//2x:i +' " " + ,8~xvi+ ei, i = 1,2 . . . . . n
where y is the dependent variable; x,, x2. . . . . xp are the independent variables;//o,//, . . . . . //p are the unknown parameters; and ~ is the traditional error component. In a search for the "best" set of independent variables, stepwise regression procedures can be employed. This search method essentially computes a sequence of regression equations where at each step an I
PLANNING 1. 2. 3.
L ~|
Problem definition Ob]ectives of study Allocation of resources
l I DATA COLLECTION AND ANALYSIS
ANALYSIS OF SYSTEM i. 2.
i. 2. 3.
Study system characteristics Effect of change in parameter values on response
T MODEL VALIDATION
Observation of system Macroanalysis of data Microanalysis of data
I
I MODEL BUILDING
I I-
Compare simulated response with actual response If discrepancies exist, make appropriate changes
2. 3.
Fig. I. Framework for simulation analysis.
Identification of significant factors affecting response Analysis of exogenous and endogenous variables Develop a model
Statistical aspects in a simulation study
197
independent variable is added or deleted, depending on whether it is a forward or a backward selection procedure. The Criteria for adding or deleting an independent variable is dependent on the error sum of squares reduction and usually measured by the -F-statistic. Often times in simulation analysis rather than identify an explicit relation between the dependent and the independent variables, the problem may be to find out whether certain variables affect the response. In such cases, the analysis of variance is more appropriate to test if a factor has an effect. Also, qualitative variables as well as quantitative variables can be handled effectively in analysis of variance. The basic assumptions of analysis of variance are that the experimental errors are normally and independently distributed with a constant variance. The procedure tests differences between means through a comparison of variances by using an -F-test. If the data to be analyzed is generated in the simulation, then the experimental errors can be made independent by using different sequences of random numbers. If the degrees of freedom is very large, then non-normality has little effect on the power of the _F-test[16]. Also, inequality of the variances has little effect on the power of the test in the case of equal number of observations per ceil. If however deviations from the assumptions are critical, nonparametric or distribution-free analysis of variance techniques should be used[17]. For a review of regression analysis and analysis of variance see [18]. Where historical data is available, testing to determine which factors statistically affect the response is often neglected. For example, where data has been gathered over time, analysis of variance procedures may be used to test if time, e.g. day of the week, time of day, etc., significantly influences the factors. If it does, then the simulation procedure should be designed to reflect this influence; namely, different distributions depending on the day and time of day should be used. The problem of analyzing and identifying empirical distributions is discussed below. Once the significant input factors (and their associated levels, if not fixed) have been identified, the distributions of these input variables must be analyzed. Historical data can be examined to identify the empirical distribution of the variables. Statistical tests may be performed to determine if the observed data are from a specified theoretical distribution. If the hypothesis, that the observations are from a specified distribution, is not rejected, then that distribution is used in the simulation. If the hypothesis is rejected, then the empirical distribution is used as the benchmark for generating random numbers for values of the variable. The chi-square goodness of fit test and the Kolmog'orov-Smirnov test are two methods for testing the above mentioned hypothesis. In the chi-square test, if the sample data is divided into k classes, with O;, i = i,2 . . . . . k representing the observed frequency in class i, and Ei, i = 1,2 . . . . . k representing the expected or theoretical frequency in cell i as determined from the specified distribution, then
k
x 2 = ~ (Oi- E32/Ei
(2)
has a chi-square distribution with k-p-i degrees of freedom where p represents the number of parameters of the specified distribution estimated from the sample statistics. This calculated ~2 is compared to the tabulated values of g 2 for a specified level of significance. The KolmogorovSmirnov test is as follows: Let F(x) denote the completely specified theoretical cumulative distribution function under the null hypothesis. Let S,(x) be the sample cumulative distribution function based on n observations. So for any x, S,(x)=k/n where k is the number of observations less than or equal to x. Let the maximum deviation between the empirical distribution function and the hypothesized theoretical distribution function be denoted by /9. = m a x l F ( x )
- S.(x)l.
(3)
x
The null hypothesis is that the empirical distribution is the same as the specified theoretical distribution. If the observed value of D, is greater than or equal to the critical value given in tables [19] for the chosen level of significance, the null hypothesis will be rejected. In the output
198
AMITAVAMITR.*,et al
of the KOLMO subroutine [20] the z statistic equals (v'n)D,, and the probability represents the probability of the statistic being greater than or equal to z. Small values of z, and correspondingly large values of probability, imply a good fit. The Kolmogorov-Smirnov test has at least two advantages over the chi-square test. First, it can be used with small sample sizes where the validity of the chi-square test would be questionable. Second, it often appears to be a more powerful test than the chi-square test for any sample size[21]. Statistical estimation methods (e.g. maximum likelihood) may be used to estimate the parameters of the specified distribution from the observed sample. In such cases the chi-square goodness of fit may still be used with an appropriate reduction in the number of degrees of freedom. (c) Model building (1) The next phase, the development of a model, is based on the results of the macroanalysis and microanalysis of data. In the previous stage significant factors affecting response were identified. Now, these factors or a subset of them may be included in the model depending on the cost and resource limitation. (2) The detailed analysis of the endogenous and exogenous variables as mentioned in the previous stage (for example, studying inter-arrival time distributions and their relationship to the time of day or day of the week, comparing empirical service time distributions to theoretical distributions, studying queue characteristics and their effect on arriving customers) should now provide the backbone for model building. Developing a model is not a one-shot approach. The steps of data analysis, formulating a model, and validating the model form a continuous process with feedback loops. (3) A typical simulation model is developed by using a flowchart describing the relationships among the various system components. The flowchart or block diagram should outline the logical sequence of events in generating the time paths of the model's variables. The advantage of a detailed block diagram is that it greatly simplifies coding of the computer program. (d) Model validation (I) The problem of validating simulation models is a difficult task. Practical, statistical, and even philosophical complexities are involved. For a more philosophical discussion (rationalism, empiricism, and positive economics in validation) see Naylor[22] and Naylor and Finger[9]. Fishman and Kiviat[13] describe verification as testing whether the model, as well as parts of the model, behaves as assumed whereas validation means testing if the model output conforms to the observed output of the real world system. This observed output or response should be different from the data that was used to build the simulation model. Several statistical measures and techniques can be used in the validation process. The chi-square goodness of fit test, described in the section on data analysis, may again be employed for testing the simulated output against observed output. The disadvantages of this test, namely, its relative sensitivity to non-normality and the problem of selecting categories, may be avoided by using the Kolmogorov-Smirnov two-sample test[19], (a minor variant of the Kolmogorov-Smirnov one-sample test). The difference is that now the test is based on the statistic 19,,., defined as D,,., = maxlFm(x)-
G.(x)i.
(4)
where F,,(x) is the sample distribution function of a sample of size m from one population and G,(x) is the sample distribution function of a sample of size n from the second population. The analysis of variance can also be utilized. Tests comparing the output characteristics of simulated responses and observed responses can be constructed to see if a significant difference exists between them. Expert opinion may also be used in such tests. Suppose the expert expects a particular factor to become significant at some specified levels. The results of a simulation model incorporating the levels mentioned by the expert can be compared to the observed response. Naylor and Finger[9] list several possible measures of goodness of fit of a simulation model. namely, number, timing and direction of turning points; average value of a simulated time path:
Statistical aspects in a simulation study
199
the whole time path itself; etc. Spectral analysis can be applied to the simulation output and the actual output to see if the spectra (time paths) are the same. For more details on spectral analysis see[4,22]. A simple regression analysis between the simulation and the actual output can also be performed, and one may test if the intercept is zero and the slope is one[22]. (2) If there is a significant difference between the simulated and the actual output, as may be found by the above mentioned procedures, then changes in the model are necessitated. Sometimes, further analysis of data is required to gain more information about the system which, in turn, will facilitate the model building procedure.
(e) Analysis of 'system (i) Once the computer simulation model has been validated, one can proceed with the study of system characteristics. For example, in the queueing framework we may study the behavior of the number of persons served in the system over a specified length of time, the maximum queue length, the average waiting time, etc. (2) Experimental design techniques can be used to determine the effects of parameters, variables and relationships. These factors are varied at several levels. For example, one may wish to consider the effect of a qualitative factor, namely, queue discipline, at three levels such as first in first out, random and shortest processing time. Procedures such as random designs, group-screening designs, factorial designs, fractional factorial designs, response surface designs, etc. may be employed [23]. The simulation user, depending on the particular problem of interest, should choose the appropriate procedure. 3. A P P L I C A T I O N : S I M U L A T I O N OF A C A F E T E R I A
The cafeteria management wanted to develop a simulation model of their newly established "mini dell" line to examine possible methods of improving service. They wished to construct a model of the present system and investigate changes within the simulation model prior to actually implementing changes to the existing system. This analysis is limited to studying the present system. The cold lunch line at a medium-sized university was simulated. Service is provided five days a week from 11:00 a.m. to 1:00 p.m. Arriving customers form a queue (the primary line) where they are processed on a first come first served basis. Once a person enters the service area, he chooses among a number of combinations of foods and makes several service stops while in the line. Figure 2 depicts schematically the lunch line area. Stations 1-4 are operated by two workers who hand out sandwiches and potato chips and soup and cheese respectively. The SERVICE PERSONNEL
D NNSm I <
Custo~re
in Service
<
Area
<
L..
)
)
P r i o r y Queue (Firsts)
TI
Secondary Queue (Seconds) Service Area: 1. 2. 3. 4.
Main Sandwiches Potato Chips Other Sandwiches Cheese & Soup
5. 6. 7. 8.
Desserts Napkins & Sugar Coffee, Hot Water, Glasses Cashier
Fig. 2. Schematic diagram of cafeteria lunch line area.
AMITAVA MITRA et
200
al.
remainder of the line consists of self-service stations until the cashier station. About 35% of the customers choose to come back for seconds (and thirds . . . . ). These seconds enter a separate (secondary) queue, which has priority over the primary queue. Seconds are served after the current customer in station 1. In the lunch line mode, familiarization with the system was achieved by interviewing the cafeteria manager and by direct observation of the system during various time periods. Some simplifying assumptions were made at this phase: (i) service time for primary customers begins when they enter the service facility and ends after paying the cashier; (ii) the term "seconds" as used in this study includes seconds, thirds, etc.; (iii) since seconds do not pay the cashier again, their service time begins when they give their plate to the service worker and ends upon receiving their plate from the server. In the macroanalysis phase, data was provided by management and included the number of primary customers served in half-hour intervals from 1!:00 a.m. to 1:00 p.m. 5 days/week over a 7-week period. Summary statistics, presented in Table 1, in conjunction with actual observation indicated a busy period in the first half-hour period of each of two hours studied. Analysis of variance was utilized in determining whether the number of primary customers served over the 7-week period differed by week (F value = 13.07, p = 0.0001), by day of week (F value = 2.26, p = 0.066), and/or by half-hour increment (F value = 11.76, p = 0.0001). Further analysis of the weekly data by plots and regression analysis (F = 4.14, p = 0.10) indicated a slight growth in the number of customers served over the time period. The regression model for the number of primary customers served in half-hour increments (Y) vs time in weeks (X) was found to be Y= 112.91 + 3.564X. The t statistic for the slope coefficient was significant (t = 2.036, p = 0.10) and the intercept was significant (t = 14.419, p = 0.0001). Management indicated that the number of customers served had increased over the 7 weeks and had now stabilized near or at present system capacity; that the customer population was identical Monday-Friday; and that the first half-hour increment of each hour seemed to be the busy period. Based on the interviews with management, direct observation of the system, and the macroanalysis of the 7-week data, five 1-hr studies were conducted. Detailed information on interarrival times, primary service time, and secondary service time was cbtained. The shape of the empirical distributions was analyzed using frequency charts and cumulative histograms. The first ten minutes of each hour comprised a busy period where the arrival pattern had a different distribution from the remaining 50rain. Based upon this analysis the interarrival time distribution for primary customers was divided into two distributions. In Table 2, various statistical measures are presented based on a sample of 50 for each of the two time periods Table I. Summary statistics of number of primary cuslomer~ served
~i~
period
Mean
Standard deviation
Minimum value
Maximum value
11:00-11:30
am
129.96
15.47
102.O0
185.60
11:30-12:00
m.
126.14
17.15
gl.OO
182.00
12:O0-12:30
pm
136.09
18.33
IO2.00
195.O0
12:30-1:00
pm
116.51
19.68
8a.O0
171.O0
Table 2. Macro statistics of primary interarrival times Statistical Measure
Time period 12:OO m . - 1 2 : l O
p.m.
12:10-1:O0
7.32
sec
1.Oq
sec
Range
~.00
sec
?0.0~ ~ s e c
I
Median
7.06
sec
17.1~
I
Mean Standard
deviation
I7.QQ 5.00
~e{:
Statistical aspects in a simulationstudy
201
12:00 noon(m.)-12:10 p.m. and 12:l0 p.m.-1:00 p.m. Likewise the same pattern was observed for the other hour. The cause of this busy period was associated with the ending of university classes on or near the hour. Next each sample distribution was compared to the normal, exponential, and uniform distribution (each with specified parameters). The Kolmogorov-Smirnov test (KOLMO subroutine from IBM Scientific Subroutine Package[20]) was utilized. A sample of 60 was selected from the 5 hr of study and compared to the theoretical distributions whose parameters were estimated from data provided by management. The results of the analysis of the primary customer interarrival time distribution for 12:l0 p.m.-1:00 p.m. are given in Table 3. The empirical distributions were not well-approximated by the specified theoretical distributions as indicated by the z statistic in Table 3. Therefore, these empirical distributions were used in the simulation for the generation of interarrival times for this time period. Similar analyses were conducted for interarrival times for the first 10-min period of the hour, and the primary and secondary service times for all time periods. Primary service time was independent of the time period, therefore one empirical distribution was used to generate these times in the simulation. The same was true for the secondary service times. Based on the preceding microanalysis, empirical distributions were the "best fit" for primary customer arrivals (both the 10-min and 50-rain periods), for secondary customer arrivals, for primary service time, and for secondary service time. A simulation model, programmed in General Purpose System Simulation (GPSS1360)[24, 25] was based on the results of the macroanalysis and microanalysis. A flowchart specifying the details of the simulation model is given in Fig. 3. The model was validated by collecting additional data over a second 5-day period and then testing the simulation output characteristics for five computer runs (using different random number seeds) against these observed system characteristics by using an analysis of variance procedure (Statistical Analysis System [26]). The response characteristics tested were the number of primary customers served, the maximum queue length, the average queue length, and the number of secondary customers served. In Table 4. the results of an analysis of variance model indicates no significant difference between the simulation output and the observed output for the number of primary customers served. Similar analysis of variance models were constructed for the maximum queue length (F = 1.15, p = 0.299); average queue length (F = 0.47, p = 0.502); and the number of secondary customers served (F = 0.07, p = 0.791); and the results indicated no significant difference between the simulated output and the actual system output. The Kolmogorov-Smirnov two-sample test was also used for validation by comparing the empirical distributions of the simulated and actual output. As an illustration, for a sample of 10 observations of the number of primary customers served, the maximum absolute difference between the empirical cumulative distributions, D,oao. was equal to 0.40. The critical values from tables[19] for the 5% level and the 1% level of significance are 0.6 and 0.7 respectively. Hence the hypothesis that the two empirical distributions are the same is not Table 3. Kolmogoro'.-Smirnov test for primary interarrival time distribution for the period 12:I0 p.m.-1:00 p.m. /
Distributions Normal
Standard
16.21 sec deviation
Mean
lb.2I
Mean
Exponential
Unifor-
:
-
Lower
limit
Upper
limit
~_. ,¢
,n.
L. n
! rohabi
se¢
I ity
.lb8
sec ;
6
sec
30
.,.~7
sec
" . (lOb
Table 4. Validationusingan analysisof varianceprocedure with simulationresults and actual systemresults as classes Dependent
vat=able:
Source
Le~rees
Model
Erlor
Number o~ 1
l~
freedom
Of
prlmar}
:us~:-el5
Sum of
squares !~.3
"3!.
5er,.e: Me,i~
~,:t~,:e
:..
.
?-,,,.~e .
='--7
202
AMITAV~Mrn~ et al.
~ENERAT~.
i
(I) Generals Ornvin~ customers according to empirical arrival distribution. ( 12:10- I OO ) (2) Test the clock time, if Qreofer than IO minutes (6OO,OOO
C 1 ~ 2
milliseconds) ¢onfir~us, else terminate T ~
R
3
(3) TrQnafsr oil entering customers to the primary queue. (4) Generate orrivlnQ customers according to empirical
GENERATE
arrival distribution (12:OO-12:10) (5) Test the clock time,if less than IOminutes (600,000 c Is-s ° ° ' / ~ ,
5
milliseconds)continue,elSe terminate. (6)AssiQn primary service time to parameter 2
(7) Join primary lunch queue.
7 (8) Seize first service location in primary service area
(9) Deport from primary bunch queue. R=peote~ (IO)Pnmory service time for orrivin~ customers at each in o Series od~nce block VIO=9895+FNSPSTO/8, of 8 (11} Seize next service location
1 RELEASEr ~ , ,2
),
(12)ReJeose present service area
(13) Transfer 35% of the customers coming ~rough the
TRANSFER
I~Wnory Service area to the seconds waitinQ line (14) Amgn seconds service time to parameter 3,
14
V9=11340+ FN$SSTD. (15) Customers retuminQ for Second ServinQs of sandv,nches hove priority ove~ primary customers (16)Jam seconds lunch queue.
(17) Seize the sandwich sorvice focil;ty
(18) Deport seconds queue
(19) Seconds terence time for sandwiches
(20) Release the service facility
T
~
21
(21) Torminote one transaction
Fig. 3, Flowchart for simulation model.
Statistical aspects in a simulation study
203
Table 5. Sample simulation results Average number o f customers in queue
Maximum queue Primary queue
length
Number served
no waiting
time
Average waiting
time
&l
20.03
1
~.q7
min
2
O.la
a
O.10
min
Secondary queue
Facility number
Number of persons who experience
1
2
3
330
23g
238
99.8
71.9
71.9
&
5
6
7
8
237
236
236
235
234
72.5
72.1
71.2
~.~
64.7
Percentage utilization
rejected at these levels. Additionally, the results of the simulation (X) and the actual response characteristic (Y) for primary number of customers served were regressed. The resulting regression model was significant (F = 74.10, p = 0.0001) and the equation is Y = - 35.50 + I. 156X. The slope coefficient was not significantly different from one (t = 1.165. p =0.30) and the intercept was not significantly different from zero (t = - 1.106, p = 0.40). After the model validation phase, the system was studied and analyzed by simulation. Sample results of queue statistics and facility statistics for a simulation of a I-hr period are shown in Table 5. The average waiting time for persons in the primary queue is about 5 rain. The stations have a high utilization with the first station being busy most of the time. Further analysis may be done by refining the model by determining ways to reduce bottlenecks in the system by possible inclusion of multiple facilities in parallel for station 1. by using multiple servers, etc. On the whole, the model was a good representation of reality. 4. C O N C L U S I O N S
The statistical aspects involved in a simulation study have been considered. Several techniques for handling the statistical issues have been mentioned and a fe~ of these have been illustrated by using an example of a simulation model. Many statistical issues in simulation models are problem dependent. This article will acquaint the user with the general statistical considerations for each stage of simulation modeling, and will assist him in selecting the particular statistical methods that conform to his particular problem.
REFERENCES I. J. F. Cox. W. N Ledbetter 2,, J. M. Smith, Simulation as an aid to corporate decision making SimMation 29. 117-120 (1977). 2. D. S. Burdick & T. H. Naylor, Design of computer simulation experiments for industrial system~ Commun. ACM 9(5), 329-339 (1966). 3. G. S. Fishman, Correlated simulation experiments, Simulation 23, 177-180 (1974). 4. G. S. Fishman & P J. Kiviat, The analysis of simulation-generated time series Management Sci. 13(7), 525-557
(1967) 5. J. P. C. Kleilnen. Design and analysis of simulations: Practical statistical techniques. Simuhm,,n 28.81-90 (1977). 6. G. A. Mihram, Some practical aspects of the verification and validation of simulation models Ops Res. Quart. 23(I), 17-29 0972). 7. G. A. Mihram, The scientific method and statistics in simulation. Simulation 24. 14-16 (19"5~ 8. T. H. Naylor. D S. Burdick & W. E. Sasser, Computer simulation experiments with economic s~,stems: The problem of experimental design. J. Am. Statis. Assoc. 62, 1315-1337 (1967). 9. T. H. Naylor & J. Mr Finger, Verification of computer simulation models. Manaeeme,~t St + 14, 2 ~. B92-B 101 11967)
204
AMITAVAMITRAet al.
10. T. H. Naylor, K. Wertz & T. H. Wonnacott, Methods of analyzing data from computer simulation experiments. Commun. ACM 10( 11), 703-710 (1967). II. W. E. Sasser & T. H. Naylor, Computer simulation of economic systems: An example model. Simulation 8, 21-32 (1967). 12. R L. Van Horn, Validation of simulation results. Management Sci. 17(5), 247-258 (1971). 13. G. S. Fishman & P. J. Kiviat, Digital computer simulation: Statistical considerations. Rand Rep. RM-5387-PR (Nov. 1967). 14. J. P. C. Kleijnen, Statistical Techniques in Simulation Part I. Marcel Dekker, New York (1974). 15. J. P. C. Kleijnen, Statistical Techniques in Simulation Part II. Marcel Dekker, New York (1975). 16. H. Scheffe, The Analysis o[ Variance. Wiley, New York (1964). 17. M. L. Purl & P. K. Sen, Nonparametric Methods in Multirariate Analysis. Wiley, New York (1971). 18. J. Neter & W. Wasserman, Applied Linear Statistical Models. irwin, Homewood, Illinois 0974). 19. B. W. Lindgren. Statistical Theory, 2nd Edn. Macmillan, New York (1968). 20. IBM application Program: System1360. Scientil~c subroutine package, version III. Programmer's Manual, Program Number 360-A-CM-03X (Aug. 1970). 21. H. W. Lilliefors, On the Kolmogorov-Smirnov lest for normality with mean and variance unknown. J. Am. Starts. Assoc. 62, 399-..402 0967). 22. T. H. Naylor, Computer Simulation Experiments with Models o/Economic Systms. Wiley, New York (1971). 23. O. L. Davies, The Design and Analysis o,f Industrial Experiments. Hafner, New York (1971). 24. H. Maisel & G. Onugnoli, Simulation o[ Discrete Stochastic Systems. Science Research Associates, Chicago (1972). 25. T. J. Schriber, Simulation Using GPSS. Wiley, New York (1974). 26. A.J. Barr & J. H. Ooodnight, A User's Guide to the Statistical Analysis System. Institute of Statistics, Raleigh Division (Aug. 1972). 27. F..I. Massey, Jr., The Kolmogorov-Smirnov test for goodness of fit. J. Am. Starts. Assoc. 46, 68-78 (1951).