A use of calibration in the development of simulation models

Mathematics and Computers North-Holland

in Simulation

A USE OF CALIBRATION

30 (1988) 27-32

27

IN THE DEVELOPMENT

OF SIMULATION

MODELS

D.G. MCCALL Whatawhata

Hill CounrT Research Station, Hamilton,

New Zealand

R.J. TOWNSLEY Agricultural

Economics & Business Department,

Massey University, Palmerston

North, New Zealand

The development of a mschanisticaZly zzilid model is often Eli objective for nnodelters of biological systems. This paper briefly corisiders problems involved i~ choosing parameter uaZues for biological mdei!s, and in determining reasons for m3det invalidity if it should faiZ a statistica lack-of-fit test. A procedure to aid n.odel devetopnretit is presented which involves parameter calibration to an appropriate data set and a statistical Zack-of-fit test to the calibratioti data. The iterative mdel revision and statistical testing process of parameter caZibration, of the mdel is presented for a determir,istic sirmilation model of pasture growth.

1.

INTRODUCTION The

and

objective

real

in much

system

illustrate

output

these

of biological

indistinguishable

criteria,

Yi, written

variable

modelling

are

consider

in the

systems

a general

"real

In equation

the

1, R(Zi)

R is the

relationship

reasons their

which real

variables values With practice of the

between

may

system

the

include:

values, will

is the mean

unknown

real

system"

a model

of similar model

with

such

mechanisms

a single

that

model

[l].

To

output

form: Yi = R(Zi)

Z i, where

is to develop

as a consequence

of Yi conditioned

of mathematical

Zi variables inability

biological

be stochastic.

system

Hence

output

a random

on the

functions

and mean

to specify

Cl1

+ vi

output

the

full

for any

component,

levels

of a vector

and parameter in the

real

system.

pi will

describing

For a number

set of Zi variables, set of values

of variables

values

for the

be associated

of

or observe (specified) with

Zi

observed

Of Yi. a deterministic we are variables

only

simulation able

in Zi, we are best

model,

to observe

system

(Ti)

is the

as Mj,

then

the corresponding

forced

available

01988,

IMACS/El

is to estimate

of Yi values

that

for R(Zi).

for R(Zi) xij

037%4754/88/$3.50

(nal)

to assume

estimate

estimate

the objective

a sample

R(Zi).

corresponding

the observed

If we denote

sample

mean

the jth

Because

in

to each

setting

from

the

simulation

real model

is:

= Mj(Zi)

sevier Science Publishers

121

B.V. (North-Holland)

28

D.G. McCall, R.J. Townsley / Use of calibration in the development of simulation models For the jth deterministic

system

output

(i)

(Yi) would

The system

model

to be valid,

be statistically

of relationships

then

ideally

indistinguishable

in R and Mj were

model

output

(Xij)

and

real

where:

identical

with

identical

parameter

settings. (ii)

The

set of Zi variables

in the

(iii)

The

values

in Zi were

Cohen

& Cyret

These

are:

A fourth

simultaneously Calibration procedure

[2] describe

specification

validation.

being

three

activity:

received

validity

the model

the model

If

IN MODEL

a model

diagnosing may

the cause

be invalid (i)

Errors

A combination

to the

literature a means

fact

may

where

appears

between

predictions

structural

reproduce omission

the of

be used

Supporting

of pasture

growth.

lack-of-fit

invalidity.

There

essential

forms

of parameter

tuning

(iii)

[6].

to evaluate

data

above)

which

the

are derived

in

structural from

a study

test,

the problem

are a number

or parameter

to have data

parameters

the

Zi variables

of general

values

from

one of

reasons

why

a model

relative

received

a range

of values

emphasis

of obtaining

precise

interpretation

of model

occur

setting

interactions of a model, final

choice

and the

of the model

variables

with

by tuning

can

the

parameter

of parameter

data

the model

be assessed

adequately, or

[7]).

incorrect

as

values aims

since judged

estimates

problem

of other within

with

as

of "poorly

parameter

parameters

acceptable

calibrated

the

sensitivity

"invalid"

by seeking

perhaps

suggested

to a given

to reproduce.

if the

from

has been

sensitivity

values

literature,

be derived

analysis

A major

model

is that

(e.g.

in the

can often

Sensitivity

of a model. importance

most

model

data

the model.

in Mj.

in an invalid

calibration

becomes

errors.

appear

of the model

essential

can

(i) to

to the

include:

to rationalise

validity

test

to reproduce.

Objections

as a result

model.

and

values

aims

4,8).

use of calibration,

model.

by some

for biological

Calibration

limits,

the

invalid

of the

of an invalid

be misleading

lack-of-fit

of the

a simulation

the model

(conditions

systems.

of parameters;

of parameter

(e.g.

simply

adequate

and modelled

DEVELOPMENT

values

parameters

the model.

data

of the above

of establishing

quantified" analysis

that

for many

data

observed

of a model

in functional

in parameter

to the

identical.

real

in developing

estimation

may mimick

simulation

These

(ii)

reference

were

estimation

of authors

of one or more

(iii)

(in Mj);

involves

view

the

of problem

forms

a number

an alternative

of model

[Sl.

Ommission

Errors due

is deemed

with

systems

between

from

a statistical

the development

2. CALIBRATION

classes

(structurally)

of a deterministic

involving

and modelled identical

calibration,

mechanistically

with

basic

attention

In this paper we present conjunction

real

of the functional

throughout has

are that

without

of variables

best Once

agreement calibrated,

model

by

some

lack-of-fit

specification

of

functions

in

biological

tails test,

to

of calibration

D.G. McCall, R.J. Townsley / Use in the model predictions This

are

for the

procedure

set and

No other

implicated. given

Zi values

appears

functional

to offer

forms

combination

of parameter

and jth model

specification.

the chance

specified

to separately

in the model,

29

in the development of simulation models

from

the

values

test

can

improve

the adequacy

adequacy

model

of the

of estimates

variable

of parameter

values. 3. EXAMPLE

APPLICATION

OF CALIBRATION

3.1 The Model The model grazing

used

in the

conditions.

Functional initially pasture

forms

obtained growth

calibrate

example

was

It is fully and parameter from

(Vi)

data

under

the model.

developed

described

reported

Each

value

for the

various

in the

literature.

regimes

of yi was

pasture

[lo].

estimates

51 grazing

to predict

by McCall

(settings

the mean

growth

components

A data

a range

of the model

set providing

of Zi) over

of either

under

2 years

three

(year

were

values

[9] was

of

of

used

1) or four

to

(year

2)

Yi observations. 3.2 Calibration The

where

objective

yi was

Procedure function

the

actual

non-linear

minimisation

function.

Eight

at the

limits

parameter

3.3

the the

from over

A test can

were

in the

calibrated

feasibility. literature.

test

calibrated question

the

(E04JAF

to the

Prime

These

were

model

which

this,

statistic

has been

remains

pooled

calibration

the q value

by ANOVA

of the

significance

which

no valid

determined

Xij. used

imposed from

A constrained

to minimise

on parameter

extreme model

values

estimates

was

based

this

of

on a

data.

Such

of Zi

follows

the

the

all estimates

resulting

variation

can

in this

F distribution

be obtained

least

squares

z(Ti_lij)2

of vi, an independent

an estimate

(q = 51,

by a constrained

is

estimate

from

the

of ~2 is

variance

(Vi)

example).

with

q and

Ci(ni-1)

degrees

of freedom

- _ F = Eni(Yi-Xij)2/q

number

of observations

calibration

level

In the absence

was

were

to the data

is whether

over

data.

settings

tuned

variance ni is the

library)

of the calibrated

the calibration

be calculated:

where

nag

observation

and constraints

Acceptance

against

c31

simulated

Test

To test

acceptable. required

the

lack-of-fit

procedure,

pooled

from

was to minimise: C = C(Yi-Xij) 2

corresponding

routine

of biological

Lack-of-fit

Although

mean

parameters

values

statistical

for calibration

data,

of F>lO%

was

chosen

of an independent

lack-of-fit

test

with

making each

up Vi.

setting

can be conducted.

of the Take

The

pooled

of Zi

for acceptance

estimate

c41

(Vi) variance

considered

as a treatment.

of the calibrated variance

for example

(Vi)

(Vi) can be obtained

(from

regression

A

model. the calibration analysis

data)

of Vi on

30

D.G. McCall, R.J. Townsley / Use of calibration in the development of simulation models With

Xi j*

regression

= xij against calibrated expect this

an alternative

model

the model

estimates

case

because

analysis

of ?i about Bij

(the test

with

such

we could

fail

to reject

goodness-of-fit

= a + b Xij.

data

by a least

for this

in the

absence

the model was

= a + b Xij)

(R(Zi)

Since

case

procedure,

R(Zi)

of the we would

? = "a + "b Rij.

to be deemed magnitude

In

invalid var iation

as the

[31).

of an independent

(hypothesis:

model

is likely

hypothesis

in the

squares

be of similar

analysis

of the

R(Zi)

estimate

= xij)

simply

is that

of 02, the danger because

the

selected

invalid.

Results

In the example

study, were

data

pooled

estimates

pooled

EMS =

ESSl

first

row of Table

and the objective estimate

function

data,

mean

square

an estimate

of the

from

ANOVA

variance

of each year's

(Vi ) for the

F statistic.

= 177.1

+ (n2-l)q2 1 shows

values

weighted

of vi in the calibration

calibration

of the error

to provide

+ ESS2

(nl-l)ql The

large

R(Zi)

to the

? = 2 + "b Xij will

criteria

a test,

hypothesis

calibration The

e.g.

tuned

hypothesis

of yi about

Hence

3.4

statistical

of a and b to be $20 and ^b=+l for the alternative

variation

alternative

the

hypothesis, has been

the alternative

the

we test

it failed

the

of both

the objective

by the number data.

F test

Despite owing

function

of observations the fact

to the

that

variation

from

associated

model

the calibration with

1 had been

C(yi-iij)2

being

each

tuned

to the

unacceptably

(SL = 1%): F = &;.; .

Table

Calibration

1:

objective

function

= 2.55

(d.f.

= 51,124)

values

for models 1 and 2

F_unction Value_ C=C(Vi-Xij)2 C1=Zni(Yi-xij)2 ____________________~~~~~~~~~~~~~~~~~~~~ Model Model

Procedures

for diagnosis

and will

almost

certainly

not well

predicted

on components

been

to a limit

The

forced

subjective

decide The

nature

which

structural

first

step

groupings

based

treatment

within

data

each

from

in order

Values

on similar

season,

Zi.

Data

were The

denoting

lax and

in model

grouped second

hard-graze

may

also

in another

by the

process

are

not clear

cut

of Zi for observations

particularly

provide

where component

used

useful

parameters

have

of the model.

in this

study

to

1. the

calibration

according step

model

settings

parameters

for an error

to categorise

of the year.

calibrated

of the

revision,

is emphasised

to change was

in the

of calibrated

to compensate

revision

23033.2 10582.3

on a study

requiring

revision

components

in model

problems

to be based

of the model

of model

season

7063.7 3037.2

of structural need

by the model.

information

1 2

was

data

to severity to plot

groupings

into

of the

broad grazing

the Vi and Ii-j‘S

separately.

Assessment

for the of the

D. G. McCall, R.J. Townsley / Use of calibration in the deoelopment of simulation models graphed for

spring

and

lax grazing The

above

and

in the model. The

(ii)

Tissue

(iii)

The daily

(iv)

Critical

There

was

rate

senescence moisture effect

rate

mass

those

respiration

below

available

and

pasture

calibrated

affecting

accumulation

values

of the major

the major

tissue

pasture

present.

flow

forced

to their

settings

pastures

DM/g

senescence

and

(MR-g DM/g

green

rates

calibrated

the maintenance

upper

was

(SR-g

tissue

green

DMlday).

DM/day).

increase

above

the

base

moisture).

initial

both

of green

by living

tissue which

soil

between

However,

new parameter grazed)

of initial

to the amount

of living

level

change

parameters.

(laxly

over-estimated

grazing.

were

in relation

rate

moisture

were

model

influenced: growth

(GM-percent

parameters of the

interest

in maintenance

a small

calibrated

by comparison

of most

senescence soil

the

it for hard

followed

These

expended

only

showed

values

of total

level

high

was

Parameter

(i)

data

under-estimated

analysis

parameters. rates

summer

31

bounds

to reduce

especially

respiration

in the

pasture

during

late

values

of the total

growth

and critical

soil

calibration

(Table

accumulation spring

2).

and

The

to a greater

degree

on

and summer.

Initial and calibrated values of major parameters in model 1 Table 2: ____________________~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_~~~~_~~___ SR CSM MR Calibrated Initial Initial Calibrated Initial Calibrated ________~___________~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_

Parametert

0.0064 0.007 0.3tt 0.1 E-3 0.2 ____________________~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_~~~~~~~~__ t See text tt At upper

for description bound limit

The above with

information

Dr Korte

from

model

had provided

in laxly

omitted

from

The effect

model

swards; The

taken

because

effective as the

starting

in grass

pasture point

and

green

data

mass

stem

on deficiencies

This

in late

identified

stem

was

the

fourth

in model step

Zi variables

spring/summer

on pasture

as a possible

important

structure

in the

of important

as photosynthetically mass

predominant

and

leaf

(leaf

pastures

in leaves.

taken

area)

material

the model

to residual

also

effective

gross

as separate

remaining

revision

being

omitted

growth

variable

and

which

mass

rate

incorporated

pasture

in laxly

after

had

grazing.

parameter

values

would grazed

lax grazing.

of detail

functions

after

to allow

green rate

in the model

level

empiricle

green

senescence

growth

variables

at this

to construct

Separate were

and hence green

growth

to reconstruct

therefore

for regrowth.

than

data.

possibility

pasture

the

stem

was

vegetative

stems

was

green

being

appropriate

green

for discussion

1.

stem

The decision

reproductive rate

pastures

of incorporating

available.

on the

of reproductive

effective

reproductive

option

basis

calibration

first

of incorporating

be to over-estimate

the

the

focused

grazed

E-2tt

of parameters

provided

The effect

1.

senescence been

who

Attention

procedure.

0.1

was

were

not

not

relating This

was

taken

for

for a higher

senescence

32

D.G. MeCaN, R.J. Revision

of model

prescribed (Table

F test

Townsley /

structure,

following

Use of calibration

if successful,

recalibration.

will

lead

to the

In the example

of simulation

resultant

study,

model

models

model

passing

2 passed

such

the

tests

1) (S.L.=25%): F = 209.5 177.1

Model

in the development

2, with

parameter

values

to subject

to validation

successful

revision

specialist

to help

set at those

against

of model

= 1.17

another

structure

interpret

the

(d.f.

derived

by calibration,

independent in this

information

data

study

made

= 51,124)

set.

was

the

available

was considered

An important

use made

appropriate

feature

of the

of a disciplinary

by calibration.

4. CONCLUSIONS A methodology test

to distinguish

associated proved

with

useful

Calibration with

is required

this

The

then

is still

of meeting

provides

lack-of-fit the

use of calibration

components value

the best estimate test

possible

agreement

this

parameters

can

prove

on model

useful

has

model

can achieve

the calibration

the calibration

doubt

those

growth.

a given

Vi) from

against

casts

from

lack-of-fit

The approach

of pasture

of 02 (variance

test,

model

functions.

model

of the model

lack-of-fit

and a suitable

of a simulation

for the

deterministic

of the calibrated

revision

point,

the model.

chance

the fails

estimates

requiring

structural of parameter

An independent

to perform model

the

of a large

values

set.

involving

in the choice

in the development

data

final

set

using

errors

incorrect

situation,

problems

outlined

of parameter

a given

calibrated

data

has been

data.

structure.

in diagnosis

data If the

In of

in the model.

however,

is that

validiation

required

before

However,

the thoroughly

validation

confidence

criteria

of a calibrated

can be expressed

developed,

than

against

in our ability

calibrated

an uncalibrated

model

model

should

an independent

to extrapolate stand

a greater

model.

REFERENCES

Cl1

Benyon,

c21

Cohen,~ K.J.;

c31

Fick,

c41

Wigan,

c51

P.R.

G.W.

Cyret,

R.M.

0. 1977:

International S.R.

Korte,

Cl01 McCall,

D.G.

In:

1984:

Unpub

of Economics

75:

112-127.

5: 137-161. 188-192. of soil water

Research

Innis,

G.S.;

dynamics.

A compendium

of recent

Centre. University

Haydock,

K.P.

of Queensland 1978:

Press.

In: Innis,

G.S.

ed.

Springer-Verlag.

Arnold,

research

Unpub

Journal

and modelling.

H.W.;

Model.

H. 1976:

1981:

simulation

Simulation Hunt,

in ecosystems C.J.

Systems 18(5):

Development

Simulation

C81 Van Keulen, analysis

Computer

1977: R.K.;

The Quarterly

1961:

Simulation

Hillel,

Steinhorst,

3: 250-56.

Agricultural

1972:

works.

Grassland

191

Search

1980:

M.R.

II61 Wilson, c71

1972:

G.W.;

Wit,

C.T.

and management.

PhD Thesis, PhD Thesis,

Massey Massey

eds. Pudoc,

University. University.

Critical

evaluation

Wageningen.

of systems